Unmanned aerial vehicle self-adaptive intelligent navigation and obstacle avoidance system based on multi-mode perception

Abstract

The invention relates to the technical field of unmanned aerial vehicle control, and discloses an unmanned aerial vehicle self-adaptive intelligent navigation and obstacle avoidance system based on multi-modal perception, and the system comprises the steps: synchronously collecting a left image and a right image; performing frequency domain analysis based on the row gray average sequence of the left image to obtain a pulse width modulation period and a corresponding stripe row spacing; performing cross-correlation operation on the line gray average sequence of the left image and the right image, and determining an acquisition deviation; performing phase positioning according to the stripe line spacing, determining a stripe reference line, performing cyclic calibration according to the stripe line spacing by taking the stripe reference line as a benchmark, and constructing a line-level mask; performing block calculation on the left image and the right image to obtain an original parallax, and performing correction processing on the original parallax based on the row-level mask to obtain a corrected parallax; distributing a confidence coefficient for each pixel according to the row-level mask, projecting the corrected parallax as a point cloud, and generating an occupancy probability according to voxel division; and constructing an obstacle potential field based on the occupancy probability, and outputting a control instruction according to the obstacle potential field to complete unmanned aerial vehicle obstacle avoidance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle control, more particularly, it relates to an unmanned aerial vehicle adaptive intelligent navigation and obstacle avoidance system based on multi-modal perception. BACKGROUND

[0002] Modern urban building facades often have high-brightness LED display screens and lighting equipment. These light sources mostly use pulse width modulation (PWM) technology for brightness adjustment, and their working frequencies usually range from hundreds of Hz to tens of kHz. At the same time, in order to reduce sensor power consumption and improve imaging speed, most lightweight unmanned aerial vehicle platforms use rolling shutter CMOS cameras as visual input devices.

[0003] When a rolling shutter binocular camera mounted on an unmanned aerial vehicle images a high-frequency PWM LED light source, the following situations may occur, as follows: 1. A rolling shutter camera does not expose the entire frame of an image at the same time, but rather captures each row from top to bottom or from left to right. The exposure time of each row may differ by several microseconds to tens of microseconds. If there is a PWM light source in the scene being imaged, the on-off state of this type of light source may be different at each exposure time. As a result, the brightness state recorded in different scanning rows may alternate between light and dark, resulting in fixed-pitch light or dark bands in the image. This banding artifact is a mixed effect caused by the time-domain dimming signal during spatial scanning.

[0004] 2. In a stereo vision system, the two cameras should be exposed synchronously to obtain parallax information. However, due to cost and hardware design constraints, the binocular cameras installed on commercial unmanned aerial vehicles often cannot be perfectly synchronized, and there may be a time difference of several microseconds between the triggering times. Since the LED light source is in a high-frequency flashing state, even a small time difference will cause the light and dark bands in the left and right images to shift in phase.

[0005] 3. Most unmanned aerial vehicle obstacle avoidance systems use parallax matching algorithms based on pixel grayscale or local texture, such as block matching, SAD (Sum of Absolute Differences), Census transformation, etc. These algorithms assume that the grayscale changes in the matching area are due to the shape or structure of the scene, and accordingly derive the parallax values of the corresponding pixels in the left and right images. However, the light and dark bands caused by rolling shutter imaging do not originate from real geometric differences in the scene, but are artificial brightness modulations. The parallax matching process will interpret these brightness bands as object edges or stereo structures, resulting in unrealistic parallax solutions, i.e., misjudging a bright surface that should be far away as a close-range obstacle.

[0006] The direct consequence of the above problems is that there are large areas of false near-obstacles in the depth map, which are distributed in an orderly manner and have depth values that are significantly smaller than the true values. In the obstacle avoidance logic, the system regards it as an obstacle that will soon be hit, triggering an avoidance action, which leads to the following problems: 1. The aircraft mistakenly avoids obstacles or stops suddenly, affecting the stability of the task; 2. The path planning is interrupted or enters a fallback loop, reducing the efficiency of the flight path; 3. Multiple false obstacle avoidance causes the aircraft attitude to oscillate, inducing control instability; 4. Actual collision risk in a small space. SUMMARY

[0007] The present application provides a UAV adaptive intelligent navigation and obstacle avoidance system based on multi-modal perception, which solves the technical problems raised in the background art.

[0008] The present application provides a UAV adaptive intelligent navigation and obstacle avoidance system based on multi-modal perception, which includes: A data acquisition module for synchronously acquiring left and right images; A data processing module for performing frequency domain analysis based on the row gray mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row distance; A time difference analysis module for taking the row gray mean sequence of the left and right images to perform cross-correlation operation to determine the acquisition deviation; An interference marking module for phase positioning according to the stripe row distance, determining the stripe reference row, and taking the stripe reference row as the reference to perform cyclic calibration according to the stripe row distance to construct a row-level mask; A parallax correction module for performing block calculation of the original parallax on the left and right images, and performing correction processing on the original parallax based on the row-level mask to obtain the corrected parallax; A confidence calculation module for assigning a confidence level to each pixel according to the row-level mask, and generating an occupancy probability after dividing the corrected parallax into voxels; A UAV obstacle avoidance module for constructing an obstacle potential field based on the occupancy probability, and outputting control instructions to complete UAV obstacle avoidance according to the obstacle potential field.

[0009] Further, the frequency domain analysis based on the row gray mean sequence of the left image obtains the pulse width modulation period and the corresponding stripe row distance, including: Calculating the gray mean value of each row of pixels of the left image to form a row gray mean sequence; Performing fast Fourier transform on the row gray mean sequence to obtain an amplitude spectrum; Determining the main frequency index ks of the amplitude spectrum, and calculating the stripe row distance Δs = 1 / ks according to the spectrum main frequency index; The prior row reading period Trow is used to calculate a pulse width modulation period TP=Δs×Trow.

[0010] Further, the row gray mean sequence of the left image and the right image is correlated to determine the acquisition deviation, including: The correlation value of the row gray mean sequence element y of the left image and the row gray mean sequence element y of the right image under the calculation of the row offset δ; The value range of the row offset δ is determined, and the value of the row offset δ is traversed to obtain the row offset δ that causes the maximum correlation value as the target row offset δ*; The acquisition deviation is calculated according to the target row offset δ, as follows: Wherein, The acquisition deviation is represented.

[0011] Further, the phase is positioned according to the stripe row distance, the stripe reference row is determined, and the stripe reference row is taken as the reference to perform cyclic calibration according to the stripe row distance, and the row-level mask is constructed, including: The local brightness contrast of the candidate row yc is calculated in the row index interval [1, Δs]; The candidate row corresponding to the maximum local brightness contrast is taken as the stripe reference row yref; The modulus value of each row pixel in the left image is calculated according to the stripe reference row; If the modulus value of the yth row pixel is < the preset stripe effective bandwidth, the yth row pixel in the left image is marked as 1; otherwise, the yth row pixel in the left image is marked as 0, to obtain the row-level mask.

[0012] Further, the block calculation original disparity is performed on the left image and the right image, and the original disparity is corrected based on the row-level mask to obtain the corrected disparity, including: The block matching of a fixed window size w×w is performed between the left image and the right image for the pixel coordinates (x, y) to obtain the original disparity; The original disparity draw(x, y) is sign-flipped using the row-level mask to obtain the corrected disparity.

[0013] Further, the row-level mask is used to configure the confidence of each pixel, including: The row-level reference confidence R0(y) is calculated according to the row-level mask; The disparity validity mark V(x, y) is generated based on the corrected disparity; The fused pixel confidence is obtained by weighted fusion based on the row-level reference confidence and the disparity validity mark for the pixel coordinates (x, y); The maximum value of the fused pixel confidence is determined, and the fused pixel confidence of each pixel coordinate is normalized according to the maximum value to obtain the confidence Rref(x,y) of each pixel coordinate.

[0014] Further, the corrected parallax is projected into a point cloud, and an occupancy probability is generated by voxel division, including: The pixel depth is calculated from the corrected parallax; The three-dimensional point cloud P(x,y) is generated according to the pixel depth; The voxel edge length svox is set, and the voxel index (a,b,c) is calculated; The confidence value S1 and the number S2 of the point cloud in each voxel are determined; The ratio of S1 and S2 is taken as the occupancy probability of the corresponding voxel.

[0015] Further, the obstacle potential field is constructed based on the occupancy probability, including: The voxel center coordinates (Xc,Yc,Zc) are determined; The distance attenuation factor is calculated according to the voxel center coordinates; The occupancy probability of each voxel is weighted according to the distance attenuation factor to obtain the original potential energy; The original potential energy is smoothed by three-dimensional Gaussian convolution to obtain the smoothed potential energy; The smoothed potential energy of the voxel is binarized to obtain the obstacle potential field Uobs.

[0016] Further, the control instruction is output according to the obstacle potential field to complete the obstacle avoidance of the unmanned aerial vehicle, including: The potential energy Up is obtained by interpolation at the spatial position of the unmanned aerial vehicle; The potential energy Up and the obstacle potential field Uobs are combined to obtain the potential energy gradient ∇Uobs by three-dimensional finite difference method; The avoidance vector is calculated based on the potential energy gradient ; The avoidance vector and the initial flight path speed vector are fused to form the expected speed; The expected speed is saturated in amplitude to obtain the saturated speed; The saturated speed is analyzed into pitch angle command, roll angle command and vertical speed command, and is executed by the flight control execution unit of the unmanned aerial vehicle.

[0017] The beneficial effects of the present application are that: by constructing the image processing and obstacle avoidance control process, the system can accurately identify and eliminate periodic false parallax signals under the conditions of LED light interference with high-frequency PWM and rolling shutter asynchronous acquisition error, effectively preventing false obstacles from triggering obstacle avoidance instructions. Based on frequency domain analysis to extract stripe parameters, cross-correlation calculation to acquire deviation, row-level mask to mark interference areas, parallax symbol correction and confidence weight distribution, the present application finally constructs a continuous obstacle potential field and outputs dynamic control instructions, realizes closed-loop control from interference detection to flight control avoidance, and significantly improves the obstacle avoidance reliability and flight path stability of the unmanned aerial vehicle in complex urban light environments. BRIEF DESCRIPTION OF DRAWINGS

[0018] Fig. 1 is a flowchart of the present application; Fig. 2 is a module diagram of the present application; Fig. 3 is an unmanned aerial vehicle obstacle avoidance flowchart of the present application. DETAILED DESCRIPTION

[0019] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is merely meant to provide a better understanding of the subject matter described herein and can be changed in function and arrangement without departing from the scope of the present description. Various processes or components can be omitted, substituted, or added according to desired implementations. Additionally, features described with respect to some examples can be combined in other examples.

[0020] As shown in Figs. 1 to 3 , the unmanned aerial vehicle adaptive intelligent navigation and obstacle avoidance system based on multi-modal perception comprises: a data acquisition module for synchronously acquiring left and right images; a data processing module for performing frequency domain analysis based on a row gray mean sequence of the left image to obtain a pulse width modulation period and a corresponding stripe row distance; a time difference analysis module for performing cross-correlation operation on the row gray mean sequences of the left and right images to determine the acquisition deviation; an interference marking module for phase positioning according to the stripe row distance, determining a stripe reference row, and taking the stripe reference row as a reference to perform cyclic marking according to the stripe row distance to construct a row-level mask; a parallax correction module for performing block calculation of the original parallax on the left and right images, and performing correction processing on the original parallax based on the row-level mask to obtain the corrected parallax; a confidence calculation module for assigning a confidence to each pixel according to the row-level mask, and generating an occupancy probability by voxel division after projecting the corrected parallax into a point cloud; The unmanned aerial vehicle obstacle avoidance module is used for constructing an obstacle potential field based on the occupancy probability and outputting a control instruction to complete unmanned aerial vehicle obstacle avoidance according to the obstacle potential field.

[0021] In one embodiment of the application, frequency domain analysis is performed based on a sequence of row gray mean values of the left image to obtain a pulse width modulation period and a corresponding stripe row distance, including: The gray mean value of each row of pixels of the left image is calculated to form a sequence of row gray mean values; Fast Fourier transform is performed on the sequence of row gray mean values to obtain an amplitude spectrum; The main frequency index ks of the amplitude spectrum is determined, and the stripe row distance is calculated as Δs = 1 / ks according to the main frequency index; The pulse width modulation period TP is calculated as Δs x Trow using the prior row reading period Trow.

[0022] In detail, each row of pixels of the left image is exposed row by row due to the rolling shutter exposure characteristic, and the exposure time is time-related to the pulse width modulation period of the high-frequency LED: when the LED is on, the gray value of the corresponding row of pixels is generally high; when the LED is off, the gray value of the corresponding row of pixels is generally low. The average value of the pixel gray value of each row of the left image is calculated to eliminate column direction noise interference and condense the gray value fluctuation between rows caused by the alternating on and off of the LED into a sequence of row gray mean values (denoted as gL(y), where y is the row index). The periodicity of the sequence directly reflects the space-time coupling relationship between the PWM signal of the LED and the rolling shutter exposure.

[0023] In detail, the periodicity of the sequence of row gray mean values is a repeating pattern of bright rows and dark rows in the time domain, and the time domain periodicity can be converted into a frequency domain amplitude spectrum through FFT. In the frequency domain, a periodic signal will present a significant peak value (main frequency), and the frequency corresponds to the periodic characteristics of the signal: If the PWM period of the LED is TP and the row reading period of the rolling shutter is Trow, then the rolling shutter will scan Δs rows (i.e., the stripe row distance) when the LED completes one on-off, and therefore the spatial period of the sequence of row gray mean values is Δs rows, and the main frequency component in the frequency domain is Δs.

[0024] In detail, in the amplitude spectrum obtained by FFT, the peak value with the maximum amplitude is searched, and the main frequency index ks corresponding to the peak value reflects the spatial frequency (unit: row negative power, i.e., the number of periods per unit row) of the sequence of row gray mean values. Since the spatial period (stripe row distance Δs) and the spatial frequency are inversely related, Δs = 1 / ks; for example, if the main frequency index ks = 0.02, then the bright-dark stripe pattern repeats every 50 rows (1 / 0.02), i.e., the stripe row distance Δs = 50 rows.

[0025] In detail, the row reading period Trow is a hardware inherent parameter (indicating the exposure time difference between two adjacent rows). Since the stripe row distance Δs corresponds to the number of rows scanned by the rolling shutter when the LED completes one PWM period (once on and once off), the relationship between the PWM period TP and Δs and Trow is TP=Δs×Trow. Through the formula, the stripe row distance in the spatial domain can be converted into the PWM period in the time domain, and the space-time correspondence between the LED signal period and the image row structure is established.

[0026] In an embodiment of the present application, the row gray mean value sequences of the left image and the right image are correlated to determine the acquisition deviation, including: The cross-correlation value of the row gray mean value sequence element y of the left image and the row gray mean value sequence element y of the right image under the row offset δ is calculated as follows: Wherein, C(δ) represents the corresponding cross-correlation value under the row offset δ, gL(y) represents the left image row gray mean value sequence element y, gR(y) represents the right image row gray mean value sequence element y, and gR(y+δ) represents the right image row gray mean value sequence element y+δ. The value range of the row offset δ is determined, and the value of the row offset δ is traversed to obtain the row offset δ that causes the maximum cross-correlation value as the target row offset δ*. The acquisition deviation is calculated according to the target row offset δ* as follows: Wherein, Indicates the acquisition deviation.

[0027] In detail, the gray mean value of each row of the left image is calculated to obtain the sequence gL(y) (y is the row index); similarly, the right image generates gR(y). The sequence has filtered out column noise and only retains the row brightness mode (light and dark period of LED stripes).

[0028] In detail, the cross-correlation value is calculated: define the row offset δ, and calculate the gray mean value product sum of all effective rows y: The logic is that when δ is equal to the real misalignment of the left and right image row modes, the brightness mode of gL(y) and gR(y+δ) is highly similar, and the product sum C(δ) reaches the global maximum value.

[0029] In detail, based on the consistency of the number of rows of the left and right images, the search range of δ is set as (H is the number of rows in the image) to ensure coverage of all possible row misalignments. Iterate through each δ within the above range, calculate the corresponding cross-correlation value, and select the δ that maximizes the cross-correlation value as the target row offset δ*. This utilizes the periodicity and correlation of row patterns, i.e., the cross-correlation value at the actual misalignment will be significantly higher than at other positions due to pattern matching, forming a recognizable peak.

[0030] In detail, the hardware has a known row read cycle Trow (i.e., the time interval between each row of pixels output from the sensor, in seconds). Since δ* represents row-level misalignment, its corresponding time difference (i.e., the deviation τ between the trigger times of the left and right cameras) is: The logic is as follows: the exposure or reading time interval of each row is Trow, so the misalignment of δ* rows corresponds to the time difference of δ*×Trow.

[0031] In one embodiment of the present invention, phase positioning is performed according to the stripe row spacing to determine the stripe reference row, and a row-level mask is constructed by cyclic calibration according to the stripe row spacing based on the stripe reference row, including: Within the row index interval [1, Δs], calculate the local brightness contrast of candidate row yc as follows: Where yc∈[1,Δs], D(yc) represents the local brightness contrast of candidate row yc, and gL(yc) represents the gray-level mean sequence element of the left image row. gL(yc+1) represents the gray-level mean sequence element of the left image row. +1, maxgL represents the maximum value in the gray-scale mean sequence of the left image row, and mingL represents the minimum value in the gray-scale mean sequence of the left image row; The candidate row corresponding to the maximum local brightness contrast is used as the stripe reference row yref; The magnitude value is calculated for each row of pixels in the left image based on the stripe reference row, as follows: in, This represents the magnitude of the pixel in the y-th row of the left image. This indicates the modulo operation; If the modulus of the y-th row pixel is less than the preset effective bandwidth of the stripe, then mark the y-th row pixel in the left image as 1; otherwise, mark the y-th row pixel in the left image as 0 to obtain the row-level mask.

[0032] In detail, the stripe row spacing Δs represents the period of the LED brightness stripes in the image row direction (i.e., the brightness pattern of the stripes repeats every Δs rows). To locate the initial phase of the stripes, candidate rows need to be searched within the row index interval of a complete stripe period.

[0033] In detail, for each candidate row yc, calculate its local brightness contrast with the next row yc+1: The grayscale difference between candidate row yc and its next row yc+1 is quantified. If yc is located at the boundary between light and dark stripes, the grayscale difference between adjacent rows will increase significantly.

[0034] It is the global grayscale range of the grayscale mean sequence of the left image row, used for normalization processing, thereby eliminating the absolute numerical influence of different lighting conditions on contrast, so that D(yc) is uniformly mapped to the [0,1] interval, ensuring comparability under different scenes.

[0035] In detail, the fringe reference row is located by searching for the maximum value of local brightness contrast. The boundary row between light and dark areas of the fringe (or the center row) has the largest gray-level difference between adjacent rows. Therefore, the candidate row yc with the largest D(yc) corresponds to the phase starting point of the fringe pattern (such as the edge row of the first bright band). Within a fringe cycle, the fringe pattern changes continuously and periodically, and there exists only one globally largest D(yc), ensuring that yref is unique and providing a fixed phase reference for subsequent cyclic calibration.

[0036] In detail, for each row y in the left image, calculate its period offset relative to the stripe reference row yref: By utilizing the periodicity of the stripes, any row y can be mapped to the interval [1, Δs]. For example, the r value of the yref+kΔs-th row (k is an integer) is the same as that of the yref-th row, corresponding to the same phase of the stripes (such as bright bands, dark bands, or boundary rows), thus achieving single-period phase → full image periodic expansion.

[0037] In detail, the preset effective stripe bandwidth ws (ws=0.3Δs) is used to mark the row-level mask based on the modulus r. The preset effective stripe bandwidth defines the range of rows in the stripe that are affected by LED brightness interference (e.g., a bright band lasting 0.3Δs). When r < ws, the row is in the interference region of the stripe (bright or dark band, depending on the stripe pattern); when r ≥ ws, the row is in the non-interference region of the stripe (stable grayscale, unaffected by LED flicker).

[0038] In one embodiment of the present invention, the original disparity is calculated by performing block calculations on the left and right images, and the original disparity is corrected based on a row-level mask to obtain the corrected disparity, including: For pixel coordinates (x, y), perform block matching with a fixed window size of w×w between the left and right images to obtain the original disparity, as follows: Where draw(x,y) represents the original disparity at pixel coordinates (x,y), IL(x+i,y+i) represents the pixel grayscale value at the left image with pixel coordinates (x,y) as the center, horizontally offset by i and vertically offset by j, IR(x+id,y+i) represents the pixel grayscale value in the right image corresponding to the position (x+i,y+i) in the left image, horizontally offset by disparity d, i and j represent the horizontal and vertical offsets within the block matching window, respectively, d represents the disparity candidate value, d∈D, and D represents the disparity search range; The original disparity draw(x,y) is sign-flipped using a row-level mask, as follows: Where draw(x,y) represents the parallax correction, and M(y) represents element y in the row-level mask.

[0039] In detail, for any pixel coordinate (x, y) in an image, its correspondence in the left and right images needs to be determined to obtain the disparity value. Specifically, a fixed-size window of size w×w is selected in both the left and right images, centered at the pixel coordinate (x, y). The purpose of the window is to improve matching stability through the grayscale information of local regions, avoiding matching errors caused by noise or isolated grayscale changes in individual pixels.

[0040] In detail, i and j represent the horizontal and vertical offsets within the window, respectively, ranging from -w / 2 to w / 2, covering the entire window area. For each position (x+i, y+j) within the left image window, its grayscale value is represented by IL(x+i, y+j); the corresponding position in the right image needs to consider the horizontal disparity candidate value d, i.e., the grayscale value of the corresponding position within the right image window is represented by IR(x+id, y+j). To find the best matching disparity, the absolute difference of the grayscale values ​​of all corresponding pixels within the left and right image windows needs to be calculated, and these absolute differences are summed. All candidate values ​​d within the disparity search range D are traversed. When the sum of the absolute grayscale differences reaches its minimum value, the corresponding candidate value d is the original disparity draw(x, y) at the pixel coordinates (x, y). This is because the smaller the sum of the absolute grayscale differences, the more similar the window areas of the left and right images are, and the more reliable the corresponding disparity value.

[0041] In detail, the parallax correction process is implemented based on row-level masks. The row-level mask M(y) is used to mark rows affected by LED stripe interference. When M(y) is 1, it indicates that the pixels in that row are affected; when M(y) is 0, it indicates that the pixels in that row are normal. The parallax correction dcorr(x,y) is calculated as follows: When M(y) is 1, -1 raised to the power of 1 equals -1, and the corrected disparity is the opposite of the original disparity, thus reversing the sign of the disparity. When M(y) is 0, -1 raised to the power of 0 equals 1, and the corrected disparity is the same as the original disparity. The necessity of sign reversal stems from the disparity errors caused by LED stripe interference. The original disparity of the interfered line will have a sign error due to mismatches between bright and dark stripes, such as misjudging a distant area as a near area (negative disparity). Sign reversal can correct this error, allowing the disparity to accurately reflect the actual depth relationship.

[0042] In one embodiment of the present invention, the confidence level is assigned to each pixel based on the row-level mask, including: The row-level baseline confidence score R0(y) is calculated as follows: in, Represents the row-level suppression coefficient. ; Generate the disparity validity label V(x,y) as follows: Where V(x,y) represents the disparity validity label of pixel coordinates (x,y), and dmin represents the minimum credible disparity threshold; For pixel coordinates (x, y), a weighted fusion is performed based on row-level baseline confidence and disparity validity label, as follows: Where R(x,y) represents the confidence score of the fused pixel, and Rrep(x,y) represents the row-level baseline confidence score of the row to which the pixel coordinate (x,y) belongs; The maximum value of the fused pixel confidence is determined, and the fused pixel confidence of each pixel coordinate is normalized based on the maximum value to obtain the confidence Rref(x,y) of each pixel coordinate.

[0043] In detail, the formula for calculating the row-level baseline confidence level R0(y) is as follows: The row-level suppression coefficient α ranges from 0 to 1. The row-level mask M(y) is used to mark rows in the image that are affected by LED stripe interference. When M(y) is 1, the corresponding row is an affected row; when M(y) is 0, the corresponding row is a normal row. The core logic of this calculation is: normal rows are not affected, so their row-level baseline confidence remains at 1 (the highest value); affected rows have their baseline confidence reduced by adjusting α. The larger α is, the more significant the decrease in confidence of affected rows, thus reflecting the unreliability of the data in affected rows.

[0044] In detail, a disparity validity label V(x,y) is generated. The determination rule is as follows: if the absolute value of the corrected disparity dcorr(x,y) is greater than or equal to the minimum reliable disparity threshold dmin, then V(x,y) is set to 1; otherwise, it is set to 0. The minimum reliable disparity threshold dmin is a preset value used to filter out excessively small disparities caused by noise or incorrect matching. That is, such disparities usually do not reflect the depth relationship of the real scene, so they are marked as invalid (V(x,y)=0); while disparities that reach or exceed dmin are considered to have basic validity (V(x,y)=1).

[0045] In detail, the weighted fusion yields the fused pixel confidence score R(x,y). Its calculation formula is the product of the disparity validity label V(x,y) and the row-level baseline confidence score Rrep(x,y) of the row to which the pixel belongs, where Rrep(x,y) is the row-level baseline confidence score R0(y) of the row to which the pixel belongs. This combines the local validity of disparity with the overall reliability at the row level: when the disparity is valid (V(x,y)=1), the pixel confidence score inherits the baseline confidence score of that row; when the disparity is invalid (V(x,y)=0), regardless of the row-level baseline confidence score, the pixel confidence score is 0, thus filtering out unreliable information corresponding to invalid disparities.

[0046] In detail, normalization processing yields the final pixel confidence score Rref(x,y). The specific operation is as follows: first, determine the maximum value among all fused pixel confidence scores R(x,y); then, divide the R(x,y) of each pixel by this maximum value, mapping the result to the range of 0 to 1. The purpose of normalization is to unify the quantization scale of the confidence scores, ensuring that the confidence scores of different pixels are comparable.

[0047] In one embodiment of the present invention, generating occupancy probabilities by voxel division after correcting the parallax projection into a point cloud includes: Pixel depth is calculated from the corrected parallax, including: Where Z(x,y) represents the pixel depth at pixel coordinates (x,y), f represents the camera focal length, and B represents the baseline length of the left and right cameras; A 3D point cloud P(x,y) is generated based on the pixel depth, as follows: The confidence score Rref(x,y) is used as a weight to weight the 3D point cloud P(x,y) to obtain the weighted 3D point cloud PR(x,y); Define the voxel side length svox and calculate the voxel index (a, b, c) as follows: Where (Xmin, Ymin, Zmin) represents the spatial location of the UAV; Determine the confidence level and value S1 of the point cloud in each voxel, and the number of point clouds S2; The ratio of S1 to S2 is used as the occupancy probability of the corresponding voxel.

[0048] In detail, pixel depth is calculated by correcting for parallax. The calculation formula is: The camera focal length *f* and baseline length *B* are inherent camera parameters, and the corrected disparity *dcorr(x,y)* reflects the horizontal offset of corresponding pixels in the left and right images. The core logic of this formula is based on the principle of stereo vision: disparity is inversely proportional to depth; that is, the greater the offset of the same spatial point in the left and right images (the greater the disparity), the closer it is to the camera (the smaller the depth); conversely, the greater the offset, the farther away it is. Through this calculation, the disparity information of pixels in a two-dimensional image can be converted into the depth information Z(x,y) of the corresponding spatial point.

[0049] In detail, a 3D point cloud P(x,y) is generated based on pixel depth. The coordinates (X,Y,Z) of the 3D point cloud are calculated from the image pixel coordinates (x,y) and depth Z(x,y): X is equal to the product of x and Z(x,y) divided by f, Y is equal to the product of y and Z(x,y) divided by f, and Z is the pixel depth Z(x,y) calculated above. This transformation is based on the perspective projection principle of camera imaging: there is a scaling relationship between the pixel coordinates (x,y) on the image plane and the coordinates (X,Y) in 3D space, with a scaling factor of Z / f. Through this relationship, 2D image pixels can be mapped to 3D space to form a point cloud composed of discrete points, which intuitively reflects the 3D structure of the scene.

[0050] In detail, the 3D point cloud is weighted. The confidence score Rref(x,y) of each 3D point is used as a weight to weight the 3D point cloud P(x,y), resulting in a weighted 3D point cloud PR(x,y). The confidence score Rref(x,y) quantifies the reliability of pixel information. The purpose of the weighting process is to give higher weight to points with high confidence in subsequent analysis, and reduce the weight of points with low confidence (such as points affected by LED stripe interference), thereby reducing the impact of unreliable data on 3D environment modeling.

[0051] In detail, the voxel side length is set and the voxel index is calculated. A voxel is a cubic unit with a side length of svox in 3D space, used to discretize a continuous 3D space into a regular mesh. The voxel index (a, b, c) is calculated as follows: subtract the UAV spatial position (Xmin, Ymin, Zmin) from the 3D point coordinates (X, Y, Z), then divide by the voxel side length svox, and take the integer part of the result. The UAV spatial position (Xmin, Ymin, Zmin) is used as the coordinate reference. This calculation determines the voxel number to which each 3D point belongs, realizing the spatial grouping of the 3D point cloud.

[0052] In detail, the confidence score and value of each voxel are statistically analyzed in relation to the number of points in the point cloud. For each voxel, the sum of the confidence scores (S1) of all its 3D points and the total number of points (S2) are calculated. S1 reflects the sum of the reliability of all points within the voxel, and S2 reflects the density of the data within the voxel. Together, they provide a basis for assessing whether the voxel is occupied by obstacles.

[0053] In detail, the occupancy probability of a voxel is calculated. The occupancy probability is the ratio of the voxel's confidence score S1 to the number of points in the point cloud S2. This ratio is essentially the average confidence score of all points within the voxel. The higher the ratio, the greater the likelihood that the voxel is occupied by a real obstacle, because more high-confidence points support this conclusion. The lower the ratio, the more likely the voxel is noise or unreliable data, thus achieving a quantitative description of obstacle distribution in the 3D environment.

[0054] It should be noted that the UAV is equipped with a Global Navigation Satellite System module, which can receive satellite signals in real time and calculate the UAV's current longitude, latitude, and altitude. After coordinate transformation, the three-dimensional spatial coordinates (Xmin, Ymin, Zmin) can be obtained. In environments with weak GNSS signals, such as indoors, the UAV can use a visual SLAM system to combine image features acquired by the camera with motion estimation to calculate its own position in three-dimensional space in real time, i.e., (Xmin, Ymin, Zmin), as a benchmark for environmental analysis.

[0055] In one embodiment of the present invention, constructing a barrier potential field based on the occupancy probability includes: Determine the voxel center coordinates (Xc, Yc, Zc) as follows: ; Calculate the distance attenuation factor: ; The original potential energy is obtained by weighting the occupancy probability of each voxel according to the distance decay factor. The original potential energy is smoothed by three-dimensional Gaussian convolution to obtain a smoothed potential energy. The smooth potential energy of the voxels is binarized to obtain the barrier potential field Uobs.

[0056] In detail, the voxel center coordinates (Xc, Yc, Zc) are determined. A voxel is a cubic unit with a side length of svox in three-dimensional space, and its center coordinates are calculated based on the voxel index and the UAV's spatial position. Specifically, Xc equals Yc and Zc are calculated in the same way, based on the UAV's Y-direction position Ymin and voxel Y-direction index b, and the UAV's Z-direction position Zmin and voxel Z-direction index c, respectively. This clarifies the precise center position of each voxel in three-dimensional space, providing a spatial reference for subsequent distance calculations and potential energy assignment.

[0057] In detail, the distance attenuation factor is calculated using the following formula: Zc reflects the distance between the voxel and the UAV in the depth direction. The closer the distance (the smaller Zc), the larger the attenuation factor; the farther the distance (the larger Zc), the smaller the attenuation factor. This makes nearby obstacles pose a more urgent threat to the UAV and require a higher weight in the potential field; distant obstacles pose a lower threat and have a correspondingly lower weight, thus allowing the potential field to reflect the difference in the urgency of obstacles.

[0058] In detail, the weighted processing yields the original potential energy, which is obtained by multiplying the voxel's occupancy probability by a distance decay factor. The occupancy probability reflects the likelihood that the voxel is occupied by a real obstacle, while the distance decay factor reflects the impact of the obstacle's proximity on the threat. Combined, the original potential energy reflects both the credibility of the obstacle's existence and the urgency of its spatial location; that is, voxels with high occupancy probabilities and close proximity have higher original potential energy, indicating a higher obstacle risk in that area.

[0059] In detail, the original potential energy is smoothed using 3D Gaussian convolution. 3D Gaussian convolution performs a weighted average of the potential energy of each voxel with that of its surrounding voxels (the weights follow a Gaussian distribution, with closer voxels having larger weights), making the originally discrete and potentially abrupt potential energy distribution continuous and smooth. This eliminates drastic fluctuations in the potential energy of individual voxels, preventing high-frequency oscillations in subsequent obstacle avoidance commands calculated based on the potential field gradient, and ensuring the stability of the UAV's movements.

[0060] In detail, the smoothed potential energy is binarized to obtain the obstacle potential field Uobs. Binarization is achieved by setting a threshold: if the smoothed potential energy exceeds the threshold, the obstacle potential field Uobs of that voxel is set to 1 (marked as an obstacle region); otherwise, it is set to 0 (marked as a safe region). The purpose of binarization is to clearly distinguish between obstacle regions and safe regions in the environment, transforming the continuous potential energy distribution into a clear obstacle identifier, providing clear spatial guidance for subsequent calculations of avoidance directions.

[0061] In one embodiment of the present invention, obstacle avoidance by the UAV is completed based on the output control command of the obstacle potential field, including: The potential energy Up is obtained by interpolation based on the spatial location of the drone, as follows: in, is (Xmin, Ymin, Zmin), express function; By combining the potential energy Up and the obstacle potential field Uobs, the potential energy gradient ∇Uobs is obtained using the three-dimensional finite difference method; Calculate the avoidance vector for the potential energy gradient. ,as follows: The desired velocity is formed by fusing the avoidance vector and the initial trajectory velocity vector, as follows: in, Indicates the track weight. This indicates an attempt to avoid weighting. Indicates the potential energy threshold. Represents the initial trajectory velocity vector; The saturation velocity is obtained by saturating the amplitude of the desired velocity. The saturation velocity is resolved into pitch angle commands, roll angle commands, and vertical velocity commands, which are then processed by the UAV's flight control execution unit.

[0062] In detail, the potential energy Up is obtained by interpolation at the UAV's spatial location. The UAV's spatial location is represented by (Xmin, Ymin, Zmin). The Interp function is used to interpolate the obstacle potential field Uobs to obtain the potential energy Up corresponding to the UAV's current location. The obstacle potential field Uobs is a discrete distribution based on voxel partitioning, while the actual location of the UAV may not be exactly at the voxel center. The purpose of the interpolation is to accurately obtain the potential energy value of the UAV's spatial location, which directly reflects the degree of obstacle risk faced by the UAV at its current location.

[0063] In detail, the potential energy gradient ∇Uobs is calculated by combining the potential energy Up and the obstacle potential field Uobs. The gradient is calculated using the three-dimensional finite difference method, which determines the direction and rate of potential energy change by comparing the potential energy difference between the target voxel and its neighboring voxels. The potential energy gradient ∇Uobs represents the direction of the fastest increase in potential energy within the obstacle potential field; this direction points towards the obstacle region, providing a basis for determining the subsequent avoidance direction.

[0064] In detail, the avoidance vector Vrep is calculated based on the potential energy gradient. When the magnitude of the potential energy gradient is greater than 0, it indicates a significant change in obstacle potential energy. In this case, the avoidance vector Vrep is the negative unit vector of the potential energy gradient, meaning its direction is opposite to the gradient and points away from the obstacle. When the magnitude of the potential energy gradient is equal to 0, it indicates no significant change in obstacle potential energy in the current area, and the avoidance vector Vrep is 0, meaning no active avoidance is needed. This ensures that the UAV can accurately adjust its movement direction towards a safe area.

[0065] In detail, the desired velocity is formed by fusing the avoidance vector and the initial trajectory velocity vector. In the formula for calculating the desired velocity, Ktrk is the trajectory weight, used to maintain the influence of the initial trajectory velocity vector vtrk, ensuring that the UAV does not deviate from the core mission path while avoiding obstacles; Krep is the avoidance weight, used to adjust the strength of the avoidance vector's influence; the min function dynamically adjusts the actual effect of the avoidance weight through the ratio of Up to the potential energy threshold τU. That is, the larger Up is (i.e., the higher the current obstacle risk), the closer this ratio is to 1, and the stronger the influence of the avoidance vector; conversely, the avoidance influence weakens. This fusion mechanism achieves a balance between trajectory maintenance and obstacle avoidance.

[0066] In detail, the initial trajectory velocity vector is the velocity vector of the UAV when it is flying along a pre-planned trajectory without detecting any obstacles. It includes the magnitude and direction of the velocity and reflects the original flight path intention of the UAV.

[0067] In detail, the avoidance vector is a vector calculated based on the potential energy gradient of the obstacle potential field. When the magnitude of the potential energy gradient is greater than 0, its direction is opposite to that of the obstacle potential field gradient. It is used to indicate the direction in which the UAV deviates from the obstacle and help the UAV avoid the obstacle area.

[0068] In detail, the desired speed is saturated to obtain the saturated speed. Amplitude saturation limits the speed to the physical performance limit of the drone (maximum flight speed). This ensures that the generated speed command is within the drone's executable range, avoiding loss of control or hardware damage due to commands exceeding its capabilities.

[0069] In detail, the saturation velocity is parsed into specific control commands and executed. Saturation velocity is a velocity vector in three-dimensional space, which needs to be converted into attitude and power commands for the UAV, namely pitch angle commands, roll angle commands, and vertical speed commands. The pitch and roll angles determine the UAV's direction of motion on the horizontal plane, while the vertical speed command determines its climb and fall. These commands are then translated by the UAV's flight control unit into specific actions such as motor speed adjustments, ultimately enabling the UAV to perform obstacle avoidance maneuvers.

[0070] For example, if the saturation velocity is a three-dimensional vector (vx = 2 m / s, vy = -1 m / s, vz = 0.3 m / s): vx represents the desired speed of the drone in the forward direction. Based on this, the pitch angle command is derived: a positive vx corresponds to a forward pitch angle (e.g., 2 degrees), and adjusting the pitch angle causes the drone to move forward; vy represents the desired speed of the drone in the right direction, and a negative vy corresponds to a left roll angle (e.g., -1 degree), and adjusting the roll angle causes the drone to move to the left; vz represents the desired speed of the drone in the vertical direction, and a positive vz corresponds to a vertical speed command (e.g., 0.3 m / s), which directly controls the motor output to make the drone ascend.

[0071] By analyzing the data, the saturation velocity is converted into attitude and velocity commands that the UAV can execute, thus enabling the motion adjustments required for obstacle avoidance.

[0072] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception, characterized in that, include: The data acquisition module is used to simultaneously acquire the left and right images; The data processing module is used to perform frequency domain analysis based on the row grayscale mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row spacing. The time difference analysis module is used to perform cross-correlation calculations on the mean grayscale values ​​of the left and right images to determine the acquisition bias. The interference marking module is used to perform phase positioning according to the stripe row spacing, determine the stripe reference row, and construct a row-level mask by cyclically calibrating according to the stripe row spacing based on the stripe reference row. The disparity correction module is used to perform block calculations on the left and right images to obtain the original disparity, and then perform correction processing on the original disparity based on the row-level mask to obtain the corrected disparity. The confidence calculation module is used to assign confidence to each pixel based on the row-level mask, and to generate occupancy probability by dividing the corrected parallax into point cloud and then dividing it into voxels. The drone obstacle avoidance module is used to construct an obstacle potential field based on the occupancy probability and output control commands based on the obstacle potential field to complete the drone obstacle avoidance.

2. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 1, characterized in that, Frequency domain analysis was performed on the row gray-level mean sequence of the left image to obtain the pulse width modulation period and the corresponding stripe row spacing, including: Calculate the average gray level for each row of pixels in the left image to form a row gray level average sequence; The amplitude spectrum is obtained by performing a fast Fourier transform on the gray-level mean sequence. Determine the dominant frequency index ks of the amplitude spectrum, and calculate the stripe row spacing Δs = 1 / ks based on the dominant frequency index; The pulse width modulation period TP = Δs × Trow is calculated using the prior readout period Trow.

3. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 2, characterized in that, Perform cross-correlation on the mean grayscale values ​​of the left and right images to determine the acquisition bias, including: Calculate the cross-correlation value between the row gray-level mean sequence element y of the left image and the row gray-level mean sequence element y of the right image under the row offset δ; Determine the range of values ​​for the row offset δ, iterate through the values ​​of the row offset δ, and find the row offset δ that results in the largest cross-correlation value as the target row offset δ*. The acquisition bias is calculated based on the target row offset δ*, as follows: in, This indicates a data acquisition bias.

4. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 3, characterized in that, Phase positioning is performed according to the stripe row spacing to determine the stripe reference row. Using the stripe reference row as a reference, cyclic calibration is performed according to the stripe row spacing to construct a row-level mask, including: Within the row index interval [1, Δs], calculate the local brightness contrast of candidate row yc; The candidate row corresponding to the maximum local brightness contrast is used as the stripe reference row yref; Calculate the modulus value for each row of pixels in the left image based on the stripe reference row; If the modulus of the y-th row pixel is less than the preset effective bandwidth of the stripe, then mark the y-th row pixel in the left image as 1; otherwise, mark the y-th row pixel in the left image as 0 to obtain the row-level mask.

5. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 4, characterized in that, The original disparity is calculated by performing block-based calculations on the left and right images, and then corrected based on row-level masks to obtain the corrected disparity, including: For pixel coordinates (x, y), perform block matching with a fixed window size of w×w between the left and right images to obtain the original disparity; The original disparity draw(x,y) is sign-flipped using a row-level mask to obtain the corrected disparity.

6. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 5, characterized in that, Confidence is assigned to each pixel based on the row-level mask, including: Calculate the row-level baseline confidence level R0(y) based on the row-level mask; Generate a disparity validity label V(x,y) based on corrected disparity; For pixel coordinates (x, y), a weighted fusion is performed based on row-level baseline confidence and disparity validity label to obtain the fused pixel confidence. The maximum value of the fused pixel confidence is determined, and the fused pixel confidence of each pixel coordinate is normalized based on the maximum value to obtain the confidence Rref(x,y) of each pixel coordinate.

7. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 6, characterized in that, After correcting the parallax projection into a point cloud, occupancy probabilities are generated by voxel partitioning, including: Pixel depth is calculated from the corrected parallax. Generate a 3D point cloud P(x,y) based on pixel depth; Set the voxel side length svox and calculate the voxel index (a,b,c); Determine the confidence level and value S1 of the point cloud in each voxel, and the number of point clouds S2; The ratio of S1 to S2 is used as the occupancy probability of the corresponding voxel.

8. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 7, characterized in that, Constructing a barrier potential field based on occupancy probability, including: Determine the coordinates of the voxel center (Xc, Yc, Zc); The distance attenuation factor is determined based on the voxel center coordinates; The original potential energy is obtained by weighting the occupancy probability of each voxel according to the distance decay factor. The original potential energy is smoothed by three-dimensional Gaussian convolution to obtain a smoothed potential energy. The smooth potential energy of the voxels is binarized to obtain the barrier potential field Uobs.

9. The UAV adaptive intelligent navigation and obstacle avoidance system based on multimodal perception according to claim 8, characterized in that, The UAV completes obstacle avoidance based on the output control commands of the obstacle potential field, including: Potential energy Up is obtained by interpolation based on the spatial position of the drone; By combining the potential energy Up and the obstacle potential field Uobs, the potential energy gradient ∇Uobs is obtained using the three-dimensional finite difference method; Calculate the avoidance vector based on the potential energy gradient. ; The desired velocity is formed by fusing the avoidance vector and the initial trajectory velocity vector; The saturation velocity is obtained by saturating the amplitude of the desired velocity. The saturation velocity is resolved into pitch angle commands, roll angle commands, and vertical velocity commands, which are then processed by the UAV's flight control execution unit.

Images