Multi-view visual data processing method and system for rapid obstacle avoidance response

Abstract

This invention relates to the field of visual data processing and discloses a multi-view visual data processing method and system for rapid obstacle avoidance response. It is applicable to the algorithm supplementary identification of small obstacles in the air by UAVs, optimizing the obstacle avoidance accuracy and flight safety of UAVs. Based on the existing visual stereo matching algorithm for spatial obstacle identification, it further filters and identifies noise data that may originate from weak information of small objects by reprocessing the visual data to eliminate noise, thereby making a faster risk assessment of small objects and achieving safer obstacle avoidance control.

Description

Technical Field

[0001] This invention relates to the field of visual data processing, specifically to a multi-view visual data processing method and system for rapid obstacle avoidance response. Background Technology

[0002] In the field of drones, obstacle avoidance has always been a very important technology, and it is a fundamental capability to ensure the safe flight of unmanned aerial vehicles. The most mature and widely adopted implementation method in the current solution is a combination of radar, vision, or radar and vision. In this way, visual or sensor images of the drone's surrounding area are acquired, and obstacles in the environment are identified and judged to complete the autonomous flight control obstacle avoidance process.

[0003] However, in existing technologies, whether it is pure vision or a hybrid approach, the recognition rate of small obstacles such as power lines and kite strings during flight is still very low. The small visual features of these objects make them easy to overlook during data processing, ultimately leading to obstacle avoidance failure and flight accidents. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-view vision data processing method and system for rapid obstacle avoidance response, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: Multi-view vision data processing methods for fast obstacle avoidance response include: Real-time synchronized array of multi-view vision data, preprocessing the multi-view vision data, and spatial calibration and alignment based on corresponding information tags, the spatial calibration and alignment including UAV spatial calibration and sensor unit relative calibration; The disparity of the multi-view visual data after spatial calibration and alignment is calculated by using a stereo matching algorithm to establish three-dimensional point cloud data of spatial objects, and the data is mapped and synchronized in the spatial coordinate system of the current UAV. The three-dimensional point cloud data is used to characterize the distribution of obstacles in the flight space. An exclusion algorithm is executed based on continuous multi-view visual data and 3D point cloud data to obtain noise data frames, which are used to characterize the distribution of obstacles in the visual projection plane. The noise data frames are spatially mapped and stacked to establish noise spatial information. The noise spatial information is fitted based on a preset obstacle feature library, and noise obstacle objects are established in the spatial coordinate system according to the fitting results.

[0006] As a further aspect of the present invention, the step of spatial calibration and alignment based on corresponding information tags specifically includes: Execute the initialization program to control the UAV and multi-view sensor unit to perform calibration actions and acquire an array of initial visual data. Each of the initial visual data is associated with an information tag, which is used to characterize sensor unit information and UAV initial action nodes. In the multi-view sensing unit, a single sensing unit is randomly selected and marked as a calibration unit. Based on the initial visual data of the calibration unit, visual feature marking is performed, and calibration feature information of the corresponding array of continuous initial action nodes is established. The calibration feature information is used to characterize the correlation features of the fixed visual feature points in the calibration unit with the change of the UAV's initial action, that is, the spatial calibration relationship of the UAV. Based on the calibration feature information, visual relative calibration is performed on multiple sensor units of the non-calibrated unit to establish spatial relative association of multi-view sensor units. The spatial relative association includes the tilt angle and orientation of a single sensor unit as well as the spatial distribution characteristics among multi-view sensor units.

[0007] As a further aspect of the present invention: in the step of performing parallax calculation to establish three-dimensional point cloud data of spatial objects and mapping and synchronizing them within the current UAV's spatial coordinate system: The disparity calculation is used to characterize the process of calculating the disparity value of the same pixel object in multi-view vision data based on the stereo matching algorithm. The disparity value characterizes the depth information of the corresponding pixel in the spatial coordinate system, that is, the spatial distance with the sensing unit. The spatial coordinate system is used to characterize the flight motion space of the UAV. The spatial coordinate system includes a wide-area spatial coordinate system and a variable control coordinate system based on the UAV as the origin, which are used to store obstacle information in the space and control the movement of the UAV, respectively.

[0008] As a further embodiment of the present invention: the step of performing a rejection algorithm based on continuous multi-view visual data and 3D point cloud data to obtain noisy data frames specifically includes: Three-dimensional spatial anomaly removal processing is performed on the consistency mapping and anomaly removal of multi-view visual data from multiple camera units based on three-dimensional point cloud data to obtain non-object abnormal noise frames. The non-object abnormal noise frames are used to characterize the noise distribution of three-dimensional point cloud data that cannot be effectively consistent between visual data from multiple camera unit perspectives. Planar multi-frame continuous noise elimination involves stacking and eliminating continuous visual data from the same camera unit to obtain continuous noise frames. These continuous noise frames are used to characterize the noise distribution of a single camera unit during continuous acquisition. The calibration noise superposition method acquires an array of dark field images of the same camera unit and superimposes the array of dark field images to establish a calibration noise frame. The calibration noise frame is used to characterize the basic noise distribution of the camera unit under the corresponding operating state. By calibrating noise frames, abnormal noise frames and continuous noise frames are superimposed to eliminate differences, resulting in noise data frames. These noise data frames are used to characterize the feature distribution of small objects.

[0009] As a further aspect of the present invention: the step of spatially mapping and stacking the noisy data frames to establish noise spatial information specifically includes: Noise data frames from multiple camera units are mapped in a spatial coordinate system. Clustering and fitting are performed on the signal distribution areas of the noise data frames to obtain connected and overlapping noise point cloud regions. Information is labeled based on multiple consecutive sets of noise point cloud regions to generate noise spatial information. The noise point cloud regions are used to characterize the distribution areas of small objects to be screened.

[0010] This invention aims to provide a multi-view vision data processing system for rapid obstacle avoidance response, comprising: The data synchronization module is used to synchronize array multi-view vision data in real time, preprocess the multi-view vision data, and perform spatial calibration and alignment based on the corresponding information tags. The spatial calibration and alignment includes UAV spatial calibration and sensor unit relative calibration. The parallax modeling module is used to perform parallax calculation on the spatially calibrated and aligned multi-view visual data through a stereo matching algorithm to establish three-dimensional point cloud data of spatial objects and map and synchronize it in the current UAV's spatial coordinate system. The three-dimensional point cloud data is used to characterize the distribution of obstacles in the flight space. The data rejection module is used to execute rejection algorithms based on continuous multi-view visual data and three-dimensional point cloud data to obtain noise data frames, which are used to characterize the distribution of obstacles in the visual projection plane. The noise stack module is used to perform spatial mapping and stacking of noise data frames to establish noise spatial information. It fits the noise spatial information based on a preset obstacle feature library and establishes noise obstacle objects in the spatial coordinate system according to the fitting results.

[0011] As a further aspect of the present invention: the data synchronization module includes: An initialization synchronization unit is used to execute an initialization program, control the UAV and multi-view sensor unit to perform calibration actions, and acquire an array of initial visual data. Each of the initial visual data is associated with an information tag, which is used to characterize sensor unit information and UAV initial action nodes. An initialization calibration unit is used to randomly select a single sensing unit in the multi-view sensing unit and mark it as a calibration unit. Based on the initial visual data of the calibration unit, visual feature marking is performed, and calibration feature information of corresponding array of continuous initial action nodes is established. The calibration feature information is used to characterize the correlation features of the fixed visual feature points in the calibration unit with the change of the UAV's initial action, that is, the spatial calibration relationship of the UAV. The calibration unit is used to perform visual relative calibration on multiple sensor units of the non-calibrated unit based on the calibration feature information, so as to establish the spatial relative association of the multi-view sensor units. The spatial relative association includes the tilt angle and orientation of a single sensor unit and the spatial distribution characteristics among the multi-view sensor units.

[0012] As a further embodiment of the present invention: in the disparity modeling module: The disparity calculation is used to characterize the process of calculating the disparity value of the same pixel object in multi-view vision data based on the stereo matching algorithm. The disparity value characterizes the depth information of the corresponding pixel in the spatial coordinate system, that is, the spatial distance with the sensing unit. The spatial coordinate system is used to characterize the flight motion space of the UAV. The spatial coordinate system includes a wide-area spatial coordinate system and a variable control coordinate system based on the UAV as the origin, which are used to store obstacle information in the space and control the movement of the UAV, respectively.

[0013] As a further embodiment of the present invention: the data rejection module includes: The spatial rejection unit is used for three-dimensional spatial rejection processing. Based on three-dimensional point cloud data, it performs consistency mapping rejection on multi-view visual data from multiple camera units to obtain non-object abnormal noise frames. The non-object abnormal noise frames are used to characterize the noise distribution of three-dimensional point cloud data that cannot be effectively consistent between visual data from multiple camera unit perspectives. A continuous noise elimination unit is used for continuous noise elimination of multiple frames in a planar manner. It stacks and eliminates continuous visual data from the same camera unit to obtain continuous noise frames. The continuous noise frames are used to characterize the noise distribution of a single camera unit in continuous acquisition. The dark field library building unit is used to calibrate noise superposition, acquire an array of dark field images of the same camera unit, and superimpose the array of dark field images to establish a calibration noise frame. The calibration noise frame is used to characterize the basic noise distribution of the camera unit under the corresponding operating state. The noise filtering unit is used to eliminate and superimpose abnormal noise frames and continuous noise frames by calibrating noise frames to obtain noise data frames, which are used to characterize the feature distribution of small objects.

[0014] As a further embodiment of the present invention: the noise stack module includes: The noise aggregation unit is used to map the noise data frames of multiple camera units in a spatial coordinate system, perform cluster fitting based on the signal distribution area of ​​the noise data frames to obtain connected and overlapping noise point cloud areas, and perform information labeling based on multiple consecutive groups of noise point cloud areas to generate noise spatial information. The noise point cloud areas are used to characterize the distribution area of ​​small objects to be screened.

[0015] Compared with the prior art, the beneficial effects of the present invention are: it is applicable to the supplementary algorithm recognition of small obstacles in the air by UAVs, optimizes the obstacle avoidance accuracy and flight safety of UAVs, and on the basis of the existing vision-based stereo matching algorithm for spatial obstacle recognition, it can further filter and identify noise data that may originate from weak information of small objects by reprocessing the noise in the visual data, and make risk judgments on small objects more quickly, thereby achieving safer obstacle avoidance control. Attached Figure Description

[0016] Figure 1 This is a flowchart of a multi-view vision data processing method for fast obstacle avoidance response.

[0017] Figure 2 This is a flowchart of the spatial calibration and alignment step in a multi-view vision data processing method for fast obstacle avoidance response.

[0018] Figure 3 This is a diagram showing the components of a multi-view vision data processing system used for rapid obstacle avoidance response. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0021] like Figure 1 The multi-view vision data processing method for rapid obstacle avoidance response, as provided in one embodiment of the present invention, includes the following steps: S10, Real-time synchronization of multi-view vision data array, preprocessing of the multi-view vision data, and spatial calibration and alignment based on corresponding information tags, the spatial calibration and alignment including UAV spatial calibration and sensor unit relative calibration; S20, the disparity calculation is performed on the multi-view vision data after spatial calibration and alignment using a stereo matching algorithm to establish three-dimensional point cloud data of spatial objects, and the data is mapped and synchronized in the current UAV's spatial coordinate system. The three-dimensional point cloud data is used to characterize the obstacle distribution in the flight space. S30, an exclusion algorithm is executed based on continuous multi-view visual data and three-dimensional point cloud data to obtain a noise data frame, which is used to characterize the distribution of obstacles in the visual projection plane. S40: Spatial mapping and stacking of noise data frames to establish noise spatial information; fitting of noise spatial information based on a preset obstacle feature library; and establishment of noise obstacle objects in the spatial coordinate system based on the fitting results.

[0022] This embodiment presents a multi-view vision data processing method for rapid obstacle avoidance response. It is applicable to supplementary algorithmic identification of small aerial obstacles by UAVs, optimizing obstacle avoidance accuracy and flight safety. Building upon existing vision-based stereo matching algorithms for spatial obstacle identification, this method reprocesses visual data to filter and identify noise data that may originate from weak information about small objects. This allows for faster risk assessment of small objects, leading to safer obstacle avoidance control. In the UAV field, obstacle avoidance is a crucial technology, a fundamental capability for ensuring safe flight (referring to UAVs capable of autonomous flight missions, not those requiring full remote human control). The most mature and widely adopted solutions currently are radar, vision, or a combination of radar and vision. This approach monitors the surrounding environment of the UAV... The system acquires visual or sensor images within a certain range to identify and judge obstacles in the environment in order to complete the obstacle avoidance process for autonomous flight control. However, existing recognition technologies may not be able to effectively identify small objects in the air, such as ropes and cables, leading to collisions. These small objects cannot be effectively used as features when calculating and building three-dimensional objects because the reflection characteristics of the images or radar are too small, resulting in missed or false judgments. Therefore, this embodiment adopts a method of calculating noise data by using a rejection algorithm on multi-view visual data based on existing obstacle avoidance technologies. The noise data that is ignored in the parallax calculation is comprehensively processed to establish a separate noise data processing flow. Noise frames are built by using multi-frame noise data from multi-view vision, and the distribution characteristics of the noise frames are fitted with the noise distribution characteristics of small objects established by training to determine the presence of small cable objects.

[0023] like Figure 2 As shown, in another preferred embodiment of the present invention, the step of spatial calibration and alignment based on corresponding information tags specifically includes: S11, execute the initialization program, control the UAV and multi-view sensor unit to perform calibration actions, and acquire an array of initial visual data. Each of the initial visual data is equipped with an information tag, and the information tag is used to characterize the sensor unit information and the initial action node of the UAV. S12, randomly select a single sensing unit in the multi-view sensing unit and mark it as a calibration unit, perform visual feature marking based on the initial visual data of the calibration unit, and establish calibration feature information of the corresponding array of continuous initial action nodes. The calibration feature information is used to characterize the correlation features of the fixed visual feature points in the calibration unit with the change of the UAV's initial action, that is, the spatial calibration relationship of the UAV. S13, based on the calibration feature information, perform visual relative calibration on multiple sensing units of the non-calibrated unit to establish spatial relative association of the multi-view sensing units. The spatial relative association includes the tilt angle and orientation of a single sensing unit as well as the spatial distribution characteristics among the multi-view sensing units.

[0024] In this embodiment, step S10 is further explained. The preprocessing of visual data from multiple sensors includes color balance, distortion and other image feature corrections to make the visual data of multiple sensors consistent, which facilitates the interaction and judgment between different visual data in subsequent use. The spatial calibration step is to obtain more accurate parallax calculation. Only on this basis can the 3D point cloud data and subsequent judgment of noisy frames be accurate. The spatial calibration alignment adopts a unified approach based on the controllable known motion of the UAV to multiple camera units. Specifically, the UAV executes a fixed motion mode, and then selects one of the sensor units as the basic object of calibration, and obtains its spatial and motion features in the picture during the UAV's movement (by selecting visual features, marking the feature objects in a certain picture). Then, the other camera units fit this spatial and motion features (i.e., calibration feature information) to zero out the calibration of several sensor units.

[0025] In another preferred embodiment of the present invention, the step of performing parallax calculation to establish three-dimensional point cloud data of spatial objects and mapping and synchronizing them within the current UAV's spatial coordinate system is as follows: The disparity calculation is used to characterize the process of calculating the disparity value of the same pixel object in multi-view vision data based on the stereo matching algorithm. The disparity value characterizes the depth information of the corresponding pixel in the spatial coordinate system, that is, the spatial distance with the sensing unit. The spatial coordinate system is used to characterize the flight motion space of the UAV. The spatial coordinate system includes a wide-area spatial coordinate system and a variable control coordinate system based on the UAV as the origin, which are used to store obstacle information in the space and control the movement of the UAV, respectively.

[0026] In this embodiment, disparity calculation is a common technique in the prior art for aligning and converting two-dimensional visual images into three-dimensional space. It is widely used in many fields such as scene reconstruction and obstacle avoidance. By calculating the disparity values ​​of corresponding pixels of the image data of the same object obtained from different viewpoints, the depth information of different pixels is obtained, that is, the spatial distance from the sensor. Thus, a dense three-dimensional point cloud representing the position and shape of the object is established in the spatial coordinate system. The wide-area spatial coordinate system here is the coordinate system used by the system for positioning and environmental storage. The UAV flies in the wide-area spatial coordinate system through the variable control coordinate system and maps the real-time acquired data in the wide-area spatial coordinate system through the mapping relationship between the coordinate systems.

[0027] In another preferred embodiment of the present invention, the step of performing a rejection algorithm based on continuous multi-view visual data and 3D point cloud data to obtain noisy data frames specifically includes: Three-dimensional spatial anomaly removal processing is performed on the consistency mapping and anomaly removal of multi-view visual data from multiple camera units based on three-dimensional point cloud data to obtain non-object abnormal noise frames. The non-object abnormal noise frames are used to characterize the noise distribution of three-dimensional point cloud data that cannot be effectively consistent between visual data from multiple camera unit perspectives. Planar multi-frame continuous noise elimination involves stacking and eliminating continuous visual data from the same camera unit to obtain continuous noise frames. These continuous noise frames are used to characterize the noise distribution of a single camera unit during continuous acquisition. The calibration noise superposition method acquires an array of dark field images of the same camera unit and superimposes the array of dark field images to establish a calibration noise frame. The calibration noise frame is used to characterize the basic noise distribution of the camera unit under the corresponding operating state. By calibrating noise frames, abnormal noise frames and continuous noise frames are superimposed to eliminate differences, resulting in noise data frames. These noise data frames are used to characterize the feature distribution of small objects.

[0028] In this embodiment, the principle of the three-dimensional spatial rejection processing is based on the inverse process of disparity calculation. The three-dimensional point cloud data is calculated from the feature pixels of the shared objects of multiple sensing units. Therefore, by using this common result, the remaining pixel information of a certain sensing unit can be calculated in reverse. This information that cannot be converted into three-dimensional point cloud data is usually considered noise data, thus obtaining corresponding abnormal noise frames. Useful object information may exist in these noise frames. For example, when a thin cable is directly facing sensing unit A, although the effective information in the visual image may be less and discontinuous, there should still be some feature image information. However, in sensing unit B, due to the change in angle, there may actually only be the area of ​​a few pixels, which may not be effectively used during the acquisition and processing. Therefore, in disparity calculation, the cable... Objects are ignored and not displayed in the 3D point cloud data. However, after reverse rejection processing, noise data containing this information can be included. Planar multi-frame continuous rejection is used to calculate the noise changes of the same sensing unit in continuous image acquisition. This change may be reflected in the change of the coverage area of ​​the small amount of useful information actually collected for the same small object in continuous image acquisition. For example, the movement of the actual effective information acquisition area caused by changes in the viewing angle and lighting environment in multiple frames. Based on the acquisition of abnormal noise frames and continuous noise frames, these noises are not mostly effective. They also include some inherent noise generated by the sensor itself. These noises are generated by the circuit itself. Therefore, although they are random, they still have a certain distribution pattern in the overall image. Therefore, a subtraction rejection method can be used to build a sensor dark field image library.

[0029] In another preferred embodiment of the present invention, the step of spatially mapping and stacking the noisy data frames to establish noise spatial information specifically includes: Noise data frames from multiple camera units are mapped in a spatial coordinate system. Clustering and fitting are performed on the signal distribution areas of the noise data frames to obtain connected and overlapping noise point cloud regions. Information is labeled based on multiple consecutive sets of noise point cloud regions to generate noise spatial information. The noise point cloud regions are used to characterize the distribution areas of small objects to be screened.

[0030] In this embodiment, the method of superimposing noise is to use a spatial mapping stack. In the image, the position of each noise is determined, and the determined position is based on the sensor extending outward to determine a cone-shaped space. This space is the location of the noise object. By performing overlapping mapping in this way, the spatial location of small objects can be determined.

[0031] like Figure 3 As shown, the present invention also provides a multi-view vision data processing system for rapid obstacle avoidance response, comprising: The data synchronization module 100 is used to synchronize array multi-view vision data in real time, preprocess the multi-view vision data, and perform spatial calibration and alignment based on the corresponding information tags. The spatial calibration and alignment includes UAV spatial calibration and sensor unit relative calibration. The parallax modeling module 200 is used to perform parallax calculation on the spatially calibrated and aligned multi-view visual data through a stereo matching algorithm to establish three-dimensional point cloud data of spatial objects and map and synchronize it in the current UAV's spatial coordinate system. The three-dimensional point cloud data is used to characterize the distribution of obstacles in the flight space. The data rejection module 300 is used to execute a rejection algorithm based on continuous multi-view visual data and three-dimensional point cloud data to obtain noise data frames, which are used to characterize the distribution of obstacles in the visual projection plane. The noise stack module 400 is used to perform spatial mapping stacking on noise data frames to establish noise spatial information, fit the noise spatial information based on a preset obstacle feature library, and establish noise obstacle objects in the spatial coordinate system according to the fitting results.

[0032] In another preferred embodiment of the present invention, the data synchronization module includes: An initialization synchronization unit is used to execute an initialization program, control the UAV and multi-view sensor unit to perform calibration actions, and acquire an array of initial visual data. Each of the initial visual data is associated with an information tag, which is used to characterize sensor unit information and UAV initial action nodes. An initialization calibration unit is used to randomly select a single sensing unit in the multi-view sensing unit and mark it as a calibration unit. Based on the initial visual data of the calibration unit, visual feature marking is performed, and calibration feature information of corresponding array of continuous initial action nodes is established. The calibration feature information is used to characterize the correlation features of the fixed visual feature points in the calibration unit with the change of the UAV's initial action, that is, the spatial calibration relationship of the UAV. The calibration unit is used to perform visual relative calibration on multiple sensor units of the non-calibrated unit based on the calibration feature information, so as to establish the spatial relative association of the multi-view sensor units. The spatial relative association includes the tilt angle and orientation of a single sensor unit and the spatial distribution characteristics among the multi-view sensor units.

[0033] In another preferred embodiment of the present invention, the disparity modeling module includes: The disparity calculation is used to characterize the process of calculating the disparity value of the same pixel object in multi-view vision data based on the stereo matching algorithm. The disparity value characterizes the depth information of the corresponding pixel in the spatial coordinate system, that is, the spatial distance with the sensing unit. The spatial coordinate system is used to characterize the flight motion space of the UAV. The spatial coordinate system includes a wide-area spatial coordinate system and a variable control coordinate system based on the UAV as the origin, which are used to store obstacle information in the space and control the movement of the UAV, respectively.

[0034] In another preferred embodiment of the present invention, the data rejection module includes: The spatial rejection unit is used for three-dimensional spatial rejection processing. Based on three-dimensional point cloud data, it performs consistency mapping rejection on multi-view visual data from multiple camera units to obtain non-object abnormal noise frames. The non-object abnormal noise frames are used to characterize the noise distribution of three-dimensional point cloud data that cannot be effectively consistent between visual data from multiple camera unit perspectives. A continuous noise elimination unit is used for continuous noise elimination of multiple frames in a planar manner. It stacks and eliminates continuous visual data from the same camera unit to obtain continuous noise frames. The continuous noise frames are used to characterize the noise distribution of a single camera unit in continuous acquisition. The dark field library building unit is used to calibrate noise superposition, acquire an array of dark field images of the same camera unit, and superimpose the array of dark field images to establish a calibration noise frame. The calibration noise frame is used to characterize the basic noise distribution of the camera unit under the corresponding operating state. The noise filtering unit is used to eliminate and superimpose abnormal noise frames and continuous noise frames by calibrating noise frames to obtain noise data frames, which are used to characterize the feature distribution of small objects.

[0035] In another preferred embodiment of the present invention, the noise stack module includes: The noise aggregation unit is used to map the noise data frames of multiple camera units in a spatial coordinate system, perform cluster fitting based on the signal distribution area of ​​the noise data frames to obtain connected and overlapping noise point cloud areas, and perform information labeling based on multiple consecutive groups of noise point cloud areas to generate noise spatial information. The noise point cloud areas are used to characterize the distribution area of ​​small objects to be screened.

[0036] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0037] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0038] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A multi-view visual data processing method for fast obstacle avoidance response, characterized in that, Include: Real-time synchronized array of multi-view vision data, preprocessing the multi-view vision data, and spatial calibration and alignment based on corresponding information tags, the spatial calibration and alignment including UAV spatial calibration and sensor unit relative calibration; The disparity of the multi-view visual data after spatial calibration and alignment is calculated by using a stereo matching algorithm to establish three-dimensional point cloud data of spatial objects, and the data is mapped and synchronized in the spatial coordinate system of the current UAV. The three-dimensional point cloud data is used to characterize the distribution of obstacles in the flight space. An exclusion algorithm is executed based on continuous multi-view visual data and 3D point cloud data to obtain noise data frames, which are used to characterize the distribution of obstacles in the visual projection plane. The noise data frames are spatially mapped and stacked to establish noise spatial information. The noise spatial information is fitted based on a preset obstacle feature library, and noise obstacle objects are established in the spatial coordinate system according to the fitting results. The steps of executing the rejection algorithm based on continuous multi-view visual data and 3D point cloud data to obtain noisy data frames specifically include: Three-dimensional spatial anomaly removal processing is performed on the consistency mapping and anomaly removal of multi-view visual data from multiple camera units based on three-dimensional point cloud data to obtain non-object abnormal noise frames. The non-object abnormal noise frames are used to characterize the noise distribution of three-dimensional point cloud data that cannot be effectively consistent between visual data from multiple camera unit perspectives. Planar multi-frame continuous noise elimination involves stacking and eliminating continuous visual data from the same camera unit to obtain continuous noise frames. These continuous noise frames are used to characterize the noise distribution of a single camera unit during continuous acquisition. The calibration noise superposition method acquires an array of dark field images of the same camera unit and superimposes the array of dark field images to establish a calibration noise frame. The calibration noise frame is used to characterize the basic noise distribution of the camera unit under the corresponding operating state. By calibrating noise frames, abnormal noise frames and continuous noise frames are superimposed to eliminate differences, resulting in noise data frames. These noise data frames are used to characterize the feature distribution of small objects.

2. The multi-view visual data processing method for fast obstacle avoidance response according to claim 1, wherein, The steps for spatial calibration and alignment based on corresponding information tags specifically include: Execute the initialization program to control the UAV and multi-view sensor unit to perform calibration actions and acquire an array of initial visual data. Each of the initial visual data is associated with an information tag, which is used to characterize sensor unit information and UAV initial action nodes. In the multi-view sensing unit, a single sensing unit is randomly selected and marked as a calibration unit. Based on the initial visual data of the calibration unit, visual feature marking is performed, and calibration feature information of the corresponding array of continuous initial action nodes is established. The calibration feature information is used to characterize the correlation features of the fixed visual feature points in the calibration unit with the change of the UAV's initial action, that is, the spatial calibration relationship of the UAV. Based on the calibration feature information, visual relative calibration is performed on multiple sensor units of the non-calibrated unit to establish spatial relative association of multi-view sensor units. The spatial relative association includes the tilt angle and orientation of a single sensor unit as well as the spatial distribution characteristics among multi-view sensor units.

3. The multi-view visual data processing method for fast obstacle avoidance response according to claim 2, wherein, In the step of performing parallax calculation to establish three-dimensional point cloud data of spatial objects and mapping and synchronizing it within the current UAV's spatial coordinate system: The disparity calculation is used to characterize the process of calculating the disparity value of the same pixel object in multi-view vision data based on the stereo matching algorithm. The disparity value characterizes the depth information of the corresponding pixel in the spatial coordinate system, that is, the spatial distance with the sensing unit. The spatial coordinate system is used to characterize the flight motion space of the UAV. The spatial coordinate system includes a wide-area spatial coordinate system and a variable control coordinate system based on the UAV as the origin, which are used to store obstacle information in the space and control the movement of the UAV, respectively.

4. The multi-view vision data processing method for rapid obstacle avoidance response according to claim 1, characterized in that, The step of spatially mapping and stacking noisy data frames to establish noise spatial information specifically includes: Noise data frames from multiple camera units are mapped in a spatial coordinate system. Clustering and fitting are performed on the signal distribution areas of the noise data frames to obtain connected and overlapping noise point cloud regions. Information is labeled based on multiple consecutive sets of noise point cloud regions to generate noise spatial information. The noise point cloud regions are used to characterize the distribution areas of small objects to be screened.

5. A multi-view vision data processing system for rapid obstacle avoidance response, characterized in that, Include: The data synchronization module is used to synchronize array multi-view vision data in real time, preprocess the multi-view vision data, and perform spatial calibration and alignment based on the corresponding information tags. The spatial calibration and alignment includes UAV spatial calibration and sensor unit relative calibration. The parallax modeling module is used to perform parallax calculation on the spatially calibrated and aligned multi-view visual data through a stereo matching algorithm to establish three-dimensional point cloud data of spatial objects and map and synchronize it in the current UAV's spatial coordinate system. The three-dimensional point cloud data is used to characterize the distribution of obstacles in the flight space. The data rejection module is used to execute rejection algorithms based on continuous multi-view visual data and three-dimensional point cloud data to obtain noise data frames, which are used to characterize the distribution of obstacles in the visual projection plane. The noise stack module is used to perform spatial mapping and stacking of noise data frames to establish noise spatial information. It fits the noise spatial information based on a preset obstacle feature library and establishes noise obstacle objects in the spatial coordinate system according to the fitting results. The data rejection module includes: The spatial rejection unit is used for three-dimensional spatial rejection processing. Based on three-dimensional point cloud data, it performs consistency mapping rejection on multi-view visual data from multiple camera units to obtain non-object abnormal noise frames. The non-object abnormal noise frames are used to characterize the noise distribution of three-dimensional point cloud data that cannot be effectively consistent between visual data from multiple camera unit perspectives. A continuous noise elimination unit is used for continuous noise elimination of multiple frames in a planar manner. It stacks and eliminates continuous visual data from the same camera unit to obtain continuous noise frames. The continuous noise frames are used to characterize the noise distribution of a single camera unit in continuous acquisition. The dark field library building unit is used to calibrate noise superposition, acquire an array of dark field images of the same camera unit, and superimpose the array of dark field images to establish a calibration noise frame. The calibration noise frame is used to characterize the basic noise distribution of the camera unit under the corresponding operating state. The noise filtering unit is used to eliminate and superimpose abnormal noise frames and continuous noise frames by calibrating noise frames to obtain noise data frames, which are used to characterize the feature distribution of small objects.

6. The multi-view vision data processing system for rapid obstacle avoidance response according to claim 5, characterized in that, The data synchronization module includes: An initialization synchronization unit is used to execute an initialization program, control the UAV and multi-view sensor unit to perform calibration actions, and acquire an array of initial visual data. Each of the initial visual data is associated with an information tag, which is used to characterize sensor unit information and UAV initial action nodes. An initialization calibration unit is used to randomly select a single sensing unit in the multi-view sensing unit and mark it as a calibration unit. Based on the initial visual data of the calibration unit, visual feature marking is performed, and calibration feature information of corresponding array of continuous initial action nodes is established. The calibration feature information is used to characterize the correlation features of the fixed visual feature points in the calibration unit with the change of the UAV's initial action, that is, the spatial calibration relationship of the UAV. The calibration unit is used to perform visual relative calibration on multiple sensor units of the non-calibrated unit based on the calibration feature information, so as to establish the spatial relative association of the multi-view sensor units. The spatial relative association includes the tilt angle and orientation of a single sensor unit and the spatial distribution characteristics among the multi-view sensor units.

7. The multi-view vision data processing system for rapid obstacle avoidance response according to claim 6, characterized in that, In the disparity modeling module: The disparity calculation is used to characterize the process of calculating the disparity value of the same pixel object in multi-view vision data based on the stereo matching algorithm. The disparity value characterizes the depth information of the corresponding pixel in the spatial coordinate system, that is, the spatial distance with the sensing unit. The spatial coordinate system is used to characterize the flight motion space of the UAV. The spatial coordinate system includes a wide-area spatial coordinate system and a variable control coordinate system based on the UAV as the origin, which are used to store obstacle information in the space and control the movement of the UAV, respectively.

8. The multi-view vision data processing system for rapid obstacle avoidance response according to claim 5, characterized in that, The noise stack module includes: The noise aggregation unit is used to map the noise data frames of multiple camera units in a spatial coordinate system, perform cluster fitting based on the signal distribution area of ​​the noise data frames to obtain connected and overlapping noise point cloud areas, and perform information labeling based on multiple consecutive groups of noise point cloud areas to generate noise spatial information. The noise point cloud areas are used to characterize the distribution area of ​​small objects to be screened.

Images