Multi-mode perception fusion machine vision cooperative control system and method

Abstract

The invention discloses a multi-mode perception fusion machine vision cooperative control system and method, and relates to the technical field of vision cooperative control, and the system comprises a dynamic positioning module which is used for collecting the multi-dimensional data of an unmanned plane and an unmanned vehicle, the multi-dimensional data specifically comprises bandwidth signal data, visual image data, geomagnetic intensity data and auxiliary data, relative positioning between the unmanned aerial vehicle and the unmanned vehicle is achieved according to the collected data, and real-time position data of the unmanned aerial vehicle and the unmanned vehicle are obtained and transmitted; through accurate positioning and adjustment, the unmanned aerial vehicle can reach a designated position more accurately, distribution failure caused by positioning deviation is avoided, the success rate of logistics distribution is improved, when the similarity is high, the unmanned aerial vehicle directly returns to a safe height without adjustment, continuous flight of the unmanned aerial vehicle in an unsafe environment is effectively avoided, and the safety of logistics distribution is improved. The safety risks of collision, falling and the like are reduced, and the safety of the unmanned aerial vehicle and the container is protected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual cooperative control, more particularly to a multi-modal perception fusion machine vision cooperative control system and method. BACKGROUND

[0002] With the acceleration of urbanization and the booming development of e-commerce, the demand for urban logistics distribution is increasing, and the traditional logistics distribution mode has many pain points in the instant distribution scene, especially in the end distribution link. These problems seriously affect the distribution efficiency and service quality. In order to meet the existing distribution demand in the city, in the face of high-rise buildings, unmanned vehicles and unmanned aerial vehicles are used for precise distribution. In the distribution process, the unmanned aerial vehicle lands on the unmanned vehicle, first moves to the area close to the distribution location through the unmanned vehicle, and then the unmanned aerial vehicle takes off for further precise distribution. In the urban environment, high-rise buildings are everywhere, forming a so-called urban canyon effect, resulting in GPS signal shielding and reflection, and positioning accuracy is greatly reduced, with a drift of 3-5 meters. For unmanned aerial vehicles, accurate landing at the end is crucial, because only accurate landing at the designated location can ensure the safe delivery of the cargo box. However, the inaccuracy of GPS positioning makes it difficult for unmanned aerial vehicles to achieve accurate landing, increasing the risk and cost of distribution. The urban traffic conditions are complex and variable, and the parking position of the distribution vehicle also often changes. This dynamic change brings great difficulty to the take-off and landing of the unmanned aerial vehicle, because the unmanned aerial vehicle needs to adjust the flight path and landing attitude in real time according to the position of the vehicle. The traditional unmanned aerial vehicle control system is difficult to adapt to this rapidly changing environment, resulting in a high failure rate of take-off and landing and affecting the distribution efficiency. SUMMARY

[0003] To solve the above problems, the present application provides a multi-modal perception fusion machine vision cooperative control system and method.

[0004] The present application provides a multi-modal perception fusion machine vision cooperative control system, which comprises a dynamic positioning module, the dynamic positioning module is used for collecting multi-dimensional data of unmanned aerial vehicles and unmanned vehicles, the multi-dimensional data specifically comprises bandwidth signal data, visual image data, geomagnetic intensity data and auxiliary data, and the relative positioning between the unmanned aerial vehicle and the unmanned vehicle is realized according to the collected data, the real-time position data of the unmanned aerial vehicle and the unmanned vehicle is obtained, and is transmitted. A path planning module is used for receiving the multi-dimensional data collected by the dynamic positioning module and the real-time position data of the unmanned aerial vehicle and the unmanned vehicle, and judging whether the cargo box of the unmanned aerial vehicle landing gear is inclined. The path planning module is further configured to adjust the operation path of the UAV when the cargo box of the UAV landing gear is tilted, so that the cargo box is transported stably; and a coordination module configured to coordinate the UAV and the unmanned vehicle according to the multi-dimensional data of the dynamic positioning module and real-time position data of the UAV and the unmanned vehicle, and coordinate the operation parameters of the UAV and the unmanned vehicle when the cargo box of the UAV landing gear is tilted.

[0005] Preferably, the dynamic positioning module comprises a data acquisition unit, a data fusion unit and a positioning calculation unit. The data acquisition unit comprises positioning base stations arranged at four corner positions of the unmanned vehicle and arranged at a front face of the UAV, and a geomagnetic intensity detector installed at a bottom position of the UAV.

[0006] Preferably, the dynamic positioning module has the following specific working steps: A local coordinate system is established with the center of mass of the UAV as the origin, and the positioning base stations at the four corners of the unmanned vehicle are marked as , , and respectively, and the coordinates of the four positioning base stations are , , and respectively. The UAV emits signals as a signal source, and the positioning base stations , , and at the four corners of the unmanned vehicle receive the signals emitted by the UAV, and record the times , , and of arrival of the signals in sequence, and the relative distances between the UAV and the four base stations are calculated. The specific calculation formula is , and the relative distances between the UAV and the positioning base stations are obtained , wherein is the speed of light, is the time of arrival of the signal at the i-th positioning base station, is the time of arrival of the signal at the j-th base station, and specifically , and . Three equations are established according to the three time differences, and the position of the UAV is obtained . According to the above method, the position of the drone and the relative position coordinates of the unmanned vehicle and the drone are calculated every time period and transmitted as the real-time position data of the drone and the unmanned vehicle.

[0007] Preferably, the specific working steps of the dynamic positioning module also include the following: When unmanned vehicles and drones arrive at designated locations for delivery, a gridded geomagnetic map is constructed in the work area, and the three components of the geomagnetic intensity are recorded at each grid point. 、 and ;It should be noted that the range of the gridded geomagnetic map is 5m×5m; Specifically ; During the delivery flight of the drone, the geomagnetic intensity is measured in real time through the geomagnetic intensity detector. , where each measurement point The corresponding coordinates are the previously obtained position of the drone , real-time measurement of geomagnetic intensity Specifically include , respectively represent the north component, east component and vertical component of the geomagnetic intensity; Then according to the real-time position of the drone Find the geomagnetic map The corresponding points in ; Calculate the similarity between the geomagnetic data currently measured by the drone and the map data ,in Specifically include , respectively represent the north component, east component and vertical component of the geomagnetic intensity; It should be noted that the similarity S is specifically expressed by the formula Calculated; If the similarity Less than or equal to 25 , it is determined that the currently measured geomagnetic data matches the map data successfully; otherwise, it is determined that the match is unsuccessful, and a signal is transmitted to the path planning module for adjustment.

[0008] Preferably, the specific working steps of the dynamic positioning module also include the following: A pressure sensor array is placed on the drone landing gear, specifically a 16 by 16 grid, to monitor the pressure distribution in real time and obtain the pressure value of each grid point. ; Then according to the formula Calculate the center of mass offset of the cargo box on the UAV landing gear ,in For the a pressure value of the grid point, and is a coordinate of the i-th grid point; a centroid offset of the cargo box is determined, if the centroid offset of the cargo box is less than 0.02m, it is determined as level one, if the centroid offset of the cargo box is less than 0.05m and greater than 0.02m, it is determined as level two, if the centroid offset of the cargo box is greater than or equal to 0.05m, it is determined as level three; the determination result is transmitted as a signal to the path planning module.

[0009] Preferably, the specific steps of the path planning module include the following: receiving the signal of the dynamic positioning module that the matching is unsuccessful, further determining the similarity whether it is between 25 and 50 , if so, the lateral offset compensation of the UAV is calculated according to the formula: , wherein is a north component of the geomagnetic intensity measured by the UAV in real time, is a north component of the corresponding grid point in the geomagnetic map, is an east component of the geomagnetic intensity measured by the UAV in real time, is an east component of the geomagnetic intensity of the corresponding grid point in the geomagnetic map; the position of the UAV is adjusted according to the lateral offset compensation and the height adjustment of the UAV. if the similarity is not between 25 and 50 , it means that the similarity is greater than or equal to 50 , a control signal is generated to make the UAV return to a safe height.

[0010] Preferably, the specific steps of the path planning module further include the following: receiving the determination result from the dynamic positioning module, if the centroid offset of the cargo box ​​​​​If the center of mass of the container is offset If it is judged as level 2, the cargo box offset compensation is performed. The specific steps are: first, obtain the center of mass offset of the cargo box from the pressure sensor array ,include Quantity and y Quantity , construct the rotation matrix according to the yaw angle of the drone , according to the formula: ; Calculate the adjusted posture of the drone ,in is the initial posture of the drone, and the posture after adjustment of the drone Adjust the current drone's attitude; If the center of mass of the container is offset If it is judged as level 3, the drone will be controlled to spiral down. The specific trajectory is as follows: ; Calculate the spiral descent trajectory of the drone 、 and It should be noted that A function representing the change in the position of the drone on the x-axis over time. As time increases, the position of the drone on the x-axis gradually moves closer to the center point and oscillates at a frequency of 4π. Also need to follow the formula , get the maximum acceleration of the drone, and limit the acceleration of the drone during the spiral descent to no more than .

[0011] Preferably, the specific steps of the collaboration module include the following: The power of the drone is provided by four motors installed at the four corners, with the center of mass offset of the cargo box. When it is judged to be level 2 or level 3, the thrust of the four motors of the drone is redistributed. The specific steps include: first, according to the formula ; The thrust of the four motors is redistributed by calculating the thrust of the four motors of the drone, where 、 、 and is the thrust vector of the four motors, It is the pseudo-inverse matrix of the power distribution matrix, which is used to distribute the control instructions to the four motors. for Control force in the axial direction, for axial direction control force, is axial direction control force, is a compensation matrix, specifically including 0.2, 0.2, 0.1 and 0 in order to adjust the centroid offset of the container .

[0012] Preferably, the specific steps of the coordination module further include the following According to the state of the unmanned aerial vehicle, the speed adjustment strategy of the unmanned vehicle is formulated, specifically: ; level2 represents the centroid offset of the container is judged as level two, level3 represents the centroid offset of the container is judged as level three, level1 represents the centroid offset of the container is judged as level one, wherein is the initial speed of the unmanned vehicle, represents the adjusted speed of the unmanned vehicle; Then according to the formula get the angle adjustment value of the unmanned vehicle , adjust the driving angle according to the angle adjustment value of the unmanned vehicle, wherein is the offset of the centroid of the container in the axial direction, represents the offset of the centroid of the container in the axial direction, specifically obtained by the centroid offset of the container .

[0013] The application also proposes a cooperative take-off and landing control method of unmanned vehicle and unmanned aerial vehicle, comprising the following steps: Step one: real-time perception of the motion state of the unmanned vehicle, prediction of the landing trajectory of the unmanned aerial vehicle, dynamic adjustment of the flight path, triggering of lateral / longitudinal offset compensation or return to a safe height when the geomagnetic matching fails, to avoid collision risk caused by positioning drift; Step two: grid pressure sensors are arranged on the landing gear of the unmanned aerial vehicle, the centroid offset of the container is calculated in real time, and the centroid offset is judged by level; Step three: when the container is tilted, adjust the thrust of the four motors through the pseudo-inverse of the power distribution matrix to offset the influence of the centroid offset, adjust the speed and driving angle of the unmanned vehicle according to the state of the container, and avoid the container from overturning caused by sharp turning or high-speed driving.

[0014] Beneficial effects: through accurate positioning and adjustment, the unmanned aerial vehicle can more accurately reach the designated position, avoid delivery failure caused by positioning deviation, improve the success rate of logistics distribution, return to a safe height without adjustment when the similarity is high, effectively avoid the continuous flight of the unmanned aerial vehicle in the unsafe environment, reduce the safety risks such as collision and falling, protect the safety of the unmanned aerial vehicle and the cargo box, and the path planning strategy based on similarity judgment can make the unmanned aerial vehicle fly stably and reliably in the complex urban environment, improve the reliability and stability of the entire logistics distribution system. By adjusting the speed and angle of the unmanned vehicle according to the centroid offset of the cargo box, the safety accidents such as collision and overturning caused by high-speed driving or wrong driving of the unmanned vehicle when the cargo box is unstable can be effectively avoided, the safety of the unmanned vehicle and the cargo box is ensured, and the unmanned vehicle can better cooperate with the unmanned aerial vehicle. When the unmanned aerial vehicle is delivering or taking off, the unmanned vehicle can adjust its driving state in time according to the state of the cargo box, ensure the accurate docking and stable cooperation between the two, and improve the overall efficiency and reliability of logistics distribution. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of the system of the application. DETAILED DESCRIPTION

[0016] As shown in Figure 1 : a multi-modal perception fusion machine vision cooperative control system, comprising a dynamic positioning module, the dynamic positioning module is used for collecting multi-dimensional data of the unmanned aerial vehicle and the unmanned vehicle, the multi-dimensional data specifically includes bandwidth signal data, visual image data, geomagnetic intensity data and auxiliary data, and the relative positioning between the unmanned aerial vehicle and the unmanned vehicle is realized according to the collected data, the real-time position data of the unmanned aerial vehicle and the unmanned vehicle are obtained, and are transmitted; the relative positioning mainly provides high-precision position information for the precise landing of the unmanned aerial vehicle and the dynamic docking of the unmanned vehicle, through the fusion of multiple positioning technologies, the dynamic positioning module can determine the relative position and attitude of the unmanned aerial vehicle and the unmanned vehicle in real time and accurately in the complex urban environment, ensure that the two always maintain accurate relative position relationship in the cooperative working process, and realize efficient and reliable logistics distribution; The positioning object of the dynamic positioning module is the relative position of the unmanned aerial vehicle and the unmanned vehicle, the unmanned aerial vehicle needs to realize precise landing and take-off in the dynamic environment of the unmanned vehicle, and the unmanned vehicle needs to be adjusted in real time according to the position information of the unmanned aerial vehicle to ensure that the two can be docked smoothly, the dynamic positioning module provides the relative coordinates, speed and attitude of the unmanned aerial vehicle and the unmanned vehicle in real time through accurate measurement and calculation, and provides key data support for the cooperative work of the two; The path planning module is configured to receive the multi-dimensional data collected by the dynamic positioning module and real-time position data of the UAV and the unmanned vehicle, and determine whether the container of the UAV landing gear is tilted. The path planning module is further configured to adjust the operation path of the UAV when the container of the UAV landing gear is tilted, so that the container is transported stably. It should be noted that the path planning module can effectively receive multi-dimensional data and real-time position data, and determine the tilt state of the container, providing key support for the safe flight of the UAV, ensuring the stability of the container during transportation, reducing the risk of container damage, improving the reliability of transportation, timely adjusting the operation path of the UAV when the container is tilted, optimizing the flight route, avoiding possible obstacles and interference areas, ensuring the UAV to reach the destination smoothly, improving the success rate and efficiency of distribution, saving flight time and energy consumption. The coordination module is configured to cooperatively control the UAV and the unmanned vehicle according to the multi-dimensional data of the dynamic positioning module and the real-time position data of the UAV and the unmanned vehicle, and cooperatively control the operation parameters of the UAV and the unmanned vehicle when the container of the UAV landing gear is tilted. It should be noted that the coordination module cooperatively controls the UAV and the unmanned vehicle according to the multi-dimensional data and real-time position data, adjusts the operation parameters of both when the container is tilted, realizes the close cooperation of both in the process of logistics distribution, ensures the smooth handover of the container from the UAV to the unmanned vehicle, improves the efficiency and accuracy of distribution, and reduces the waiting time and operation complexity. It should be noted that in the urban environment, due to the influence of buildings, signal interference and other factors, the traditional positioning technology is difficult to realize the precise landing of the UAV, and the dynamic positioning module can realize the centimeter-level positioning accuracy in the complex urban environment by fusing ultra-wideband positioning, visual auxiliary positioning and geomagnetic anomaly matching and other technologies, During the driving process of the unmanned vehicle, its position and attitude will change constantly, which puts higher requirements on the positioning and landing of the UAV. The dynamic positioning module can sense the motion state of the unmanned vehicle in real time and adjust the positioning and flight path of the UAV in time according to the changes, so as to ensure that the UAV can still maintain accurate relative position in the dynamic environment of the unmanned vehicle, realize dynamic docking and cooperative work; In the complex urban environment, a single positioning technology often cannot meet the requirements of high precision and high stability. The dynamic positioning module can provide reliable and stable positioning information under various complex conditions by fusing multi-source data, combining the high precision of positioning, the real-time performance of visual positioning and the stability of geomagnetic matching, ensuring that the cooperative work of the UAV and the unmanned vehicle is not affected by environmental factors, and improving the overall performance and reliability of the system.

[0017] As an optional embodiment, the dynamic positioning module includes a data acquisition unit, a data fusion unit and a positioning solution unit. The data acquisition unit includes positioning base stations arranged at the four corner positions of the unmanned vehicle, arranged at the front of the unmanned aerial vehicle, and further includes a geomagnetic intensity detector installed at the bottom position of the unmanned aerial vehicle. It should be noted that the ultra-wideband positioning base station calculates the position of the unmanned aerial vehicle by transmitting and receiving UWB signals, realizes high-precision relative positioning, the geomagnetic intensity detector is composed of a 3-axis magnetometer, which is used to detect the geomagnetic intensity in real time, construct a geomagnetic intensity map, and realize high-precision positioning through geomagnetic anomaly matching technology, the inertial measurement device contains a gyroscope and an accelerometer, which is used to measure the acceleration and angular velocity of the unmanned aerial vehicle; The multi-source data of the unmanned vehicle and the unmanned aerial vehicle is collected to provide a basis for subsequent positioning and navigation.

[0018] As an optional embodiment, the specific working steps of the dynamic positioning module are as follows: A local coordinate system is established with the center of mass of the unmanned aerial vehicle as the origin, and the positioning base stations at the four corners of the unmanned vehicle are marked as , , and respectively, and the coordinates of the four positioning base stations are , , and ; The unmanned aerial vehicle acts as a signal source to emit signals, and the positioning base stations at the four corners of the unmanned vehicle , , and receive the signals emitted by the unmanned aerial vehicle, and record the times , , and of arrival of the signals in sequence, and calculate the relative distances between the unmanned aerial vehicle and the four base stations; The specific calculation formula is , and the relative distances between the unmanned aerial vehicle and the positioning base stations are obtained , wherein is the speed of light, is the time of arrival of the signal at the i-th positioning base station, is the time of arrival of the signal at the j-th base station, specifically , and ; According to the three time differences, three equations are established, and the position of the unmanned aerial vehicle is obtained ; According to the above manner, the position of the unmanned aerial vehicle and the relative position coordinates of the unmanned vehicle and the unmanned aerial vehicle are calculated every other time period, and are transmitted as real-time position data of the unmanned aerial vehicle and the unmanned vehicle. It should be noted that the above manner is mainly used to solve the problem of insufficient accuracy of the traditional GPS positioning technology in complex environments such as urban canyon effect, so as to ensure that the unmanned aerial vehicle can accurately land at the specified position of the unmanned vehicle, thereby improving the efficiency and safety of logistics distribution; The advantage of using positioning base station signals to determine the relative distance is that it does not rely on satellite signals, but directly receives the signals transmitted by the unmanned aerial vehicle through the ground base station to calculate the distance. This method can effectively avoid the shielding and reflection of signals by high-rise buildings, reduce positioning errors, and through multiple base stations receiving signals at the same time, multiple equations can be formed for solving; In view of the problem of GPS positioning accuracy decline caused by urban canyon effect, the dynamic positioning module using positioning base station signals to determine the relative distance has significant beneficial effects. It can provide high-precision positioning information for the unmanned aerial vehicle in the environment where GPS signal is limited, and ensure the accurate take-off and dynamic docking of the unmanned aerial vehicle in the city. This is crucial for logistics distribution, because only when the unmanned aerial vehicle accurately lands on the unmanned vehicle, the safety of the delivery of the cargo box can be ensured, and the failure or delay of the delivery caused by inaccurate positioning can be avoided, thereby reducing the risk and cost of delivery, and improving the efficiency and reliability of the entire logistics system.

[0019] As an optional embodiment, the specific working steps of the dynamic positioning module further include the following: It should be noted that the geomagnetic field is a stable natural physical field that is not affected by human interference and shielding, and can provide independent positioning reference to enhance the robustness of the positioning system. In long-time flight or complex environment, the error of the inertial navigation system will gradually accumulate, and the geomagnetic matching can correct these accumulated errors by matching with the geomagnetic reference map, thereby improving the navigation accuracy. Especially in the environment where satellite signals are limited or missing, such as urban canyons and tunnels, by installing a geomagnetic intensity detector, a geomagnetic intensity map is constructed and the geomagnetic intensity data measured by the unmanned aerial vehicle is matched in real time to calibrate the position of the unmanned aerial vehicle and improve the accuracy and reliability of the positioning; When the unmanned vehicle and the unmanned aerial vehicle arrive at the specified position for distribution, a grid geomagnetic map is constructed in the working area, and the three components of the geomagnetic intensity are recorded at each grid point 、 and ; It should be noted that the range of the grid geomagnetic map is 5m x 5m; Specifically ; In the process of distribution flight of the unmanned aerial vehicle, the geomagnetic intensity detector is used to measure the geomagnetic intensity in real time, where each measurement point The corresponding coordinates are the previously obtained position of the drone , real-time measurement of geomagnetic intensity Specifically include , respectively represent the north component, east component and vertical component of the geomagnetic intensity; Then according to the real-time position of the drone Find the geomagnetic map The corresponding points in ; Calculate the similarity between the geomagnetic data currently measured by the drone and the map data ,in Specifically include , respectively represent the north component, east component and vertical component of the geomagnetic intensity; It should be noted that the similarity S is specifically expressed by the formula Calculated; If the similarity Less than or equal to 25 , it is determined that the currently measured geomagnetic data matches the map data successfully; otherwise, it is determined that the match is unsuccessful, and the signal is transmitted to the path planning module for adjustment. It should be noted that by comparing the geomagnetic intensity measured by the drone in real time with the data in the geomagnetic map, the current position of the drone can be determined, thereby achieving high-precision positioning. When the match is unsuccessful, the signal is transmitted to the path planning module for adjustment, ensuring that the drone can adjust its flight path based on real-time geomagnetic information, avoid possible interference areas or obstacles, and ensure the safety and smoothness of flight. The matching of geomagnetic data can help the drone perceive changes in the surrounding environment, such as the influence of buildings and metal structures on geomagnetism, so as to better adapt to complex urban environments and improve the reliability of delivery. In the case of urban canyon effects causing GPS signals to be blocked and reflected, the use of geomagnetic matching technology can effectively improve the positioning accuracy of the drone, avoid delivery failures or delays caused by inaccurate GPS signals, and ensure that the cargo boxes can be delivered accurately and safely. By matching geomagnetic data with map data, the drone can fly and locate stably in complex and changing urban environments, reducing the risk of system failures caused by environmental factors and improving the reliability and stability of the entire logistics distribution system.

[0020] As an optional embodiment: the specific working steps of the dynamic positioning module also include the following: A pressure sensor array is placed on the drone landing gear, specifically a 16 by 16 grid, to monitor the pressure distribution in real time and obtain the pressure value of each grid point. ; Then according to the formula Calculate the center of mass offset of the cargo box on the UAV landing gear ,in For the The pressure value at each grid point, and is the coordinate of the i-th grid point; Offset to the center of mass of the container To judge, if the center of mass of the cargo box is offset If the center of mass of the cargo box is less than 0.02m, it is judged as level one. If the center of mass of the container is less than 0.05m and greater than 0.02m, it is judged as level 2. If it is greater than or equal to 0.05m, it is judged as level three; The judgment result is transmitted as a signal to the path planning module. It should be noted that by deploying a pressure sensor array on the drone landing gear, monitoring the pressure distribution in real time and calculating the center of mass offset of the cargo box, it is possible to achieve real-time monitoring and evaluation of the drone's cargo status. Specifically, the role and purpose of this judgment include obtaining the pressure distribution of the cargo box on the drone landing gear in real time through the pressure sensor array, being able to promptly detect whether the cargo box is offset or tilted during transportation, ensuring the stability and safety of the cargo box, and making graded judgments based on the size of the center of mass offset, so as to provide different levels of early warning information. Level 1 indicates that the cargo box is in good condition, level 2 indicates that attention is needed, and level 3 indicates that immediate action needs to be taken. This hierarchical warning mechanism helps to adjust the flight attitude or path in a timely manner to avoid flight accidents caused by cargo box deviation. The judgment results are transmitted to the path planning module, so that the drone can adjust the flight parameters such as flight attitude and speed according to the real-time status of the cargo box to ensure flight stability and safety and improve the success rate of delivery. By real-time monitoring of the center of mass deviation of the cargo box, the cargo box deviation problem can be discovered and handled in a timely manner to avoid the drone imbalance or fall caused by the instability of the cargo box, thereby ensuring the safety of the cargo box during transportation.

[0021] As an optional embodiment, the specific steps of the path planning module include the following: Receive the signal from the dynamic positioning module that the match is unsuccessful, and further determine the similarity Is it located at 25 and 50 Between, if yes, then according to the formula; ; Calculate the lateral offset compensation of the drone , longitudinal offset compensation and height adjustment ,in is the north component of the geomagnetic intensity measured by the drone in real time, is the north component of the corresponding grid point in the geomagnetic map, The east component of the geomagnetic intensity measured by the UAV in real time, The east component of the geomagnetic intensity corresponding to the grid point in the geomagnetic map; According to the lateral offset compensation , longitudinal offset compensation and height adjustment of the UAV, the position of the UAV is adjusted correspondingly; If the similarity is not between 25 and 50 , it means that the similarity is greater than or equal to 50 , and a control signal is generated to make the UAV return to a safe height. It should be noted that the safe height is set in advance according to the height of the delivery area, which is generally higher than the average height; It should be further pointed out that through accurate positioning and adjustment, the UAV can more accurately reach the designated position, avoid delivery failure caused by positioning deviation, and improve the success rate of logistics delivery. When the similarity is high, no adjustment is made and the UAV directly returns to the safe height, effectively avoiding the UAV continuing to fly in an unsafe environment, reducing the risk of collision, falling and other safety risks, and protecting the safety of the UAV and the cargo box. This path planning strategy based on similarity judgment can make the UAV fly stably and reliably in a complex and variable urban environment, improving the reliability and stability of the entire logistics delivery system.

[0022] As an optional embodiment, the specific steps of the path planning module further include the following: Receive the judgment result from the dynamic positioning module. If the centroid offset of the cargo box is judged to be level one, no adjustment is needed; If the centroid offset of the cargo box is judged to be level two, offset compensation of the cargo box is performed. The specific steps are as follows: first, obtain the centroid offset of the cargo box from the pressure sensor array, including the x component and the y component , construct a rotation matrix according to the yaw angle of the UAV, and calculate the adjusted attitude of the UAV according to the formula: ; , wherein is the initial attitude of the UAV, and the attitude of the current UAV is adjusted according to the adjusted attitude of the UAV ; If the centroid offset of the cargo box is judged to be level three, longitudinal offset compensation is performed. The specific steps are as follows: first, obtain the centroid offset of the cargo box from the pressure sensor array, includingIf the level is three, the unmanned aerial vehicle is controlled to spiral down, and the specific trajectory is: ; The spiral down trajectory of the unmanned aerial vehicle is calculated 、 and ; it should be noted that represents a function of the position of the unmanned aerial vehicle on the x-axis with respect to time, and as time increases, the position of the unmanned aerial vehicle on the x-axis gradually approaches the center point while oscillating at a frequency of 4π; It should be noted that represents a function of the position of the unmanned aerial vehicle on the y-axis with respect to time, and as time increases, the position of the unmanned aerial vehicle on the y-axis gradually approaches the center point while oscillating at a frequency of 4π; represents a function of the position of the unmanned aerial vehicle on the z-axis with respect to time. As time increases, the position of the unmanned aerial vehicle on the z-axis gradually decreases until it reaches the target height; wherein is the radius of the spiral down, is the total time of the spiral down, is the current time, 、 and are the initial positions of the unmanned aerial vehicle, which are obtained by the real-time position of the unmanned aerial vehicle before spiral down ; The maximum acceleration of the unmanned aerial vehicle is also obtained according to the formula ， , which limits the acceleration of the unmanned aerial vehicle during spiral down to . It should be noted that the initial attitude of the unmanned aerial vehicle is obtained by installing a gyroscope on the unmanned aerial vehicle. The scheme for adjusting the attitude of the unmanned aerial vehicle is as follows: by comparing the current attitude of the unmanned aerial vehicle with the target attitude, the deviation between the two is calculated by a PID control algorithm to obtain a suitable control amount to drive the motor to adjust the attitude of the unmanned aerial vehicle; The attitude of the unmanned aerial vehicle at each moment can be represented by the roll angle, pitch angle, and yaw angle. For each degree of freedom, it is a second-order system, and PID control is used for each of the three angles. For roll angle control, the attitude error signal and error rate of the multi-rotor unmanned aerial vehicle are first obtained, and then the control amount of each motor is obtained through the improved PID control algorithm and transmitted to the corresponding motor. The speed of the motor is changed to adjust the attitude of the unmanned aerial vehicle, and the attitude error is eliminated as much as possible, thereby forming a two-level closed-loop control.

[0023] As an optional embodiment, the specific steps of the coordination module include the following: ​The power of the UAV is provided by four motors installed at the four corners. When the offset of the center of mass of the cargo box is level 2 or level 3, the thrust of the four motors of the UAV is redistributed, including the following steps: first, according to the formula: ; The thrust of the four motors is redistributed, wherein 、 、 and is the thrust vector of the four motors, is the pseudo-inverse matrix of the power distribution matrix, which is used to distribute the control command to the four motors, is the control force in the axis direction, is the control force in the axis direction, is the control force in the axis direction, is the compensation matrix, which specifically includes 0.2, 0.2, 0.1 and 0 in order, for adjusting the impact of the offset of the center of mass of the cargo box . It should be noted that when the center of mass of the UAV is offset, the compensation term will adjust the control command to make the UAV generate additional force or torque to offset the impact of the offset, for example, if the center of mass is offset forward, the will increase, making the UAV tilt backward to restore balance.

[0024] As an optional embodiment, the specific steps of the coordination module further include the following According to the state of the UAV, the speed adjustment strategy of the unmanned vehicle is formulated, specifically: ; level 2 represents the offset of the center of mass of the cargo box is level 2, level 3 represents the offset of the center of mass of the cargo box is level 3, level 1 represents the offset of the center of mass of the cargo box is level 1, wherein is the initial speed of the unmanned vehicle, represents the adjusted speed of the unmanned vehicle; Then, according to the formula the angle adjustment value of the unmanned vehicle is obtained , and the driving angle is adjusted according to the angle adjustment value of the unmanned vehicle, wherein is the offset of the center of mass of the cargo box in the axis direction, indicates the offset of the center of mass of the cargo box in the axis direction, which is specifically calculated by the offset of the center of mass of the cargo box Obtained. It should be noted that by adjusting the speed and angle of the unmanned vehicle according to the centroid offset of the cargo box, collisions, rollovers and other safety accidents caused by high-speed driving or incorrect driving of the unmanned vehicle when the cargo box is unstable can be effectively avoided, the safety of the unmanned vehicle and the cargo box is ensured, and the unmanned vehicle can better cooperate with the unmanned aerial vehicle. When the unmanned aerial vehicle is flying or landing, the unmanned vehicle can adjust its driving state in time according to the state of the cargo box, ensure accurate docking and stable cooperation between the two, improve the overall efficiency and reliability of logistics distribution, and appropriate speed and angle adjustment can reduce the impact and shaking of the cargo box caused by unstable driving of the unmanned vehicle, reduce the risk of damage to the cargo box during transportation, and protect the integrity and safety of the cargo box. In the urban environment, the road conditions and the surrounding environment are complex and changeable, and through this flexible speed and angle adjustment strategy, the unmanned vehicle can better adapt to various complex situations and ensure stable driving and smooth completion of the cargo box distribution task in different environments.

[0025] The present application also provides a cooperative landing control method for unmanned vehicles and unmanned aerial vehicles, comprising the following steps: Step one: real-time perception of the motion state of the unmanned vehicle, prediction of the landing trajectory of the unmanned aerial vehicle, dynamic adjustment of the flight path, triggering of lateral / longitudinal offset compensation or return to a safe height when the geomagnetic matching fails, to avoid collision risks caused by positioning drift; Step two: arranging a 16x16 grid pressure sensor array on the landing gear of the unmanned aerial vehicle, real-time calculation of the centroid offset of the cargo box, and hierarchical determination of the centroid offset; Step three: when the cargo box is tilted, adjusting the thrust of the four motors through the pseudo-inverse of the power distribution matrix to offset the influence of the centroid offset, adjusting the speed and driving angle of the unmanned vehicle according to the state of the cargo box, and avoiding the overturning of the cargo box caused by sharp turns or high-speed driving.

[0026] Working principle: The dynamic positioning module interacts with the UWB signal of the unmanned aerial vehicle through the UWB base stations arranged at the four corners of the unmanned vehicle, adopts a time difference ranging algorithm to calculate the relative distance in real time, and fuses the geomagnetic matching technology; The module real-time perceives the motion state of the unmanned vehicle, predicts the landing trajectory of the unmanned aerial vehicle, dynamically compensates for the positioning deviation caused by building obstruction and multipath effect, triggers lateral / longitudinal offset compensation or controls the unmanned aerial vehicle to return to a safe height when the geomagnetic matching fails, to avoid collision risks caused by positioning drift; A 16x16 grid pressure sensor array is arranged on the landing gear of the unmanned aerial vehicle, the centroid offset of the cargo box is calculated in real time, and hierarchical determination is performed: first-order offset: the cargo box is stable and does not need to be adjusted; second-order offset: the attitude of the unmanned aerial vehicle is adjusted through a rotation matrix to compensate for the tilting of the cargo box; third-order offset: a spiral descent trajectory is triggered, and the acceleration is limited to be less than or equal to 5 m / s² to ensure safe landing; The cooperative module adopts a power distribution matrix pseudo-inverse to redistribute the thrust of the four motors of the unmanned aerial vehicle, offsets the influence of the centroid offset, adjusts the speed and driving angle of the unmanned vehicle according to the state of the cargo box, avoids the cargo box overturning caused by sharp turning or high-speed driving, ensures the system robustness through double-redundancy communication link to guarantee data synchronization, triggers hovering or emergency braking when the main link is interrupted, and ensures the system robustness. The scheme significantly improves the reliability and efficiency of urban logistics distribution through high-precision dynamic positioning, hierarchical response mechanism and multi-platform cooperative optimization.

[0027] The above is only the preferred embodiment of the present application, the protection scope of the present application is not limited to the above-mentioned examples only, any technical scheme belonging to the idea of the present application is within the protection scope of the present application. It should be pointed out that for ordinary technical staff in this technical field, some improvements and decorations without departing from the principle of the present application are also regarded as the protection scope of the present template.

Claims

1. A multimodal perception fusion machine vision collaborative control system, characterized by: It includes a dynamic positioning module, which is used to collect multi-dimensional data of the UAV and the unmanned vehicle, the multi-dimensional data specifically including bandwidth signal data, visual image data, geomagnetic intensity data and auxiliary data, and realize the relative positioning between the UAV and the unmanned vehicle based on the collected data, obtain the real-time position data of the UAV and the unmanned vehicle, and transmit it; A path planning module, which is used to receive the multi-dimensional data collected by the dynamic positioning module and the real-time position data of the UAV and the unmanned vehicle, and determine whether the cargo box of the UAV landing gear is tilted; The path planning module is also used to adjust the flight path of the drone when the cargo box of the drone landing gear is tilted, so that the cargo box is transported smoothly; A collaborative module is used to collaboratively control the drone and the unmanned vehicle based on the multi-dimensional data of the dynamic positioning module and the real-time position data of the drone and the unmanned vehicle, and to collaboratively control the operating parameters of the unmanned vehicle and the drone when the cargo box of the drone landing gear is tilted.

2. The multimodal perception fusion machine vision collaborative control system according to claim 1, characterized in that: The dynamic positioning module includes a data acquisition unit, a data fusion unit and a positioning solution unit: The data acquisition unit includes a positioning base station, which is arranged at the four corners of the unmanned vehicle and the front of the drone, and also includes a geomagnetic intensity detector, which is installed at the bottom of the drone.

3. The multimodal perception fusion machine vision collaborative control system according to claim 2, characterized in that: The specific working steps of the dynamic positioning module are as follows: With the center of mass of the UAV as the origin, a local coordinate system is established, and the positioning base stations at the four corners of the UAV are marked as 、 、 and , the coordinates of the four positioning base stations are 、 、 and ; Drone as signal source Signal, positioning base stations at the four corners of the unmanned vehicle 、 、 and Receive the signals transmitted by the drones respectively and record the arrival time of the signals in turn 、 、 and , calculate the relative distances between the drone and the four base stations; The specific calculation formula is: , calculate and obtain the relative distance between the drone and the positioning base station ,in is the speed of light, is the time when the signal reaches the i-th positioning base station, is the time it takes for the signal to reach the jth base station, Specifically 、 and ; According to the three time differences, three equations are established to solve the position of the drone. ; According to the above method, the position of the drone and the relative position coordinates of the unmanned vehicle and the drone are calculated every time period and transmitted as the real-time position data of the drone and the unmanned vehicle.

4. The multimodal perception fusion machine vision collaborative control system according to claim 3, characterized in that: The specific working steps of the dynamic positioning module also include the following: When unmanned vehicles and drones arrive at designated locations for delivery, a gridded geomagnetic map is constructed in the work area, and the three components of the geomagnetic intensity are recorded at each grid point. 、 and ;It should be noted that the range of the gridded geomagnetic map is 5m×5m; Specifically ; During the delivery flight of the drone, the geomagnetic intensity is measured in real time through the geomagnetic intensity detector. , where each measurement point The corresponding coordinates are the previously obtained position of the drone , real-time measurement of geomagnetic intensity Specifically include , respectively represent the north component, east component and vertical component of the geomagnetic intensity; Then according to the real-time position of the drone Find the geomagnetic map The corresponding points in ; Calculate the similarity between the geomagnetic data currently measured by the drone and the map data ,in Specifically include , respectively represent the north component, east component and vertical component of the geomagnetic intensity; If the similarity Less than or equal to 25 , it is determined that the currently measured geomagnetic data matches the map data successfully; otherwise, it is determined that the match is unsuccessful, and a signal is transmitted to the path planning module for adjustment.

5. The multimodal perception fusion machine vision collaborative control system according to claim 3, characterized in that: The specific working steps of the dynamic positioning module also include the following: A pressure sensor array is placed on the drone landing gear, specifically a 16 by 16 grid, to monitor the pressure distribution in real time and obtain the pressure value of each grid point. ; Then according to the formula Calculate the center of mass offset of the cargo box on the UAV landing gear ,in For the The pressure value at each grid point, and is the coordinate of the i-th grid point; Offset to the center of mass of the container To judge, if the center of mass of the cargo box is offset If the center of mass of the cargo box is less than 0.02m, it is judged as level one. If the center of mass of the container is less than 0.05m and greater than 0.02m, it is judged as level 2. If it is greater than or equal to 0.05m, it is judged as level three; The judgment result is transmitted as a signal to the path planning module.

6. The multimodal perception fusion machine vision collaborative control system according to claim 4, characterized in that: The specific steps of the path planning module include the following: Receive the signal from the dynamic positioning module that the match is unsuccessful, and further determine the similarity Is it located at 25 and 50 If so, the lateral offset compensation of the UAV is calculated. , longitudinal offset compensation and height adjustment ; Compensation for lateral deviation of the drone , longitudinal offset compensation and height adjustment , to adjust the position of the drone accordingly; If the similarity Not located at 25 and 50 The similarity between Greater than or equal to 50 , a control signal is generated to make the drone return to a safe altitude.

7. The multimodal perception fusion machine vision collaborative control system according to claim 5, characterized in that: The specific steps of the path planning module also include the following: Receive the judgment result from the dynamic positioning module, if the center of mass offset of the cargo box If it is judged as level one, no adjustment is required; If the center of mass of the container is offset If it is judged as level 2, the cargo box offset compensation is performed. The specific steps are: first, obtain the center of mass offset of the cargo box from the pressure sensor array ,include Quantity and y Quantity , construct the rotation matrix according to the yaw angle of the drone , Calculate the adjusted posture of the drone , according to the attitude of the drone after adjustment Adjust the current drone's attitude; If the center of mass of the container is offset If it is judged as level three, the drone will be controlled to spiral down; Calculate the spiral descent trajectory of the drone 、 and .

8. The multimodal perception fusion machine vision collaborative control system according to claim 7, characterized in that: The specific steps of the collaborative module include the following: The power of the drone is provided by four motors installed at the four corners, with the center of mass offset of the cargo box. When it is judged to be level 2 or level 3, the thrust of the four motors of the drone is redistributed. The specific steps include: first, according to the formula ； The thrust of the four motors is redistributed by calculating the thrust of the four motors of the drone, where 、 、 and is the thrust vector of the four motors, It is the pseudo-inverse matrix of the power distribution matrix, which is used to distribute the control instructions to the four motors. for Control force in the axial direction, for Control force in the axial direction, for Control force in the axial direction, The compensation matrix includes 0.2, 0.2, 0.1 and 0 in order to adjust the center of mass offset of the cargo box. impact.

9. The multimodal perception fusion machine vision collaborative control system according to claim 8, characterized in that: The specific steps of the collaborative module also include the following Formulate the speed adjustment strategy of the unmanned vehicle according to the status of the drone, specifically: ； Level 2 represents the center of mass offset of the cargo box Level 2, level 3 represents the center of mass offset of the cargo box It is judged as level 3, level 1 represents the center of mass offset of the cargo box It is judged as level one, is the initial speed of the unmanned vehicle, Represents the adjusted speed of the autonomous vehicle; Then according to the formula Get the angle adjustment value of the unmanned vehicle , adjust the driving angle according to the angle adjustment value of the unmanned vehicle, where The center of mass of the container is Axis offset, The center of mass of the container is Axis offset, specifically the offset of the center of mass of the container get.

10. A coordinated take-off and landing control method for an unmanned vehicle and a drone, applicable to a multimodal perception fusion machine vision coordinated control system as described in any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: Real-time perception of the UAV's motion status, prediction of the UAV's landing trajectory, and dynamic adjustment of the flight path. When geomagnetic matching fails, triggering lateral / longitudinal offset compensation or returning to a safe altitude to avoid collision risks caused by positioning drift. Step 2: Deploy grid pressure sensors on the drone landing gear to calculate the center of mass offset of the cargo box in real time and determine the center of mass offset in a graded manner; Step 3: When the cargo box tilts, the thrust of the four motors is adjusted through pseudo-inversion of the power distribution matrix to offset the impact of the center of mass offset. The speed and driving angle of the unmanned vehicle are adjusted according to the state of the cargo box to avoid the cargo box from overturning due to sharp turns or high-speed driving.

Images