A collision avoidance method and system for boarding bridges based on multi-sensor fusion

Abstract

A collision avoidance method for boarding bridges based on multi-sensor fusion is disclosed. Multiple sensors, including cameras, millimeter-wave radar, and ultrasonic radar, are installed around the moving wheels of the boarding bridge. The method includes: calibrating the spatial position consistency of the multiple sensors and collecting obstacle information around the boarding bridge; acquiring image data from the cameras and using the YOLOv5 algorithm to obtain obstacle category information from the image data; obtaining obstacle information based on the data from the millimeter-wave radar and ultrasonic radar, and the correspondence of coordinate systems; generating a dataset based on the obstacle information and dividing the dataset into a training set and a test set; constructing a backpropagation (BP) neural network algorithm to train the training set data and generate a collision risk assessment network; and inputting the test set data into the collision risk assessment network to obtain a result indicating whether a collision risk has occurred. This method improves the accuracy of collision avoidance for boarding bridges.

Description

Technical Field

[0001] This invention relates to the field of collision avoidance technology for boarding bridges, and in particular to a collision avoidance method and system for boarding bridges based on multi-sensor fusion. Background Technology

[0002] With the emergence of the "smart airport" concept, the intelligentization of airport facilities has become a new direction for airport development. Boarding bridges, as the link between the airport and aircraft, play a crucial role in airport operations. As the most convenient and efficient way to connect the terminal and flights, boarding bridges have become one of the busiest pieces of equipment in airports due to the increasing number of flights and the rapid expansion of passenger flow. Whenever a flight arrives or departs, the boarding bridge needs to be manually operated to dock with the aircraft door and evacuate from the door. This docking and evacuation process is becoming increasingly frequent with the increase in flights and passenger traffic, making it essential to ensure the safety of airport staff and passengers during the use of boarding bridges.

[0003] Currently, the main method of preventing dangers relies on manual visual supervision, which is inefficient and prone to false alarms and omissions. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a multi-sensor fusion-based boarding bridge collision avoidance method that monitors and reports various possible situations during the boarding bridge docking or disengagement process in real time, thereby improving the efficiency and accuracy of monitoring and early warning of dangerous situations and protecting the safety of boarding bridges, aircraft, and personnel.

[0005] To achieve the above objectives, the present invention provides a collision avoidance method for boarding bridges based on multi-sensor fusion. Multiple sensors, including cameras, millimeter-wave radar, and ultrasonic radar, are installed around the moving wheels of the boarding bridge. The method includes the following steps:

[0006] The spatial positional consistency of the multiple sensors is calibrated, and obstacle information around the boarding bridge is collected;

[0007] Acquire image data from the camera, and use the YOLOv5 algorithm to obtain the category information of obstacles in the image data;

[0008] Based on the data from the millimeter-wave radar and ultrasonic radar, and the correspondence of the coordinate system, the location, type, direction of movement, speed of movement, and distance information of the obstacle are obtained.

[0009] A dataset is generated based on the location, category, direction of movement, speed of movement, and distance information, and the dataset is divided into a training set and a test set according to a predetermined ratio.

[0010] A BP neural network algorithm is constructed to train the training set data and generate a collision risk judgment network.

[0011] The data from the test set is input into the collision risk assessment network to obtain the result of whether a collision risk has occurred.

[0012] Furthermore, the step of calibrating the spatial position consistency of the multiple sensors and collecting obstacle information around the boarding bridge also includes:

[0013] Let the radar coordinate system be O r -X r Y r Z r The camera coordinate system is O c -X c Y c Z c The world coordinate system is O W -X W Y W Z W The radar is installed at a height of H. The transformation relationship from a point (R, α) in the radar coordinate system to the world coordinate system is as follows:

[0014] X = R * sinα,

[0015] Y = H,

[0016] Z = Z r +R*cosα,

[0017] The distance between the radar and the camera on the Y-axis is H. If there is a point P(R, α) in the millimeter-wave radar coordinate system, how can its coordinates be transformed to the camera coordinate system (X, α) using coordinate system transformation? c Y c Z c )for:

[0018]

[0019] The camera has a focal length of f. If there exists a point P(X) in the three-dimensional world shown in the image... c Y c Z c The image data is captured by the camera and projected onto the image plane P′(x, y). The geometric relationship between the camera coordinate system and the image coordinate system is as follows:

[0020]

[0021] Furthermore, the obstacles include: personnel, vehicles, and pipelines connected to the aircraft.

[0022] Furthermore, the step of acquiring image data from the camera and using the YOLOv5 algorithm to obtain obstacle category information in the image data further includes:

[0023] Use image annotation tools to annotate obstacles in the image in YOLO format;

[0024] The labeled images are input into the YOLOv5 algorithm for training, which identifies obstacles and outputs the category information of the obstacles.

[0025] Furthermore, the step of obtaining the location, type, direction of movement, speed, and distance information of the obstacle based on the data from the millimeter-wave radar and the ultrasonic radar, and the correspondence of the coordinate systems, further includes:

[0026] The target information acquired by the millimeter-wave radar is read through the CAN protocol, and the location, speed, and direction of movement of different obstacles are associated with the category information through coordinate transformation.

[0027] Read ultrasonic radar data to obtain distance information for different obstacles.

[0028] Furthermore, the ratio of the training set to the test set is 8:2.

[0029] Furthermore, the step of training the training set data using the BP neural network algorithm to obtain the collision risk judgment network further includes:

[0030] The data in the training set is normalized.

[0031] Set the number of layers, the number of hidden neurons, and the corresponding transfer function of the BP neural network algorithm. Feed the test set into the BP neural network algorithm for iterative training and use the adaptive gradient descent method to adjust the learning rate.

[0032] The collision risk assessment network is obtained by taking the obstacle information, such as its type, distance, direction of movement, speed, and distance to nearby obstacles, as inputs and whether a collision risk occurs as outputs. After backpropagation of errors, the network is trained.

[0033] To achieve the above objectives, the present invention also provides a boarding bridge anti-collision system, employing the multi-sensor fusion-based boarding bridge anti-collision method described above, comprising:

[0034] The camera module uses multiple cameras to acquire images of obstacles around the boarding bridge and transmits them to the image processing module.

[0035] The millimeter-wave radar module uses multiple millimeter-wave radars to acquire information on the number, location, speed, and direction of movement of the obstacles from different locations, and transmits this information to the data processing module.

[0036] The ultrasonic radar module uses multiple ultrasonic radars to acquire distance information of the obstacle from different locations.

[0037] The image processing module identifies the obstacle image, annotates the image data in YOLO format, outputs the obstacle category information, and sends it to the fusion network module.

[0038] The data processing module reads the number, location, speed, and direction of obstacles sent by the millimeter-wave radar module via the CAN protocol. It then uses coordinate transformation to associate the location, speed, and direction of different obstacles with the obstacle category information. Finally, it reads the distance information of obstacles sent by the ultrasonic radar module and sends the category, location, speed, direction, and distance information to the collision risk assessment module.

[0039] The collision risk assessment module takes into account the category, orientation, speed, direction, and distance information, and uses a collision risk assessment network to determine whether there is a risk of collision with the corresponding obstacle, and sends an early warning command to the anti-collision warning module.

[0040] The collision avoidance warning module responds to the received warning instructions.

[0041] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program stored in the memory to implement the multi-sensor fusion-based boarding bridge collision avoidance method as described above.

[0042] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the multi-sensor fusion-based boarding bridge collision avoidance method as described above.

[0043] The multi-sensor fusion-based anti-collision method for boarding bridges provided by this invention has the following advantages compared with existing technologies:

[0044] Beneficial effects:

[0045] 1) This invention uses multiple sensors to collect on-site information of the boarding bridge. Compared with the traditional method of judging whether there is a risk of collision by relying on cameras and personnel observation, the perception of the environment is more comprehensive and specific, thereby reducing the probability of danger during the docking process of the boarding bridge.

[0046] 2) This invention uses artificial neural networks to achieve multi-sensor information fusion. By combining the multi-dimensional perception of environmental obstacle information from multiple sensors, it comprehensively judges whether a collision risk will occur and has a certain early warning function for potential risks.

[0047] 3) The sensor setup has a certain degree of redundancy, which greatly reduces the possibility of missed or false alarms, improves the accuracy of early warning, and prevents the occurrence of single sensor failure.

[0048] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 This is a flowchart of a multi-sensor fusion-based collision avoidance method for boarding bridges according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram showing the installation positions of multiple sensors according to an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the architecture of the YOLOv5 algorithm according to the present invention;

[0053] Figure 4 This is a schematic diagram of the collision risk assessment network structure according to an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the anti-collision system for boarding bridges according to an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of an electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0056] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0057] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0058] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0059] It should be noted that the concepts of "first" and "second" may be mentioned in this invention only to distinguish different devices, components or parts, and are not used to limit the order of the functions performed by these devices, components or parts or their interdependence.

[0060] It should be noted that the terms "one" and "multiple" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "Multiple" should be understood as two or more.

[0061] Figure 1 The following is a flowchart of a multi-sensor fusion-based collision avoidance method for boarding bridges according to an embodiment of the present invention, which will be referred to below. Figure 1 The embodiments of the present invention will be described in detail.

[0062] like Figure 1 As shown, the multi-sensor fusion-based boarding bridge collision avoidance method of the present invention includes the following steps:

[0063] In step S1, multiple sensors are installed around the moving wheels of the boarding bridge, and the spatial positional consistency of these sensors is calibrated.

[0064] In this embodiment of the invention, in order to achieve real-time monitoring of the environment around the moving wheels of the boarding bridge, multiple sensors are installed around them, including two cameras, two millimeter-wave radars and six ultrasonic radars. After installation, the spatial position consistency of these sensors is calibrated to ensure that the data information is in the same coordinate system, thereby accurately collecting relevant information about obstacles around the boarding bridge during its movement.

[0065] Figure 2 This is a schematic diagram of the installation positions of multiple sensors according to an embodiment of the present invention, such as... Figure 2 The diagram illustrates the installation positions of the cameras, millimeter-wave radar, and ultrasonic radar in an embodiment of the present invention. Specifically, a camera is installed at the front and rear of the section between the moving wheels 41 and 42; a millimeter-wave radar 2 is installed below each camera; and an ultrasonic radar is installed on the outer side and front and rear sides of the moving wheels 41 and 42, for a total of six ultrasonic radars. All the millimeter-wave radars and ultrasonic radars are at the same height. By installing these sensors at these key locations, obstacles around the boarding bridge can be comprehensively perceived, eliminating blind spots.

[0066] In this embodiment of the invention, the radar coordinate system and the pixel coordinate system are associated by transforming the radar coordinate system with the world coordinate system and the pixel coordinate system with the world coordinate system, and the coordinates of the radar target on the pixel plane are obtained by combining the relative position of the radar and the camera.

[0067] Let the radar coordinate system be O r -X r Y r Z r The camera coordinate system is O c -X c Y c Z c The world coordinate system is O W -X W Y W Z W The radar is installed at height H. The transformation relationship from a point (R, α) in the radar coordinate system to the world coordinate system is as follows:

[0068] X = R * sinα,

[0069] Y = H,

[0070] Z = Z r +R*cosα,

[0071] The distance between the radar and the camera on the Y-axis is H. If there is a point P(R, α) in the millimeter-wave radar coordinate system, how can its coordinates be transformed to the camera coordinate system (X, α) using coordinate system transformation? c Y c Z c )for:

[0072]

[0073] The camera has a focal length of f. If there exists a point P(X) in the three-dimensional world shown in the image... c Y c Z c The image data is captured by the camera and projected onto the image plane P′(x, y). The geometric relationship between the camera coordinate system and the image coordinate system is as follows:

[0074]

[0075] In step S2, the YOLOv5 algorithm is used to train the image data from the camera to obtain obstacle category information.

[0076] In this embodiment of the invention, the image data obtained by the camera is labeled in YOLO format using LabelImg (an image annotation tool). After labeling, the YOLOv5 algorithm is used for training, which enables real-time monitoring of surrounding obstacles.

[0077] Furthermore, in step S21, the on-site images of the airport apron captured by the camera are labeled. The labeled objects include all obstacles that may pose a collision risk with the boarding bridge, such as airport ground staff, special vehicles, and various pipelines connected to the aircraft.

[0078] In step S22, the labeled images are input into the YOLOv5 algorithm for training. After training, obstacles in the scene image can be identified in real time and their category information can be output.

[0079] Figure 3 This is a schematic diagram of the architecture of the YOLOv5 algorithm according to the present invention.

[0080] In step S3, obstacle information is obtained based on the data from multiple radars and the correspondence between coordinate systems.

[0081] In this embodiment of the invention, millimeter-wave radar data is mapped to specific obstacles identified by the camera through a coordinate system correspondence, obtaining the correspondence between the obstacle's location, category, and its direction and speed of movement. Ultrasonic radar data returns data containing distance information to the nearest obstacles around the moving wheels, offering high accuracy and a short detection range, serving as the last layer of protection for collision avoidance warnings.

[0082] Further, in step S31, the target information acquired by the millimeter-wave radar is read through the CAN protocol, including the number, orientation, speed and direction of the target objects. The orientation, speed and direction information of different target objects are transformed by coordinates and associated with the category information obtained in step S2.

[0083] Step S32: Read the ultrasonic radar data to obtain the precise distance information of obstacles around the walking wheels, and combine this information with the data obtained by the millimeter-wave radar to eliminate the blind spots that may exist between the millimeter-wave radar and the camera.

[0084] In step S4, a dataset is generated based on the obtained obstacle information, and the dataset is divided into a training set and a test set.

[0085] In this embodiment of the invention, after all sensors are installed and calibrated, data is collected on the airport tarmac. Different obstacles are used to simulate possible situations under real conditions, thereby obtaining image data and radar data in the same spatial coordinate system. The acquired image data and radar data are processed separately. The YOLOv5 algorithm is used to identify obstacles in the images. After obtaining the obstacle category, the data is combined with the radar data through coordinate system transformation to form a four-dimensional data set containing category, orientation, direction of movement, and speed of movement. This data set is then combined with the distance data of nearby obstacles from the microwave radar to form a five-dimensional data set, which serves as the dataset for subsequent network training.

[0086] In this embodiment of the invention, image data is combined with radar data, and spatial and category information of obstacles is associated through coordinate system mapping to form a set of obstacle feature data. The system then determines whether the obstacle poses a risk of collision with the boarding bridge. After multiple data collections, a dataset of feature data is obtained, and this dataset is divided into a training set and a test set according to a predetermined ratio. In one embodiment, the ratio of the training set to the test set is 8:2.

[0087] In step S5, a BP neural network algorithm is constructed to train the training set data and generate a collision risk judgment network.

[0088] In this embodiment of the invention, a BP neural network is constructed, with the obstacle information category, distance, direction of movement, speed of movement, and distance to nearby obstacles as five-dimensional inputs, and whether there is a risk of collision as a one-dimensional output. After training through error backpropagation, a collision risk judgment network is obtained.

[0089] Step S51: Normalize the data.

[0090] Step S52: Set the number of neural network layers, the number of hidden neurons, and the corresponding transfer function. Send the test set into the network for iterative training and use adaptive gradient descent to automatically adjust the learning rate.

[0091] The specific BP neural network algorithm is as follows:

[0092] A backpropagation (BP) neural network consists of three main functional layers: the input layer, hidden layers, and the output layer, from front to back. The input layer receives the input. After receiving the input, each input neuron weights and passes the value to the neurons in the hidden layers. The hidden neurons in the first hidden layer receive the value from the input neurons, sum it with their own threshold b, pass it through an activation function, and then weight it before passing it to the second hidden layer. The second hidden layer processes this similarly, and so on, until the last hidden layer, which then passes the value to the output layer. The output neurons sum the values ​​from all the hidden neurons with their own threshold b to obtain the output value.

[0093] Configure an input layer, a hidden layer, and an output layer, and set the hidden layer transfer function to the tansig function:

[0094]

[0095] The output layer is set to the purelin function:

[0096] y = x;

[0097] Assuming the number of nodes in the three layers of a BP neural network is [2, 3, 1], the mathematical expression of the BP neural network can be obtained from the model as follows:

[0098]

[0099] Where w represents the weight and b represents the threshold. This means that the weight is the weight from the second node in the second layer to the first node in the third layer. This means that the threshold is the threshold of the first node in the second layer.

[0100] For a three-layer BP neural network, its expression can be written in a general matrix form as follows:

[0101] f(x) = W (o) tansig(W (h) x+b (h) )+b (o) ,

[0102] Where W is the weight matrix, b is the threshold vector, and the superscripts (o) and (h) represent the output layer and hidden layer, respectively.

[0103] After determining the model expression, the mean squared error function between the network predicted values ​​and the actual sample values ​​can be obtained as follows:

[0104]

[0105] Where m is the number of training samples and k is the number of output samples. Let y be the predicted value of the k-th output of the i-th sample. ij This corresponds to the true value. After obtaining the mean squared error function, the process of minimizing the mean squared error function is the training process of the neural network.

[0106] To find the minimum value of the mean squared error function, gradient descent is typically used. The idea behind gradient descent is to first initialize an initial solution, and then continuously adjust x according to the gradient descent direction of the objective function L, eventually reaching a local optimum.

[0107] The gradient descent algorithm is as follows:

[0108] (1) First initialize W and b;

[0109] (2) Calculate the gradient according to the gradient formula;

[0110] (3) Adjust W and b towards the negative gradient direction;

[0111] (4) Repeat (1) and (2) until the termination condition is met.

[0112] The termination condition is: reaching the maximum number of iterations, or the error is small enough.

[0113] Based on the above network structure, the weight gradient of the output layer can be obtained:

[0114]

[0115] Output layer threshold gradient:

[0116]

[0117] Hidden layer weight gradient:

[0118]

[0119] Threshold gradient of the hidden layer Y:

[0120]

[0121] Where M and K are the number of samples and the number of outputs, respectively, and E km =f(x) km -y km Let A be the error of the k-th output of the m-th sample. im =tansig im Let x be the activation value of the i-th hidden node in the m-th sample. im This is the i-th input of the m-th sample.

[0122] Figure 4This is a schematic diagram of the collision risk assessment network structure according to an embodiment of the present invention, such as... Figure 4 As shown, the collision risk assessment network of the present invention has five input terminals, which respectively input category, distance, direction, speed, and distance; and one output terminal, which outputs a warning command.

[0123] In step S6, after training is completed, the data in the test set is fed into the trained collision risk judgment network to obtain the result of whether a collision risk has occurred.

[0124] In this embodiment of the invention, a collision avoidance method for boarding bridges based on multi-sensor information fusion is implemented. Compared with the prior art, its advantages are as follows:

[0125] 1) This invention uses multiple sensors to collect on-site information of the boarding bridge. Compared with the traditional method of judging whether there is a risk of collision by relying on cameras and personnel observation, the perception of the environment is more comprehensive and specific, thereby reducing the probability of danger during the docking process of the boarding bridge.

[0126] 2) This invention uses artificial neural networks to achieve multi-sensor information fusion. By combining the multi-dimensional perception of environmental obstacle information from multiple sensors, it comprehensively judges whether a collision risk will occur and has a certain early warning function for potential risks. Furthermore, the sensor settings have a certain degree of redundancy, which greatly reduces the possibility of missed or false alarms, improves the accuracy of early warning, and prevents the occurrence of single sensor failure.

[0127] In embodiments of the present invention, a boarding bridge anti-collision system based on multi-sensor fusion is also provided. Figure 5 This is a schematic diagram of the anti-collision system structure of a boarding bridge according to an embodiment of the present invention, as shown below. Figure 5 As shown, the multi-sensor fusion-based boarding bridge anti-collision system of the present invention includes: a camera module 501, a millimeter-wave radar module 502, an ultrasonic radar module 503, an image processing module 504, a data processing module 505, a collision risk judgment module 506, and an anti-collision warning output module 507.

[0128] The camera module 501 uses two cameras to acquire images of obstacles around the moving wheels of the boarding bridge and transmit them to the image processing module 504.

[0129] The millimeter-wave radar module 502 uses two millimeter-wave radars to obtain information such as the number, location, speed and direction of movement of obstacles from different positions around the moving wheels of the boarding bridge, and transmits it to the data processing module 505.

[0130] The ultrasonic radar module 503 uses six ultrasonic radars to obtain distance information of obstacles from different positions around the moving wheels of the boarding bridge.

[0131] The image processing module 504 identifies obstacle images, annotates the image data in YOLO format, outputs obstacle category information, and sends it to the collision risk assessment module 506.

[0132] The data processing module 505 reads information such as the number, location, speed, and direction of obstacles sent by the millimeter-wave radar module 502 via the CAN protocol, and associates the location, speed, and direction of different obstacles with the category information of the obstacles through coordinate transformation; it also reads the distance information of obstacles sent by the ultrasonic radar module 503; and sends the category, location, speed, direction, and distance information to the collision risk assessment module 506.

[0133] The collision risk assessment module 506 takes into account the type, location, speed, direction and distance information of the obstacle, uses the collision risk assessment network to determine whether there is a risk of collision with the corresponding obstacle, and sends an early warning command to the anti-collision warning module 507.

[0134] The collision avoidance warning module 507 responds to the received warning command.

[0135] In summary, this invention utilizes sensors installed at various key locations on the boarding bridge to monitor and report on various possible situations during the boarding bridge docking process in real time. Through multi-sensor data fusion, it achieves the function of monitoring and early warning of sudden dangerous situations, thereby protecting the safety of the boarding bridge, the aircraft, and all relevant personnel.

[0136] In embodiments of the present invention, an electronic device is also provided. Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, such as... Figure 6 As shown, the electronic device of the present invention includes a processor 601 and a memory 602, wherein,

[0137] The memory 602 stores a computer program, which, when read and executed by the processor 601, performs the steps described above in the embodiment of the multi-sensor fusion-based boarding bridge collision avoidance method.

[0138] In embodiments of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the steps in the embodiments of the multi-sensor fusion-based boarding bridge collision avoidance method described above when running.

[0139] In this embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0140] It will be understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A collision avoidance method for boarding bridges based on multi-sensor fusion, wherein multiple sensors, including cameras, millimeter-wave radar, and ultrasonic radar, are installed around the moving wheels of the boarding bridge, characterized in that... The method includes the following steps: The spatial positional consistency of the multiple sensors is calibrated, and obstacle information around the boarding bridge is collected; Acquire image data from the camera, and use the YOLOv5 algorithm to obtain the category information of obstacles in the image data; Based on the data from the millimeter-wave radar and ultrasonic radar, and the correspondence of the coordinate system, the location, type, direction of movement, speed of movement, and distance information of the obstacle are obtained. A dataset is generated based on the location, category, direction of movement, speed of movement, and distance information, and the dataset is divided into a training set and a test set according to a predetermined ratio. A BP neural network algorithm is constructed to train the training set data and generate a collision risk judgment network. The data from the test set is input into the collision risk assessment network to obtain the result of whether a collision risk has occurred.

2. The anti-collision method for boarding bridges based on multi-sensor fusion according to claim 1, characterized in that, The step of calibrating the spatial position consistency of the multiple sensors and collecting obstacle information around the boarding bridge further includes: Let the radar coordinate system be O r -X r Y r Z r The camera coordinate system is O c -X c Y c Z c The world coordinate system is O w -X w Y w Z w The radar is installed at a height of H. The transformation relationship from a point (R, α) in the radar coordinate system to the world coordinate system is as follows: X = R * sinα, Y = H, Z=Z r +R*cosα, The distance between the radar and the camera on the Y-axis is H. If there is a point P(R, α) in the millimeter-wave radar coordinate system, how can its coordinates be transformed to the camera coordinate system (X, α) using coordinate system transformation? c Y c Z c )for: The camera has a focal length of f. If there exists a point P(X) in the three-dimensional world shown in the image... c Y c Z c The image data is captured by the camera and projected onto the image plane P′(x, y). The geometric relationship between the camera coordinate system and the image coordinate system is as follows:

3. The anti-collision method for boarding bridges based on multi-sensor fusion according to claim 1, characterized in that, The obstacles include: personnel, vehicles, and pipelines connected to the aircraft.

4. The anti-collision method for boarding bridges based on multi-sensor fusion according to claim 1, characterized in that, The step of acquiring image data from the camera and using the YOLOv5 algorithm to obtain obstacle category information in the image data further includes: The obstacles in the image were annotated in YOLO format using image annotation software. The labeled images are input into the YOLOv5 algorithm for training, which identifies obstacles and outputs the category information of the obstacles.

5. The boarding bridge collision avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The step of obtaining the location, type, direction of movement, speed, and distance information of the obstacle based on the data from the millimeter-wave radar and ultrasonic radar, and the correspondence of the coordinate system, further includes: The target information acquired by the millimeter-wave radar is read through the CAN protocol, and the location, speed, and direction of movement of different obstacles are associated with the category information through coordinate transformation. Read ultrasonic radar data to obtain distance information for different obstacles.

6. The anti-collision method for boarding bridges based on multi-sensor fusion according to claim 1, characterized in that, The ratio of the training set to the test set is 8:

2.

7. The anti-collision method for boarding bridges based on multi-sensor fusion according to claim 1, characterized in that, The step of constructing a BP neural network algorithm to train the training set data and generate a collision risk judgment network further includes: The data in the training set is normalized. Set the number of layers, the number of hidden neurons, and the corresponding transfer function of the BP neural network algorithm. Feed the test set into the BP neural network algorithm for iterative training and use the adaptive gradient descent method to adjust the learning rate. The collision risk assessment network is obtained by taking the obstacle information, such as its type, distance, direction of movement, speed, and distance to nearby obstacles, as inputs and whether a collision risk occurs as outputs. After backpropagation of errors, the network is trained.

8. A boarding bridge collision avoidance system, employing the multi-sensor fusion-based boarding bridge collision avoidance method according to any one of claims 1-7, comprising: The camera module uses multiple cameras to acquire images of obstacles around the boarding bridge and transmits them to the image processing module. The millimeter-wave radar module uses multiple millimeter-wave radars to acquire information on the number, location, speed, and direction of movement of the obstacles from different locations, and transmits this information to the data processing module. The ultrasonic radar module uses multiple ultrasonic radars to acquire distance information of the obstacle from different locations. The image processing module identifies the obstacle image, annotates the image data in YOLO format, outputs the obstacle category information, and sends it to the fusion network module. The data processing module reads the number, location, speed, and direction of obstacles sent by the millimeter-wave radar module via the CAN protocol. It then uses coordinate transformation to associate the location, speed, and direction of different obstacles with the obstacle category information. Finally, it reads the distance information of obstacles sent by the ultrasonic radar module and sends the category, location, speed, direction, and distance information to the collision risk assessment module. The collision risk assessment module takes into account the category, orientation, speed, direction, and distance information, and uses a collision risk assessment network to determine whether there is a risk of collision with the corresponding obstacle, and sends an early warning command to the anti-collision warning module. The collision avoidance warning module responds to the received warning instructions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being configured to execute the computer program stored in the memory to implement the multi-sensor fusion-based boarding bridge collision avoidance method according to any one of claims 1-7.

10. A computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the multi-sensor fusion-based boarding bridge collision avoidance method according to any one of claims 1-7.

Images