Airport facility anomaly detection method and device based on multi-source feature fusion

Abstract

Embodiments of the present disclosure disclose an airport facility anomaly detection method and device based on multi-source feature fusion. A specific implementation of the method comprises: generating a current collection time period and a UAV planning path; controlling a target UAV to take pictures of an airport area to obtain a first image sequence and a second image sequence; determining a shooting area identifier corresponding to a second area image of the first image sequence and the second image sequence to obtain a shooting area identifier sequence; performing image reorganization processing on the first image sequence and the second image sequence to obtain a first area image pair sequence and a second area image pair sequence; and performing image recognition according to the first area image pair sequence and the second area image pair sequence to generate a facility anomaly detection result. The implementation can comprehensively detect the airport facilities of the airport area to be detected.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computer technology, the field of airport three-dimensional model construction, the field of airport facility anomaly identification, and in particular to an airport facility anomaly detection method and device based on multi-source feature fusion. BACKGROUND

[0002] The anomaly detection of airport facilities is conducive to ensuring the safe operation of the airport. At present, when performing airport facility anomaly detection, the commonly used method is to use a fixed camera on the ground to take pictures of the airport facilities to perform anomaly detection. However, the observation angle of the fixed camera on the ground is limited by the position of the camera, and there is a blind area for shooting, making it difficult to shoot the complete airport detection area, thereby resulting in that the range of the airport facility anomaly detection in the airport detection area is not comprehensive enough. SUMMARY

[0003] The summary part of the present disclosure is used to introduce the concepts in a brief form, which will be described in detail in the specific embodiment part. The summary part of the present disclosure is not intended to identify the key features or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of the present disclosure propose an airport facility anomaly detection method and device based on multi-source feature fusion to solve the technical problems mentioned in the background part.

[0005] In a first aspect, some embodiments of the present disclosure provide an airport facility anomaly detection method based on multi-source feature fusion, the method comprising: in response to obtaining a UAV calling instruction, generating a current collection time period and a UAV planned path according to a pre-established airport three-dimensional facility model and current airport flight information set; based on the current collection time period and the UAV planned path, controlling a target UAV to take pictures of the airport area to obtain a first image sequence and a second image sequence, wherein the first image is a temperature distribution map taken by a thermal infrared camera, the second image is an infrared image taken by an infrared camera, and the first image and the second image correspond one-to-one; determining the shooting area identifier corresponding to the second area image of the first image sequence and the second image sequence, to obtain a shooting area identifier sequence, wherein the shooting area identifier corresponds to at least one of the following: an airport road area, an airport vegetation area, and an airport electrical equipment area; based on the shooting area identifier sequence, performing image reorganization processing on the first image sequence and the second image sequence to obtain a first area image pair sequence and a second area image pair sequence; performing image recognition on the first area image pair sequence and the second area image pair sequence to generate a facility anomaly detection result, wherein the facility anomaly detection result includes a first facility anomaly information set, a second facility anomaly information set, and / or a third facility anomaly information set, the first facility anomaly information represents a road anomaly, the second facility anomaly information represents a vegetation anomaly, and the third facility anomaly information represents an electrical equipment anomaly.

[0006] In a second aspect, some embodiments of the present disclosure provide an airport facility anomaly detection device based on multi-source feature fusion, the device comprising: a generation unit configured to, in response to obtaining a UAV calling instruction, generate a current collection time period and a UAV planned path according to a pre-established airport three-dimensional facility model and a current airport flight information set; a shooting unit configured to control a target UAV to shoot an airport area based on the current collection time period and the UAV planned path to obtain a first image sequence and a second image sequence, wherein the first image is a temperature distribution map shot by a thermal infrared camera, the second image is an infrared image shot by an infrared camera, and the first image and the second image correspond one by one; a determination unit configured to determine a shooting area identifier corresponding to a second area image of the first image sequence and the second image sequence to obtain a shooting area identifier sequence, wherein the shooting area identifier corresponds to at least one of the following: an airport road area, an airport vegetation area, and an airport electrical equipment area; an image recombination processing unit configured to perform image recombination processing on the first image sequence and the second image sequence based on the shooting area identifier sequence to obtain a first area image pair sequence and a second area image pair sequence; and an image recognition unit configured to perform image recognition on the first area image pair sequence and the second area image pair sequence to generate a facility anomaly detection result, wherein the facility anomaly detection result comprises a first facility anomaly information set, a second facility anomaly information set, and / or a third facility anomaly information set, the first facility anomaly information set represents a road anomaly, the second facility anomaly information set represents a vegetation anomaly, and the third facility anomaly information set represents an electrical equipment anomaly.

[0007] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementations of the first aspect.

[0008] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method described in any of the implementations of the first aspect.

[0009] The above various embodiments of the present disclosure have the following beneficial effects: through the airport facility anomaly detection method based on multi-source feature fusion of some embodiments of the present disclosure, the airport facilities in the airport to be detected region can be comprehensively detected. Specifically, the reason why the airport facility anomaly detection in the airport to be detected region is not comprehensive enough is that the observation angle of the fixed camera on the ground is limited by the camera position, and there is a blind area, making it difficult to shoot the complete airport to be detected region. Based on this, the airport facility anomaly detection method based on multi-source feature fusion of some embodiments of the present disclosure, first, in response to obtaining the unmanned aerial vehicle calling instruction, generates the current collection time period and the unmanned aerial vehicle planning path according to the pre-established airport three-dimensional facility model and the current airport flight information set. Then, based on the above current collection time period and the above unmanned aerial vehicle planning path, the target unmanned aerial vehicle is controlled to shoot the airport region to obtain a first image sequence and a second image sequence, wherein the first image is a temperature distribution map shot by a thermal infrared camera, the second image is an infrared image shot by an infrared camera, and the first image and the second image correspond one by one. In practice, the fixed camera on the ground of the airport often only collects visible light images with color. When identifying abnormal situations such as airport road regions, vegetation regions, or electrical equipment, only the surface features can be identified from the visible light images with color, and it is difficult to detect potential safety hazards (for example, road hollow hazards) in advance. Therefore, the present application replaces the ground fixed camera scheme with a scheme of unmanned aerial vehicle carrying a thermal infrared camera and an infrared camera, so as to monitor all-weather according to the airport three-dimensional facility model and the flight information. Thus, not only can the movable characteristics of the unmanned aerial vehicle be used to shoot more comprehensive images, but also the thermal infrared camera and the infrared camera carried by the unmanned aerial vehicle can be used to shoot temperature distribution maps and infrared images simultaneously. Therefore, compared with commonly used visible light images, the temperature state and material characteristics (for example, the temperature state of the region where there is a road hollow is different from that of other normal road regions, and for example, the temperature state of the region where there is icing is different from that of the region where there is no icing) of the shooting region are highlighted, so as to detect potential risks. Then, the shooting region identifiers corresponding to the second region images of the first image sequence and the second image sequence are determined to obtain a shooting region identifier sequence, wherein the shooting region identifier corresponds to at least one of the following: an airport road region, an airport vegetation region, and an airport electrical equipment region. Then, based on the shooting region identifier sequence, the first image sequence and the second image sequence are subjected to image reorganization processing to obtain a first region image pair sequence and a second region image pair sequence. Here, by determining the shooting region identifier, the actual shooting region of the unmanned aerial vehicle can be represented, so that the image reorganization can be performed according to the airport road region, the airport vegetation region, and the airport electrical equipment region, thereby reducing the interference error of cross-region detection and improving the recognition accuracy.Finally, image recognition is performed on the first sequence of region image pairs and the second sequence of region image pairs to generate a facility anomaly detection result, wherein the facility anomaly detection result comprises a first facility anomaly information set, a second facility anomaly information set and / or a third facility anomaly information set, the first facility anomaly information set characterizes a road anomaly, the second facility anomaly information set characterizes a vegetation anomaly, and the third facility anomaly information set characterizes an electrical equipment anomaly. Thus, the airport facility anomaly detection can be more comprehensive, and because the temperature distribution feature is introduced, hidden facility anomalies can be detected. Furthermore, the airport facilities in the to-be-detected region of the airport can be comprehensively detected. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, aspects and advantages of various embodiments of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, similar or common elements are denoted by like reference numerals. It is to be understood that the drawings are schematic, and elements and features are not necessarily to scale.

[0011] Figure 1 is a flowchart of some embodiments of a multi-source feature fusion based airport facility anomaly detection method of some embodiments of the present disclosure;

[0012] Figure 2 is an initial detection route schematic diagram;

[0013] Figure 3 is a structure schematic diagram of a road facility detection model;

[0014] Figure 4 is an airport road icing anomaly scene diagram;

[0015] Figure 5 is a structure schematic diagram of some embodiments of a multi-source feature fusion based airport facility anomaly detection apparatus according to the present disclosure;

[0016] Figure 6 is a structure schematic diagram of an electronic device suitable for use to implement some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0018] In addition, it needs to be noted that only parts related to the present application are shown in the drawings for the convenience of description. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0019] It should be noted that the concepts of "first", "second" and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative and not restrictive, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0022] The present disclosure will be described in detail below with reference to the drawings and in conjunction with embodiments.

[0023] Figure 1 is a flowchart of some embodiments of the airport facility anomaly detection method based on multi-source feature fusion of some embodiments of the present disclosure. The airport facility anomaly detection method based on multi-source feature fusion includes the following steps:

[0024] Step 101, in response to acquiring the unmanned aerial vehicle calling instruction, generating the current collection time period and the unmanned aerial vehicle planning path according to the pre-established airport three-dimensional facility model and the current airport flight information set.

[0025] In some embodiments, the execution subject (e.g., a computing device) of the airport facility anomaly detection method based on multi-source feature fusion can generate a current collection time period and a UAV planning path according to a pre-established airport three-dimensional facility model and a current airport flight information set in response to obtaining a UAV calling instruction. Among them, a time period other than a preset flight peak period can be determined as an initial time period. Then, an idle time period with a duration exceeding a preset duration is selected from the initial time period as the current collection time period. Secondly, the A* algorithm can be used to perform grid route planning in the airport three-dimensional facility model with the starting position of the target UAV as the starting point and the ending position as the ending point to obtain the UAV planning path. Here, the grid of the grid route planning can be the shooting range of the target UAV at the target height, for example, the target UAV shoots at a height of 50 meters, and the plane at a height of 50 meters in the airport three-dimensional facility model is grid divided. Here, the size of each grid is the inscribed rectangle of the target UAV shooting area. Thus, the grid route planning can make the UAV shooting area fully cover the airport area. In practice, the airport three-dimensional facility model can be a preselected airport three-dimensional simulation model constructed according to the collected images. The target UAV is a UAV authenticated by the airport airspace management end.

[0026] It should be noted that the above computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the above-mentioned hardware devices. It can be implemented as, for example, multiple software or software modules for providing distributed services, or as a single software or software module. Herein, no specific limitation is made.

[0027] Optionally, before the above-mentioned execution subject generates the current collection time period and the UAV planning path according to the pre-established airport three-dimensional facility model and the current airport flight information in response to obtaining the UAV calling instruction, it further includes:

[0028] Step S1, in response to determining that the target time period arrives at the preset UAV collection period, issuing a UAV enabling instruction. Among them, the target time period is a time length for determining in advance whether the UAV can be enabled for airport area shooting, so as to perform advance authentication. For example, the target time period can be 1 hour.

[0029] Step S2, obtaining airport weather information from an airport weather station. Among them, the airport weather information can include wind power value, rainfall value, snowfall value, fog value, etc. of the airport after the target time period.

[0030] In step S3, the device selection identifier is determined by using the airport weather information. The device selection identifier represents at least one of a remote sensing satellite, a UAV or a ground monitoring device, and different airport weather information corresponds to different device selection identifiers.

[0031] In practice, considering that the UAV is disturbed by bad weather, when the wind value is greater than the preset wind threshold, the device selection identifier is determined as the identifier representing the remote sensing satellite and / or the identifier representing the ground monitoring device (i.e., the ground camera). For example, strong wind or thunderstorm weather can easily cause damage to the UAV, so the remote sensing satellite and / or the ground fixed camera can be selected for airport facility anomaly detection. When the rainfall value is greater than the preset rainfall threshold (i.e., rainy weather), the snowfall value is greater than the preset snowfall threshold (i.e., snowy weather), and the fog value is greater than the preset fog threshold (i.e., foggy weather), the ground fixed camera is easily disturbed by detection. Therefore, the UAV can be used for fine-grained patrol detection. Thus, the device damage rate is reduced, and the accuracy of anomaly detection is improved.

[0032] In step S4, in response to the device selection identifier representing the UAV, the UAV calling application information is sent to obtain the UAV calling instruction. The UAV calling application information can be used to apply to enable the UAV to perform anomaly detection and provide a UAV authentication request to the target terminal. When the UAV calling instruction is obtained, it can represent that the UAV authentication request is passed. The UAV calling instruction can include the UAV identifier corresponding to the target UAV.

[0033] Optionally, the current airport flight information includes a road identifier occupied and a corresponding road occupation time period.

[0034] In some optional implementations of some embodiments, the execution subject, in response to obtaining the UAV calling instruction, generates a current collection time period and a UAV planning path according to a pre-established airport three-dimensional facility model and the current airport flight information set, including:

[0035] In step S1, the road idle time periods corresponding to each road identifier in the airport are determined according to the road occupation time periods and the road identifiers occupied by each current airport flight information in the current airport flight information set, and a road idle time period set is obtained. The idle time length between the road occupation time periods corresponding to the same road identifier can be determined as the road idle time period.

[0036] In step S2, the road identifiers corresponding to the road idle time periods greater than the preset detection time length in the road idle time period set are determined as the detectable road identifiers, and a detectable road identifier set is obtained. Here, the road idle time periods less than the preset detection time length in the road idle time period set are eliminated, which can avoid the interference of the UAV flight on the aircraft take-off.

[0037] Step S3, for the detectable road sign in the above-mentioned detectable road sign set, the following processing steps are performed in the above-mentioned airport three-dimensional facility model:

[0038] Firstly, a detection route matching the airport road corresponding to each detectable road sign is selected from the preset detection route set as an initial detection route, thereby obtaining an initial detection route set, wherein the two ends of the initial detection route correspond to the first detection point and the second detection point. Here, the preset detection route set can be a route selected in the airport three-dimensional facility model according to the airport regional road structure in advance. In addition, one matching detection route can be selected as the initial detection route for each airport road. Specifically, the detection route is in the vegetation area in the airport region, thereby greatly avoiding the interference of the UAV route to the aircraft. Even if the UAV fails, it can also land in the vegetation area to avoid affecting the airport road. Secondly, the detectable road sign corresponds to the airport road. The vegetation area (i.e., the green isolation belt) on both sides of the airport road is the matching detection route.

[0039] Secondly, path planning is performed on each initial detection route in the initial detection route set to obtain an initial planning path. Wherein the initial planning path passes through the first observation point and the second observation point of each initial detection route. Here, first, the first observation point and the second observation point corresponding to each initial detection route in the airport three-dimensional facility model are determined. Then, the initial planning path is obtained by performing path planning on the starting point and the ending point of the target UAV and the first observation point and the second observation point corresponding to each initial detection route. Specifically, the initial planning path passes through each initial detection route and the corresponding first observation point and second observation point, and the first observation point and the second observation point of each initial detection route are adjacent in the initial planning path.

[0040] As an example, refer to the initial detection route schematic diagram shown in Figure 2 In the airport three-dimensional facility model in Figure 2 , a plurality of airport roads are shown. The initial detection route 201 (i.e., the red dotted line in Figure 2 ) is located at the center line position of the vegetation area on both sides of the airport road. The two ends of each initial detection route 201 correspond to the first detection point 202 and the second detection point 203 (i.e., the red dot in Figure 2 ). Here, the order of the first detection point 202 and the second detection point 203 is not specifically limited. In practice, the Dijkstra algorithm can be used to connect the head and tail of each initial detection route 201, thereby generating an initial planning path. In the initial planning path, the initial detection route 201 is not repeated planning, and the initial planning path passes through each first detection point 202 and second detection point 203.

[0041] In the third step, the detection time period of the road corresponding to each detectable road mark in the initial planning path is determined to obtain a detection time period sequence. The first timestamp and the second timestamp of the target UAV when reaching each first observation point and second observation point can be determined according to the predetermined flight speed of the target UAV and the initial planning path. Then, the time period between the first timestamp and the second timestamp corresponding to each detectable road mark can be determined as the detection time period.

[0042] In step S4, in response to the fact that there is no time conflict between the detection time period sequence and the road occupation time period included in the current airport flight information set in the current airport flight information set, the initial planning path is determined as the UAV planning path, and the current collection time period is generated using the road idle time period set, and the processing ends. For each detection time period, if there is no intersection between the detection time period and the road occupation time period after adding a safety time (for example, 20 minutes) before and after the detection time period, it is determined that there is no time conflict. Then, the union of each road idle time period can be determined as the current collection time period.

[0043] In practice, when detecting airport facility abnormalities, if the UAV path is planned without restrictions, it is easy to plan a UAV path that crosses the airport road multiple times. As a result, it is not only easy to cause signal interference to the aircraft taking off and landing, but also difficult to capture continuous road areas and vegetation areas, which leads to feature isolation and difficulty in splicing, affecting the accuracy of subsequent airport facility anomaly detection. Therefore, the above-mentioned embodiments of the present application first establish a detection route in the airport three-dimensional facility model, which can facilitate the selection of the initial detection route corresponding to the airport road. Here, by introducing the initial detection route and the first detection point and the second detection point at both ends, not only can the UAV path planning provide accurate route reference, compared with the path planning method of wireless route reference and coordinate reference, it greatly reduces the calculation consumption of path planning, improves the efficiency of path planning, and also can avoid the UAV planning path crossing the airport road as much as possible, reducing the spatial interaction between the UAV planning path and the airport road. Thus, it reduces the risk of suddenness and improves the emergency response capability. Finally, it can also ensure that the UAV takes pictures according to the road structure of the airport, so as to facilitate the identification of airport facility abnormalities. Further, it improves the accuracy of airport facility anomaly detection.

[0044] Optionally, the execution subject, in response to obtaining the UAV calling instruction, generates the current collection time period and the UAV planning path according to the pre-established airport three-dimensional facility model and the current airport flight information set, and further comprises:

[0045] Step S1, in response to detecting that there is a time conflict between the road occupation time period included in the current airport flight information set in the time period sequence, record the current execution round. Wherein, detecting that there is a time conflict between the road occupation time period included in the current airport flight information set in the time period sequence, it means that the current unmanned aerial vehicle planning path is used for shooting, which will conflict with the airport aircraft landing time period, so it is necessary to re-plan the path. Secondly, the current execution round is less than or equal to the preset round, and the above processing steps can be executed again.

[0046] In practice, re-planning the path can increase the priority of the initial detection route with time conflict, so as to prioritize planning the initial detection route. Thus, when the unmanned aerial vehicle shoots according to the planned route, the shooting of the route is completed preferentially, avoiding the influence on the safety of aircraft take-off and landing.

[0047] Step S2, in response to the current execution round being greater than the preset round, removing the detectable road identification corresponding to the detection time period with time conflict, and executing the above processing steps again. Wherein, the current execution round is greater than the preset round, which can represent the case that there is still a time conflict after multiple planning. Therefore, it can be determined that the available time of the initial detection route with time conflict is short, and in order to avoid the influence of unmanned aerial vehicle shooting on aircraft take-off and landing, the planning of the initial detection route can be removed. In addition, in order to further ensure the safety of aircraft take-off and landing, the initial detection routes adjacent to the initial detection route can also be removed.

[0048] Step 102, based on the current acquisition time period and the unmanned aerial vehicle planning path, control the target unmanned aerial vehicle to shoot the airport area to obtain a first image sequence and a second image sequence.

[0049] In some embodiments, the above execution subject can control the target unmanned aerial vehicle to shoot the airport area based on the current acquisition time period and the unmanned aerial vehicle planning path to obtain a first image sequence and a second image sequence. Wherein, the first image is a temperature distribution map shot by a thermal infrared camera, the second image is an infrared image shot by an infrared camera, and the first image and the second image correspond one by one. Secondly, the target unmanned aerial vehicle can be controlled to move according to the unmanned aerial vehicle planning path to shoot at a fixed height and a fixed downward angle to obtain a first image sequence and a second image sequence.

[0050] In practice, considering that the infrared camera can capture the difference in material reflection caused by the decrease in road base density, thereby reflecting the change in road material, and the thermal infrared camera can be used to capture the low temperature problem caused by the rapid heat dissipation of the road void, thereby reflecting the temperature change of the road, and the road void is often accompanied by road cracking, road settlement, and thus causes road waterlogging, road icing and other problems. At the same time, for the case of road foreign matter, the temperature difference between the foreign matter and the road can also be identified. Therefore, the thermal infrared camera and the infrared camera are introduced at the same time to take images of the airport area. In addition, the thermal infrared camera and the infrared camera are carried on the target unmanned aerial vehicle in the same direction and side by side through the gimbal. There is a pre-labeled positional relationship between the thermal infrared camera and the infrared camera.

[0051] Optionally, the execution subject based on the current collection time period and the unmanned aerial vehicle planning path controls the target unmanned aerial vehicle to take images of the airport area to obtain a first image sequence and a second image sequence, including the following steps:

[0052] Step S1, determining a sub-route corresponding to the initial detection route in the unmanned aerial vehicle planning path. In practice, since the initial detection sub-route is selected according to the adjacent route of the airport road, when the target unmanned aerial vehicle moves and takes pictures according to the unmanned aerial vehicle planning path, the vegetation area and the corresponding airport road where the unmanned aerial vehicle is located can be taken at the same time. Therefore, the length of the unmanned aerial vehicle planning path can be reduced when planning the path, so as to complete the shooting task more quickly and avoid conflicts with the aircraft take-off and landing period.

[0053] In addition, during the moving and shooting process of the target unmanned aerial vehicle, the unmanned aerial vehicle shooting wide angle can also be adjusted under the condition of ensuring the definition, and the images of multiple airport roads and multiple vegetation areas can be taken at the same time.

[0054] Step S2, for each sub-route, determining the shooting height and shooting angle of the target unmanned aerial vehicle. In order to take pictures of the vegetation area and the corresponding airport road where the unmanned aerial vehicle is located at the same time when the unmanned aerial vehicle moves and takes pictures, the shooting angle of the unmanned aerial vehicle needs to be adjusted according to the shooting height of the target unmanned aerial vehicle. Here, the shooting height of the target unmanned aerial vehicle corresponding to each position on the unmanned aerial vehicle planning path can be the same or different. Here, since the height limit of different areas in the airport is different, different shooting heights can be correspondingly set. Specifically, for each sub-route, the width of the corresponding airport road and vegetation area can be determined. Then, the width can be determined as the visual angle width of the unmanned aerial vehicle shooting. Finally, the angle of the coordinates on the sub-route to the center of the visual angle width can be taken as the shooting angle.

[0055] Step S3, using each shooting height and shooting angle, controlling the target unmanned aerial vehicle to shoot along the unmanned aerial vehicle planning path, and determining the temperature distribution map shot by the thermal infrared camera as the first image and the infrared image shot by the infrared camera as the second image, to obtain the first image sequence and the second image sequence.

[0056] In practice, when detecting airport facilities, the ground fixed camera often only carries a traditional optical camera for shooting, and it is difficult to extract more deep features. For example, for road detection, it is difficult to identify whether the ground is hollow through the traditional optical image. Therefore, by introducing a thermal infrared camera and an infrared camera, the present application can simultaneously shoot infrared images and temperature distribution images. Thus, it is convenient to detect whether the ground is abnormal in terms of the temperature state of the ground hollow area and the normal area, so as to detect potential safety hazards of airport facilities in advance and further improve the safety of the airport.

[0057] Step 103, determining the shooting area identifier corresponding to the second area image of the first image sequence and the second image sequence, to obtain a shooting area identifier sequence.

[0058] In some embodiments, the above execution subject can determine the shooting area identifier corresponding to the second area image of the first image sequence and the second image sequence, to obtain a shooting area identifier sequence. The shooting area identifier corresponds to at least one of the following: an airport road area, an airport vegetation area, and an airport electrical equipment area.

[0059] Here, first, the unmanned aerial vehicle coordinates corresponding to each group of first images and second area images can be converted to the coordinate system of the airport three-dimensional facility model to obtain converted unmanned aerial vehicle coordinates. Second, the corresponding airport simulation area in the airport three-dimensional facility model can be determined according to the same perspective of the target unmanned aerial vehicle shooting the first image and the second image. In practice, the airport three-dimensional facility model can be pre-marked according to the structure of the airport facility, and each facility mark has a corresponding facility identifier. For example, the airport road is marked with a road identifier and a corresponding boundary line, the vegetation area is marked with a vegetation identifier and a corresponding green belt boundary line, and the electrical equipment is marked with an equipment identifier and a corresponding area. Thus, the corresponding facility identifier in the airport simulation area can be determined as the shooting area identifier. For example, the airport simulation area corresponds to a road identifier and a vegetation identifier, so the road identifier and the vegetation identifier can be determined as the shooting area identifier.

[0060] Optionally, for the first image or the second image can be input to the pre-trained convolutional neural network, the facility identification and the corresponding semantic detection frame recognized from the image are output. For example, there is an electrical equipment arranged in a vegetation area, and the second image corresponds to a vegetation area identification and a device identification, and the vegetation area identification and the device identification can be determined as the corresponding shooting area identification.

[0061] In step 104, based on the shooting area identification sequence, the first image sequence and the second image sequence are subjected to image reorganization processing to obtain a first region image pair sequence and a second region image pair sequence.

[0062] In some embodiments, the above execution subject can perform image reorganization processing on the above first image sequence and the above second image sequence based on the above shooting area identification sequence to obtain a first region image pair sequence and a second region image pair sequence. Each first region image in the first region image pair corresponds to a shooting area identification representing an airport road area, and each second region image in the second region image pair corresponds to a shooting area identification representing an airport vegetation area. In addition, the first region image or the second region image can also correspond to a shooting area identification representing an airport electrical equipment area. Here, the first image including the same road identification can be determined as the first region image pair, and the second image including the same shooting area identification can be determined as the second region image pair. In practice, if the electrical equipment is arranged in the road area, the corresponding first region image pair has a corresponding shooting area identification representing the road and a shooting area identification representing the electrical equipment. If the electrical equipment is arranged in the vegetation area, the corresponding second region image pair has a corresponding shooting area identification representing the vegetation and a shooting area identification representing the electrical equipment.

[0063] In some optional implementations of some embodiments, the above execution subject performs image reorganization processing on the above first image sequence and the above second image sequence based on the above shooting area identification sequence to obtain a first region image pair sequence and a second region image pair sequence, including:

[0064] Step S1, according to the above-mentioned photographing region identification sequence, the above-mentioned second image sequence is segmented to obtain a first segmented sub-image sequence and a second segmented sub-image sequence. Wherein, the first segmented sub-image corresponds to the airport road region, and the second segmented sub-image corresponds to the airport vegetation region. Here, for each second image, the boundary line of the corresponding region can be selected from the airport three-dimensional facility model according to the unmanned aerial vehicle photographing direction, and then the boundary line is converted from the airport three-dimensional facility model to the camera coordinate system through coordinate conversion, and then projected into the second image. Thus, image segmentation can be performed according to the projected boundary line. After image segmentation, the sub-image representing the airport road region can be taken as the first segmented sub-image, and the sub-image representing the airport vegetation region can be taken as the second segmented sub-image. In addition, the corresponding boundary line can also be selected by matching a specific coordinate, that is, matching the position relationship between the target number of feature points in the second image and the feature points in the airport three-dimensional facility model, and then projected into the image to realize image segmentation.

[0065] In practice, in order to avoid the mutual influence of different region features, the traditional image recognition method is often to first perform image semantic recognition, then perform image segmentation, and finally perform image target recognition on the segmented region. However, due to the large size of the airport region and the large number of unmanned aerial vehicle images, directly performing image segmentation and image recognition on all the photographed images requires a large amount of computing resources. Therefore, for the airport region with obvious structural features, the application first introduces the structural features of the corresponding actual airport region in the airport three-dimensional facility model. At the same time, combined with a large number of fixed unmanned aerial vehicle photographing routes (i.e. initial detection routes) and photographing angles and other parameters, the consistency of the images photographed by the unmanned aerial vehicle is greatly ensured, so that the application can combine the actual structure of the introduced airport region, complete the image segmentation by matching + boundary line segmentation (or a small amount of image recognition + boundary line segmentation), and finally perform image equipment abnormality recognition. In comparison, the application significantly reduces the processing steps before image segmentation, thereby reducing the occupation of computing resources.

[0066] In addition, since the unmanned aerial vehicle photographing region is pre-set, the second image can be segmented into two sub-images according to the boundary line between the airport road region and the airport vegetation region as the first segmented sub-image and the second segmented sub-image. In addition, if the pre-set unmanned aerial vehicle photographing region is larger, it can be segmented into more sub-images, so that each sub-image region corresponds to a photographing region identification, and the sub-image is taken as the first segmented sub-image or the second segmented sub-image. Thus, direct cross-region recognition of the image is avoided, the interference error between different regions is reduced, and the accuracy of subsequent image recognition is improved.

[0067] Step S2, according to the segmentation boundary corresponding to the first segmented sub-image sequence and the second segmented sub-image sequence, synchronously segmenting the first image sequence to obtain a third segmented sub-image sequence and a fourth segmented sub-image sequence. Wherein, the third segmented sub-image corresponds to the airport road area, and the fourth segmented sub-image corresponds to the airport vegetation area. Here, since the first image and the second image are synchronously photographed, the corresponding image areas are the same, and therefore the image segmentation can be performed according to the same segmentation boundary.

[0068] Step S3, taking the corresponding first segmented sub-image and third segmented sub-image in the first segmented sub-image sequence and the third segmented sub-image sequence as a first region image pair to obtain a first region image pair sequence. Here, the first region image pair can include a temperature distribution map and an infrared image corresponding to the same shooting area (i.e. the airport road area).

[0069] Step S4, taking the corresponding second segmented sub-image and fourth segmented sub-image in the second segmented sub-image sequence and the fourth segmented sub-image sequence as a second region image pair to obtain a second region image pair sequence. Here, the second region image pair can include a temperature distribution map and an infrared image corresponding to the same shooting area (i.e. the airport vegetation area).

[0070] In practice, considering that the power equipment in the airport is relatively dispersed (for example, some power equipment is in the vegetation area, and some power equipment is in the road area), and the corresponding feature range of the power equipment is smaller relative to other areas, if separate region segmentation is performed, it is difficult to determine the equipment abnormality according to the overall characteristics. Therefore, without separate segmentation, the corresponding image (such as the first region image or the second region image) in the shooting area containing the power equipment can be used for power equipment abnormality identification. For example, the power equipment can include: transformer, cable bridge, air conditioning unit of terminal building, and aid navigation light equipment in the airport.

[0071] Step 105, performing image recognition according to the first region image pair sequence and the second region image pair sequence to generate a facility anomaly detection result.

[0072] In some embodiments, the above execution subject can perform image recognition according to the first region image pair sequence and the second region image pair sequence to generate a facility anomaly detection result. Wherein, the facility anomaly detection result includes a first facility anomaly information set, a second facility anomaly information set and / or a third facility anomaly information set, the first facility anomaly information represents road anomaly, the second facility anomaly information represents vegetation anomaly, and the third facility anomaly information represents power equipment anomaly.

[0073] Here, each coordinate and temperature value in the temperature distribution map in the first region image pair can be clustered by a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to generate a group of abnormal temperature center coordinates representing a coarse label of an abnormal temperature and a coordinate range corresponding to each abnormal temperature center coordinate. Here, the coarse label representing an abnormal temperature can be pre-set. Then, the coordinate range corresponding to each abnormal temperature center coordinate can be marked in the corresponding infrared image in the first region image pair to obtain a marked first image. And the temperature distribution map marked with the coordinate range corresponding to each abnormal temperature center coordinate is taken as a marked second image. Finally, each marked first image and each marked second image can be taken as a facility anomaly detection result.

[0074] In some optional implementations of some embodiments, the above execution subject performs image recognition on the above first region image pair sequence and the above second region image pair sequence to generate a facility anomaly detection result, including:

[0075] In step S1, road region anomaly detection is performed on each first region image pair in the above first region image pair sequence to generate first facility anomaly information, and a first facility anomaly information set is obtained. The first facility anomaly information includes a road anomaly detection label representing a road anomaly or a road normality and a corresponding road anomaly region. The anomaly detection label representing a road anomaly corresponds to at least one of the following road anomaly types: road hollow, road cracking, road waterlogging, road icing and road foreign matter. Here, each first facility anomaly detection information can include an anomaly detection label corresponding to one road anomaly type, a detection region and a corresponding detection confidence.

[0076] As an example, the five road anomaly types can be represented by numbers. For example, road hollow: "0", road cracking: "1", road waterlogging: "2", road icing: "3", and road foreign matter: "4".

[0077] Specifically, when performing anomaly detection on airport roads, for multiple abnormal situations (e.g., road hollow, road cracking, road waterlogging, road icing and road foreign matter) belonging to airport roads, if detection is performed separately, there is a high time cost, data processing redundancy and poor coordination (e.g., road hollow is usually accompanied by road cracking, and separate detection causes the feature of road cracking to be ignored while detecting the hollow). Therefore, for multiple abnormal situations belonging to airport roads, infrared images and thermal infrared images can be used simultaneously for anomaly recognition to improve feature correlation and detection efficiency.

[0078] For example, referring toFigure 3 A structural schematic diagram of the road facility detection model is shown. Each first region image pair input value can be input into the pre-trained road facility detection model to perform facility anomaly recognition to generate first facility anomaly information. Here, the road facility detection model can include the following modules: an input layer, a double-branch feature extraction layer, a cross-modal feature fusion layer, a multi-scale detection head, and an output layer.

[0079] Specifically, for the first segmentation sub-image and the third segmentation sub-image in the first region image pair, the input layer can first scale the image ratio of the first segmentation sub-image and the third segmentation sub-image to obtain a scaled first sub-image and a scaled third sub-image. For example, scaled to 640 × 640 size. Secondly, the pixel value of the scaled first sub-image can be normalized to the [0, 1] interval, and the temperature value of the scaled third sub-image can be normalized to the [0, 1] interval. In addition, the single channel of the scaled third sub-image can also be repeated to fill 3 channels, so as to facilitate the synchronization of the features of the two images.

[0080] The double-branch feature extraction layer can be used to extract texture, morphology, etc. features of the scaled first sub-image in parallel, and extract hierarchical features of the temperature distribution in the scaled third sub-image. The double-branch feature extraction layer can be composed of a first branch and a second branch. Specifically, the first branch is used for feature extraction on the scaled first sub-image, so the input of the first branch can be the scaled first sub-image with a scale of 640 × 640 × 3, and the output is three feature maps of different scales. Here, the first branch is composed of five groups of convolution modules and five groups of feature extraction modules connected alternately. Among them, a fast spatial pyramid pooling layer is added before the fifth group of feature extraction modules in order to increase the receptive field of the features and facilitate the detection of large-scale targets. The above three feature maps of different scales are generated by the third, fourth and fifth feature extraction modules, respectively. The third feature extraction module outputs a feature map with a scale of 80 × 80 × 256 to adapt to small targets (e.g., road cracks, smaller foreign objects, etc.). The fourth feature extraction module outputs a feature map with a scale of 40 × 40 × 512 to adapt to medium-sized abnormal targets (e.g., local road water accumulation, medium-sized foreign objects, etc.). The fifth feature extraction module outputs a feature map with a scale of 20 × 20 × 1024 to adapt to large detection targets (e.g., larger road hollow areas and large foreign objects). The second branch is used for feature extraction on the scaled third sub-image of 3 channels, and the input scale is also 640 × 640 × 3, and the output is the same as that of the first branch. Here, the second branch is composed of five groups of convolution modules and five groups of feature enhancement modules connected alternately. Specifically, the feature enhancement module includes a temperature feature enhancement module and a feature extraction module. The temperature feature enhancement module includes a one-dimensional convolution layer, a 3 × 3 atrous convolution layer and a residual connection, so as to expand the receptive field of the thermal infrared image and facilitate the capture of temperature gradient features. In particular, a fast spatial pyramid pooling layer is also inserted before the fifth feature enhancement module. Finally, three feature maps of different scales are output by the third, fourth and fifth feature enhancement modules.

[0081] The cross-modal feature fusion layer is used for feature fusion of the same scale feature maps output by the first branch and the second branch respectively, to obtain three fused feature maps. Specifically, the cross-modal feature fusion layer can fuse two 20 × 20 × 1024 scale feature maps through a cross-modal collaborative attention fusion module to obtain a first fused feature map. The other two scale feature maps are fused through bilinear interpolation and the cross-modal collaborative attention fusion module respectively to obtain a second fused feature map and a third fused feature map. Here, the cross-modal collaborative attention fusion module includes global average pooling, an activation function, feature weighting, and cross-modal interaction. Among them, feature weighting can be to multiply the pooled feature map by the weight through element-wise multiplication to obtain a weighted first image and a weighted second image. Cross-modal interaction can be to fuse the weighted first image and the weighted second image of the same scale through linear combination and nonlinear combination to obtain a fused feature map. Linear combination is to directly add the feature maps of the two modalities. For example, through linear combination, for the detection of water accumulation areas, the specular reflection features of water in the weighted first image and the low-temperature area features of water in the weighted second image can be superimposed to highlight the core features of water accumulation. Nonlinear combination is to multiply the abnormal features at the same position in the feature maps of the two modalities, and finally, the stability of the nonlinear features is restored through one-dimensional convolution.

[0082] The multi-scale detection head includes a classification branch, a regression branch, and a confidence branch. The classification branch, the regression branch, and the confidence branch have the same structure and include a convolution module, a feature extraction module, and a one-dimensional convolution layer. The difference is that the output channel of the classification branch is 5, corresponding to five types of facility anomalies in the airport road area. The output channel of the regression branch is 4, corresponding to the coordinates of the top of the detection area frame. The output channel of the confidence branch is 1, corresponding to the confidence of the anomaly judgment. Finally, the first facility anomaly detection information is output through the output layer.

[0083] In practice, by introducing the above road facility detection model, the infrared image and the thermal infrared image can be extracted in parallel and cooperatively fused, solving the missed detection problem in single-modal detection, realizing blind area coverage of five types of anomalies of road holes, cracking, water accumulation, icing, and foreign matter, and cooperating with the multi-scale detection head and the decoupling branch to improve the accuracy of anomaly detection in the airport road area while meeting the stringent requirements of the airport on detection accuracy.

[0084] In step S2, vegetation region anomaly detection is performed on each second region image pair in the second region image pair sequence to generate second facility anomaly information, thereby obtaining a second facility anomaly information set. The second facility anomaly information includes vegetation anomaly identification and a corresponding vegetation anomaly region, which represent vegetation anomaly or normal vegetation. The vegetation anomaly identification corresponding to at least one of the following vegetation anomaly types: high temperature, pest, drought, and disease.

[0085] As an example, the four vegetation anomaly types can be represented by numbers. For example, high temperature: "6", pest: "7", drought: "8", and disease: "9". Here, each second region image pair can be input into a vegetation anomaly detection model to output second facility anomaly information. Here, the vegetation anomaly detection model can have the same model structure as the road facility detection model, and the model parameters, loss function, and output channel are adjusted according to the actual vegetation characteristics. Thus, without reconstructing the model code, only updating the configuration file and training data in the original training framework can simultaneously improve the accuracy of vegetation anomaly recognition in the vegetation region. In addition, the trained YOLO-v8 detection model can also be used to perform vegetation region anomaly detection on the second region image pair to obtain the second facility anomaly information.

[0086] In practice, the vegetation region of the airport can be used to replace the hardened region to reduce construction costs, while avoiding soil hardening, dust raising, and facilitating the establishment of a better air environment. In addition, it can also provide a buffer for the forced landing of an aircraft. Therefore, the maintenance of the vegetation region is also a key item of facility anomaly detection. The vegetation region often has abnormal risks such as high temperature, pest, drought, and disease. If only infrared images are used for identification, only obvious features such as smoke and fire can be identified. In addition, it is difficult to comprehensively identify all hazards using visible light images, such as hidden feature changes in the early detection of disease. Thus, there is a great safety hazard. Here, considering that the abnormal situation of the vegetation region can also reflect abnormal changes in the thermal infrared image and the infrared image, the same model structure can be used simultaneously. The present application can be used to locate the high temperature area, low temperature area, or temperature anomaly area in the vegetation region in advance by simultaneously detecting the infrared image and the thermal infrared image, thereby facilitating the early detection of vegetation anomalies. Specifically, the first branch can be used to extract the shape, texture, and leaf state of the vegetation, and the second branch can be used to simultaneously extract the temperature distribution characteristics of the vegetation. Then, through cross-modal feature fusion, the temperature distribution characteristics and the shape and texture characteristics can be deeply associated, thereby more accurately determining the anomaly type of the vegetation region.

[0087] Step S3, device anomaly detection is performed on the second region image pairs in the above-mentioned second region image pair sequence to generate a third facility anomaly information set, and the first facility anomaly information set, the second facility anomaly information set and the third facility anomaly information set are determined as the facility anomaly detection result. For the second region image pairs including the shooting region identifiers representing the electrical equipment, the second partition sub-images in the second region image pairs are subjected to semantic segmentation according to the semantic detection boxes corresponding to the shooting region identifiers to obtain electrical equipment detection boxes. Then, the regions of the same electrical equipment detection boxes in the fourth partition sub-images in the second region image pairs are subjected to temperature detection to obtain the third facility anomaly information. Specifically, the temperature detection can be whether the number of coordinate points corresponding to the temperature values greater than a preset temperature threshold in the region exceeds a preset number threshold, and if so, the third facility anomaly information representing the electrical equipment anomaly is generated.

[0088] Optionally, the execution subject further includes:

[0089] Step S1, in response to the first facility anomaly information, the second facility anomaly information and / or the third facility anomaly information, anomaly description information sets are generated from the above-mentioned first region image pair sequence and the above-mentioned second region image pair sequence. The anomaly description information includes anomaly region size information and anomaly region association information. Here, the anomaly regions corresponding to the first facility anomaly information, the second facility anomaly information and / or the third facility anomaly information can be marked in the first region image or the second region image.

[0090] If the confidence of the first region image corresponding to the road anomaly type representing road icing is greater than a preset threshold, it is determined that the airport road is in the icing condition. Then, the corresponding icing region is determined from the third partition sub-image in the first region image through a threshold segmentation algorithm, and then if the icing region is not in the detected road hollow region, the icing region is marked in the airport three-dimensional facility model. Then, the nearest drainage outlet to the icing region in the airport three-dimensional facility model is found, and the distance value is less than a preset distance threshold (for example, 3 meters). Finally, the drainage outlet position coordinates corresponding to the drainage outlet, the size information of the icing region and the corresponding anomaly detection identifier are determined as the anomaly region association information. Thus, the anomaly region size information and the anomaly region association information can be determined as the anomaly description information.

[0091] As an example, the roads on both sides of the aircraft taking off and landing in the airport are provided with drainage ditches, and the parking area is also provided with drainage outlets. If the drainage outlets are blocked or obstructed by foreign matters (such as garbage, leaves, etc.), water accumulation is easy to occur, and even icing occurs when the weather is cold. For example, see Figure 4As shown in the airport road icing abnormal scene diagram, there is a water accumulation area or icing area 402 on the road 401, which indicates that there is a depression in this area. At the same time, if there is a drain 403 within a certain distance of this area, there is also a drain blockage. Therefore, by screening the coordinates of the drain position, more detailed related information can be provided to the airport facility safety detection department to facilitate timely cleaning of the drain and repair of the abnormal situation of ground depression. Thus, avoid the accumulation or icing caused by ground depression or drain blockage. Further, reduce the safety hazards of airport facilities.

[0092] In addition, if the icing area is in the detected road cavity area, the size range of the road cavity and the corresponding abnormal detection mark are determined as the abnormal description information. Thus, more detailed facility abnormal information can be provided to the airport facility safety detection department, avoiding only dealing with the icing or water accumulation problem in this area while ignoring the potential road cavity risk. Further, the safety hazards of airport facilities can be further reduced.

[0093] Step S2, mark each abnormal description information in the above abnormal description information set to the airport three-dimensional facility model to obtain a marked three-dimensional facility model. Wherein, the abnormal description information can be marked to the corresponding abnormal area in the airport three-dimensional facility model to obtain a marked three-dimensional facility model. Thus, the airport facility abnormal situation can be directly reflected through the marked three-dimensional facility model.

[0094] Optionally, the above execution subject further comprises:

[0095] Step S1, in response to the above facility abnormal detection result satisfying the preset abnormal condition, the corresponding first image and / or second image is added to the above airport three-dimensional facility model to obtain a current three-dimensional facility model. Wherein, the preset abnormal condition can be that the facility abnormal detection result includes first facility abnormal information, second facility abnormal information and / or third facility abnormal information with a confidence greater than a preset threshold. Secondly, the corresponding first image and / or second image can be saved and associated with the marked abnormal area in the above airport three-dimensional facility model to obtain a current three-dimensional facility model.

[0096] Step S2, according to the above current three-dimensional facility model, a facility abnormal warning is sent to a target terminal. Wherein, the target terminal can be a terminal corresponding to the airport facility safety detection department or the facility maintenance department. Secondly, by associating the first image and the second image with the abnormal in the airport three-dimensional facility model, manual checking of the specific abnormal scene can be facilitated.

[0097] The above various embodiments of the present disclosure have the following beneficial effects: through the airport facility anomaly detection method based on multi-source feature fusion of some embodiments of the present disclosure, the airport facilities in the airport to be detected region can be comprehensively detected. Specifically, the reason why the airport facility anomaly detection in the airport to be detected region is not comprehensive enough is that the observation angle of the fixed camera on the ground is limited by the camera position, and there is a blind area, making it difficult to shoot the complete airport to be detected region. Based on this, the airport facility anomaly detection method based on multi-source feature fusion of some embodiments of the present disclosure, first, in response to obtaining the unmanned aerial vehicle calling instruction, generates the current collection time period and the unmanned aerial vehicle planning path according to the pre-established airport three-dimensional facility model and the current airport flight information set. Then, based on the above current collection time period and the above unmanned aerial vehicle planning path, the target unmanned aerial vehicle is controlled to shoot the airport region to obtain a first image sequence and a second image sequence, wherein the first image is a temperature distribution map shot by a thermal infrared camera, the second image is an infrared image shot by an infrared camera, and the first image and the second image correspond one by one. In practice, the fixed camera on the ground of the airport often only collects visible light images with color. When identifying abnormal situations such as airport road regions, vegetation regions, or electrical equipment, only the surface features can be identified from visible light images with color, and it is difficult to detect potential safety hazards of airport facilities in advance (for example, the hazards of road hollows). Therefore, the present application replaces the ground fixed camera scheme with a scheme of unmanned aerial vehicles carrying thermal infrared cameras and infrared cameras, so as to monitor all-weather according to the airport three-dimensional facility model and the flight information. Thus, not only can the movable characteristics of the unmanned aerial vehicle be used to shoot more comprehensive images, but also the thermal infrared camera and the infrared camera carried by the unmanned aerial vehicle can be used to shoot temperature distribution maps and infrared images simultaneously. Therefore, compared with commonly used visible light images, the temperature state and material characteristics of the shooting area (for example, the temperature state of the area where there is a road hollow is different from that of other normal road areas, and for example, the temperature state of the area where there is icing is different from that of the area where there is no icing) are highlighted, so as to detect potential risks. Then, the shooting area identifiers corresponding to the second region images of the first image sequence and the second image sequence are determined to obtain a shooting area identifier sequence, wherein the shooting area identifier corresponds to at least one of the following: an airport road region, an airport vegetation region, and an airport electrical equipment region. Then, based on the shooting area identifier sequence, the first image sequence and the second image sequence are subjected to image reorganization processing to obtain a first region image pair sequence and a second region image pair sequence. Here, by determining the shooting area identifier, the actual shooting area of the unmanned aerial vehicle can be represented, so that image reorganization can be performed according to the airport road region, the airport vegetation region, and the airport electrical equipment region, thereby reducing the interference error of cross-region detection and improving the recognition accuracy.Finally, image recognition is performed on the first sequence of region image pairs and the second sequence of region image pairs to generate a facility anomaly detection result, wherein the facility anomaly detection result includes a first facility anomaly information set, a second facility anomaly information set, and / or a third facility anomaly information set, the first facility anomaly information set characterizing a road anomaly, the second facility anomaly information set characterizing a vegetation anomaly, and the third facility anomaly information set characterizing an electrical equipment anomaly. Thus, the airport facility anomaly detection can be more comprehensive, and because the temperature distribution feature is introduced, hidden facility anomalies can be detected. Further, the airport facilities in the to-be-detected region of the airport can be comprehensively detected.

[0098] Further reference Figure 5 , as an implementation of the method shown in the above figures, the present disclosure provides some embodiments of an airport facility anomaly detection device based on multi-source feature fusion, which corresponds to the method embodiments shown in Figure 1 , and the airport facility anomaly detection device based on multi-source feature fusion can be applied to various electronic devices.

[0099] As Figure 5As shown, the airport facility anomaly detection device 500 based on multi-source feature fusion of some embodiments includes a generation unit 501, a shooting unit 502, a determination unit 503, an image reorganization processing unit 504, and an image recognition unit 505. Among them, the generation unit is configured to, in response to obtaining a UAV calling instruction, generate a current collection time period and a UAV planned path according to a pre-established airport three-dimensional facility model and a current airport flight information set; the shooting unit is configured to control the target UAV to shoot the airport area based on the above-mentioned current collection time period and the above-mentioned UAV planned path, to obtain a first image sequence and a second image sequence, wherein the first image is a temperature distribution map shot by a thermal infrared camera, the second image is an infrared image shot by an infrared camera, and the first image and the second image correspond one by one; the determination unit is configured to determine the shooting area identifier corresponding to the second area image of the first image sequence and the second image sequence, to obtain a shooting area identifier sequence, wherein the shooting area identifier corresponds to at least one of the following: an airport road area, an airport vegetation area, and an airport electrical equipment area; the image reorganization processing unit is configured to perform image reorganization processing on the first image sequence and the second image sequence based on the shooting area identifier sequence, to obtain a first area image pair sequence and a second area image pair sequence; and the image recognition unit is configured to perform image recognition according to the first area image pair sequence and the second area image pair sequence, to generate a facility anomaly detection result, wherein the facility anomaly detection result includes a first facility anomaly information set, a second facility anomaly information set, and / or a third facility anomaly information set, the first facility anomaly information represents a road anomaly, the second facility anomaly information represents a vegetation anomaly, and the third facility anomaly information represents an electrical equipment anomaly.

[0100] It can be understood that the units described in the airport facility anomaly detection device 500 based on multi-source feature fusion correspond to the respective steps in the method described above. Figure 1 The operations, features, and advantages described above for the method also apply to the airport facility anomaly detection device 500 based on multi-source feature fusion and the units contained therein, and will not be repeated here.

[0101] Reference is made below to Figure 6 which shows a structural schematic diagram of an electronic device 600 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0102] As Figure 6As shown, the electronic device 600 can include a processing device 601 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. Various programs and data required for the operation of the electronic device 600 are also stored in the random access memory 603. The processing device 601, the read-only memory 602, and the random access memory 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0103] In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that all of the illustrated devices are not required, and fewer or different devices can alternatively be implemented. Figure 6 Each block shown in the flowcharts can represent a device, or a plurality of devices, as needed.

[0104] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for performing the methods shown in the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the read-only memory 602. When the computer program is executed by the processing device 601, the above-described functions defined in the methods of some embodiments of the present disclosure are performed.

[0105] Note that the computer readable medium in some embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus or device, or that can be used by or in connection with an instruction execution system, apparatus or device. In some embodiments of the present disclosure, the computer readable signal medium can include a computer readable program code propagated on or through a carrier wave, in baseband or as part of a carrier wave. Such propagated signals can take a variety of forms, including but not limited to electro-magnetic signals, optical signals or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0106] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0107] The computer readable medium can be included in the electronic device, or can exist separately from the electronic device. The computer readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: in response to obtaining a UAV calling instruction, generate a current collection time period and a UAV planning path according to a pre-established airport three-dimensional facility model and a current airport flight information set; control a target UAV to capture an airport area based on the current collection time period and the UAV planning path to obtain a first image sequence and a second image sequence, where the first image is a temperature distribution map captured by a thermal infrared camera, the second image is an infrared image captured by an infrared camera, and the first image and the second image correspond one by one; determine a shooting area identifier corresponding to a second area image of the first image sequence and the second image sequence to obtain a shooting area identifier sequence, where the shooting area identifier corresponds to at least one of the following: an airport road area, an airport vegetation area, and an airport electrical equipment area; perform image reorganization processing on the first image sequence and the second image sequence based on the shooting area identifier sequence to obtain a first area image pair sequence and a second area image pair sequence; and perform image recognition according to the first area image pair sequence and the second area image pair sequence to generate a facility anomaly detection result, where the facility anomaly detection result includes a first facility anomaly information set, a second facility anomaly information set, and / or a third facility anomaly information set, the first facility anomaly information represents a road anomaly, the second facility anomaly information represents a vegetation anomaly, and the third facility anomaly information represents an electrical equipment anomaly.

[0108] Computer program code for carrying out operations of some embodiments of the disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0109] The computer program product of the first aspect can further include one or more of the following features. The computer program product can include a computer readable medium. The computer readable medium can include a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include tangible storage medium. The computer readable signal medium can include a propagated data signal with computer readable program code embodied therein. The computer readable program code can be downloaded into a working memory of a computer from the computer readable signal medium or from the computer readable storage medium. The computer readable program code can cause the computer to perform the steps of the first aspect. The computer readable program code can be executed by one or more processors associated with the computer. The computer readable program code can be executed by one or more hardware processors.

[0110] The functions described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, non-limiting examples of hardware logic components that can be used include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0111] The above description is only some preferred embodiments of the present disclosure and an explanation of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features and the technical features disclosed in the embodiments of the present disclosure (but not limited to) with similar functions are replaced with each other to form a technical solution.

Claims

1. A method for detecting anomalies in airport facilities based on multi-source feature fusion, characterized in that, include: In response to receiving the drone call command, the system generates the current data collection time period and the drone's planned path based on the pre-established 3D airport facility model and the current airport flight information set. Based on the current data collection time period and the UAV's planned path, the target UAV is controlled to take pictures of the airport area, resulting in a first image sequence and a second image sequence. The first image is a temperature distribution map taken by a thermal infrared camera, and the second image is an infrared image taken by an infrared camera. The second image is used to reflect the road cavity status information and surface icing status information. The first image and the second image correspond one-to-one and are used to jointly characterize the temperature status of the cavity area and the surface icing area. Determine the shooting area identifiers corresponding to the second region images of the first image sequence and the second image sequence to obtain a shooting area identifier sequence, wherein the shooting area identifiers correspond to at least one of the following: airport road area, airport vegetation area, and airport electrical equipment area; Based on the shooting area identification sequence, the first image sequence and the second image sequence are subjected to image recombination processing to obtain a first region image pair sequence and a second region image pair sequence. The three-dimensional facility model includes equipment markers and boundary line markers generated according to the airport facility structure. The first image and the second image are synchronously segmented based on the equipment markers and boundary line markers in the three-dimensional facility model. Image recognition is performed on the first region image pair sequence and the second region image pair sequence to generate facility anomaly detection results. The facility anomaly detection results include a first facility anomaly information set, a second facility anomaly information set and / or a third facility anomaly information set. The first facility anomaly information set represents road anomalies, the second facility anomaly information set represents vegetation anomalies, and the third facility anomaly information set represents electrical equipment anomalies.

2. The method according to claim 1, characterized in that, The method further includes: In response to the facility anomaly detection result meeting the preset anomaly conditions, the corresponding first image and / or second image are added to the airport 3D facility model to obtain the current 3D facility model; Based on the current 3D facility model, a facility anomaly warning is issued to the target terminal.

3. The method according to claim 1, characterized in that, Current airport flight information includes road markings and corresponding road occupancy time periods. In response to receiving a drone invocation command, the system generates the current data collection time period and the drone's planned path based on a pre-established 3D airport facility model and the current airport flight information set, including: Based on the road occupancy time periods and road signs included in the current airport flight information set, determine the road idle time periods corresponding to each road sign within the airport, and obtain a set of road idle time periods; The road identifiers corresponding to road idle time periods longer than the preset detection time period in the set of road idle time periods are identified as detectable road identifiers, thus obtaining a set of detectable road identifiers; For the airport roads corresponding to the detectable road signs in the set of detectable road signs, the following processing steps are performed in the airport 3D facility model: The initial detection route is selected from the preset detection route set and the detection route that matches the airport road corresponding to each detectable road sign is selected as the initial detection route. The two ends of the initial detection route correspond to the first detection point and the second detection point. Path planning is performed on each initial detection route in the initial detection route set to obtain the initial planned path, wherein the initial planned path passes through each initial detection route and the corresponding first observation point and second observation point. Determine the detection time period for each detectable road sign in the initial planning path to obtain a sequence of detection time periods; In response to the absence of a time conflict between the detected time period sequence and the road occupancy time period included in the current airport flight information set, the initial planned path is determined as the UAV planning path, and the current collection time period is generated using the road idle time period set, and the processing steps are terminated.

4. The method according to claim 3, characterized in that, The response to receiving the UAV call command, based on the pre-established 3D airport facility model and the current airport flight information set, generates the current data collection time period and the UAV planned path, and also includes: In response to a time conflict between the detected time period sequence and the road occupancy time period included in the current airport flight information set, the current execution round is recorded. In response to the current execution round being greater than the preset round, detectable road signs corresponding to the detection time periods with time conflicts are removed, and the processing steps are executed again.

5. The method according to claim 3 or 4, characterized in that, The step of performing image recombination processing on the first image sequence and the second image sequence based on the shooting area identification sequence to obtain a first region image pair sequence and a second region image pair sequence includes: Based on the shooting area identification sequence, the second image sequence is segmented to obtain a first segmented sub-image sequence and a second segmented sub-image sequence, wherein the first segmented sub-image corresponds to the airport road area and the second segmented image corresponds to the airport vegetation area. According to the segmentation boundaries corresponding to the first segmented sub-image sequence and the second segmented sub-image sequence, the first image sequence is synchronously segmented to obtain a third segmented sub-image sequence and a fourth segmented sub-image sequence, wherein the third sub-segmented image corresponds to the airport road area and the fourth segmented image corresponds to the airport vegetation area. The first segmented image and the third segmented image in the first segmented image sequence and the third segmented image sequence are taken as the first region image pair to obtain the first region image pair sequence; The corresponding second and fourth segmented images in the second and fourth segmented image sequences are used as second region image pairs to obtain a second region image pair sequence.

6. The method according to claim 5, characterized in that, The step of performing image recognition based on the first region image sequence and the second region image sequence to generate facility anomaly detection results includes: For each first region image pair in the first region image pair sequence, road region anomaly detection is performed to generate first facility anomaly information, resulting in a first facility anomaly information set. The first facility anomaly information includes a road anomaly detection identifier representing a road anomaly or a normal road and a corresponding road anomaly region. The anomaly detection identifier representing a road anomaly corresponds to at least one of the following road anomaly types: road cavity, road crack, road water accumulation, road icing, and road foreign object. For each second region image pair in the second region image pair sequence, vegetation region anomaly detection is performed to generate second facility anomaly information, resulting in a second facility anomaly information set. The second facility anomaly information includes vegetation anomaly identifiers representing vegetation anomalies or vegetation normality and corresponding vegetation anomaly regions. The vegetation anomaly identifiers representing vegetation anomalies correspond to at least one of the following vegetation anomaly types: high temperature, insect pests, drought, and diseases. Equipment anomaly detection is performed on the second region image pairs in the second region image pair sequence to generate a third facility anomaly information set, and the first facility anomaly information set, the second facility anomaly information set, and the third facility anomaly information set are determined as facility anomaly detection results.

7. The method according to claim 6, characterized in that, The method further includes: In response to the detection of a first facility anomaly, a second facility anomaly, and / or a third facility anomaly, an anomaly description information set is generated based on the first region image pair sequence and the second region image pair sequence, wherein the anomaly description information includes anomaly region size information and anomaly region association information; Each anomaly description in the anomaly description information set is marked into the airport 3D facility model to obtain the marked 3D facility model.

8. An airport facility anomaly detection device based on multi-source feature fusion, characterized in that, include: The generation unit is configured to, in response to receiving a drone call command, generate the current data collection time period and the drone's planned path based on a pre-established 3D airport facility model and the current airport flight information set. The shooting unit is configured to control the target drone to shoot the airport area based on the current collection time period and the drone's planned path, and obtain a first image sequence and a second image sequence. The first image is a temperature distribution map captured by a thermal infrared camera, and the second image is an infrared image captured by an infrared camera. The second image is used to reflect the cavity status information and surface icing status information of the road. The first image and the second image correspond one-to-one and are used to jointly characterize the temperature status of the cavity area and the surface icing area. The determining unit is configured to determine the shooting area identifiers corresponding to the second region images of the first image sequence and the second image sequence, and obtain a shooting area identifier sequence, wherein the shooting area identifiers correspond to at least one of the following: airport road area, airport vegetation area, and airport electrical equipment area; An image reconstruction processing unit is configured to perform image reconstruction processing on the first image sequence and the second image sequence based on the shooting area identifier sequence to obtain a first region image pair sequence and a second region image pair sequence, wherein the three-dimensional facility model includes equipment markers and boundary line markers generated according to the airport facility structure, and the first image and the second image are synchronously segmented based on the equipment markers and boundary line markers in the three-dimensional facility model; The image recognition unit is configured to perform image recognition based on the first region image pair sequence and the second region image pair sequence to generate facility anomaly detection results, wherein the facility anomaly detection results include a first facility anomaly information set, a second facility anomaly information set and / or a third facility anomaly information set, the first facility anomaly information representing road anomalies, the second facility anomaly information representing vegetation anomalies, and the third facility anomaly information representing electrical equipment anomalies.

9. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images