Airport runway abnormal target identification methods, devices and electronic equipment

Abstract

This disclosure presents a method, apparatus, and electronic device for identifying abnormal targets on airport runways. One specific implementation of the method includes: performing panoramic image stitching on a real-time image set; identifying runway regions from the real-time panoramic images to obtain runway region images and runway description information; performing directional segmentation of the runway region images based on the runway description information to obtain a first local runway region image and a second local runway region image; identifying rubber attachments in the first local runway region image to obtain first attachment description information; identifying rubber attachments in the second local runway region image based on the first attachment description information to obtain second attachment description information; and generating airport runway anomaly warning information based on a set of reference point locations, the first attachment description information, and the second attachment description information. This implementation achieves all-weather, automatic, and accurate identification of rubber attachments on airport runway surfaces.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer technology and image processing, and specifically to methods, apparatus and electronic devices for identifying abnormal targets on airport runways. Background Technology

[0002] An airport runway is a long, narrow area for aircraft to take off and land. During landing, the aircraft tires experience high-speed friction with the runway, causing the rubber on the tire surface to soften due to heat and adhere to the runway surface. If these rubber particles (rubber deposits) are not cleaned in time, they reduce the runway's friction, increasing the aircraft's braking distance and thus affecting landing safety.

[0003] Currently, regular manual inspections and cleaning are commonly used to identify and treat rubber deposits on airport runways. Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure propose methods, apparatus, and electronic devices for identifying abnormal targets on airport runways to address the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide a method for identifying abnormal targets on an airport runway. The method includes: stitching together a real-time image set into a panoramic view to generate a real-time panoramic image, wherein the real-time image set is acquired by a high-magnification full-color camera facing the airport runway; identifying a runway region from the real-time panoramic image to obtain a runway region image and runway description information, wherein the runway description information includes a runway direction vector and a set of reference point positions, the reference point positions corresponding to runway lights installed within the airport runway; and performing directional segmentation on the runway region image based on the runway description information to obtain a first local segment. The system generates a runway area image and a second partial runway area image. It then identifies rubber attachments in the first partial runway area image to obtain first attachment description information. Based on the first attachment description information, it identifies rubber attachments in the second partial runway area image to obtain second attachment description information. Both the first and second attachment description information include: rubber attachment confidence level, rubber attachment location information, and rubber attachment severity. Finally, based on the reference point location set, the first attachment description information, and the second attachment description information, it generates an airport runway anomaly warning information.

[0007] Secondly, some embodiments of this disclosure provide an airport runway abnormal target identification device. The device includes: a panoramic image stitching unit configured to stitch together a set of real-time images to generate a real-time panoramic image, wherein the set of real-time images is acquired by a high-magnification full-color camera facing the airport runway; a runway area identification unit configured to identify the runway area in the real-time panoramic image to obtain a runway area image and runway description information, wherein the runway description information includes: a runway direction vector and a set of reference point positions, the reference point positions corresponding to runway lights installed within the airport runway; and a directional segmentation unit configured to perform directional segmentation on the runway area image based on the runway description information to obtain a first local runway area. The system comprises: a domain image and a second local runway area image; a first rubber attachment identification unit configured to identify rubber attachments in the first local runway area image to obtain first attachment description information; a second rubber attachment identification unit configured to identify rubber attachments in the second local runway area image based on the first attachment description information to obtain second attachment description information, wherein both the first and second attachment description information include: rubber attachment confidence, rubber attachment location information, and rubber attachment severity; and a generation unit configured to generate airport runway anomaly warning information based on the reference point location set, the first attachment description information, and the second attachment description information.

[0008] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0009] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0010] The various embodiments of this disclosure have the following beneficial effects: The airport runway abnormal target identification method of some embodiments of this disclosure achieves all-weather automatic and accurate identification of rubber attachments on the airport runway surface. In practice, existing methods typically use manual periodic inspections to identify and process rubber attachments within the airport runway, which is often lagging. Furthermore, due to the large area of ​​airport runways and the high priority of aircraft takeoff and landing missions, it is difficult to conduct all-weather inspections of rubber attachments manually. Therefore, the airport runway abnormal target identification method of some embodiments of this disclosure first performs panoramic image stitching on a real-time image set to generate a real-time panoramic image. This real-time image set is acquired by a high-magnification full-color camera facing the airport runway. In practice, this disclosure uses a high-magnification full-color camera positioned at a high location to achieve real-time image acquisition of the airport runway, and uses image stitching to remove redundant parts between real-time images and reduce subsequent invalid data processing. Secondly, runway region identification is performed on the aforementioned real-time panoramic image to obtain runway region images and runway description information. The runway description information includes: runway direction vectors and a set of reference point positions, where the reference point positions correspond to runway lights installed within the airport runway. In practice, due to factors such as shooting angle, real-time panoramic images often include areas outside the airport runway. Since flight operations primarily occur on airport runways, it is unnecessary to identify rubber attachments in areas outside the runway. Therefore, this disclosure further eliminates images corresponding to areas outside the airport runway in the real-time panoramic image through runway region identification. Next, based on the aforementioned runway description information, the runway region image is directionally segmented to obtain a first local runway region image and a second local runway region image. Furthermore, rubber attachment identification is performed on the first local runway region image to obtain first attachment description information. In addition, based on the aforementioned first attachment description information, rubber attachments are identified in the aforementioned second local runway area image to obtain second attachment description information. Both the first and second attachment description information include: rubber attachment confidence, rubber attachment location information, and rubber attachment severity. In practice, runway areas often exhibit symmetry, and the rubber attachments generated by aircraft during landing also tend to be symmetrical. Therefore, this disclosure, through directional segmentation and prior rubber attachment identification of local runway area images, reduces data processing volume and increases identification speed compared to directly identifying the runway area image. Finally, based on the aforementioned reference point location set, the aforementioned first attachment description information, and the aforementioned second attachment description information, airport runway anomaly warning information is generated. In summary, this achieves all-weather automatic and accurate identification of rubber attachments on the airport runway surface. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0012] Figure 1 This is a flowchart of some embodiments of the airport runway abnormal target identification method according to the present disclosure;

[0013] Figure 2 This is a schematic diagram of the sub-image patch matrix extraction process;

[0014] Figure 3 This is a schematic diagram illustrating the process of determining the runway area boundary;

[0015] Figure 4 This is a schematic diagram of the network structure of a rubber adhesion recognition network;

[0016] Figure 5 This is a schematic diagram of the structure of some embodiments of the airport runway abnormal target identification device according to the present disclosure;

[0017] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of an airport runway anomaly target identification method according to the present disclosure. This airport runway anomaly target identification method, applied to the identification of rubber attachments within an airport runway, includes the following steps:

[0025] Step 101: Perform panoramic image stitching on the real-time image set to generate a real-time panoramic image.

[0026] In some embodiments, the execution subject of the airport runway abnormal target identification method (e.g., a computing device) can perform panoramic image stitching on a real-time image set to generate a real-time panoramic image.

[0027] The real-time image set is acquired by high-magnification full-color cameras facing the airport runway. These high-magnification full-color cameras can be full-color cameras with high-magnification optical zoom capabilities. They can be positioned to overlook the entire airport runway (e.g., on the top of the terminal building). The real-time panoramic image is a stitched image obtained from the real-time images in the real-time image set.

[0028] In practice, rubber residue is primarily generated by the friction between the aircraft tires and the ground during landing. Therefore, in response to an aircraft landing on an airport runway, a high-magnification full-color camera can be triggered to capture a set of real-time images of the runway. Specifically, the camera can be rotated using a pan-tilt-zoom (PTZ) mechanism to continuously capture real-time images, resulting in the aforementioned set of images. Alternatively, the high-magnification full-color camera can be triggered to capture real-time images at regular intervals. For example, images can be captured daily at midday when sunlight intensity is optimal. Furthermore, the real-time image capture cycle of the high-magnification full-color camera can be dynamically adjusted based on the frequency of aircraft takeoffs and landings at the airport.

[0029] In practice, firstly, the aforementioned execution entity can extract feature points from each real-time image in the real-time image set. For example, the execution entity can extract feature points from the real-time images using the SIFT (Scale Invariant Feature Transform) algorithm. Then, by calculating the feature point similarity, the stitching points between two adjacent real-time images are calculated. Finally, using the stitching points as the stitching positions, adjacent real-time images in the real-time image set are stitched together to obtain the aforementioned real-time panoramic image.

[0030] It should be noted that the aforementioned computing devices can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed on the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0031] In some optional implementations of certain embodiments, the execution entity performs panoramic image stitching on a set of real-time images to generate a real-time panoramic image, including:

[0032] Step S1: Perform image distortion correction on each real-time image in the above real-time image set to generate a corrected image, thus obtaining a corrected image set.

[0033] In practice, high-magnification full-color cameras are used for long-distance real-time image acquisition, and high-magnification telephoto lenses are prone to pincushion distortion. Therefore, it is necessary to perform corresponding distortion correction on the real-time images. Specifically, this disclosure uses the Brown-Conrady polynomial model to perform image distortion correction on the real-time images to generate corrected images.

[0034] Step S2: Determine the global brightness gain information based on the above-mentioned set of corrected images.

[0035] In practice, high-magnification full-color cameras require pan-tilt-zoom (PTZ) to rotate during real-time image acquisition to ensure the acquired image set covers the entire airport runway. However, the brightness of different images may vary at different shooting angles. To ensure that all corrected images in the corrected image set have the same brightness parameters, overall brightness unification of the corrected image set is necessary. Specifically, firstly, the image histogram corresponding to each corrected image in the corrected image set is read. The image histogram represents the number of pixels corresponding to different brightness levels. Then, the brightness histograms corresponding to the corrected image set are weighted and summed to obtain a weighted brightness histogram, which serves as the global brightness gain information.

[0036] Step S3: Based on the global brightness gain information mentioned above, perform brightness compensation on each corrected image in the corrected image set to generate a brightness-compensated image, thus obtaining a brightness-compensated image set.

[0037] In practice, firstly, the histogram difference between the luminance histogram of the corrected image and the histogram of the global luminance gain information (weighted luminance histogram) is determined. Then, based on the histogram difference, luminance compensation is performed on the corrected image to generate a luminance-compensated image. This method achieves unified global luminance adjustment for the set of corrected images.

[0038] Step S4: Reduce the resolution of each brightness-compensated image in the brightness-compensated image set to generate a low-resolution image, thus obtaining a low-resolution image set.

[0039] The low-resolution image has a resolution that is 3 / 4 of the resolution of the brightness-compensated image. The image size (physical size) of the low-resolution image is the same as that of the brightness-compensated image.

[0040] In practice, physical size × image resolution = pixel size. Here, pixel size represents the total number of pixels contained in the image. When the image resolution decreases while the physical size remains constant, the pixel size decreases accordingly. Specifically, image resolution reduction can be achieved through pixel weighting and merging. Since subsequent detection of the interface edges between the airport runway and other areas is required, and edge features have distinct boundaries, reducing the image resolution decreases the number of pixels processed, thereby improving the speed of subsequent edge feature extraction.

[0041] Step S5: Extract edge features from each low-resolution image in the above low-resolution image set to obtain an edge feature group.

[0042] The edge feature group includes edge features corresponding to the four edges of the low-resolution image.

[0043] In practice, due to the large area of ​​airport runways, real-time image acquisition often requires not only horizontal movement of the camera via a pan-tilt-zoom (PTZ) system linked to a high-magnification full-color camera, but also potentially vertical movement. Therefore, constructing real-time panoramic images involves stitching both horizontal and vertical real-time images. This disclosure therefore extracts edge features from the four edges of the low-resolution image. Specifically, the SIFT algorithm is used to extract feature points from the four image edges of the low-resolution image, resulting in edge feature groups. Each edge feature can consist of multiple feature points extracted from the corresponding image edge. These feature points can be represented by a 128-dimensional feature vector. Since adjacent real-time images mainly overlap at the image edges, this disclosure only extracts feature points at the edge locations to avoid the large data processing volume associated with extracting feature points from the entire low-resolution image, thereby improving the feature point extraction speed.

[0044] Step S6: Based on the edge feature group corresponding to the low-resolution image, perform image stitching on the brightness-compensated images in the above brightness-compensated image set to obtain the above real-time panoramic image.

[0045] In practice, the aforementioned execution entity can stitch together two adjacent brightness-compensated images in the brightness-compensated image set according to the acquisition order of the real-time images in the real-time image set and the edge feature groups corresponding to the low-resolution images, through feature point matching (such as the similarity calculation of the feature vectors corresponding to the feature points), to obtain the aforementioned real-time panoramic image.

[0046] Step 102: Perform runway region identification on the real-time panoramic image to obtain runway region image and runway description information.

[0047] In some embodiments, the aforementioned execution entity can perform runway region identification on the real-time panoramic image to obtain runway region images and runway description information.

[0048] The runway area image contains only the airport runway. Runway description information includes the runway direction vector and a set of reference point positions. The runway direction vector represents the runway direction within the runway area image. Because the high-magnification full-color camera performs long-distance real-time image acquisition from a high-angle overhead view, the direction of the airport runway within the runway area image is not horizontal to the image boundary of the real-time panoramic image due to the positional relationship between the airport runway and the high-magnification full-color camera. Therefore, it is necessary to extract the runway direction vector within the runway area image. The reference point positions correspond to the runway lights installed within the airport runway. Runway lights are used to guide aircraft takeoff and landing on the airport runway; their positions are fixed and often symmetrically arranged at equal intervals on both sides of the runway. Therefore, the runway lights can be used as references for subsequent transformations between the image coordinate system and the 3D coordinate system. The reference point positions represent the pixel coordinates of the runway lights within the runway area image. The 3D coordinates corresponding to the runway lights can be pre-acquired, thus obtaining the mapping relationship between the 3D coordinates of the runway lights and the image coordinates.

[0049] In practice, firstly, a model such as YOLO (You Only Look Once) can be used to identify the runway area in real-time panoramic images, thereby determining the runway area image. Then, the YOLO model is used to further identify runway lights and road guide lines within the runway area image, thereby obtaining a set of reference point locations and determining the road guide lines. Since road guide lines are often horizontal with the airport runway, the direction vector corresponding to the direction of the road guide lines can be used as the runway direction vector.

[0050] In some optional implementations of certain embodiments, the execution entity performs runway region identification on the real-time panoramic image to obtain a runway region image and runway description information, including:

[0051] Step S1: Divide the above panoramic image into image blocks to obtain an image block matrix.

[0052] The image blocks in the aforementioned image block matrix have the same size, and the matrix dimension of the aforementioned image block matrix is ​​N×M.

[0053] As an example, see Figure 2 The diagram illustrates the extraction process of the sub-image block matrix. The aforementioned execution entity can uniformly divide the panoramic image 201 into image blocks, obtaining an image block matrix 202. The image block matrix 202 consists of N×M image blocks. Figure 2 For example, N can be 15 and M can be 5.

[0054] Step S2: Extract the sub-image block matrix located at the beginning of the above image block matrix as a local image block matrix.

[0055] The matrix dimension of the aforementioned local image patch matrix is ​​S×M, where S is less than N.

[0056] As an example, see further. Figure 2 The panoramic image 201 is often a rectangular image. Therefore, a sub-image patch matrix can be extracted from either side of the long side of the panoramic image 201 as a local image patch matrix 203. The local image patch matrix 203 has a matrix dimension of S×M. Figure 2 For example, S can take the value 4, and M can take the value 5. In particular, 0.2 ≤ S / N ≤ 0.4, where N, M, and S are all positive integers. By limiting the range of S / N, the extracted local image patch matrix is ​​guaranteed to contain airport runways, while avoiding the extraction of too many image patches that would reduce the algorithm's execution efficiency.

[0057] Step S3: Perform runway edge recognition on each image block in the above local image block matrix to generate first image block recognition information and obtain the first image block recognition information set.

[0058] The first image patch identification information includes: image patch position, runway edge position, and runway edge identifier. The image patch position represents the location of the image patch within the image patch matrix. Since the image patch matrix is ​​a two-dimensional matrix, the image patch position can be represented by two-dimensional coordinates. For example, the image patch position can be represented as (n, m), where "n" represents the horizontal coordinate of the image patch, and "m" represents the vertical coordinate of the image patch. The runway edge position represents the location of the runway edge within the image patch. For example, the runway edge position can be composed of a set of coordinate points located on the runway edge. The runway edge identifier indicates whether the image patch contains a runway edge. For example, the runway edge identifier can be represented by "0" or "1". When the runway edge identifier is "1", it indicates that the image patch contains a runway edge. When the runway edge identifier is "0", it indicates that the image patch does not contain a runway edge.

[0059] In practice, the aforementioned execution entity can extract edge features from image patches using a lightweight convolutional neural network model, and output runway edge identifiers through a binary classifier following the lightweight convolutional neural network model. When the runway edge identifier is "1", the edge features are mapped to pixel coordinates to obtain the runway edge location. Specifically, airport runways are mainly composed of cement, asphalt, or a mixture of both. In image representation, airport runways and areas outside the runway have significant feature differences. Therefore, this disclosure achieves rapid runway edge identification with low model cost by training a weak classifier (composed of a lightweight convolutional neural network model and a binary classifier). The weak classifier is trained using a supervised training method.

[0060] Step S4: Determine the runway area recognition direction based on the second image block recognition information set mentioned above.

[0061] Among them, the runway region recognition direction represents the direction of runway region recognition along the image block matrix.

[0062] In practice, due to the influence of the image acquisition position of the high-magnification full-color camera, the runway edges in the image are not parallel as in reality, but rather exhibit an approximately trumpet-shaped shape. Simultaneously, the local image patch matrix is ​​located at either end of the image patch matrix. Therefore, when combining the local image patch matrix for runway edge recognition, not only the runway edges corresponding to the short side of the airport runway will be identified, but also the runway edges corresponding to the long side. Specifically, firstly, the aforementioned execution entity can perform runway edge fitting based on the runway edge positions included in the second image patch recognition information set. Ideally, this will result in three local runway edges. Among them, two runway edges correspond to the long side of the airport runway, which can be determined by the number of intersection points between the local runway edges. For example, the local runway edge corresponding to the short side intersects with both of the local runway edges corresponding to the two long sides, while the local runway edge corresponding to the long side only intersects with the local runway edge corresponding to the short side. Therefore, the two local runway edges corresponding to the long side can be selected from the three local runway edges based on the number of intersection points. Then, using the local runway edges corresponding to the two long sides as the recognition direction, the above-mentioned runway area recognition direction is obtained, that is, the runway area recognition direction is composed of two runway area recognition direction vectors.

[0063] As an example, see Figure 3 The diagram illustrates the runway area boundary determination process. First, the executing entity uses a weak classifier to perform runway edge recognition on each of the 20 image blocks included in the local image block matrix 203, generating a first image block recognition information set containing the recognition information of the 20 first image blocks. Figure 3 For example, the six image blocks in the lower right corner of the local image block matrix 203 contain runway edges. Therefore, the first image block recognition information corresponding to the six image blocks in the lower right corner of the local image block matrix 203 meets the filtering conditions. Thus, the first image block recognition information corresponding to the six image blocks in the lower right corner of the local image block matrix 203 is used as the second image block recognition information set (containing six second image block recognition information). Based on the runway edge positions included in the six second image block recognition information, runway edge fitting is performed to obtain the local runway edges corresponding to the two long sides (airport runway) and the local runway edge corresponding to the one short side (airport runway). And based on the runway region recognition direction vector of the local runway edges corresponding to the two long sides, the runway region recognition direction is generated.

[0064] Step S5: Based on the second image block recognition information set, perform runway edge recognition on the image blocks in the image block matrix along the runway region recognition direction to obtain the third image block recognition information set.

[0065] The third image block recognition information includes the image block location, runway edge location, and runway edge marker.

[0066] In practice, firstly, the aforementioned execution entity can construct two rays along the local runway edges corresponding to the two fitted long sides of the second image block recognition information set, using the two runway region recognition direction vectors included in the runway region recognition direction as directions. Then, the aforementioned execution entity only performs runway edge recognition on the image blocks through which the two rays pass, obtaining a third image block recognition information set, wherein the runway edge recognition still uses the weak classifier in step S3 above.

[0067] As an example, see further. Figure 3 Excluding the image blocks included in the local image block matrix, the aforementioned execution entity only needs to perform runway edge recognition on the 24 image blocks traversed by the two rays to obtain the third image block recognition information set. It can be seen that... Figure 3 Taking the image block matrix 202 containing 75 image blocks as an example, in the process of runway area recognition, only 44 (20+24) of the image blocks need to be feature extracted and runway edge recognized, while the remaining 41.3% of the panoramic image does not need to be feature processed. At the same time, due to the use of a weak classifier, the amount of data processing in the runway edge recognition process is also relatively small.

[0068] Step S6: Determine the runway area boundary based on the second image block recognition information set and the third image block recognition information set.

[0069] In practice, the aforementioned execution entity can fit and construct a closed runway region boundary based on the runway edge position included in the second image block recognition information set and the runway edge position included in the third image block recognition information set.

[0070] Step S7: Based on the runway area boundary, crop the real-time panoramic image to obtain the runway area image.

[0071] In practice, conventional methods often use the entire image as input for region recognition, requiring deep and complex network structures to capture image features for accurate recognition. This also results in extremely high training and deployment costs for the models. However, this disclosure fully considers the characteristics of airport runways, effectively optimizing the recognition process and significantly reducing data processing volume while ensuring both recognition accuracy and speed.

[0072] Step S8: Perform runway light recognition on the above runway area image to obtain the set of reference point locations included in the above runway description information.

[0073] In practice, runway lights are primarily used to guide aircraft takeoffs and landings. They mainly use white, green, red, yellow, and blue, exhibiting a significant color difference compared to airport runways. Furthermore, runway lights often have circular color boundaries. Therefore, firstly, the aforementioned implementing entity can determine the regions of interest (ROIs) for the runway lights using a circle search (e.g., Hough circle detection), obtaining a set of ROIs. Then, ROIs that meet the region selection criteria are selected from this set and used as the runway light locations, serving as reference point locations. The region selection criteria are: the average color value corresponding to the ROI of the runway light must correspond to any one of the colors white, green, red, yellow, or blue.

[0074] Step S9: Based on the above set of reference point locations, determine the runway direction vector included in the above runway description information.

[0075] In practice, runway lights are often symmetrically distributed on both sides of the airport runway. Therefore, the runway orientation vector can be determined by straight-line fitting, i.e., the line corresponding to the runway orientation vector contains the most reference point positions. Specifically, the reason for re-determining the runway orientation vector based on the set of reference point positions is that: due to the strict requirements on the placement of runway lights (e.g., GB / T 7256.2-2023, Civil Airport Navigation Lights Part 2: Sequential Flashing Lights and Runway Threshold Identification Lights), the runway orientation vector obtained from the set of reference point positions can more accurately represent the orientation of the airport runway compared to the runway area boundary.

[0076] Step 103: Based on the runway description information, perform directional segmentation on the runway region image to obtain a first local runway region image and a second local runway region image.

[0077] In some embodiments, the aforementioned execution entity may perform directional segmentation of the runway region image based on runway description information to obtain a first local runway region image and a second local runway region image.

[0078] In practice, since the runway lights are symmetrically distributed on both sides of the airport runway, the runway area image can be segmented using the runway direction vector as the segmentation direction and the centerline of the symmetrical multiple sets of runway lights as the segmentation position, to obtain the first local runway area image and the second local runway area image mentioned above.

[0079] In some optional implementations of certain embodiments, the execution entity performs directional segmentation on the runway region image based on the runway description information to obtain a first local runway region image and a second local runway region image, including:

[0080] Step S1: Along the direction corresponding to the above runway direction vector, perform position matching on the reference point positions in the above reference point position set to obtain a set of reference point position pairs.

[0081] In this case, the line connecting the reference point positions included in each reference point position pair in the reference point position pair set forms the same angle with the direction of the runway direction vector.

[0082] In practice, since runway lights are symmetrically positioned on both sides of the airport runway, and the angle between the line connecting the two reference points corresponding to the two runway lights and the runway direction vector is a fixed value, the reference point position corresponding to the runway light on the same side of the airport runway can be determined based on the runway direction vector.

[0083] As an example, the set of reference point positions may include: Reference Point Position 1, Reference Point Position 2, Reference Point Position 3, Reference Point Position 4, Reference Point Position 5, and Reference Point Position 6. Reference Point Position 1, Reference Point Position 2, and Reference Point Position 3 lie on the same straight line parallel to the runway direction vector. Reference Point Position 4, Reference Point Position 5, and Reference Point Position 6 lie on the same straight line parallel to the runway direction vector. Next, the angles between the line connecting Reference Point Position 1 and Reference Point Positions 4, 5, and 6 and the runway direction vector are calculated, resulting in [Angle 11, Angle 12, Angle 13]. Similarly, the angles between the line connecting Reference Point Position 2 and Reference Point Positions 4, 5, and 6 and the runway direction vector are calculated, resulting in [Angle 21, Angle 22, Angle 23]. Calculate the angles between the line connecting reference point positions 3, 4, 5, and 6 and the runway direction vector, obtaining [direction angle 31, direction angle 32, direction angle 33]. Since the angles between the line connecting the two reference point positions in each reference point pair and the runway direction vector are the same, the reference point positions can be matched based on the previously calculated direction angles. For example, if direction angles 11, 22, and 33 are the same and the distance between the two reference point positions at the same direction angle is the shortest, then reference point positions 1 and 4 form a reference point pair. Reference point positions 2 and 5 form a reference point pair. Reference point positions 3 and 6 form a reference point pair. In particular, since airport runways are often long (for example, the runway length often varies depending on the airport class), multiple pairs of runway lights are distributed on both sides of the airport runway. Since the line connecting the two reference points corresponding to each matched runway light forms the same angle with the runway direction vector, to reduce computational complexity and time consumption, the direction angles between some position reference points and other position reference points can be calculated. Based on the constraint that the line connecting the two reference points corresponding to each matched runway light forms the same angle with the runway direction vector, the specific value of the direction angle is determined. This allows for the rapid construction of reference point position pairs based on the specific direction angles.

[0084] Step S2: For each pair of reference point positions in the above set of reference point positions, determine the midpoint position based on the above reference point position pair.

[0085] The midpoint position represents the midpoint of the line segment formed by the reference point position pair, with the reference point position as the endpoint.

[0086] In practice, the average of the positions of the two reference points can be taken as the midpoint position.

[0087] Step S3: Based on the obtained set of midpoint positions, fit and generate image segmentation lines.

[0088] In practice, linear fitting can be performed based on the obtained set of midpoint positions, for example, by using the least squares method to obtain the above image segmentation lines.

[0089] Step S4: Based on the above image segmentation lines, perform image segmentation on the above runway region image to obtain the above first local runway region image and the above second local runway region image.

[0090] Step 104: Identify rubber attachments in the first local runway area image to obtain description information of the first attachments.

[0091] In some embodiments, the aforementioned executing entity may perform rubber attachment identification on the first local runway area image to obtain first attachment description information.

[0092] The first description of the attachment includes: rubber attachment confidence, rubber attachment location information, and rubber attachment severity. The rubber attachment confidence represents the confidence that a rubber attachment exists at the location corresponding to the rubber attachment location information. The rubber attachment confidence value ranges from [0,1]. A higher rubber attachment confidence value indicates a higher confidence in the existence and severity of the rubber attachment at the location corresponding to the rubber attachment location information. The rubber attachment location information represents the identified location where a rubber attachment exists. The rubber attachment severity represents the severity of the rubber attachment present at the location corresponding to the rubber attachment location information. The rubber attachment severity value ranges from [0,1]. A higher rubber attachment severity value indicates a higher severity of the rubber attachment present at the location corresponding to the rubber attachment location information.

[0093] In practice, since rubber residue is generated by the friction between aircraft tires and airport runways during landing, multiple factors, such as aircraft type, weather conditions, tire material, and runway material, can affect the location and severity of the residue. Therefore, traditional segmentation methods are insufficient to guarantee segmentation accuracy. This disclosure employs a machine learning approach, using the YOLOv3-Tiny model to identify rubber residue in a first local runway region image, obtaining descriptive information about the residue. Specifically, since the severity of the rubber residue also needs to be output, a multi-classifier is added to the YOLOv3-Tiny model to output the severity. The YOLOv3-Tiny model is trained in a supervised manner using training samples labeled with rubber residue location and severity information.

[0094] In some optional implementations of certain embodiments, the execution entity performs rubber attachment identification on the first local runway area image to obtain first attachment description information, including:

[0095] Step S1: Using a multi-scale feature extraction network, perform multi-scale feature extraction on the first local runway region image to obtain the first image feature map.

[0096] As an example, see Figure 4The diagram shows the network structure of the rubber attachment recognition network. The multi-scale feature extraction network consists of a downsampling network and an upsampling network. Specifically, the downsampling network comprises downsampling layers A1, A2, A3, and A4. The input to downsampling layer A1 is either a first local runway region image or a second local runway region image. The input to downsampling layer A2 is the output of downsampling layer A1. The input to downsampling layer A3 is the output of downsampling layer A2. The input to downsampling layer A4 is the output of downsampling layer A3. The upsampling network consists of upsampling layers A1, A2, A3, and A4. A convolutional layer A1 is placed between downsampling layer A1 and upsampling layer A1. A convolutional layer A2 is placed between downsampling layer A2 and upsampling layer A2. A convolutional layer A3 is placed between downsampling layer A3 and upsampling layer A3. A convolutional layer A4 is placed between downsampling layer A4 and upsampling layer A4. Convolutional layers A1, A2, A3, and A4 are all 1×1 convolutional layers. The output of downsampling layer A1 is convolved by convolutional layer A1 and then superimposed with the output of upsampling layer A2, serving as the input to upsampling layer A1. The output of downsampling layer A2 is convolved by convolutional layer A2 and then superimposed with the output of upsampling layer A3, serving as the input to upsampling layer A2. The output of downsampling layer A3 is convolved by convolutional layer A3 and then superimposed with the output of upsampling layer A4, serving as the input to upsampling layer A3. The output of downsampling layer A4 is convolved by convolutional layer A4 and then serves as the input to upsampling layer A4. The output of upsampling layer A1 is either a first image feature map or a second image feature map. Assuming the image size of the first local runway region image is W×L, then the output size of downsampling layer A1 is W / 2×L / 2. The output size of downsampling layer A2 is W / 4×L / 4. The output size of downsampling layer A3 is W / 8 × L / 8. The output size of downsampling layer A4 is W / 16 × L / 16. The output size of upsampling layer A4 is W / 8 × L / 8. The output size of upsampling layer A3 is W / 4 × L / 4. The output size of upsampling layer A2 is W / 2 × L / 2. The output size of upsampling layer A1 is W × L. Multi-scale feature extraction is used to extract features from different receptive fields, thereby improving the expressive power of the features. Simultaneously, feature overlay and upsampling ensure that the output feature map has the same image size as the original image (either the first or second local runway region image). Furthermore, since only one feature map (either the first or second image feature map) is output after overlay, subsequent texture feature extraction networks, the aforementioned brightness feature extraction networks, and the aforementioned color feature extraction networks no longer need to set up independent feature extraction networks for features in different receptive fields, thus simplifying the network structure.

[0097] Step S2: Generate the first texture feature based on the first image feature map and texture feature extraction network described above.

[0098] Step S3: Generate the first brightness feature based on the first image feature map and the brightness feature extraction network described above.

[0099] Step S4: Generate the first color feature based on the first image feature map and the color feature extraction network described above.

[0100] As an example, see further. Figure 4 The texture feature extraction network, brightness feature extraction network, and color feature extraction network are configured in parallel. Specifically, the network structures of the texture feature extraction network, brightness feature extraction network, and color feature extraction network are consistent, all using the ResNet18 network. The reason for using the ResNet18 network instead of a deeper network structure is that the multi-scale feature extraction network has already performed multi-scale feature extraction. To avoid invalid feature extraction, the lightweight ResNet18 network is used. Furthermore, the identical network structure facilitates model training and updates.

[0101] Step S5: Based on the first texture feature, the first brightness feature, the first color feature, and the rubber attachment locator, generate a first confidence level and the rubber attachment location information included in the first attachment description information.

[0102] The rubber attachment locator employs an RPN (Region Proposal Network). The first confidence level represents the confidence level that a rubber attachment exists at the location corresponding to the rubber attachment location information.

[0103] In practice, further refer to Figure 4Since the texture feature extraction network, brightness feature extraction network, and color feature extraction network have the same network structure and input, the output first texture feature, first brightness feature, and first color feature have the same feature size. Therefore, the first texture feature, first brightness feature, and first color feature can be directly superimposed as input to the rubber attachment locator. In particular, during feature superposition, a 1×3 weight matrix controls the feature influence of the first texture feature, first brightness feature, and first color feature. Specifically, since the texture feature extraction network, brightness feature extraction network, and color feature extraction network have the same network structure, they do not assign weights to different features during feature extraction. However, in reality, factors such as weather, tire material, aircraft weight, and airport runway material can affect the performance of rubber attachments in the three dimensions of texture, brightness, and color. Therefore, this disclosure controls the feature influence by setting a 1×3 weight matrix. During model training, the initial 1×3 weight matrix is ​​[0.33, 0.33, 0.34], and the weight matrix is ​​updated during model training. This method allows the model to automatically learn the weights of features, while avoiding the inaccuracies of manually adjusting the model depth to affect the influence of features.

[0104] Step S6: Based on the first texture feature, the first brightness feature, the first color feature, and the rubber attachment severity classifier, generate a second confidence score and the rubber attachment severity included in the first attachment description information.

[0105] The aforementioned multi-scale feature extraction network, texture feature extraction network, brightness feature extraction network, color feature extraction network, rubber attachment locator, and rubber attachment severity classifier are all included in the rubber attachment recognition network. The rubber attachment severity classifier consists of multiple fully connected layers connected in series and one Softmax layer. The multiple fully connected layers are used to perform feature shaping on the input of the rubber attachment severity classifier. The second confidence level represents the confidence level of the rubber attachment severity.

[0106] As an example, see further. Figure 4 The first texture feature, the first brightness feature, and the first color feature, after feature superposition, are used as inputs to the rubber adhesion severity classifier. Specifically, during feature superposition, a 1×3 weight matrix controls the feature influence of the first texture feature, the first brightness feature, and the first color feature.

[0107] Step S7: The first confidence level and the second confidence level are weighted and summed to obtain the confidence level of the rubber attachment included in the first attachment description information.

[0108] In practice, the aforementioned implementing entity can average and weight the first confidence level and the second confidence level to obtain the confidence level of the rubber attachments included in the first attachment description information.

[0109] Step 105: Based on the first attachment description information, identify the rubber attachments in the second local runway area image to obtain the second attachment description information.

[0110] In some embodiments, the aforementioned execution entity can identify rubber attachments in the second local runway area image based on the first attachment description information to obtain the second attachment description information.

[0111] The aforementioned description information of the second type of attachment includes: rubber attachment confidence, rubber attachment location information, and rubber attachment severity. The rubber attachment confidence represents the confidence that a rubber attachment exists at the location corresponding to the rubber attachment location information. The rubber attachment confidence value ranges from [0,1]. A higher rubber attachment confidence value indicates a higher confidence in the existence and severity of the rubber attachment at the location corresponding to the rubber attachment location information. The rubber attachment location information represents the identified location where a rubber attachment exists. The rubber attachment severity represents the severity of the rubber attachment present at the location corresponding to the rubber attachment location information. The rubber attachment severity value ranges from [0,1]. A higher rubber attachment severity value indicates a higher severity of the rubber attachment present at the location corresponding to the rubber attachment location information.

[0112] In practice, since the first and second local runway region images are approximately symmetrical along the image segmentation line, and when the aircraft lands, the symmetrical set of tires rubs against the airport runway, producing an approximately symmetrical set of rubber attachments, the location of the rubber attachments in the second local runway region image corresponding to the first attachment information can be taken as the region of interest along the image segmentation line. Rubber attachment identification is then performed on the second local runway region image to obtain the second attachment descriptor information. The rubber attachment identification also employs a rubber attachment identification network; the network structure of the rubber attachment identification network is described in step 104 and will not be repeated here.

[0113] In some optional implementations of certain embodiments, the execution entity identifies rubber attachments in the second local runway area image based on the first attachment description information to obtain second attachment description information, including:

[0114] Step S1: Using the image segmentation line as the axis of symmetry, determine the symmetrical position of the rubber attachment location information included in the first attachment description information in the second local runway area image, and use it as the target position.

[0115] The target location can represent the region of interest of the rubber attachments present in the second local runway area image.

[0116] Step S2: Using the multi-scale feature extraction network described above, multi-scale feature extraction is performed on the second local runway region image to obtain the second image feature map.

[0117] The process of the multi-scale feature extraction network extracting multi-scale features from the second local runway region image to generate the second image feature map can be found in step S1 of step 104, and will not be repeated here.

[0118] Step S3: The local image features corresponding to the target position in the second image feature map are enhanced to obtain the third image feature map.

[0119] In practice, firstly, the aforementioned executing entity can generate an influence factor feature map, where the feature size of the influence factor feature map is consistent with the feature size of the second image feature map. The value range of the influence factors within the influence factor feature map is [0.5, 1]. The influence factor value at the location corresponding to the target location in the influence factor feature map is 1. Then, taking the location corresponding to the target location in the influence factor feature map as the center, the values ​​of the remaining influence factors are determined radially. Specifically, the closer the location is to the location corresponding to the target location in the influence factor feature map, the closer the influence factor value is to 1; the farther the location is from the location corresponding to the target location in the influence factor feature map, the closer the influence factor value is to 0.5. For example, assuming the distance from the location corresponding to the target location in the influence factor feature map to the nearest boundary in the influence factor feature map is S1, and the distance from the location corresponding to the target location in the influence factor feature map to the influence factor to be valued is S2, then: S1 / (1 - 0.5) = S2 / (the specific value of the influence factor to be valued), thus the values ​​of all influence factors within the influence factor feature map can be determined. Then, the values ​​at corresponding positions in the influence factor feature map and the second image feature map are multiplied to obtain the third image feature map.

[0120] Step S4: Based on the third image feature map and the rubber attachment recognition network including the texture feature extraction network, the brightness feature extraction network, the color feature extraction network, the rubber attachment locator, and the rubber attachment severity classifier, generate a third confidence score, a fourth confidence score, and the rubber attachment location information and rubber attachment severity information included in the second attachment description information.

[0121] In practice, firstly, the aforementioned execution entity can input the third image feature map in parallel into the aforementioned texture feature extraction network, brightness feature extraction network, and color feature extraction network to obtain the second texture feature, the second brightness feature, and the second color feature. Then, the execution entity can perform feature superposition on the second texture feature, the second brightness feature, and the second color feature, using this as input to the rubber attachment locator to obtain the third confidence score and the rubber attachment location information included in the second attachment description information. Specifically, during the feature superposition process, a 1×3 weight matrix controls the feature influence of the second texture feature, the second brightness feature, and the second color feature. Next, the superimposed feature is used as input to the rubber attachment severity classifier to obtain the fourth confidence score and the rubber attachment severity included in the second attachment description information.

[0122] Step S5: Perform a weighted summation of the third confidence level and the fourth confidence level to obtain the confidence level of the rubber attachments included in the second attachment description information.

[0123] In practice, the aforementioned implementing entity can average and weight the third confidence level and the aforementioned fourth confidence level to obtain the confidence level of the rubber attachments included in the aforementioned second attachment description information.

[0124] Step 106: Generate airport runway anomaly warning information based on the reference point location set, the first attachment description information, and the second attachment description information.

[0125] In some embodiments, the aforementioned executing entity may generate airport runway anomaly warning information based on the reference point location set, the first attachment descriptor information, and the second attachment descriptor information.

[0126] Among them, airport runway anomaly warning information can be warning information indicating the presence of rubber deposits within the airport runway.

[0127] In practice, since the reference point location corresponds to the runway light, and the three-dimensional coordinates of the runway light and the corresponding reference point location (image coordinates) are known, the reference point location set can be used as a reference to perform coordinate transformation on the rubber attachment location information (image coordinates) corresponding to the first attachment description information and the second attachment description information to obtain the three-dimensional coordinates corresponding to the rubber attachment location information. Based on the rubber attachment severity and rubber attachment confidence included in the first attachment description information and the second attachment description information, and according to the pre-set triggering rules, airport runway anomaly warning information of different severity levels can be automatically generated.

[0128] In some optional implementations of certain embodiments, the execution entity generates airport runway anomaly warning information based on the reference point location set, the first attachment description information, and the second attachment description information, including:

[0129] Step S1: In response to the fact that the confidence level of the rubber attachment included in the first attachment description information is greater than the preset confidence level and the confidence level of the rubber attachment included in the second attachment description information is greater than the preset confidence level, according to the reference point location set, the position coordinates of the rubber attachment location information included in the first attachment description information and the rubber attachment location information included in the second attachment description information are transformed to obtain the first transformed position information and the second transformed position information.

[0130] In practice, since the reference point position corresponds to the runway light, and the three-dimensional coordinates of the runway light and the corresponding reference point position (two-dimensional image coordinates) are known, that is, the mapping relationship between the three-dimensional coordinate system and the two-dimensional image coordinate system is known, the position coordinate transformation can be performed on the rubber attachment position information included in the first attachment description information and the rubber attachment position information included in the second attachment description information according to the set of reference point positions, to obtain the first transformed position information and the second transformed position information.

[0131] Step S2: In response to the fact that the severity of the rubber attachments included in the first attachment description information is greater than a preset severity and the severity of the rubber attachments included in the second attachment description information is less than or equal to the preset severity, the airport runway anomaly warning information is generated based on the first converted location information.

[0132] Step S3: In response to the fact that the severity of the rubber attachment included in the first attachment description information is less than or equal to a preset severity and the severity of the rubber attachment included in the second attachment description information is greater than the preset severity, the airport runway anomaly warning information is generated based on the second converted location information.

[0133] Step S4: In response to the fact that the severity of the rubber attachments included in the first attachment description information is greater than a preset severity and the severity of the rubber attachments included in the second attachment description information is greater than a preset severity, the airport runway anomaly warning information is generated based on the first converted location information and the second converted location information.

[0134] The airport runway anomaly warning information may include: anomaly location and anomaly level. The anomaly location can represent the first-transformed location information and / or the first-transformed location information. The anomaly level can be mapped based on the severity of rubber adhesions.

[0135] The various embodiments of this disclosure have the following beneficial effects: The airport runway abnormal target identification method of some embodiments of this disclosure achieves all-weather automatic and accurate identification of rubber attachments on the airport runway surface. In practice, existing methods typically use manual periodic inspections to identify and process rubber attachments within the airport runway, which is often lagging. Furthermore, due to the large area of ​​airport runways and the high priority of aircraft takeoff and landing missions, it is difficult to conduct all-weather inspections of rubber attachments manually. Therefore, the airport runway abnormal target identification method of some embodiments of this disclosure first performs panoramic image stitching on a real-time image set to generate a real-time panoramic image. This real-time image set is acquired by a high-magnification full-color camera facing the airport runway. In practice, this disclosure uses a high-magnification full-color camera positioned at a high location to achieve real-time image acquisition of the airport runway, and uses image stitching to remove redundant parts between real-time images and reduce subsequent invalid data processing. Secondly, runway region identification is performed on the aforementioned real-time panoramic image to obtain runway region images and runway description information. The runway description information includes: runway direction vectors and a set of reference point positions, where the reference point positions correspond to runway lights installed within the airport runway. In practice, due to factors such as shooting angle, real-time panoramic images often include areas outside the airport runway. Since flight operations primarily occur on airport runways, it is unnecessary to identify rubber attachments in areas outside the runway. Therefore, this disclosure further eliminates images corresponding to areas outside the airport runway in the real-time panoramic image through runway region identification. Next, based on the aforementioned runway description information, the runway region image is directionally segmented to obtain a first local runway region image and a second local runway region image. Furthermore, rubber attachment identification is performed on the first local runway region image to obtain first attachment description information. In addition, based on the aforementioned first attachment description information, rubber attachments are identified in the aforementioned second local runway area image to obtain second attachment description information. Both the first and second attachment description information include: rubber attachment confidence, rubber attachment location information, and rubber attachment severity. In practice, runway areas often exhibit symmetry, and the rubber attachments generated by aircraft during landing also tend to be symmetrical. Therefore, this disclosure, through directional segmentation and prior rubber attachment identification of local runway area images, reduces data processing volume and increases identification speed compared to directly identifying the runway area image. Finally, based on the aforementioned reference point location set, the aforementioned first attachment description information, and the aforementioned second attachment description information, airport runway anomaly warning information is generated. In summary, this achieves all-weather automatic and accurate identification of rubber attachments on the airport runway surface.

[0136] Further reference Figure 5As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an airport runway abnormal target identification device, which are similar to... Figure 1 Corresponding to the method embodiments shown, the airport runway abnormal target identification device can be specifically applied to various electronic devices.

[0137] like Figure 5 As shown, an airport runway abnormal target identification device 500 in some embodiments includes: a panoramic image stitching unit 501, a runway area identification unit 502, a directional segmentation unit 503, a first rubber attachment identification unit 504, a second rubber attachment identification unit 505, and a generation unit 506. The panoramic image stitching unit 501 is configured to stitch together a set of real-time images to generate a real-time panoramic image. The set of real-time images is acquired by a high-magnification full-color camera facing the airport runway. The runway area identification unit 502 is configured to identify the runway area from the real-time panoramic image to obtain a runway area image and runway description information. The runway description information includes a runway direction vector and a set of reference point positions, where the reference point positions correspond to runway lights installed within the airport runway. The directional segmentation unit 503 is configured to... The system uses runway description information to perform directional segmentation on the runway area image to obtain a first local runway area image and a second local runway area image. A first rubber attachment identification unit 504 is configured to identify rubber attachments in the first local runway area image to obtain first attachment description information. A second rubber attachment identification unit 505 is configured to identify rubber attachments in the second local runway area image based on the first attachment description information to obtain second attachment description information. Both the first and second attachment description information include: rubber attachment confidence, rubber attachment location information, and rubber attachment severity. A generation unit 506 is configured to generate airport runway anomaly warning information based on the reference point location set, the first attachment description information, and the second attachment description information.

[0138] It is understandable that the units recorded in the airport runway abnormal target identification device 500 are related to the reference Figure 1 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the airport runway abnormal target identification device 500 and the units contained therein, and will not be repeated here.

[0139] The following is for reference. Figure 6 It shows a schematic diagram of the structure of an electronic device (e.g., a computing device) 600 suitable for implementing some embodiments of the present disclosure. Figure 6The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0140] like Figure 6 As shown, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 or a program loaded from a storage device 608 into a random access memory 603. The random access memory 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, the read-only memory 602, and the random access memory 603 are interconnected via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0141] Typically, the following devices can be connected to the input / output interface 605: input devices 606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 608 including, for example, magnetic tape, hard disk, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0142] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a read-only memory 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0143] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0144] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0145] The aforementioned computer-readable medium may be included within the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: perform panoramic image stitching on a set of real-time images to generate a real-time panoramic image, wherein the set of real-time images is acquired by a high-magnification full-color camera facing the airport runway; perform runway region identification on the real-time panoramic image to obtain a runway region image and runway description information, wherein the runway description information includes: a runway direction vector and a set of reference point positions, the reference point positions corresponding to runway lights installed within the airport runway; and perform directional segmentation on the runway region image based on the runway description information. A first local runway area image and a second local runway area image are obtained; rubber attachments are identified in the first local runway area image to obtain first attachment description information; based on the first attachment description information, rubber attachments are identified in the second local runway area image to obtain second attachment description information, wherein both the first and second attachment description information include: rubber attachment confidence, rubber attachment location information, and rubber attachment severity; based on the reference point location set, the first attachment description information, and the second attachment description information, an airport runway anomaly warning information is generated.

[0146] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0149] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for identifying abnormal targets on an airport runway, characterized in that, include: A real-time panoramic image is generated by stitching together a set of real-time images, wherein the set of real-time images is acquired by a high-magnification full-color camera facing the airport runway. Runway area identification is performed on the real-time panoramic image to obtain runway area image and runway description information. The runway description information includes: runway direction vector and reference point position set, where the reference point position corresponds to the runway lights set in the airport runway. Based on the runway description information, the runway region image is directionally segmented to obtain a first local runway region image and a second local runway region image, wherein the first local runway region image and the second local runway region image are approximately symmetrical along the image segmentation line; Rubber attachments are identified in the first local runway area image to obtain description information of the first attachments; Based on the first attachment description information, rubber attachments are identified in the second local runway area image to obtain second attachment description information. Both the first attachment description information and the second attachment description information include: rubber attachment confidence, rubber attachment location information, and rubber attachment severity. Based on the reference point location set, the description information of the first attachment, and the description information of the second attachment, an airport runway anomaly warning information is generated. The step of identifying rubber attachments in the first local runway area image to obtain first attachment description information includes: A multi-scale feature extraction network is used to extract multi-scale features from the first local runway region image to obtain a first image feature map. The multi-scale feature extraction network consists of a downsampling network and an upsampling network. The downsampling network consists of downsampling layer A1, downsampling layer A2, downsampling layer A3, and downsampling layer A4. The upsampling network consists of upsampling layer A1, upsampling layer A2, upsampling layer A3, and upsampling layer A4. A convolutional layer A1 is set between downsampling layer A1 and upsampling layer A1. A convolutional layer A2 is set between downsampling layer A2 and upsampling layer A2. A convolutional layer A3 is set between downsampling layer A3 and upsampling layer A3. A convolutional layer A4 is set between downsampling layer A4 and upsampling layer A4. Based on the first image feature map and the texture feature extraction network, a first texture feature is generated; Based on the first image feature map and the brightness feature extraction network, a first brightness feature is generated; Based on the first image feature map and the color feature extraction network, a first color feature is generated, wherein the texture feature extraction network, the brightness feature extraction network and the color feature extraction network are set in parallel. Based on the first texture feature, the first brightness feature, the first color feature, and the rubber attachment locator, a first confidence level and the rubber attachment location information included in the first attachment description information are generated. Based on the first texture feature, the first brightness feature, the first color feature, and the rubber attachment severity classifier, a second confidence score and the rubber attachment severity included in the first attachment description information are generated. The multi-scale feature extraction network, the texture feature extraction network, the brightness feature extraction network, the color feature extraction network, the rubber attachment locator, and the rubber attachment severity classifier are all included in the rubber attachment recognition network. The first texture feature, the first brightness feature, and the first color feature are superimposed, and a 1×3 weight matrix controls the feature influence of the first texture feature, the first brightness feature, and the first color feature, serving as the input to the rubber attachment locator. The first confidence level and the second confidence level are weighted and summed to obtain the confidence level of the rubber attachment included in the first attachment description information.

2. The method for identifying abnormal targets on an airport runway according to claim 1, characterized in that, The step of stitching together a set of real-time images to generate a real-time panoramic image includes: Image distortion correction is performed on each real-time image in the real-time image set to generate a corrected image, thus obtaining a corrected image set; Based on the corrected image set, determine the global brightness gain information; Based on the global brightness gain information, brightness compensation is performed on each corrected image in the corrected image set to generate a brightness-compensated image, thus obtaining a brightness-compensated image set. Each brightness-compensated image in the brightness-compensated image set is reduced in resolution to generate a low-resolution image, resulting in a low-resolution image set, wherein the image resolution of the low-resolution image is 3 / 4 of the image resolution of the brightness-compensated image. Edge features are extracted from each low-resolution image in the set of low-resolution images to obtain an edge feature group; Based on the edge feature groups corresponding to the low-resolution image, the brightness-compensated images in the brightness-compensated image set are stitched together to obtain the real-time panoramic image.

3. The airport runway abnormal target identification method according to claim 2, characterized in that, The step of performing directional segmentation of the runway region image based on the runway description information to obtain a first local runway region image and a second local runway region image includes: Along the direction corresponding to the runway direction vector, the reference point positions in the reference point position set are matched to obtain a reference point position pair set, wherein the line connecting the reference point positions included in each reference point position pair in the reference point position pair set has the same angle as the direction of the runway direction vector. For each pair of reference point positions in the set of reference point positions, the midpoint position is determined based on the reference point position pair, wherein the midpoint position represents the midpoint of the line segment formed by the reference point positions included in the reference point position pair as endpoints; Based on the obtained set of midpoint positions, image segmentation lines are fitted and generated. Based on the image segmentation lines, the runway region image is segmented to obtain the first local runway region image and the second local runway region image.

4. The method for identifying abnormal targets on an airport runway according to claim 3, characterized in that, The step of performing runway region identification on the real-time panoramic image to obtain runway region images and runway description information includes: The panoramic image is divided into image blocks to obtain an image block matrix, wherein the image blocks in the image block matrix have the same size, and the matrix dimension of the image block matrix is ​​N×M; Extract the sub-image block matrix located at the beginning of the image block matrix as a local image block matrix, wherein the matrix dimension of the local image block matrix is ​​S×M, and S is less than N; Runway edge recognition is performed on each image block in the local image block matrix to generate first image block recognition information, resulting in a first image block recognition information set, wherein the first image block recognition information includes: image block position, runway edge position, and runway edge identifier; First image block recognition information that meets the filtering conditions is selected from the first image block recognition information set and used as second image block recognition information to obtain a second image block recognition information set. The filtering condition is: the runway edge identifier included in the first image block recognition information indicates that the corresponding image block contains a runway edge. Based on the second image block recognition information set, the runway area recognition direction is determined; Based on the second image block recognition information set, runway edge recognition is performed on the image blocks in the image block matrix along the runway region recognition direction to obtain the third image block recognition information set. The runway area boundary is determined based on the second image block recognition information set and the third image block recognition information set; Based on the runway area boundary, the real-time panoramic image is cropped to obtain the runway area image; Runway light recognition is performed on the runway area image to obtain the set of reference point locations included in the runway description information; Based on the set of reference point locations, the runway direction vector included in the runway description information is determined.

5. The airport runway abnormal target identification method according to claim 4, characterized in that, The step of identifying rubber attachments in the second local runway area image based on the first attachment description information to obtain the second attachment description information includes: Using the image segmentation line as the axis of symmetry, determine the symmetrical position of the rubber attachment location information included in the first attachment description information in the second local runway area image, and use it as the target position; The second local runway region image is subjected to multi-scale feature extraction through the multi-scale feature extraction network to obtain the second image feature map. The local image features corresponding to the target location in the second image feature map are enhanced to obtain a third image feature map; Based on the third image feature map and the rubber attachment recognition network including the texture feature extraction network, the brightness feature extraction network, the color feature extraction network, the rubber attachment locator, and the rubber attachment severity classifier, a third confidence score, a fourth confidence score, and the rubber attachment location information and rubber attachment severity included in the second attachment description information are generated. The third confidence level and the fourth confidence level are weighted and summed to obtain the confidence level of the rubber attachment included in the second attachment description information.

6. The method for identifying abnormal targets on an airport runway according to claim 5, characterized in that, The step of generating airport runway anomaly warning information based on the reference point location set, the first attachment description information, and the second attachment description information includes: In response to the fact that the confidence level of the rubber attachment included in the first attachment description information is greater than a preset confidence level and the confidence level of the rubber attachment included in the second attachment description information is greater than a preset confidence level, the position coordinates of the rubber attachment position information included in the first attachment description information and the rubber attachment position information included in the second attachment description information are transformed according to the reference point position set to obtain the first transformed position information and the second transformed position information. In response to the fact that the severity of the rubber attachment included in the first attachment description information is greater than a preset severity and the severity of the rubber attachment included in the second attachment description information is less than or equal to the preset severity, the airport runway anomaly warning information is generated based on the first converted location information. In response to the fact that the severity of the rubber attachment included in the first attachment description information is less than or equal to a preset severity and the severity of the rubber attachment included in the second attachment description information is greater than the preset severity, the airport runway anomaly warning information is generated based on the second converted location information. In response to the fact that the severity of the rubber attachments included in the first attachment description information is greater than a preset severity and the severity of the rubber attachments included in the second attachment description information is greater than a preset severity, the airport runway anomaly warning information is generated based on the first converted location information and the second converted location information.

7. An airport runway abnormal target identification device, characterized in that, include: A panoramic image stitching unit is configured to stitch together a set of real-time images to generate a real-time panoramic image, wherein the set of real-time images is acquired by a high-magnification full-color camera facing the airport runway. The runway area identification unit is configured to identify the runway area in the real-time panoramic image to obtain a runway area image and runway description information. The runway description information includes a runway direction vector and a set of reference point positions, where the reference point positions correspond to runway lights installed within the airport runway. The directional segmentation unit is configured to perform directional segmentation on the runway region image according to the runway description information to obtain a first local runway region image and a second local runway region image, wherein the first local runway region image and the second local runway region image are approximately symmetrical along the image segmentation line. The first rubber attachment identification unit is configured to identify rubber attachments in the first local runway area image to obtain first attachment description information. The second rubber attachment identification unit is configured to identify rubber attachments in the second local runway area image based on the first attachment description information to obtain second attachment description information. Both the first attachment description information and the second attachment description information include: rubber attachment confidence, rubber attachment location information, and rubber attachment severity. The generation unit is configured to generate airport runway anomaly warning information based on the reference point location set, the first attachment description information, and the second attachment description information. The step of identifying rubber attachments in the first local runway area image to obtain first attachment description information includes: A multi-scale feature extraction network is used to extract multi-scale features from the first local runway region image to obtain a first image feature map. The multi-scale feature extraction network consists of a downsampling network and an upsampling network. The downsampling network consists of downsampling layer A1, downsampling layer A2, downsampling layer A3, and downsampling layer A4. The upsampling network consists of upsampling layer A1, upsampling layer A2, upsampling layer A3, and upsampling layer A4. A convolutional layer A1 is set between downsampling layer A1 and upsampling layer A1. A convolutional layer A2 is set between downsampling layer A2 and upsampling layer A2. A convolutional layer A3 is set between downsampling layer A3 and upsampling layer A3. A convolutional layer A4 is set between downsampling layer A4 and upsampling layer A4. Based on the first image feature map and the texture feature extraction network, a first texture feature is generated; Based on the first image feature map and the brightness feature extraction network, a first brightness feature is generated; Based on the first image feature map and the color feature extraction network, a first color feature is generated, wherein the texture feature extraction network, the brightness feature extraction network and the color feature extraction network are set in parallel. Based on the first texture feature, the first brightness feature, the first color feature, and the rubber attachment locator, a first confidence level and the rubber attachment location information included in the first attachment description information are generated. Based on the first texture feature, the first brightness feature, the first color feature, and the rubber attachment severity classifier, a second confidence score and the rubber attachment severity included in the first attachment description information are generated. The multi-scale feature extraction network, the texture feature extraction network, the brightness feature extraction network, the color feature extraction network, the rubber attachment locator, and the rubber attachment severity classifier are all included in the rubber attachment recognition network. The first texture feature, the first brightness feature, and the first color feature are superimposed, and a 1×3 weight matrix controls the feature influence of the first texture feature, the first brightness feature, and the first color feature, serving as the input to the rubber attachment locator. The first confidence level and the second confidence level are weighted and summed to obtain the confidence level of the rubber attachment included in the first attachment description information.

8. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images