Methods and devices for identifying unidentified objects intruding into airspace

Abstract

This disclosure presents a method and apparatus for identifying unidentified intruders in airspace. One specific implementation of the method includes: acquiring a sequence of real-time image groups; for each real-time image group in the sequence, performing cross-image enhancement on a first real-time image, a second real-time image, and a third real-time image included in the real-time image group to obtain an enhanced real-time image group; performing anomaly detection based on the obtained enhanced real-time image group sequence to determine a target image group sequence; performing image fusion on each target image group in the target image group sequence to generate a fused image, obtaining a fused image sequence; and identifying unidentified intruders based on the fused image sequence to generate unidentified intruder identification information. This implementation achieves accurate identification of unidentified intruders.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer technology, target recognition, and image processing, and specifically to a method and apparatus for identifying unidentified intruders in flight zones. Background Technology

[0002] The airfield refers to the area used for aircraft takeoff, landing, taxiing, and parking, mainly including runways, taxiways, aprons, takeoff and landing strips, runway end safety zones, and areas housing instrument landing systems and approach lighting systems. The large grassy areas surrounding the airfield, as well as pipelines (e.g., drainage pipes, cable conduits), provide habitats for felines and canines, especially stray cats and dogs. Simultaneously, the small birds and rodents that congregate around the airfield provide food sources for stray cats and dogs. Furthermore, stray cats and dogs have a very high reproductive rate. This has led to a steady increase in the number of stray cats and dogs around airports. On the positive side, stray cats and dogs have a certain inhibitory effect on bird and rodent populations, thus reducing the risk of bird strikes and rodent damage to some extent; however, on the negative side, the increase in the number of stray cats and dogs also significantly increases the frequency and probability of intrusions into the airfield. Therefore, identifying intrusions by stray cats and dogs within the airfield plays a positive role in aircraft safety within the airfield. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of this disclosure propose methods and apparatus for identifying unidentified intruders in airspace to address the technical problems mentioned in the background section above.

[0005] In a first aspect, some embodiments of this disclosure provide a method for identifying unidentified intruders in a flight zone. The method includes: acquiring a sequence of real-time image groups, wherein the real-time image group consists of a first real-time image, a second real-time image, and a third real-time image from the same image acquisition location within the corresponding flight zone; the first real-time image is an infrared image, the second real-time image is a color image, and the third real-time image is a depth image; for each real-time image group in the above-mentioned real-time image group sequence, performing cross-image enhancement on the first real-time image, the second real-time image, and the third real-time image included in the above-mentioned real-time image group sequence to obtain an enhanced real-time image group; performing anomaly detection based on the obtained enhanced real-time image group sequence to determine a target image group sequence, wherein the target image group is the enhanced real-time image group in the above-mentioned enhanced real-time image group sequence that exhibits positional anomalies within the corresponding image acquisition location; performing image fusion on each target image group in the above-mentioned target image group sequence to generate a fused image, obtaining a fused image sequence; and identifying unidentified intruders based on the above-mentioned fused image sequence to generate unidentified intruder identification information, wherein the unidentified intruder identification information includes: intruding animal type, animal movement tendency, animal identification characteristics, intrusion area identifier, and intrusion time.

[0006] Secondly, some embodiments of this disclosure provide an intrusion identification device for an airspace. The device includes: a data acquisition unit configured to acquire a sequence of real-time image groups, wherein the real-time image group consists of a first real-time image, a second real-time image, and a third real-time image corresponding to the same image acquisition location within the airspace, wherein the first real-time image is an infrared image, the second real-time image is a color image, and the third real-time image is a depth image; a cross-image enhancement unit configured to perform cross-image enhancement on the first real-time image, the second real-time image, and the third real-time image included in each of the real-time image group sequences to obtain an enhanced real-time image group; and a determination unit. The system is configured to perform anomaly detection based on the obtained enhanced real-time image group sequence to determine the target image group sequence, wherein the target image group is the enhanced real-time image group in the above enhanced real-time image group sequence that has positional anomalies within the corresponding image acquisition location; the image fusion unit is configured to perform image fusion on each target image group in the above target image group sequence to generate a fused image, thereby obtaining a fused image sequence; the generation unit is configured to perform intrusion unidentified object identification based on the above fused image sequence to generate intrusion unidentified object identification information, wherein the intrusion unidentified object identification information includes: intrusion animal type, animal movement tendency, animal identification characteristics, intrusion area identifier, and intrusion time.

[0007] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0008] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0009] The above embodiments of this disclosure have the following beneficial effects: The method for identifying unidentified intruders in flight zones, based on some embodiments of this disclosure, achieves accurate identification of unidentified intruders. Specifically, firstly, a sequence of real-time image groups is acquired. Each real-time image group consists of a first real-time image, a second real-time image, and a third real-time image from the same image acquisition location within the corresponding flight zone. The first real-time image is an infrared image, the second real-time image is a color image, and the third real-time image is a depth image. In practice, stray cats and dogs are nocturnal, and flight zones have strict light control requirements. Therefore, to ensure subsequent identification accuracy, this disclosure uses three different image sources as the basis for identification. Secondly, for each real-time image group in the above real-time image group sequence, cross-image enhancement is performed on the first, second, and third real-time images included in the real-time image group to obtain an enhanced real-time image group. In practice, different image types have different advantages and disadvantages. For example, infrared images mainly reflect thermal radiation characteristics but lack texture and geometric details; color images show differences in texture detail accuracy under different light intensities; and depth images mainly reflect geometric details but lack texture details. Therefore, this disclosure improves image quality through cross-image enhancement. Next, anomaly detection is performed on the obtained enhanced real-time image group sequence to determine the target image group sequence. The target image group is the enhanced real-time image group within the aforementioned enhanced real-time image group sequence that exhibits positional anomalies at the corresponding image acquisition location. In practice, stray cats and dogs exhibit significant displacement changes when entering the flight zone, and the frequency and probability of stray cats and dogs entering the flight zone are uncertain. This results in a large number of invalid images in the continuously acquired real-time images. Therefore, this disclosure uses anomaly detection to quickly extract useful images. Furthermore, image fusion is performed on each target image group in the aforementioned target image group sequence to generate a fused image, resulting in a fused image sequence. Image fusion transforms images of different graphic types into the same feature representation space. Finally, intrusion identification is performed based on the aforementioned fused image sequence to generate intrusion identification information, which includes: intruding animal type, animal movement tendency, animal identification characteristics, intrusion area identifier, and intrusion time. In summary, this disclosure achieves accurate intrusion identification. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0011] Figure 1This is a flowchart of some embodiments of the method for identifying unidentified intruders in airspace according to the present disclosure;

[0012] Figure 2 This is a schematic diagram of the real-time image group sequence acquisition process;

[0013] Figure 3 This is a schematic diagram of the texture detail feature generation process;

[0014] Figure 4 This is a schematic diagram illustrating the relationship between the enhanced real-time image group sequence and the target image group sequence;

[0015] Figure 5 This is a schematic diagram of the anomaly detection process;

[0016] Figure 6 This is a structural schematic diagram of some embodiments of the unidentified intrusion identification device for airspace based on the present disclosure;

[0017] Figure 7 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of a method for identifying unidentified intruders in airspaces according to this disclosure. This method for identifying unidentified intruders in airspaces, applied to the identification of stray cats and dogs intruding, includes the following steps:

[0025] Step 101: Acquire real-time image group sequence.

[0026] In some embodiments, the entity executing the method for identifying unidentified intruders in the flight zone (e.g., a computing device) can acquire a sequence of real-time images using a camera assembly located within the flight zone.

[0027] The real-time image group consists of a first real-time image, a second real-time image, and a third real-time image from the same image acquisition location within the corresponding flight area. The first real-time image is an infrared image. The second real-time image is a color image. The third real-time image is a depth image. The camera assembly may include an infrared camera, a full-color camera, and a depth camera. The infrared camera, full-color camera, and depth camera within the camera assembly are clock-synchronized to ensure that the real-time image group, including the first, second, and third real-time images, is acquired simultaneously. Due to the large area of ​​the flight area, multiple camera assemblies can be set up at the boundary of the flight area to acquire the real-time image group. The first, second, and third real-time images in the real-time image group have the same image size.

[0028] In particular, the stray cats and dogs referred to in this disclosure are semi-wild populations formed by abandoned or lost cats and dogs, as well as wild populations that have regained their natural ecological niches through self-reproduction.

[0029] As an example, see Figure 2 The diagram illustrates the acquisition process of a real-time image sequence, where multiple camera components 203 are positioned at the boundary of the flight area 202. The camera components 203 face areas within the flight area 202, including the runway, taxiway, apron, takeoff and landing strip, and runway end safety zone. The camera components 203 acquire real-time image sequences in real time, resulting in a real-time image sequence 204, which is then transmitted to a computing device 201 via a local area network 205. Specifically, the camera components 203 and the computing device 201 can communicate and transmit data via wired or wireless connections.

[0030] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (Ultra Wide Band) connections, and other currently known or future wireless connection methods.

[0031] It should be noted that the aforementioned computing devices can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed on the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0032] Step 102: For each real-time image group in the real-time image group sequence, perform cross-image enhancement on the first real-time image, the second real-time image, and the third real-time image included in the real-time image group to obtain the enhanced real-time image group.

[0033] In some embodiments, for each real-time image group in the real-time image group sequence, the aforementioned execution entity may perform cross-image enhancement on the first real-time image, the second real-time image, and the third real-time image included in the real-time image group to obtain an enhanced real-time image group.

[0034] The enhanced real-time image group consists of an enhanced first real-time image, an enhanced second real-time image, and an enhanced third real-time image. The enhanced first real-time image is the first real-time image after image enhancement. The enhanced second real-time image is the second real-time image after image enhancement. The enhanced third real-time image is the third real-time image after image enhancement.

[0035] In practice, infrared images (the first real-time image) primarily reflect thermal radiation characteristics and are very effective at distinguishing different radiation sources, but they lack texture and geometric details. Color images (the second real-time image) primarily reflect texture details, but the quality of texture detail representation varies significantly due to factors such as lighting. Depth images (the third real-time image) primarily reflect geometric details, but lack texture details and representation of thermal radiation characteristics.

[0036] As an example, taking a set of real-time images as an example, firstly, wavelet transform can be used to transform and decompose the first, second, and third real-time images in the set, respectively, to obtain sub-bands of different frequencies for the first, second, and third real-time images. Then, the high-frequency sub-bands corresponding to the first and second real-time images are weighted and fused; the high-frequency sub-bands corresponding to the first and third real-time images are weighted and fused; and the low-frequency sub-bands corresponding to the first, second, and third real-time images are weighted and fused, to obtain updated sub-bands of different frequencies for the first, second, and third real-time images. Finally, by performing inverse wavelet transform on the updated sub-bands of different frequencies of the first real-time image, the second real-time image, and the third real-time image, the enhanced real-time image group corresponding to the real-time image group is obtained, which includes the enhanced first real-time image, the enhanced second real-time image, and the enhanced third real-time image. Since the low-frequency sub-bands mainly correspond to geometric representation, and the high-frequency sub-bands correspond to edge and texture representation, a sub-band weighted fusion method is used to assign the texture details corresponding to the color image to the infrared image and the depth image, and to assign the geometric details of the infrared image and the depth image to the color image, thereby achieving the purpose of cross-image enhancement.

[0037] In some optional implementations of some embodiments, the execution entity performs cross-image enhancement on each real-time image group in the real-time image group sequence, including the first real-time image, the second real-time image, and the third real-time image of the real-time image group, to obtain an enhanced real-time image group, including:

[0038] Step S1: Determine the real-time light intensity corresponding to the above real-time image group.

[0039] The aforementioned real-time light intensity characterizes the ambient light intensity of the aforementioned real-time image group when the images are acquired.

[0040] In practice, the camera assembly may also include a light sensor to determine the ambient light intensity of the real-time image group when the images are captured.

[0041] Step S2: In response to the real-time light intensity being less than a preset light intensity threshold, the following first processing step is performed:

[0042] Step S21: Determine the first geometric edge features, the first thermal radiation source edge features, and the edge correction coefficient based on the above real-time image set.

[0043] In practice, the aforementioned executing entity determines the first geometric edge features, the first thermal radiation source edge features, and the edge correction coefficients based on the aforementioned real-time image set, including:

[0044] Step S211: Extract geometric edges from the third real-time image included in the above real-time image group to obtain the first geometric edge features.

[0045] In practice, the aforementioned execution entity can use a pre-trained feature extractor to extract the geometric edges of the third real-time image included in the real-time image group, thereby obtaining the first geometric edge features. Among them, the feature extractor adopts the LV-UNet model. The reason for adopting the LV-UNet model is that since a large number of camera components are often deployed in the flight area to achieve image acquisition with as few blind spots as possible, a large number of real-time images will be generated. Although a complex feature extraction model can extract high-quality feature representations, it requires a large amount of computing resources. Compared with the task of identifying intrusive unidentified objects, the proportion of computing resources invested is relatively large. (2) The feature extractor with a symmetrical structure can ensure the scale consistency between input and output, which is convenient for subsequent feature processing between different features. In particular, the feature extractor is trained by transfer learning.

[0046] Step S212: Extract the edge of the thermal radiation source from the first real-time image included in the above real-time image group to obtain the edge features of the first thermal radiation source.

[0047] In practice, compared to depth images, infrared images exhibit more pronounced thermal radiation edge representation. Therefore, to improve data processing speed, an edge detection algorithm based on the Sober operator can be used to extract the thermal radiation source edges of the first real-time image, obtaining initial thermal radiation source edge features. Next, feature projection is performed on the initial thermal radiation source edge features, and feature shaping is achieved through multiple serially connected fully connected layers, ensuring that the output first thermal radiation source edge features and the first geometric edge features have the same feature scale.

[0048] Step S213: Determine the edge correction coefficient based on the first geometric edge features and the first thermal radiation edge features described above.

[0049] In practice, infrared targets in infrared images (first real-time images) exhibit thermal radiation differences from the background. Due to thermal expansion and contraction, the thermal radiation edges often shift, necessitating edge correction. Considering that the geometric edge features corresponding to the depth image (third real-time image) are unaffected by thermal radiation and possess stability, an edge correction coefficient is determined based on the first geometric edge features. Specifically, since the features exhibit scale invariance, and the thermal radiation edges uniformly contract or expand according to thermal radiation characteristics, the first geometric edge features and their corresponding similar edges can be calculated using feature similarity. Then, the edge correction coefficient is determined based on the positional relationship between the thermal radiation center, the similar edges included in the first thermal radiation edge, and the similar edges included in the first geometric edge features. Specifically, when the similar edges included in the first geometric edge features are located between the thermal radiation center and the similar edges included in the first thermal radiation edge, it indicates "thermal expansion" of the thermal radiation edge. Therefore, the edge distance between the similar edges included in the first geometric edge features and the similar edges included in the first thermal radiation edge can be determined and normalized to form the edge correction coefficient for contraction (edge ​​correction coefficient for contraction = normalized edge distance). When the similar edge included in the first thermal radiation edge is located between the thermal radiation center and the similar edge included in the first geometric edge feature, it indicates that there is "cold shrinkage" in the thermal radiation edge. Therefore, the edge distance between the similar edge included in the first geometric edge feature and the similar edge included in the first thermal radiation edge can be determined and normalized to an edge correction coefficient for amplification (edge ​​correction coefficient for amplification = normalized edge distance).

[0050] Step S22: Based on the edge correction coefficient and the edge features of the first thermal radiation source, perform thermal radiation correction on the first real-time image included in the real-time image group to obtain the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group.

[0051] In practice, the edge correction coefficient can be used as the shrinkage or magnification coefficient to shrink or magnify the edge features of the first thermal radiation source toward the corresponding thermal radiation center, and then reprojected onto the first real-time image to obtain the enhanced first real-time image.

[0052] Step S23: Extract the target edge from the second real-time image included in the above real-time image group to obtain the target edge features.

[0053] In practice, since color images (second real-time images) often use RGB (Red Green Blue) channels to express texture details, continuing to use feature extractors for edge extraction requires data processing in all three channels, resulting in a large amount of data processing. Therefore, for the target edge extraction of the second real-time images included in the real-time image group, the Sober operator-based edge detection method is still used, and feature projection and multiple serially connected fully connected layers are used for feature shaping to obtain the target edge features.

[0054] Step S24: Extract feature points from the above target edge features to obtain the first feature point set.

[0055] The first feature point includes a feature point descriptor. The feature point descriptor can be represented by a 128-dimensional feature vector.

[0056] In practice, since the target edge features are reprojected into the second real-time image and the edge representation is enhanced by edge extraction, this disclosure uses the FAST (Features from Accelerated Segment Test) algorithm to extract feature points from the above-mentioned target edge features to obtain the first feature point set.

[0057] Step S25: Extract thermal radiation edges from the enhanced first real-time image included in the enhanced real-time image group corresponding to the above real-time image group to obtain the second thermal radiation edge features.

[0058] In practice, the method of extracting the edge features of the first thermal radiation source in step S212 of step 102 can be used to extract the thermal radiation edge of the enhanced first real-time image included in the enhanced real-time image group corresponding to the above real-time image group, and obtain the second thermal radiation edge features, which will not be elaborated here.

[0059] Step S26: Extract feature points from the above-mentioned second thermal radiation edge features to obtain a second feature point set.

[0060] The second feature point includes a feature point descriptor. The feature point descriptor can be represented by a 128-dimensional feature vector.

[0061] In practice, since the second thermal radiation edge features and the target edge features use the same feature extraction method, in order to ensure that the extracted first feature points and second feature points use the same form of feature expression, this disclosure uses the FAST (Features from Accelerated Segment Test) algorithm to extract feature points from the above-mentioned second thermal radiation edge features to obtain the second feature point set.

[0062] Step S27: Perform alignment point matching based on the first feature point set and the second feature point set to obtain the first alignment point group set.

[0063] The first alignment point group includes a first feature point and a second feature point that have a matching relationship.

[0064] In practice, the first set of feature points and the second set of feature points can be aligned by calculating cosine similarity to obtain the first set of alignment points.

[0065] Step S28: Based on the first alignment point set and the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group, perform image edge enhancement on the second real-time image included in the real-time image group to obtain the enhanced second real-time image included in the enhanced real-time image group corresponding to the real-time image group.

[0066] In practice, the first and second feature points included in the first alignment point group can be used as alignment points. The thermal radiation edge included in the enhanced first real-time image can be projected onto the second real-time image included in the real-time image group to obtain the enhanced second real-time image included in the enhanced real-time image group corresponding to the aforementioned real-time image group. Specifically, since the second thermal radiation edge feature corresponding to the enhanced first real-time image has a specific feature position expression, and the first and second real-time images have the same image size, a direct projection method can be used to project the thermal radiation edge included in the enhanced first real-time image onto the second real-time image included in the real-time image group.

[0067] Step S29: Determine the third real-time image included in the above real-time image group as the enhanced third real-time image included in the enhanced real-time image group corresponding to the above real-time image group.

[0068] Step S3: In response to the above real-time light intensity being greater than or equal to a preset light intensity threshold, the following second processing step is executed:

[0069] Step S31: Extract texture details from the second real-time image included in the above real-time image group to obtain texture detail features.

[0070] In practice, a lightweight Feature Pyramid Network (FPN) with three convolutional layers can be used as the backbone to extract features from the second real-time image. The output of the FPN with its three convolutional layers is then continuously upsampled to reconstruct a feature map of the same size as the second real-time image, which serves as the texture detail feature. For details, see [link to documentation]. Figure 3The diagram illustrates the texture detail feature generation process. The FPN network, containing three convolutional modules, includes convolutional modules A1, A2, and A3. Assume the image size of the second real-time image is H×W×3. Then, the input to convolutional module A1 is the second real-time image, and the feature map size of its output feature map is H / 2×W / 2×3. The input to convolutional module A2 is the output of convolutional module A1, and the feature map size of its output feature map is H / 4×W / 4×3. The input to convolutional module A3 is the output of convolutional module A2, and the feature map size of its output feature map is H / 8×W / 8×3. The output of convolutional module A3 is upsampled through sampling layer U1, and the feature map size of the output feature map of sampling layer U1 is H / 4×W / 4×3. The feature map output by sampling layer U1 and the feature map output by convolutional module A2 are superimposed as the input to sampling layer U2, and the feature map size of the output feature map is H / 2×W / 2×3. The feature map output by sampling layer U2 and the feature map output by convolution module A1 are superimposed as the input to sampling layer U3. The feature map size of the output feature map (texture detail features) is (H×W×3). Among them, sampling layer U1, sampling layer U2 and sampling layer U3 are all upsampling layers.

[0071] Step S32: Extract the target edge from the second real-time image included in the above real-time image group to obtain the target edge features.

[0072] In practice, since color images (second real-time images) often use RGB (Red Green Blue) channels to express texture details, continuing to use feature extractors for edge extraction requires data processing in all three channels, resulting in a large amount of data processing. Therefore, for the target edge extraction of the second real-time images included in the real-time image group, the Sober operator-based edge detection method is still used, and feature projection and multiple serially connected fully connected layers are used for feature shaping to obtain the target edge features.

[0073] Step S33: Extract feature points from the above target edge features to obtain the first feature point set.

[0074] The first feature point includes a feature point descriptor. The feature point descriptor can be represented by a 128-dimensional feature vector.

[0075] In practice, since the target edge features are reprojected into the second real-time image and the edge representation is enhanced by edge extraction, this disclosure uses the FAST (Features from Accelerated Segment Test) algorithm to extract feature points from the above-mentioned target edge features to obtain the first feature point set.

[0076] Step S34: Extract geometric edges from the third real-time image included in the above real-time image group to obtain the first geometric edge features.

[0077] In practice, the aforementioned execution entity can use a pre-trained feature extractor to extract geometric edges from the third real-time image included in the real-time image group, thereby obtaining the first geometric edge features. The feature extractor employs the LV-UNet model.

[0078] Step S35: Extract feature points from the first geometric edge features to obtain a third set of feature points.

[0079] In practice, in order to ensure that the extracted third feature points and the first feature points use the same form of feature representation, this disclosure uses the FAST (Features from Accelerated Segment Test) algorithm to extract feature points from the above-mentioned first geometric edge features to obtain the third feature point set.

[0080] Step S36: Perform alignment point matching based on the first feature point set and the third feature point set to obtain the second alignment point set.

[0081] The second alignment point group includes the first feature point and the third feature point that have a matching relationship.

[0082] In practice, the first set of feature points and the third set of feature points can be aligned by calculating cosine similarity to obtain the second set of alignment points.

[0083] Step S37: Based on the above-mentioned second alignment point set and the above-mentioned texture detail features, perform texture detail enhancement on the third real-time image included in the above-mentioned real-time image set to obtain the enhanced third real-time image included in the enhanced real-time image set corresponding to the above-mentioned real-time image set.

[0084] In practice, the first and third feature points included in the second alignment point group can be used as alignment points to project the texture detail features onto the third real-time image included in the real-time image group, thereby obtaining the enhanced third real-time image included in the enhanced real-time image group corresponding to the aforementioned real-time image group. Specifically, since the second and third real-time images have the same image size, a direct projection method can be used to project the texture detail features corresponding to the first real-time image onto the second real-time image.

[0085] Step S38: Perform geometric edge features on the enhanced third real-time image included in the enhanced real-time image group corresponding to the above real-time image group to obtain the second geometric edge features.

[0086] In practice, the aforementioned execution entity can use a pre-trained feature extractor to perform geometric edge feature extraction on the enhanced third real-time image included in the enhanced real-time image group corresponding to the real-time image group, thereby obtaining the second geometric edge feature. The feature extractor employs the LV-UNet model.

[0087] Step S39: Based on the second geometric edge features mentioned above, perform edge projection enhancement on the first real-time image included in the real-time image group to obtain the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group.

[0088] In practice, since the feature size of the second geometric edge feature is the same as the image size of the first real-time image, this disclosure uses direct projection to project the second geometric edge feature onto the first real-time image included in the aforementioned real-time image group, thereby obtaining the enhanced first real-time image included in the enhanced real-time image group corresponding to the aforementioned real-time image group. Since the enhanced third real-time image, in addition to its own geometric details, also has enhanced texture details through the second real-time image, edge projection enhancement of the first real-time image is performed based on the enhanced third real-time image using the extracted second geometric edge features. This is equivalent to combining the second and third real-time images to jointly perform edge projection enhancement on the first real-time image.

[0089] Step S310: Determine the third real-time image included in the above real-time image group as the enhanced third real-time image included in the enhanced real-time image group corresponding to the above real-time image group.

[0090] In practice, the content of "in some optional implementations of some embodiments" in step 103 above is a core inventive point of this disclosure. Specifically, regarding steps S21 to S29 in step 103, since the texture detail expression quality of the color image (second real-time image) varies greatly due to factors such as illumination, the texture detail expression quality of the color image decreases when the ambient light intensity is weak. Meanwhile, considering that infrared images mainly reflect thermal radiation characteristics, and the stable expression characteristics of geometric details corresponding to depth images (third real-time images) under different ambient light intensities, the thermal radiation edges present in the first real-time image are first corrected using the third real-time image. Then, combined with the enhanced first real-time image, edge detail supplementation is performed on the second real-time image, which has poor texture detail expression quality in low-light environments, thereby achieving the purpose of image cross-enhancement. Regarding steps S31 to S310 in step 103, when the ambient light intensity is strong (i.e., the lighting is excellent), the texture detail expression of the color image is excellent. Therefore, based on the color image (second real-time image), the texture detail of the depth image (third real-time image) is first enhanced. Meanwhile, considering that when the ambient light intensity is strong, the surface radiation may cause the difference in thermal radiation between the infrared source and the background in the infrared image (first real-time image) to be insignificant, the edge features of the infrared image are enhanced by combining the enhanced third real-time image.

[0091] Step 103: Perform anomaly detection based on the obtained enhanced real-time image group sequence to determine the target image group sequence.

[0092] In some embodiments, the aforementioned execution entity can perform anomaly detection based on the obtained enhanced real-time image group sequence to determine the target image group sequence.

[0093] The target image group is the enhanced real-time image group in the above-mentioned enhanced real-time image group sequence that has positional changes within the corresponding image acquisition location.

[0094] In practice, a combination of object detection (e.g., SSD (Single Shot MultiBox Detector) model) and optical flow (e.g., Lucas Kanade algorithm) can be used to detect anomalies in the enhanced real-time image group sequence and determine the target image group sequence.

[0095] As an example, see Figure 4The diagram illustrates the relationship between the enhanced real-time image group sequence and the target image group sequence. Anomaly detection is performed on the enhanced real-time image group sequence to extract the moving portions, which are then used as the target image group sequence. Specifically, the reason for not directly identifying unidentified intruders based on the enhanced real-time image group sequence is that the intrusion of wild cats and dogs into flight zones is uncertain, and camera components often require long periods of image acquisition. This results in uncertain positions and relatively small proportions of images containing wild animals within the real-time image group sequence. Directly identifying unidentified intruders requires extracting information such as the type of intruding animal, its movement tendency, animal identification characteristics, and intrusion area markers, often necessitating deep feature extraction and processing of the images themselves. This leads to a massive amount of data processing and a high proportion of invalid processing. Therefore, this disclosure first performs binary classification for anomaly detection to filter out enhanced real-time image groups containing anomalies before proceeding with subsequent unidentified intruder identification, which can reduce the amount of data processing to some extent.

[0096] In some optional implementations of some embodiments, the execution entity performs anomaly detection based on the obtained enhanced real-time image group sequence to determine the target image group sequence, including:

[0097] Step S1: Divide the above-mentioned enhanced real-time image group sequence into image groups to obtain the enhanced first real-time image sequence, the enhanced second real-time image sequence, and the enhanced third real-time image sequence.

[0098] In practice, because infrared images, color images, and depth images have different details, it is often necessary to fuse images of the three different modalities in order to perform effective anomaly detection. However, as mentioned above, this leads to the need to fuse a large number of invalid images, resulting in a surge in data processing volume. Therefore, this disclosure bypasses the image fusion step in the anomaly detection stage and instead performs image grouping separately.

[0099] Step S2: Based on the enhanced first real-time image sequence, the enhanced second real-time image sequence, and the enhanced third real-time image sequence, perform anomaly detection in parallel to obtain the first detection information sequence, the second detection information sequence, and the third detection information sequence.

[0100] The first, second, and third detection information each include: image frame number, anomaly location, and anomaly confidence level. The anomaly location can be represented by the corner coordinates of the anomaly point (e.g., the coordinates of two diagonally opposite corners). Specifically, the image frame number included in the first detection information represents the image frame index of the enhanced first real-time image. The image frame number included in the second detection information represents the image frame index of the enhanced second real-time image. The image frame number included in the third detection information represents the image frame index of the enhanced third real-time image. The anomaly location included in the first detection information represents the anomaly point present in the enhanced first real-time image. The anomaly location included in the second detection information represents the anomaly point present in the enhanced second real-time image. The anomaly location included in the third detection information represents the anomaly point present in the enhanced third real-time image. The anomaly confidence level included in the first detection information represents the confidence level of the existence of anomaly in the enhanced first real-time image. The anomaly confidence level included in the second detection information represents the confidence level of the existence of anomaly in the enhanced second real-time image. The anomaly confidence level included in the third detection information represents the confidence level of the existence of anomaly in the enhanced third real-time image.

[0101] In practice, this disclosure employs a first anomaly detection model, a second anomaly detection model, and a third anomaly detection model for anomaly detection. These three models are configured in parallel. To reduce model training costs, all three models utilize the Tiny-YOLO model. The first anomaly detection model is used to detect anomalies in the enhanced first real-time image sequence. The second anomaly detection model is used to detect anomalies in the enhanced second real-time image sequence. The third anomaly detection model is used to detect anomalies in the enhanced third real-time image sequence. However, if the first, second, and third anomaly detection models perform anomaly detection independently, the correlation between the infrared image, color image, and depth image is ignored. Therefore, the core issue is how to simultaneously consider the correlation between infrared, color, and depth images for anomaly detection while controlling or even reducing the amount of data processing as much as possible.

[0102] As an example, see Figure 5The diagram illustrating the anomaly detection process shows that this disclosure optimizes the algorithm as follows: For enhanced first real-time images, enhanced second real-time images, and enhanced third real-time images with the same image frame number, the enhanced first real-time image is first input into the corresponding first anomaly detection model for anomaly detection to obtain the corresponding first detection information. When the anomaly confidence level included in the corresponding first detection information is greater than a preset confidence level, the corresponding enhanced second and third real-time images are no longer subjected to anomaly detection through the second and third anomaly detection models. Simultaneously, the corresponding first detection information is directly used as the second and third detection information for the corresponding enhanced second and third real-time images. When the anomaly confidence level included in the corresponding first detection information is less than or equal to the preset confidence level, the corresponding enhanced second and third real-time images are then input into the corresponding second and third anomaly detection models for anomaly detection to obtain the second and third detection information, respectively. The reason for this design is that, in the previous image cross-enhancement stage, enhancement of texture and geometric details is performed on the basis of the infrared image, thereby enhancing the robustness of the infrared image. Meanwhile, wild cats and dogs exhibit significant displacement changes during movement, which is clearly visible in infrared images. Therefore, we first combine infrared images for movement detection. When the confidence level of the corresponding movement is greater than the preset confidence level, we skip the movement detection of the corresponding color and depth images, thereby further reducing the amount of data processing.

[0103] Step S3: Perform weighted adjustments on the first detection information sequence, the second detection information sequence, and the third detection information sequence to obtain the fourth detection information sequence.

[0104] In practice, firstly, the anomaly confidence scores of the first, second, and third detection information corresponding to the same image frame number can be weighted and summed to obtain the anomaly confidence score of the corresponding fourth detection information. Then, the intersection of the anomaly locations included in the first, second, and third detection information corresponding to the same image frame number is taken as the anomaly location included in the corresponding fourth detection information. Finally, the image frame numbers included in the first, second, and third detection information corresponding to the same image frame number are taken as the image frame numbers included in the corresponding fourth detection information.

[0105] Step S4: Based on the fourth detection information sequence mentioned above, perform image group filtering on the enhanced real-time image group sequence to obtain the target image group sequence mentioned above.

[0106] In practice, the enhanced real-time image group whose anomaly confidence level in the corresponding fourth detection information is greater than the screening threshold can be used as the target image group to obtain the target image group sequence.

[0107] Step 104: Perform image fusion on each target image group in the target image group sequence to generate a fused image, thus obtaining a fused image sequence.

[0108] In some embodiments, the execution entity may perform image fusion on each target image group in the target image group sequence to generate a fused image, thereby obtaining a fused image sequence.

[0109] The image size of the fused image is consistent with the image sizes of the enhanced first real-time image, the enhanced second real-time image, and the enhanced third real-time image.

[0110] In practice, firstly, a three-branch CNN (Convolutional Neural Network) model can be used to extract image features in parallel from the enhanced first real-time image, the enhanced second real-time image, and the enhanced third real-time image of the target image group. Then, a stacking layer is used to stack the three feature maps output by the three-branch CNN model. Finally, the output of the stacking layer is input into multiple serially connected fully connected layers for feature shaping to obtain the fused image.

[0111] Step 105: Based on the above fused image sequence, identify the intrusion object and generate intrusion object identification information.

[0112] In some embodiments, the aforementioned execution entity can perform intrusion identification based on the aforementioned fused image sequence and generate intrusion identification information.

[0113] The identification information for intruding unidentified objects includes: the type of intruding animal, its movement tendency, its identification characteristics, the intrusion area marker, and the intrusion time. Specifically, the animal movement tendency indicates whether the stray cats or dogs are moving towards the flight zone. The intrusion area marker indicates the area intruded by the stray cats or dogs. The intrusion time indicates the time of the intrusion.

[0114] In practice, since the real-time image sequence is acquired by multiple camera components positioned in the flight area, the intrusion area can be identified based on the location of the camera components corresponding to the fused image. Similarly, the image acquisition time of the real-time image sequence corresponding to the fused image can be used as the intrusion time.

[0115] In practice, models such as Tiny-YOLO can be used to identify intruding objects based on the aforementioned fused image sequence, generating identification information. Specifically, to generate this information, the Tiny-YOLO model's tail needs modification. Specifically, a fully connected layer needs to be added to the tail of the Tiny-YOLO model to shape the image region corresponding to the located intruding animal into a one-dimensional feature vector, serving as the animal's identity feature. A multi-classifier is also added to output the intruding animal type. Furthermore, a parallel binary classifier is needed to output the animal's movement tendency.

[0116] In some optional implementations of certain embodiments, the execution entity performs intrusion object identification based on the fused image sequence to generate intrusion object identification information, including:

[0117] Step S1: Perform coarse image feature extraction on each fused image in the above fused image sequence to obtain a coarse fused image feature sequence.

[0118] In practice, the aforementioned execution entity can use an FPN network (e.g., including a 5-layer convolutional layer structure) to perform coarse image feature extraction on each fused image in the aforementioned fused image sequence to obtain a coarse fused image feature sequence.

[0119] Step S2: Based on the above coarse fusion image feature sequence, target localization is performed to obtain the local image feature sequence and the intrusion area identifier included in the above intrusion unidentified object identification information.

[0120] Among them, local image features correspond to the image features of local regions containing stray cats and dogs.

[0121] In practice, a positioning head based on an anchor frame can be used to locate the target based on the coarsely fused image feature sequence mentioned above, thereby obtaining the local image feature sequence and the intrusion area identifier included in the aforementioned intrusion unidentified object identification information.

[0122] Step S3: Generate body shape features based on the above local image feature sequence and body shape feature extraction module.

[0123] The body shape feature extraction module consists of a lightweight, five-layer serially connected convolutional layer and feature stacking layer.

[0124] In practice, local image features in the local image feature sequence are extracted sequentially through 5 convolutional layers connected in series, and the extracted feature maps are superimposed through a feature overlay layer to serve as body shape features.

[0125] Step S4: Generate pattern features based on the above local image feature sequence and pattern feature extraction module.

[0126] Step S5: Generate fur color features based on the above local image feature sequence and fur color feature extraction module.

[0127] Step S6: Generate facial features based on the above local image feature sequence and facial feature extraction module.

[0128] Among them, the pattern feature extraction module, the fur color feature extraction module, and the facial feature extraction module all adopt the same model structure, that is, they are all composed of 7 serially connected convolutional layers.

[0129] Step S7: Based on the above-mentioned body shape characteristics, pattern characteristics, fur color characteristics and facial characteristics, generate the animal identity characteristics included in the above-mentioned intrusion unidentified object identification information.

[0130] In practice, the body shape features, the aforementioned pattern features, the aforementioned coat color features, and the aforementioned facial features are first concatenated. Then, the concatenated features are input into multiple serially connected fully connected layers to generate a one-dimensional animal identity feature.

[0131] Step S8: Based on the coarse fusion image feature sequence and optical flow tracing module, generate the animal movement tendency included in the above-mentioned intrusion unidentified object identification information.

[0132] In practice, the optical flow tracing module can use the Lucas Kanade algorithm for optical flow tracing and output the animal's movement tendency through a binary classifier.

[0133] Step S9: Determine the image acquisition time corresponding to the above target image group sequence as the intrusion time included in the above intrusion unidentified object identification information.

[0134] In practice, the FPN network, anchor-frame-based positioning head, body shape feature extraction module, pattern feature extraction module, fur color feature extraction module, facial feature extraction module, multiple serially connected fully connected layers, and optical flow tracing module included in steps S1 to S9 are trained as a whole in supervised model training. The reason for this design is that by decoupling and independently defining each module, the model structure can be flexibly adjusted according to computing resources, thus facilitating model deployment in scenarios with different hardware resources.

[0135] In some optional implementations of some embodiments, the above method further includes:

[0136] Step S1: Perform identity matching based on the animal identification characteristics and identity information database included in the above-mentioned intrusion unidentified object identification information.

[0137] The aforementioned identity information database stores a set of animal identity information, which represents wild animals that have entered the flight zone. Animal identity information may include: animal type, animal characteristics, and an intrusion record chain. The intrusion record chain stores intrusion records in a chained manner. Intrusion records may include: intrusion time and intrusion area identifier.

[0138] In practice, to improve retrieval speed, the identity information database pre-builds query indexes based on different animal types. Specifically, it performs identity matching by calculating the similarity between animal identity features included in the intrusion unidentified object identification information and animal identity features included in the animal identity information.

[0139] Step S2: In response to successful identity matching, the identity information database is updated based on the aforementioned intrusion identification information to obtain the updated identity information database.

[0140] In practice, when the identity is successfully matched, the intrusion record can be updated based on the intrusion identification information of the unidentified intruder, including the intrusion record chain.

[0141] Step S4: In response to the target trigger, select the target area.

[0142] The target trigger is preset with a sampling period for triggering, and the target area is the sampling area within the flight zone.

[0143] In practice, stray cats and dogs may experience changes in their activity range due to the influence of their living and foraging environments. At the same time, stray cats and dogs have a very high reproductive capacity, and the population size of stray cats and dogs within the same activity range may also change. Therefore, this disclosure uses a target trigger to sample in a fixed periodic manner.

[0144] Step S5: Extract animal identity information that meets the filtering criteria from the updated identity information database and use it as the target animal identity information to obtain the target animal identity information set.

[0145] The above-mentioned screening criteria are: the invasion area corresponding to the invasion record included in the animal identity information matches the target area, and the invasion time corresponding to the invasion record included in the animal identity information is within the sampling period.

[0146] Step S6: Determine the animal size information based on the above set of target animal identity information and target area.

[0147] In practice, animal size information can be determined by combining the sampled population size estimation method with the aforementioned target animal identity information set and target area.

[0148] As an example, the estimated population size = (total number of times the animal of this type appeared during the sampling period × area of ​​the flight zone) / (average activity frequency of the animal of this type × sampling period × ratio of the area of ​​the target area to the area of ​​the flight zone).

[0149] Step S7: In response to the above animal size information indicating abnormal animal size, initiate an animal size abnormality warning.

[0150] In practice, an animal size anomaly warning is issued when the estimated population size included in the animal size information exceeds the warning threshold or when the estimated population size shows a significant upward trend. This enables periodic sampling and automated warning, thereby allowing for reasonable and effective control of the wild cat and dog population within the flight zone.

[0151] In some optional implementations of some embodiments, the above method further includes:

[0152] Step S1: Determine the intrusion risk level based on the above-mentioned intrusion unidentified object identification information.

[0153] In practice, multiple triggering rules can be set for different invasion risk levels based on the type of invading animal and its movement tendency. The corresponding invasion risk level can be determined based on the triggering rules triggered by the identification information of the invading unidentified object.

[0154] Step S2: In response to the above-mentioned intrusion risk level being greater than the first preset risk level, create an intrusive wildlife removal task and update the task status corresponding to the above-mentioned intrusive wildlife removal task to task pending execution.

[0155] Step S3: In response to the above-mentioned intrusion risk level being greater than the first preset risk level and less than the second preset risk level, broadcast the above-mentioned task of driving away the intruding wild animals.

[0156] In practice, tasks to drive away invasive wildlife can be issued to ground crew working in the flight area.

[0157] Step S4: In response to the above-mentioned intrusion risk level being greater than the second preset risk level, the above-mentioned task of driving away the intruding wild animals is assigned in a targeted manner.

[0158] In practice, tasks to remove invasive wildlife can be sent to dedicated wildlife removal personnel at airports.

[0159] Step S5: In response to the above-mentioned task of removing invasive wild animals being received, update the task status corresponding to the above-mentioned task of removing invasive wild animals to task in progress.

[0160] In practice, when staff members accept tasks to remove invasive wild animals through their mobile terminals, the task status is updated to "task in progress," and the task is linked to the staff member who accepted the task.

[0161] Step S6: In response to the completion of the above-mentioned task of removing invasive wild animals, cancel the above-mentioned task of removing invasive wild animals, generate a task record, and store the above-mentioned task record in the private blockchain.

[0162] In practice, in response to the task-bound staff, the task completion feedback is initiated through the corresponding mobile terminal, the aforementioned task of driving away invasive wild animals is cancelled, a task record is generated, and the task record is stored in the private blockchain.

[0163] The above embodiments of this disclosure have the following beneficial effects: The method for identifying unidentified intruders in flight zones, based on some embodiments of this disclosure, achieves accurate identification of unidentified intruders. Specifically, firstly, a sequence of real-time image groups is acquired. Each real-time image group consists of a first real-time image, a second real-time image, and a third real-time image from the same image acquisition location within the corresponding flight zone. The first real-time image is an infrared image, the second real-time image is a color image, and the third real-time image is a depth image. In practice, stray cats and dogs are nocturnal, and flight zones have strict light control requirements. Therefore, to ensure subsequent identification accuracy, this disclosure uses three different image sources as the basis for identification. Secondly, for each real-time image group in the above real-time image group sequence, cross-image enhancement is performed on the first, second, and third real-time images included in the real-time image group to obtain an enhanced real-time image group. In practice, different image types have different advantages and disadvantages. For example, infrared images mainly reflect thermal radiation characteristics but lack texture and geometric details; color images show differences in texture detail accuracy under different light intensities; and depth images mainly reflect geometric details but lack texture details. Therefore, this disclosure improves image quality through cross-image enhancement. Next, anomaly detection is performed on the obtained enhanced real-time image group sequence to determine the target image group sequence. The target image group is the enhanced real-time image group within the aforementioned enhanced real-time image group sequence that exhibits positional anomalies at the corresponding image acquisition location. In practice, stray cats and dogs exhibit significant displacement changes when entering the flight zone, and the frequency and probability of stray cats and dogs entering the flight zone are uncertain. This results in a large number of invalid images in the continuously acquired real-time images. Therefore, this disclosure uses anomaly detection to quickly extract useful images. Furthermore, image fusion is performed on each target image group in the aforementioned target image group sequence to generate a fused image, resulting in a fused image sequence. Image fusion transforms images of different graphic types into the same feature representation space. Finally, intrusion identification is performed based on the aforementioned fused image sequence to generate intrusion identification information, which includes: intruding animal type, animal movement tendency, animal identification characteristics, intrusion area identifier, and intrusion time. In summary, this disclosure achieves accurate intrusion identification.

[0164] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an intrusion detection device for airspace, which are similar to... Figure 1 Corresponding to the method embodiments shown, this intrusion detection device for airspace can be specifically applied to various electronic devices.

[0165] like Figure 6As shown, some embodiments of the unidentified intrusion identification device 600 for flight zones include: an acquisition unit 601, a cross-image enhancement unit 602, a determination unit 603, an image fusion unit 604, and a generation unit 605. The acquisition unit 601 is configured to acquire a sequence of real-time image groups. Each real-time image group consists of a first real-time image, a second real-time image, and a third real-time image from the same image acquisition location within the flight zone. The first real-time image is an infrared image, the second real-time image is a color image, and the third real-time image is a depth image. The cross-image enhancement unit 602 is configured to perform cross-image enhancement on each real-time image group in the above-mentioned real-time image group sequence, considering the first real-time image, the second real-time image, and the third real-time image included in the real-time image group. The enhancement process yields an enhanced real-time image set. A determination unit 603 is configured to perform anomaly detection based on the obtained enhanced real-time image set sequence to determine a target image set sequence, wherein the target image set is the enhanced real-time image set in the sequence that exhibits positional anomalies at the corresponding image acquisition location. An image fusion unit 604 is configured to perform image fusion on each target image set in the target image set sequence to generate a fused image, resulting in a fused image sequence. A generation unit 605 is configured to perform intrusion identification based on the fused image sequence, generating intrusion identification information, wherein the intrusion identification information includes: intruding animal type, animal movement tendency, animal identification characteristics, intrusion area identifier, and intrusion time.

[0166] It is understandable that the units described in the unidentified intrusion identification device 600 targeting the airspace are similar to those in the reference device. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the unidentified intrusion detection device 600 for airspace and the units contained therein, and will not be repeated here.

[0167] The following is for reference. Figure 7 It shows a schematic diagram of the structure of an electronic device (e.g., a computing device) 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0168] like Figure 7As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage device 708 into a random access memory 703. The random access memory 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, the read-only memory 702, and the random access memory 703 are interconnected via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0169] Typically, the following devices can be connected to the input / output interface 705: input devices 706 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 708 including, for example, magnetic tape, hard disk, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.

[0170] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a read-only memory 702. When the computer program is executed by the processing device 701, it performs the functions defined in the methods of some embodiments of this disclosure.

[0171] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0172] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0173] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs. When the aforementioned one or more programs are executed by the electronic device, the electronic device causes the following actions: It acquires a sequence of real-time image groups, wherein each real-time image group consists of a first real-time image, a second real-time image, and a third real-time image from the same image acquisition location within a corresponding flight area. The first real-time image is an infrared image, the second real-time image is a color image, and the third real-time image is a depth image. For each real-time image group in the aforementioned real-time image group sequence, it performs cross-image enhancement on the first real-time image, the second real-time image, and the third real-time image included in the aforementioned real-time image group sequence to obtain an enhanced real-time image group. It performs anomaly detection based on the obtained enhanced real-time image group sequence to determine a target image group sequence, wherein the target image group is the enhanced real-time image group in the aforementioned enhanced real-time image group sequence that exhibits positional anomalies within its corresponding image acquisition location. It performs image fusion on each target image group in the aforementioned target image group sequence to generate a fused image, thus obtaining a fused image sequence. It performs intrusion identification based on the aforementioned fused image sequence to generate intrusion identification information, wherein the intrusion identification information includes: intruding animal type, animal movement tendency, animal identification characteristics, intrusion area identifier, and intrusion time.

[0174] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0176] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0177] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for identifying unidentified intrusive objects in airspace, characterized in that, include: Acquire a sequence of real-time images, wherein the real-time image group consists of a first real-time image, a second real-time image, and a third real-time image from the same image acquisition location within the corresponding flight area. The first real-time image is an infrared image, the second real-time image is a color image, and the third real-time image is a depth image. For each real-time image group in the real-time image group sequence, cross-image enhancement is performed on the first real-time image, the second real-time image, and the third real-time image included in the real-time image group to obtain an enhanced real-time image group, wherein the enhanced real-time image group consists of an enhanced first real-time image, an enhanced second real-time image, and an enhanced third real-time image. Anomaly detection is performed based on the obtained enhanced real-time image group sequence to determine the target image group sequence, wherein the target image group is the enhanced real-time image group in the enhanced real-time image group sequence that has positional anomalies within the corresponding image acquisition position; Image fusion is performed on each target image group in the target image group sequence to generate a fused image, resulting in a fused image sequence; Based on the fused image sequence, intrusion object identification is performed to generate intrusion object identification information, which includes: intruding animal type, animal movement tendency, animal identification characteristics, intrusion area identifier, and intrusion time. For each real-time image group in the real-time image group sequence, cross-image enhancement is performed on the first real-time image, the second real-time image, and the third real-time image included in the real-time image group to obtain an enhanced real-time image group, including: Determine the real-time light intensity corresponding to the real-time image group, wherein the real-time light intensity characterizes the ambient light intensity of the real-time image group when the images are acquired; In response to the real-time light intensity being less than a preset light intensity threshold, the following first processing step is performed: The first geometric edge feature, the first thermal radiation source edge feature, and the edge correction coefficient are determined based on the real-time image set. Based on the edge correction coefficient and the edge features of the first thermal radiation source, thermal radiation correction is performed on the first real-time image included in the real-time image group to obtain the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group. Target edge extraction is performed on the second real-time image included in the real-time image group to obtain target edge features; Feature points are extracted from the target edge features to obtain a first set of feature points; Thermal radiation edge extraction is performed on the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group to obtain the second thermal radiation edge feature; Feature points are extracted from the second thermal radiation edge features to obtain a second feature point set; Alignment point matching is performed based on the first feature point set and the second feature point set to obtain the first alignment point group set; Based on the first alignment point set and the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group, image edge enhancement is performed on the second real-time image included in the real-time image group to obtain the enhanced second real-time image included in the enhanced real-time image group corresponding to the real-time image group. The third real-time image included in the real-time image group is determined as the enhanced third real-time image included in the enhanced real-time image group corresponding to the real-time image group.

2. The method for identifying unidentified intruders in flight zones according to claim 1, characterized in that, The method further includes: The identification of the intruding unidentified object is matched with the animal identification features and the identification information database, wherein the identification information database is used to store a set of animal identification information, and the animal identification information represents wild animals that have entered the flight area. In response to successful identity matching, the identity information database is updated based on the identification information of the intruding unidentified object to obtain the updated identity information database. In response to a target trigger, a target region is selected, wherein the target trigger has a preset sampling period for triggering, and the target region is a sampling region within the flight area; Animal identity information that meets the filtering criteria is extracted from the updated identity information database and used as target animal identity information to obtain a set of target animal identity information. The filtering criteria are: the intrusion area corresponding to the intrusion record included in the animal identity information matches the target area, and the intrusion time corresponding to the intrusion record included in the animal identity information is within the sampling period. Based on the target animal identity information set and the target area, determine the animal size information; In response to the animal size information indicating an abnormal animal size, an animal size anomaly warning is initiated.

3. The method for identifying unidentified intruders in flight zones according to claim 1, characterized in that, The step of detecting anomalies based on the obtained enhanced real-time image group sequence and determining the target image group sequence includes: The enhanced real-time image group sequence is divided into image groups to obtain the enhanced first real-time image sequence, the enhanced second real-time image sequence, and the enhanced third real-time image sequence. Based on the enhanced first real-time image sequence, the enhanced second real-time image sequence, and the enhanced third real-time image sequence, anomaly detection is performed in parallel to obtain a first detection information sequence, a second detection information sequence, and a third detection information sequence. The first detection information, the second detection information, and the third detection information all include: image frame number, anomaly location, and anomaly confidence level. The first detection information sequence, the second detection information sequence, and the third detection information sequence are weighted differently to obtain the fourth detection information sequence. Based on the fourth detection information sequence, the enhanced real-time image group sequence is filtered to obtain the target image group sequence.

4. The method for identifying unidentified intruders in flight zones according to claim 1, characterized in that, The step of identifying unidentified intruders based on the fused image sequence and generating unidentified intruder identification information includes: Coarse image feature extraction is performed on each fused image in the fused image sequence to obtain a coarse fused image feature sequence; Target localization is performed based on the coarsely fused image feature sequence to obtain a local image feature sequence and the intrusion area identifier included in the intrusion unidentified object identification information; Based on the local image feature sequence and the body shape feature extraction module, body shape features are generated; Based on the local image feature sequence and the pattern feature extraction module, pattern features are generated; Based on the local image feature sequence and the fur color feature extraction module, fur color features are generated; Based on the local image feature sequence and the facial feature extraction module, facial features are generated; Based on the body shape characteristics, pattern characteristics, fur color characteristics, and facial characteristics, the animal identity characteristics included in the identification information of the intruding unidentified object are generated; Based on the coarse fusion image feature sequence and the optical flow tracing module, the animal movement tendency included in the identification information of the intruding unidentified object is generated; The image acquisition time corresponding to the target image group sequence is determined as the intrusion time included in the intrusion identification information of the unidentified intruder.

5. The method for identifying unidentified intruders in flight zones according to claim 2, characterized in that, The method further includes: Based on the identification information of the intruding unidentified object, the intrusion risk level is determined; In response to the intrusion risk level being greater than a first preset risk level, an intrusive wildlife removal task is created, and the task status corresponding to the intrusive wildlife removal task is updated to task pending execution. In response to the fact that the intrusion risk level is greater than a first preset risk level and less than a second preset risk level, the task of driving away the intruding wild animals is broadcast. In response to the intrusion risk level being greater than the second preset risk level, the task of repelling the intruding wild animals is assigned in a targeted manner; In response to the receipt of the task to remove the invasive wildlife, the task status corresponding to the task to remove the invasive wildlife is updated to "task in progress"; In response to the completion of the invasive wildlife removal task, the invasive wildlife removal task is cancelled, a removal task record is generated, and the removal task record is stored in a private blockchain.

6. A device for identifying unidentified intruders in airspace, characterized in that, include: The acquisition unit is configured to acquire a sequence of real-time images. The real-time image group consists of a first real-time image, a second real-time image, and a third real-time image at the same image acquisition location within the corresponding flight area. The first real-time image is an infrared image, the second real-time image is a color image, and the third real-time image is a depth image. The cross-image enhancement unit is configured to perform cross-image enhancement on the first real-time image, the second real-time image, and the third real-time image included in each real-time image group sequence to obtain an enhanced real-time image group, wherein the enhanced real-time image group consists of the enhanced first real-time image, the enhanced second real-time image, and the enhanced third real-time image. The determining unit is configured to perform anomaly detection based on the obtained enhanced real-time image group sequence and determine the target image group sequence, wherein the target image group is the enhanced real-time image group in the enhanced real-time image group sequence that has positional anomalies within the corresponding image acquisition position; An image fusion unit is configured to perform image fusion on each target image group in the target image group sequence to generate a fused image, thereby obtaining a fused image sequence; The generation unit is configured to identify intruding unidentified objects based on the fused image sequence and generate intruding unidentified object identification information, wherein the intruding unidentified object identification information includes: intruding animal type, animal movement tendency, animal identification characteristics, intrusion area identifier, and intrusion time. For each real-time image group in the real-time image group sequence, cross-image enhancement is performed on the first real-time image, the second real-time image, and the third real-time image included in the real-time image group to obtain an enhanced real-time image group, including: Determine the real-time light intensity corresponding to the real-time image group, wherein the real-time light intensity characterizes the ambient light intensity of the real-time image group when the images are acquired; In response to the real-time light intensity being less than a preset light intensity threshold, the following first processing step is performed: The first geometric edge feature, the first thermal radiation source edge feature, and the edge correction coefficient are determined based on the real-time image set. Based on the edge correction coefficient and the edge features of the first thermal radiation source, thermal radiation correction is performed on the first real-time image included in the real-time image group to obtain the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group. Target edge extraction is performed on the second real-time image included in the real-time image group to obtain target edge features; Feature points are extracted from the target edge features to obtain a first set of feature points; Thermal radiation edge extraction is performed on the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group to obtain the second thermal radiation edge feature; Feature points are extracted from the second thermal radiation edge features to obtain a second feature point set; Alignment point matching is performed based on the first feature point set and the second feature point set to obtain the first alignment point group set; Based on the first alignment point set and the enhanced first real-time image included in the enhanced real-time image group corresponding to the real-time image group, image edge enhancement is performed on the second real-time image included in the real-time image group to obtain the enhanced second real-time image included in the enhanced real-time image group corresponding to the real-time image group. The third real-time image included in the real-time image group is determined as the enhanced third real-time image included in the enhanced real-time image group corresponding to the real-time image group.

7. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images