A UAV public safety data management platform

Abstract

The present application relates to a UAV public safety data management platform. It includes: extracting foreground high-frequency components of regional monitoring images collected by UAVs based on deep learning image processing technology, and performing multi-level feature extraction and feature gradient mask enhancement processing on the extracted regional foreground high-frequency object images to obtain shallow and deep significant feature representations of regional foreground high-frequency objects, and then intelligently identifying whether the high-frequency object is a human object based on the main component matching fusion features of the two, and generating corresponding personnel activity prompt signals. In this way, it is possible to achieve rapid and accurate recognition of human targets in UAV monitoring images, thereby improving the efficiency of public safety management and the timeliness of emergency response.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and in particular, to a drone public safety data management platform. Background Art

[0002] With the development of science and technology and the growth of social needs, drones are being used more and more widely in various fields, especially in public safety management. Drones have become one of the important tools due to their flexibility and efficiency. Traditional public safety monitoring systems mainly rely on fixed cameras and manual monitoring. Although this method can provide a certain degree of security, it has limitations in dealing with emergencies or when large-scale monitoring is required. For example, in scenes such as natural disaster relief and large-scale event security, fixed cameras have limited viewing angles and it is difficult to cover all key areas, while manual monitoring may reduce efficiency due to factors such as visual fatigue.

[0003] As an emerging monitoring method, drones can overcome some of the shortcomings of traditional monitoring systems and have become an indispensable tool for urban monitoring, disaster relief, traffic management, environmental monitoring and other aspects. Drones can be quickly deployed to any location that needs to be monitored and can move freely in the air to achieve all-round monitoring without blind spots. However, due to the huge amount of image data collected by drones, relying solely on human analysis is not only time-consuming and laborious, but also unable to respond in real time.

[0004] Therefore, we look forward to an intelligent drone public safety data management platform that can automatically process and analyze large amounts of image data collected by drones to achieve efficient and real-time monitoring and management. Summary of the invention

[0005] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] In a first aspect, the present application provides a drone public safety data management platform, the platform comprising:

[0007] A face recognition module, used to identify and verify the identity of the drone operator;

[0008] Fireworks recognition module, used to detect and identify fireworks events in drone footage;

[0009] License plate recognition module, used to identify vehicle license plate information in the drone footage;

[0010] The vehicle identification module is used to identify vehicles in the drone footage and count the number of vehicles;

[0011] The bare land detection module is used to detect the bare land area in the drone footage;

[0012] The personnel identification module is used to detect whether there are personnel activities in the drone footage and generate corresponding prompt signals;

[0013] Wherein, the personnel identification module includes:

[0014] A monitoring image acquisition unit, used to acquire regional monitoring images collected by a drone;

[0015] a foreground high-frequency object extraction unit, configured to extract the foreground high-frequency objects in the regional monitoring image to obtain a regional foreground high-frequency object image;

[0016] A shallow-deep feature extraction unit is used to perform multi-scale feature extraction and feature saliency enhancement processing on the regional foreground high-frequency object image to obtain a shallow-enhanced feature map of the regional foreground high-frequency object and a deep-enhanced feature map of the regional foreground high-frequency object;

[0017] A high-frequency object recognition unit, configured to generate a recognition result based on the principal component matching fusion features of the shallow layer enhancement feature map of the regional foreground high-frequency object and the deep layer enhancement feature map of the regional foreground high-frequency object, wherein the recognition result is used to indicate whether the high-frequency object is a human object;

[0018] The prompt signal generating unit is used to generate a prompt signal indicating that there are human activities around in response to the recognition result that the high-frequency object is a human object.

[0019] Optionally, the foreground high-frequency object extraction unit includes: a foreground area extraction subunit, used to input the regional monitoring image into a foreground-background segmentation network to obtain a regional monitoring foreground image; and a high-frequency component extraction subunit, used to extract high-frequency components in the regional monitoring foreground image to obtain the regional foreground high-frequency object image.

[0020] Optionally, the shallow-depth feature extraction unit comprises: a multi-scale feature extraction sub-unit, used to input the regional foreground high-frequency object image into an object multi-scale feature extractor based on a dilated pyramid network to obtain a shallow feature map of the regional foreground high-frequency object and a deep feature map of the regional foreground high-frequency object; a gradient mask sub-unit, used to input the shallow feature map of the regional foreground high-frequency object and the deep feature map of the regional foreground high-frequency object into a gradient mask feature saliency enhancement module to obtain a shallow enhanced feature map of the regional foreground high-frequency object and a deep enhanced feature map of the regional foreground high-frequency object.

[0021] Optionally, the gradient mask subunit includes: a gradient amplitude calculation secondary subunit, which is used to calculate the multi-directional gradient value distribution of each position in the shallow feature map of the regional foreground high-frequency object, and determine the gradient amplitude value of each position in the shallow feature map of the regional foreground high-frequency object based on the multi-directional gradient value distribution of each position to obtain the gradient amplitude distribution map of the shallow feature of the regional foreground high-frequency object; a local significance measurement secondary subunit, which is used to calculate the gradient amplitude local description operator of each position in the gradient amplitude distribution map of the shallow feature of the regional foreground high-frequency object to obtain the local significant distribution map of the gradient amplitude of the shallow feature of the regional foreground high-frequency object; a gated mask secondary subunit, which is used to input the local significant distribution map of the gradient amplitude of the shallow feature of the regional foreground high-frequency object into a gated masker based on the GELU function to obtain a gradient amplitude local significant gated mask map; a mask application secondary subunit, which is used to calculate the position point multiplication between the gradient amplitude local significant gated mask map and the shallow feature map of the regional foreground high-frequency object to obtain the shallow enhanced feature map of the regional foreground high-frequency object.

[0022] Optionally, the local saliency measurement secondary sub-unit is used to: determine the scale of the local neighborhood, calculate the average of the differences between the gradient amplitude value of a predetermined position in the shallow feature gradient amplitude distribution map of the regional foreground high-frequency object and the gradient amplitude values ​​of other positions in the local neighborhood to obtain a gradient amplitude local description operator corresponding to the predetermined position.

[0023] Optionally, the high-frequency object recognition unit includes: a principal component matching and fusion subunit, which is used to input the shallow enhanced feature map of the regional foreground high-frequency object and the deep enhanced feature map of the regional foreground high-frequency object into a significant fusion network based on feature principal component query matching to obtain a shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object; and a recognition result generation subunit, which is used to input the shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object into a classifier-based object identifier to obtain the recognition result.

[0024] Optionally, the principal component matching and fusion subunit includes: a feature flattening secondary subunit, which is used to perform feature flattening processing on the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object to obtain a shallow enhancement feature vector of the regional foreground high-frequency object and a deep enhancement feature vector of the regional foreground high-frequency object; a standardization secondary subunit, which is used to perform standardization processing on the shallow enhancement feature vector of the regional foreground high-frequency object and the deep enhancement feature vector of the regional foreground high-frequency object to obtain a standardized shallow enhancement feature vector of the regional foreground high-frequency object and a standardized deep enhancement feature vector of the regional foreground high-frequency object; a sample covariance calculation secondary subunit, which is used to calculate the sample covariance matrix of the standardized shallow enhancement feature vector of the regional foreground high-frequency object and the standardized deep enhancement feature vector of the regional foreground high-frequency object respectively to obtain a shallow sample covariance matrix of the regional foreground high-frequency object and a deep sample covariance matrix of the regional foreground high-frequency object; a principal component feature extraction secondary subunit, which is used to perform standardization processing on the shallow sample covariance matrix of the regional foreground high-frequency object and the deep sample covariance matrix of the regional foreground high-frequency object The difference matrix is ​​used to extract feature vectors based on matrix decomposition to obtain a set of shallow principal component feature vectors of regional foreground high-frequency objects and a set of deep principal component feature vectors of regional foreground high-frequency objects; the query matching secondary subunit is used to input the set of shallow principal component feature vectors of regional foreground high-frequency objects and the set of deep principal component feature vectors of regional foreground high-frequency objects into the maximum approximate query matching network to obtain a set of best matching pairs of shallow principal component feature vectors of regional foreground high-frequency objects and deep principal component feature vectors of regional foreground high-frequency objects; the fine-grained semantic joint secondary subunit is used to input each best matching pair in the set of best matching pairs of shallow principal component feature vectors of regional foreground high-frequency objects and deep principal component feature vectors of regional foreground high-frequency objects into the semantic fine-grained joint analysis module to obtain a set of shallow-deep principal component joint feature vectors of regional foreground high-frequency objects; the principal component matching feature global aggregation secondary subunit is used to cascade the set of shallow-deep principal component joint feature vectors of regional foreground high-frequency objects to obtain a significant fusion feature vector of shallow-deep principal components of regional foreground high-frequency objects.

[0025] Optionally, the query matching secondary sub-unit is used to: calculate the cosine similarity between the i-th regional foreground high-frequency object shallow principal component feature vector in the set of regional foreground high-frequency object shallow principal component feature vectors and the deep principal component feature vectors of each regional foreground high-frequency object in the set of regional foreground high-frequency object deep principal component feature vectors to obtain a set of cosine similarities; and form the best matching pair with the i-th regional foreground high-frequency object shallow principal component feature vector.

[0026] Optionally, the fine-grained semantic joint secondary sub-unit is used to: calculate the position point addition, position point subtraction and position point multiplication between the shallow principal component feature vector of the regional foreground high-frequency object and the deep principal component feature vector of the regional foreground high-frequency object in the best matching pair to obtain a first matching fusion result, a second matching fusion result and a third matching fusion result; cascade the first matching fusion result, the second matching fusion result and the third matching fusion result into a shallow-deep principal component multi-scale fusion feature vector of the regional foreground high-frequency object, and then perform one-dimensional convolution processing and maximum pooling processing on it to obtain the shallow-deep principal component joint feature vector of the regional foreground high-frequency object.

[0027] By adopting the above technical solution, the high-frequency foreground components of the regional monitoring images collected by the drone are extracted through deep learning-based image processing technology, and the extracted regional foreground high-frequency object images are subjected to multi-level feature extraction and feature gradient mask enhancement processing to obtain the shallow and deep significant feature representations of the regional foreground high-frequency objects, and then based on the main component matching fusion features of the two, intelligently identify whether the high-frequency object is a human object, and generate corresponding personnel activity prompt signals. In this way, it is possible to achieve rapid and accurate recognition of human targets in drone monitoring images, thereby improving the efficiency of public safety management and the timeliness of emergency response.

[0028] Other features and advantages of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0030] Figure 1 It is a block diagram of a drone public safety data management platform according to an exemplary embodiment.

[0031] Figure 2 The figure is a schematic diagram of a UAV public safety data management platform according to an exemplary embodiment.

[0032] Figure 3 It is a block diagram of the personnel identification module of a drone public safety data management platform according to an exemplary embodiment.

[0033] Figure 4 The present invention is a flowchart of a method for managing public safety data of a drone according to an exemplary embodiment.

[0034] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0036] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0037] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0038] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0041] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0042] Figure 1 FIG. 1 is a block diagram of a drone public safety data management platform according to an exemplary embodiment. Figure 1 As shown, the platform 100 includes:

[0043] A face recognition module 101 is used to identify and verify the identity information of the drone operator;

[0044] Fireworks recognition module 102, used to detect and recognize fireworks events in the images taken by the drone;

[0045] The license plate recognition module 103 is used to recognize the vehicle license plate information in the image captured by the drone;

[0046] The vehicle identification module 104 is used to identify the vehicles in the image captured by the drone and count the number of vehicles;

[0047] The bare land detection module 105 is used to detect the bare land area in the image taken by the drone;

[0048] The personnel identification module 106 is used to detect whether there is any personnel activity in the image captured by the drone and generate a corresponding prompt signal.

[0049] Among them, Figure 2 As shown in the figure, the UAV public safety data management platform is a comprehensive online management system for UAVs. It provides users with a one-stop solution through the cloud, realizing functions such as real-time monitoring of UAVs, historical records, team and personnel management, and equipment management.

[0050] The platform focuses on breaking down the data barriers of various types of equipment and information systems, realizing the sharing of drone equipment and resources, forming a pattern of construction in one place and global sharing, and forming a command pattern of "rapid response, high integration, unified command, coordinated operations, and information sharing."

[0051] The UAV public safety data management platform can be connected with multi-brand UAV systems, 3D map platforms, etc., and can access data such as the location and image transmission data of police UAVs, and realize UAV gimbal control. It can achieve seamless protocol access with domestic mainstream UAV brands; use the native APP of UAV manufacturers to avoid insurance claims risks; support various types of UAV video access platforms; support access to various types of automatic charging and battery replacement airports; support other single police equipment video and GPS data.

[0052] Specifically, the UAV public safety data management platform can perform online digital display of UAVs; real-time video broadcast of UAVs to determine whether there are debris on the airport roof, whether there are human activities in the surrounding area, whether the weather is suitable for takeoff, etc.; view basic information and flight information of UAVs, click on the current list of UAVs to view the corresponding name of the UAV, department, UAV model manufacturer, operator, battery power, signal strength, SN number and flight status (the airport part is still being adapted and not displayed in this interface); intelligent detection to realize the recognition of people, vehicles and objects during the flight (recognition during flight will cause the image transmission video to freeze), and output the recognition statistics within the recognition time period; command flight function, after the platform takes over the UAV and the airport, the platform remotely gives commands for one-key takeoff, remote control, and flight guidance operations.

[0053] More specifically, the drone public safety data management platform can realize pilot management and registration management of pilots in the whole city: including unit, name, mobile phone number, ID number, and driver's license number; mount management, realize the management of the mounted equipment within the police aviation, and can independently add mounted equipment, including equipment name, equipment picture, equipment details and other information; drone management, realize the management of drone equipment within the police aviation, including drone model, brand and type and other information. Collect and register data such as drone equipment type, flight control SN code, drone name, Civil Aviation Administration registration number, aircraft type, flight time, weight level, etc. Realize the life cycle management of drones, mainly including body status, usage time, battery life, etc. Set the cumulative use and maintenance time limit of drones and major consumables, and issue alarm prompts for overtime or illegal use; flight plan management, realize the management of daily and temporary flight tasks within the police aviation. Realize the application and approval of daily flight tasks and temporary flight tasks. Realize the approval management of civil drone flight applications and generate drone flight application reports. It can approve the flight applications submitted by civil drone pilots on the official account, or the units or individuals involved in large-scale activities or drone cluster flights must file with the public security department; historical video playback, drone historical flight video playback, can archive and store live video streams; can archive and store pictures and videos uploaded on the APP according to the team; support playback or downloading of archived pictures and video resources to the local; historical trajectory viewing, support display of drone number, pilot name, flight time, flight altitude, flight distance, flight trajectory playback, etc. Flight trajectory playback supports displaying the location on the map, as well as the latitude and longitude, speed and altitude information of the corresponding time point, and trajectory playback supports multiple speed playback; route planning, the platform can realize the function of intelligent route automatic planning, route planning automatically plans routes according to terrain, national no-fly zones, restricted areas, civil aviation clear air zones, custom no-fly zones and other data, can display the relative height of the flight route and the ground to automatically avoid, and the route data is uniformly saved on the platform to form a route library. It supports the planning of flight routes based on waypoints. Supports different waypoints to specify drones to perform specific actions, such as taking photos, recording videos, adjusting the gimbal angle, etc.; data statistics, displaying drone flight history. Flight log information: drone equipment type, flight crew account, drone SN code, take-off time, maximum altitude, flight time, flight distance, etc. Total number of flights, total flight time and average flight time within a specific statistical time. Statistical display of flight number bar charts based on flight time range. Supports flight log statistical analysis based on specific teams or members.

[0054] It should be understandable that the drone public safety data management platform can collect various flight data and images of drones of various brands, including Zero Degree and DJI, in real time, and grasp the situation of drones in real time. The drone flight process can be visually traced, and the camera angle and distance can be freely adjusted. Fine-tune the field of view of the captured image. The drone equipment, drone pilot team, flight video, and dispatch during the execution of tasks can be managed in a streamlined manner. Managers obtain the basics and real-time status of civil drones of the public and drone companies through detection equipment and official accounts, and realize standardized management of civil drones.

[0055] Accordingly, in the personnel identification module, in view of the limitations of traditional manual analysis and processing of image data collected by drones, this application uses deep learning-based image processing technology to extract foreground high-frequency components from regional monitoring images collected by drones, and performs multi-level feature extraction and feature gradient mask enhancement processing on the extracted regional foreground high-frequency object images to obtain shallow and deep significant feature representations of regional foreground high-frequency objects, and then intelligently identify whether the high-frequency object is a human object based on the main component matching fusion features of the two, and generate corresponding personnel activity prompt signals. In this way, it is possible to achieve rapid and accurate recognition of human targets in drone monitoring images, thereby improving the efficiency of public safety management and the timeliness of emergency response.

[0056] Figure 3 is a block diagram of the personnel identification module of a drone public safety data management platform according to an exemplary embodiment, such as Figure 3 As shown, the personnel identification module 106 includes:

[0057] The monitoring image acquisition unit 1061 is used to acquire the regional monitoring image collected by the drone;

[0058] A foreground high-frequency object extraction unit 1062, configured to extract the foreground high-frequency objects in the regional monitoring image to obtain a regional foreground high-frequency object image;

[0059] A shallow-deep feature extraction unit 1063 is used to perform multi-scale feature extraction and feature saliency enhancement processing on the regional foreground high-frequency object image to obtain a shallow-enhanced feature map of the regional foreground high-frequency object and a deep-enhanced feature map of the regional foreground high-frequency object;

[0060] A high-frequency object recognition unit 1064 is used to generate a recognition result based on the principal component matching fusion features of the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object, wherein the recognition result is used to indicate whether the high-frequency object is a human object;

[0061] The prompt signal generating unit 1065 is used to generate a prompt signal indicating that there are human activities around in response to the recognition result that the high-frequency object is a human object.

[0062] Based on this, in the technical solution of the present application, first, a regional monitoring image collected by a drone is obtained. Then, considering that the regional monitoring image usually contains a large amount of background information, this information is redundant for identifying specific objects (such as human bodies). Therefore, in order to remove or weaken the background information and only retain or highlight the foreground object, thereby reducing the amount of data for subsequent processing and improving processing efficiency, the present application uses a foreground-background segmentation network to process the regional monitoring image to extract a regional monitoring foreground image. In this way, factors such as complex textures and lighting changes in the background can be avoided from interfering with the recognition of the target object, and attention can be focused on the foreground object itself, thereby improving recognition accuracy.

[0063] In one embodiment of the present application, the foreground high-frequency object extraction unit includes: a foreground area extraction subunit, used to input the regional monitoring image into a foreground-background segmentation network to obtain a regional monitoring foreground image; and a high-frequency component extraction subunit, used to extract high-frequency components in the regional monitoring foreground image to obtain the regional foreground high-frequency object image.

[0064] Next, considering that the human target object usually corresponds to the high-frequency components in the image, that is, it has a higher spatial frequency change, therefore, in order to further narrow the processing range, in the technical solution of the present application, based on the frequency domain analysis method, the high-frequency components in the regional monitoring foreground image are extracted. Specifically, first, the regional monitoring foreground image is subjected to a fast Fourier transform to convert the image from the spatial domain to the frequency domain. In the frequency domain, the high-frequency components usually correspond to the edge, contour, shape and other detail information in the image, while the low-frequency components correspond to the smooth area in the image. By setting a suitable threshold, the low-frequency components are filtered out, and only the high-frequency components are retained, and then the high-frequency components are converted from the frequency domain back to the spatial domain through an inverse Fourier transform, thereby obtaining a regional foreground high-frequency object image. In this way, background interference can be effectively removed, key information in the image can be extracted, and more accurate basic data can be provided for subsequent feature extraction and target recognition.

[0065] In one embodiment of the present application, the shallow-depth feature extraction unit includes: a multi-scale feature extraction sub-unit, which is used to input the regional foreground high-frequency object image into an object multi-scale feature extractor based on a hollow pyramid network to obtain a shallow feature map of the regional foreground high-frequency object and a deep feature map of the regional foreground high-frequency object; a gradient mask sub-unit, which is used to input the shallow feature map of the regional foreground high-frequency object and the deep feature map of the regional foreground high-frequency object into a gradient mask feature saliency enhancement module to obtain a shallow enhanced feature map of the regional foreground high-frequency object and a deep enhanced feature map of the regional foreground high-frequency object.

[0066] Next, in order to more comprehensively understand the characteristics of the target object to determine whether it is a human object, the present application uses a dilated pyramid network (ASPP) as an object multi-scale feature extractor to perform multi-scale feature extraction on the regional foreground high-frequency object image. Among them, the dilated pyramid network is an effective multi-scale feature extraction method, which captures features of different scales by introducing dilated convolution kernels of different sizes in the convolution layer, and can simultaneously capture shallow local detail information and deep overall context information of the target object, so as to obtain a shallow feature map of the regional foreground high-frequency object and a deep feature map of the regional foreground high-frequency object, thereby achieving a comprehensive description of the target object.

[0067] Then, in order to further highlight the key features of the target object and improve the accuracy of feature expression, the present application introduces a gradient mask feature saliency enhancement module to process the shallow feature map of the regional foreground high-frequency object and the deep feature map of the regional foreground high-frequency object respectively, and generate a corresponding gradient mask by calculating the gradient distribution of the feature map, thereby highlighting the key areas in the feature map and improving the feature discrimination. Specifically, the module first quantifies the local change intensity of the pixels at each position by calculating the gradient amplitude value at each position in the input feature map. Then, the gradient amplitude local description operator is calculated by measuring the significance of the gradient amplitude value at each position relative to the gradient amplitude value at other positions in the local neighborhood, so as to represent the feature importance of each position. Then, the GELU function is used to nonlinearly activate the gradient amplitude local description operator at each position to generate a gated mask to guide the enhancement of key features in the feature map. Finally, the feature map is element-by-element weighted based on the generated gated mask, so as to focus on the key feature area in the feature map and remove background noise to obtain a clearer and more discriminative shallow enhanced feature map of the regional foreground high-frequency object and a deep enhanced feature map of the regional foreground high-frequency object.

[0068] In one embodiment of the present application, the gradient mask subunit includes: a gradient amplitude calculation secondary subunit, which is used to calculate the multi-directional gradient value distribution of each position in the shallow feature map of the regional foreground high-frequency object, and determine the gradient amplitude value of each position in the shallow feature map of the regional foreground high-frequency object based on the multi-directional gradient value distribution of each position to obtain the gradient amplitude distribution map of the shallow feature of the regional foreground high-frequency object; a local significance measurement secondary subunit, which is used to calculate the gradient amplitude local description operator of each position in the gradient amplitude distribution map of the shallow feature of the regional foreground high-frequency object to obtain the local significant distribution map of the gradient amplitude of the shallow feature of the regional foreground high-frequency object; a gated mask secondary subunit, which is used to input the local significant distribution map of the gradient amplitude of the shallow feature of the regional foreground high-frequency object into a gated masker based on the GELU function to obtain a gradient amplitude local significant gated mask map; a mask application secondary subunit, which is used to calculate the position point multiplication between the gradient amplitude local significant gated mask map and the shallow feature map of the regional foreground high-frequency object to obtain the shallow enhanced feature map of the regional foreground high-frequency object.

[0069] Furthermore, in one embodiment of the present application, the local saliency measurement secondary sub-unit is used to: determine the scale of the local neighborhood, calculate the average of the difference between the gradient amplitude value of a predetermined position in the shallow feature gradient amplitude distribution map of the regional foreground high-frequency object and the gradient amplitude values ​​of other positions in the local neighborhood to obtain a gradient amplitude local description operator corresponding to the predetermined position.

[0070] Specifically, the shallow feature map of the regional foreground high-frequency object is processed by the following feature distribution gradient mask formula to obtain the shallow enhanced feature map of the regional foreground high-frequency object, wherein the feature distribution gradient mask formula is:

[0071] g i (i,j,k)=F(i+1,j,k)-F(i-1,j,k)

[0072] g j (i,j,k)=F(i,j+1,k)-F(i,j-1,k)

[0073] g k (i,j,k)=F(i,j,k+1)-F(i,j,k-1)

[0074]

[0075] {G(i,j,k)}=GELU(GMLD(i,j,k))

[0076] F1=G⊙F

[0077] Among them, F(i+1,j,k), F(i-1,j,k), F(i,j+1,k), F(i,j-1,k), F(i,j,k+1) and F(i,j,k-1) respectively represent the grayscale values ​​of the corresponding positions of the shallow feature map of the foreground high-frequency object in the region, and g i (i, j, k) represents the horizontal coordinate direction gradient value of the pixel point at position (i, j, k) in the shallow feature map of the foreground high-frequency object in the region, g j (i, j, k) represents the vertical coordinate direction gradient value of the pixel point at position (i, j, k) in the shallow feature map of the foreground high-frequency object in the region, g k (i,j,k) represents the channel direction gradient value of the pixel point at position (i,j,k) in the shallow feature map of the foreground high-frequency object in the region, g(i,j,k) represents the gradient amplitude value at position (i,j,k) of the shallow feature map of the foreground high-frequency object in the region, A represents the gradient amplitude value set in the local neighborhood centered at position (i,j,k) of the shallow feature gradient amplitude distribution map of the foreground high-frequency object in the region, count(A) represents the number of gradient amplitude values ​​in the gradient amplitude value set in the local neighborhood, Δi, Δj and Δk are respectively represents the offset in the horizontal, vertical and channel directions, GMLD(i,j,k) represents the gradient amplitude local description operator of the (i,j,k)th position of the shallow feature map of the foreground high-frequency object in the region, GELU represents the GELU function, G(i,j,k) represents the gradient amplitude local description operator of the (i,j,k)th position of the gradient amplitude local significant gated mask map, G represents the gradient amplitude local significant gated mask map, ⊙ represents the point multiplication by position, and F1 represents the shallow enhanced feature map of the foreground high-frequency object in the region.

[0078] In one embodiment of the present application, the high-frequency object recognition unit includes: a principal component matching fusion subunit, which is used to input the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object into a significant fusion network based on feature principal component query matching to obtain a shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object; and a recognition result generation subunit, which is used to input the shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object into an object recognizer based on a classifier to obtain the recognition result.

[0079] Furthermore, in order to make full use of the information complementarity of the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object to achieve more accurate target object recognition, the present application introduces a saliency fusion network based on feature principal component query matching to fuse the shallow features and deep features of the regional foreground high-frequency object, and achieves deep fusion of shallow and deep features through principal component analysis and matching fusion mechanism. Specifically, the network first flattens the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object into a one-dimensional vector, and standardizes it to eliminate the influence of different dimensions and magnitudes. Then, the principal component analysis algorithm is used to calculate the sample covariance and extract the principal component features of the shallow enhancement feature vector of the regional foreground high-frequency object and the deep enhancement feature vector of the regional foreground high-frequency object after standardization, and the main components are mined and redundant information is removed to generate a set of shallow principal component feature vectors of the regional foreground high-frequency object and a set of deep principal component feature vectors of the regional foreground high-frequency object. Then, based on the maximum approximate query matching between feature vectors, the best matching pair is found by calculating and comparing the similarity between two principal component feature vector sets, and the significant correlation features between the two are extracted by performing fine-grained semantic joint analysis on each best matching pair. Finally, the feature joint results of each best matching pair are cascaded to aggregate the shallow features of the regional foreground high-frequency objects and the multiple core component joint features of the deep features to generate a global shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency objects.

[0080] In one embodiment of the present application, the principal component matching and fusion subunit includes: a feature flattening secondary subunit, which is used to perform feature flattening processing on the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object to obtain a shallow enhancement feature vector of the regional foreground high-frequency object and a deep enhancement feature vector of the regional foreground high-frequency object; a standardization secondary subunit, which is used to perform standardization processing on the shallow enhancement feature vector of the regional foreground high-frequency object and the deep enhancement feature vector of the regional foreground high-frequency object to obtain a standardized shallow enhancement feature vector of the regional foreground high-frequency object and a standardized deep enhancement feature vector of the regional foreground high-frequency object; a sample covariance calculation secondary subunit, which is used to calculate the sample covariance matrix of the standardized shallow enhancement feature vector of the regional foreground high-frequency object and the standardized deep enhancement feature vector of the regional foreground high-frequency object respectively to obtain a shallow sample covariance matrix of the regional foreground high-frequency object and a deep sample covariance matrix of the regional foreground high-frequency object; a principal component feature extraction secondary subunit, which is used to perform standardization processing on the shallow sample covariance matrix of the regional foreground high-frequency object and the deep sample covariance matrix of the regional foreground high-frequency object The sample covariance matrix performs matrix decomposition-based feature vector extraction to obtain a set of shallow principal component feature vectors of regional foreground high-frequency objects and a set of deep principal component feature vectors of regional foreground high-frequency objects; a query matching secondary subunit is used to input the set of shallow principal component feature vectors of regional foreground high-frequency objects and the set of deep principal component feature vectors of regional foreground high-frequency objects into a maximum approximate query matching network to obtain a set of best matching pairs of shallow principal component feature vectors of regional foreground high-frequency objects and deep principal component feature vectors of regional foreground high-frequency objects; a fine-grained semantic joint secondary subunit is used to input each best matching pair in the set of best matching pairs of shallow principal component feature vectors of regional foreground high-frequency objects and deep principal component feature vectors of regional foreground high-frequency objects into a semantic fine-grained joint analysis module to obtain a set of shallow-deep principal component joint feature vectors of regional foreground high-frequency objects; a principal component matching feature global aggregation secondary subunit is used to cascade the set of shallow-deep principal component joint feature vectors of regional foreground high-frequency objects to obtain a significant fusion feature vector of shallow-deep principal components of regional foreground high-frequency objects.

[0081] Furthermore, in one embodiment of the present application, the query matching secondary sub-unit is used to: calculate the cosine similarity between the i-th regional foreground high-frequency object shallow principal component feature vector in the set of the regional foreground high-frequency object shallow principal component feature vectors and the deep principal component feature vectors of each regional foreground high-frequency object in the set of the regional foreground high-frequency object deep principal component feature vectors to obtain a set of cosine similarities; and form the best matching pair with the i-th regional foreground high-frequency object shallow principal component feature vector.

[0082] Furthermore, in one embodiment of the present application, the fine-grained semantic joint secondary sub-unit is used to: calculate the position point addition, position point subtraction and position point multiplication between the shallow principal component feature vector of the regional foreground high-frequency object and the deep principal component feature vector of the regional foreground high-frequency object in the best matching pair to obtain a first matching fusion result, a second matching fusion result and a third matching fusion result; after cascading the first matching fusion result, the second matching fusion result and the third matching fusion result into a shallow-deep principal component multi-scale fusion feature vector of the regional foreground high-frequency object, one-dimensional convolution processing and maximum pooling processing are performed on it to obtain the shallow-deep principal component joint feature vector of the regional foreground high-frequency object.

[0083] Specifically, the shallow enhanced feature vector of the regional foreground high-frequency object and the deep enhanced feature vector of the regional foreground high-frequency object are fused using the following principal component query matching formula to obtain the shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object, wherein the principal component query matching formula is:

[0084]

[0085] C1=U1Λ1U1 T

[0086] C2=U2Λ2U2 T

[0087] U1=[v 11 ,v 12 ,…,v 1m ]

[0088] Λ1=diag(λ 11 ,λ 12 ,…,λ 1m )

[0089] U2=[v 21 ,v 22 ,…,v 2m ]

[0090] Λ2=diag(λ 21 ,λ 22 ,…,λ 2m )

[0091]

[0092]

[0093] V=[v p1 ;v p2 ;...v pm]

[0094] Wherein, V1 represents the shallow enhancement feature vector of the foreground high-frequency object in the region, μ1 and σ1 represent the mean and standard deviation of the shallow enhancement feature vector of the foreground high-frequency object in the region, respectively. 1nor represents the shallow enhancement feature vector of the foreground high-frequency object in the standardized region, V2 represents the deep enhancement feature vector of the foreground high-frequency object in the region, μ2 and σ2 represent the mean and standard deviation of the deep enhancement feature vector of the foreground high-frequency object in the region, respectively, V 2no represents the standardized regional foreground high-frequency object deep enhancement feature vector, the value of n1 is the length of the regional foreground high-frequency object shallow enhancement feature vector, the value of n2 is the length of the regional foreground high-frequency object deep enhancement feature vector, C1 represents the regional foreground high-frequency object shallow sample covariance matrix, C2 represents the regional foreground high-frequency object deep sample covariance matrix, U1 represents the matrix composed of the set of regional foreground high-frequency object shallow principal component feature vectors obtained by performing eigenvalue decomposition on the regional foreground high-frequency object shallow sample covariance matrix, Λ1 represents the diagonal matrix composed of the set of regional foreground high-frequency object shallow principal component eigenvalues ​​obtained by performing eigenvalue decomposition on the regional foreground high-frequency object shallow sample covariance matrix, v 11 、v 12 、v 1i and v 1m Respectively represent the first, second, i-th and m-th shallow principal component feature vectors of the foreground high-frequency object in the set of shallow principal component feature vectors of the foreground high-frequency object in the region, λ 11 , 12 and λ 1m respectively represent the first, second and mth shallow principal component eigenvalues ​​of the regional foreground high-frequency object in the set of shallow principal component eigenvalues ​​of the regional foreground high-frequency object, the value of m is the number of shallow principal component eigenvalues ​​of the regional foreground high-frequency object, U2 represents the matrix composed of the set of deep principal component eigenvectors of the regional foreground high-frequency object obtained by performing eigenvalue decomposition on the deep sample covariance matrix of the regional foreground high-frequency object, Λ2 represents the diagonal matrix composed of the set of deep principal component eigenvalues ​​of the regional foreground high-frequency object obtained by performing eigenvalue decomposition on the deep sample covariance matrix of the regional foreground high-frequency object, v 21 、v 22 、v 2j 、v 2k and v 2m respectively represent the first, second, jth, kth and mth regional foreground high-frequency object deep principal component feature vectors in the set of regional foreground high-frequency object deep principal component feature vectors, λ 21 , 22 and λ 2mdenote the first, second and mth regional foreground high-frequency object deep principal component eigenvalues ​​in the set of deep principal component eigenvalues ​​of the regional foreground high-frequency object, respectively; diag is a diag function used to construct a diagonal matrix; <·> denotes an inner product; ‖·‖ denotes the modulus of a eigenvector; argmax(·) denotes the index corresponding to the maximum value; k denotes the index of the regional foreground high-frequency object deep principal component eigenvector that forms the best matching pair with the i-th regional foreground high-frequency object shallow principal component eigenvector; [·,·] denotes a cascade operation; MaxPool(·) denotes a maximum pooling operation; conv1D(·) denotes a one-dimensional convolution operation; ⊙ denotes a dot product; Indicates point addition, Indicates point minus, v pi represents the joint feature vector of the shallow-deep principal components of the foreground high-frequency object in the i-th region, and V represents the significant fusion feature vector of the shallow-deep principal components of the foreground high-frequency object in the said region.

[0095] The shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object contains multi-level target object information. In order to further convert it into a specific target object recognition result, in the technical solution of the present application, a classifier-based object identifier is used to classify the shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object to obtain a recognition result indicating whether the high-frequency object is a human object.

[0096] Secondly, in response to the recognition result that the high-frequency object is a human object, a prompt signal indicating that there are human activities around is generated.

[0097] In a preferred example, considering that the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object respectively represent the image shallow semantic features and image deep structural semantic features of the regional foreground high-frequency object image, therefore, when performing significant fusion based on feature principal component query matching, due to the difference in receptive field and depth of the image spatial domain feature distribution, the position information is lost when the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object are superimposed and expanded into a feature vector. The shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object will have a diversified set expression distribution of image semantic features. Therefore, it is expected to improve the balance between the accuracy and completeness of the regression mapping when the shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object is input into a classifier-based object identifier for classification regression, thereby improving the accuracy of the recognition result.

[0098] Based on this, in the preferred example, inputting the shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object into the classifier-based object identifier to obtain the identification result specifically includes:

[0099] Determine the maximum eigenvalue of the regional foreground high-frequency object shallow-deep principal component significant fusion and the minimum eigenvalue of the regional foreground high-frequency object shallow-deep principal component significant fusion of the regional foreground high-frequency object shallow-deep principal component significant fusion feature vector, and calculate the regional foreground high-frequency object shallow-deep principal component significant fusion mean and regional foreground high-frequency object shallow-deep principal component significant fusion standard deviation of the feature set of the regional foreground high-frequency object shallow-deep principal component significant fusion feature vector;

[0100] Calculating the quotient of the significant fusion mean of the shallow-deep principal components of the foreground high-frequency object in the region and the significant fusion standard deviation of the shallow-deep principal components of the foreground high-frequency object in the region to obtain a statistical normalization value of the significant fusion of the shallow-deep principal components of the foreground high-frequency object in the region;

[0101] Calculate the inverse of each eigenvalue of the shallow-deep principal component significant fusion eigenvector of the regional foreground high-frequency object, multiply it by the difference between the maximum eigenvalue of the shallow-deep principal component significant fusion of the regional foreground high-frequency object and the minimum eigenvalue of the shallow-deep principal component significant fusion of the regional foreground high-frequency object, and then perform point subtraction from the statistical normalization value of the shallow-deep principal component significant fusion of the regional foreground high-frequency object to obtain the shallow-deep principal component significant fusion distribution approximation vector of the regional foreground high-frequency object;

[0102] Calculate an exponential function with a natural constant as the base, where each eigenvalue of the shallow-deep principal component significant fusion distribution approximation vector of the regional foreground high-frequency object is an exponent, and perform point addition with the statistical normalization value of the shallow-deep principal component significant fusion of the regional foreground high-frequency object to obtain a shallow-deep principal component significant fusion probability approximation vector of the regional foreground high-frequency object;

[0103] Calculating the base-two logarithm of the absolute value of each eigenvalue of the regional foreground high-frequency object shallow-deep principal component significant fusion probability approximation vector to obtain an optimized regional foreground high-frequency object shallow-deep principal component significant fusion eigenvector; and

[0104] The optimized regional foreground high-frequency object shallow-deep principal component significant fusion feature vector is input into a classifier-based object identifier to obtain a recognition result.

[0105] Among them, the optimization expression of the shallow-deep principal component significant fusion feature vector of the foreground high-frequency object in the region, denoted as V, is:

[0106]

[0107] α, β, v min and v max are the feature sets v of the shallow-deep principal component significant fusion feature vectors of the foreground high-frequency objects in the region i ∈V a The mean, standard deviation, minimum and maximum values ​​of a is the shallow-deep principal component significant fusion feature vector of the foreground high-frequency object in the region, v max is the maximum eigenvalue of the shallow-deep principal component significant fusion eigenvector of the regional foreground high-frequency object shallow-deep principal component significant fusion, v min is the minimum eigenvalue of the shallow-deep principal component significant fusion eigenvector of the regional foreground high-frequency object shallow-deep principal component significant fusion, V a ⊙-1 is the reciprocal of each eigenvalue of the shallow-deep principal component significant fusion eigenvector of the foreground high-frequency object in the region, α is the mean of the shallow-deep principal component significant fusion of the foreground high-frequency object in the region, β is the quotient of the standard deviation of the shallow-deep principal component significant fusion of the foreground high-frequency object in the region, exp[·] is an exponential function calculated with a natural constant as the base, and each eigenvalue of the vector is an exponential function, log is a logarithmic value with a base of two, V′ a is the optimized regional foreground high-frequency object shallow-deep principal component significant fusion feature vector, ⊙ is the position point multiplication, It is added by location point. It is subtracted by position point.

[0108] That is, by performing random statistical normalization on the diversified feature sets of the shallow-deep principal component significant fusion feature vectors of the regional foreground high-frequency objects, the confidence space constructed based on the overall eigenvalues ​​of the shallow-deep principal component significant fusion feature vectors of the regional foreground high-frequency objects is approximated by the standardized continuous probability density distribution relative to the response hypothesis test of each eigenvalue of the shallow-deep principal component significant fusion feature vectors of the regional foreground high-frequency objects, thereby establishing the target reachability of the discrete feature distribution of the shallow-deep principal component significant fusion feature vectors of the regional foreground high-frequency objects to the unified regression target, so as to achieve the balanced executableness between mapping accuracy and mapping completeness in the classification regression process based on the discrete feature distribution of the shallow-deep principal component significant fusion feature vectors of the regional foreground high-frequency objects, and improve the accuracy of the recognition result obtained by inputting the shallow-deep principal component significant fusion feature vectors of the regional foreground high-frequency objects into the classifier-based object identifier.

[0109] In summary, the above scheme is adopted to extract the foreground high-frequency components of the regional monitoring images collected by drones based on deep learning image processing technology, and multi-level feature extraction and feature gradient mask enhancement processing are performed on the extracted regional foreground high-frequency object images to obtain the shallow and deep significant feature representations of the regional foreground high-frequency objects, and then based on the main component matching fusion features of the two, intelligently identify whether the high-frequency object is a human object, and generate corresponding personnel activity prompt signals. In this way, it is possible to achieve rapid and accurate recognition of human targets in drone monitoring images, thereby improving the efficiency of public safety management and the timeliness of emergency response.

[0110] Figure 4 is a flow chart of a method for managing public safety data of a drone according to an exemplary embodiment. Figure 4 As shown, the method includes:

[0111] Step S201, identifying and verifying the identity information of the drone operator;

[0112] Step S202, detecting and identifying fireworks events in the image captured by the drone;

[0113] Step S203, identifying the vehicle license plate information in the image taken by the drone;

[0114] Step S204, identifying vehicles in the image taken by the drone and counting the number of vehicles;

[0115] Step S205, detecting the exposed land area in the image taken by the drone;

[0116] Step S206, detecting whether there is human activity in the image taken by the drone, and generating a corresponding prompt signal;

[0117] Among them, step S206, detecting whether there is human activity in the image taken by the drone and generating a corresponding prompt signal, includes:

[0118] Step S2061, obtaining a regional monitoring image collected by a drone;

[0119] Step S2062, extracting the foreground high-frequency objects in the regional monitoring image to obtain a regional foreground high-frequency object image;

[0120] Step S2063, performing multi-scale feature extraction and feature saliency enhancement processing on the regional foreground high-frequency object image to obtain a regional foreground high-frequency object shallow enhancement feature map and a regional foreground high-frequency object deep enhancement feature map;

[0121] Step S2064, generating a recognition result based on the principal component matching fusion features of the shallow layer enhancement feature map of the regional foreground high-frequency object and the deep layer enhancement feature map of the regional foreground high-frequency object, wherein the recognition result is used to indicate whether the high-frequency object is a human object;

[0122] Step S2065: In response to the recognition result that the high-frequency object is a human object, a prompt signal is generated indicating that there are human activities around.

[0123] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0124] like Figure 5 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0125] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0126] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0127] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0128] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0129] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0130] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The modules involved in the embodiments described in this application can be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the test parameter acquisition module can also be described as a "module for acquiring device test parameters corresponding to the target device".

[0133] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0134] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] In the application scenario of the drone public safety data management platform, first, the regional monitoring image collected by the drone is obtained; then, the obtained regional monitoring image is input into a server deployed with the drone public safety data management platform algorithm, wherein the server can process the regional monitoring image based on the drone public safety data management platform algorithm to generate a prompt signal that there are human activities around.

[0136] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0137] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0138] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely exemplary forms of implementation. With respect to the apparatus in the above-mentioned embodiments, the specific manner in which each module performs the operation has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

1. A UAV public safety data management platform, characterized in that: include: A face recognition module, used to identify and verify the identity of the drone operator; Fireworks recognition module, used to detect and identify fireworks events in drone footage; License plate recognition module, used to identify vehicle license plate information in the drone footage; The vehicle identification module is used to identify vehicles in the drone footage and count the number of vehicles; The bare land detection module is used to detect the bare land area in the drone footage; The personnel identification module is used to detect whether there are personnel activities in the drone footage and generate corresponding prompt signals; Wherein, the personnel identification module includes: A monitoring image acquisition unit, used to acquire regional monitoring images collected by a drone; a foreground high-frequency object extraction unit, configured to extract the foreground high-frequency objects in the regional monitoring image to obtain a regional foreground high-frequency object image; A shallow-deep feature extraction unit is used to perform multi-scale feature extraction and feature saliency enhancement processing on the regional foreground high-frequency object image to obtain a shallow-enhanced feature map of the regional foreground high-frequency object and a deep-enhanced feature map of the regional foreground high-frequency object; A high-frequency object recognition unit, configured to generate a recognition result based on the principal component matching fusion features of the shallow layer enhancement feature map of the regional foreground high-frequency object and the deep layer enhancement feature map of the regional foreground high-frequency object, wherein the recognition result is used to indicate whether the high-frequency object is a human object; a prompt signal generating unit, configured to generate a prompt signal indicating that there are human activities around the high-frequency object in response to the recognition result indicating that the high-frequency object is a human object; The depth feature extraction unit comprises: A multi-scale feature extraction subunit, used for inputting the regional foreground high-frequency object image into an object multi-scale feature extractor based on a dilated pyramid network to obtain a regional foreground high-frequency object shallow feature map and a regional foreground high-frequency object deep feature map; A gradient mask subunit, used for inputting the shallow feature map of the regional foreground high-frequency object and the deep feature map of the regional foreground high-frequency object into a gradient mask feature saliency enhancement module to obtain the shallow enhanced feature map of the regional foreground high-frequency object and the deep enhanced feature map of the regional foreground high-frequency object; The gradient mask subunit comprises: A gradient amplitude calculation secondary subunit is used to calculate the multi-directional gradient value distribution of each position in the shallow feature map of the regional foreground high-frequency object, and determine the gradient amplitude value of each position in the shallow feature map of the regional foreground high-frequency object based on the multi-directional gradient value distribution of each position to obtain a gradient amplitude distribution map of the shallow feature of the regional foreground high-frequency object; A local saliency measurement secondary subunit is used to calculate the local description operator of the gradient amplitude at each position in the regional foreground high-frequency object shallow feature gradient amplitude distribution map to obtain a local saliency distribution map of the regional foreground high-frequency object shallow feature gradient amplitude; A gated mask secondary subunit, used for inputting the local significant distribution map of shallow feature gradient amplitude of the foreground high-frequency object in the region into a gated masker based on a GELU function to obtain a gradient amplitude local significant gated mask map; The mask applying secondary subunit is used to calculate the position point multiplication between the gradient amplitude local significant gated mask map and the shallow feature map of the regional foreground high-frequency object to obtain the shallow enhanced feature map of the regional foreground high-frequency object.

2. The UAV public safety data management platform according to claim 1, characterized in that: The foreground high-frequency object extraction unit comprises: A foreground region extraction subunit, used for inputting the regional monitoring image into a foreground-background segmentation network to obtain a regional monitoring foreground image; The high-frequency component extraction subunit is used to extract the high-frequency components in the regional monitoring foreground image to obtain the regional foreground high-frequency object image.

3. The UAV public safety data management platform according to claim 2, characterized in that: The local saliency measurement secondary sub-unit is used to: The scale of the local neighborhood is determined, and the mean value of the difference between the gradient amplitude value of a predetermined position in the shallow feature gradient amplitude distribution map of the foreground high-frequency object in the region and the gradient amplitude values ​​of other positions in the local neighborhood is calculated to obtain a gradient amplitude local description operator corresponding to the predetermined position.

4. The UAV public safety data management platform according to claim 3, characterized in that: The high-frequency object recognition unit comprises: A principal component matching fusion subunit is used to input the shallow enhancement feature map of the regional foreground high-frequency object and the deep enhancement feature map of the regional foreground high-frequency object into a significant fusion network based on feature principal component query matching to obtain a shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object; The recognition result generating subunit is used to input the shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object into the classifier-based object identifier to obtain the recognition result.

5. The UAV public safety data management platform according to claim 4, characterized in that: The principal component matching fusion subunit includes: A feature flattening secondary subunit is used to perform feature flattening processing on the regional foreground high-frequency object shallow enhancement feature map and the regional foreground high-frequency object deep enhancement feature map to obtain a regional foreground high-frequency object shallow enhancement feature vector and a regional foreground high-frequency object deep enhancement feature vector; A standardization secondary subunit is used to perform standardization processing on the shallow enhancement feature vector of the regional foreground high-frequency object and the deep enhancement feature vector of the regional foreground high-frequency object to obtain a standardized shallow enhancement feature vector of the regional foreground high-frequency object and a standardized deep enhancement feature vector of the regional foreground high-frequency object; A sample covariance calculation secondary subunit is used to calculate the sample covariance matrix of the standardized regional foreground high-frequency object shallow layer enhancement feature vector and the standardized regional foreground high-frequency object deep layer enhancement feature vector to obtain the regional foreground high-frequency object shallow layer sample covariance matrix and the regional foreground high-frequency object deep layer sample covariance matrix; A principal component feature extraction secondary subunit is used to perform feature vector extraction based on matrix decomposition on the shallow sample covariance matrix of the regional foreground high-frequency object and the deep sample covariance matrix of the regional foreground high-frequency object to obtain a set of shallow principal component feature vectors of the regional foreground high-frequency object and a set of deep principal component feature vectors of the regional foreground high-frequency object; A query matching secondary subunit, used for inputting the set of shallow principal component feature vectors of the regional foreground high-frequency objects and the set of deep principal component feature vectors of the regional foreground high-frequency objects into a maximum approximate query matching network to obtain a set of best matching pairs of shallow principal component feature vectors of the regional foreground high-frequency objects and deep principal component feature vectors of the regional foreground high-frequency objects; A fine-grained semantic joint secondary subunit is used to input each best matching pair in the set of best matching pairs of the shallow principal component feature vectors of the regional foreground high-frequency objects and the deep principal component feature vectors of the regional foreground high-frequency objects into a semantic fine-grained joint analysis module to obtain a set of shallow-deep principal component joint feature vectors of the regional foreground high-frequency objects; The principal component matching feature global aggregation secondary subunit is used to cascade the set of shallow-deep principal component joint feature vectors of the regional foreground high-frequency object to obtain the shallow-deep principal component significant fusion feature vector of the regional foreground high-frequency object.

6. The UAV public safety data management platform according to claim 5, characterized in that: The query matching secondary subunit is used to: Calculating the cosine similarity between the i-th regional foreground high-frequency object shallow principal component feature vector in the set of the regional foreground high-frequency object shallow principal component feature vectors and each regional foreground high-frequency object deep principal component feature vector in the set of the regional foreground high-frequency object deep principal component feature vectors to obtain a set of cosine similarities; The deep principal component feature vector of the regional foreground high-frequency object corresponding to the maximum value in the set of cosine similarities and the shallow principal component feature vector of the i-th regional foreground high-frequency object form the best matching pair.

7. The UAV public safety data management platform according to claim 6, characterized in that: The fine-grained semantic joint secondary sub-unit is used to: Calculate the position point addition, position point subtraction and position point multiplication between the shallow principal component feature vector of the regional foreground high-frequency object and the deep principal component feature vector of the regional foreground high-frequency object in the best matching pair to obtain a first matching fusion result, a second matching fusion result and a third matching fusion result; After the first matching fusion result, the second matching fusion result and the third matching fusion result are cascaded into a shallow-deep principal component multi-scale fusion feature vector of the regional foreground high-frequency object, one-dimensional convolution processing and maximum pooling processing are performed on them to obtain the shallow-deep principal component joint feature vector of the regional foreground high-frequency object.

Images