A method and system for air-ground collaborative target detection driven by cognitive semantic communication

Abstract

The present invention discloses a method and system for air-ground collaborative target detection driven by cognitive semantic communication, including: image multi-scale semantic feature extraction; multi-scale semantic feature compression encoding; channel semantic transmission; multi-scale semantic feature decoding and reconstruction; knowledge graph node embedding; knowledge graph driven target detection; input aerial image target detection training set to train the air-ground collaborative target detection model driven by cognitive semantic communication; input aerial image target detection test set to verify the task accuracy and performance of the air-ground collaborative target detection model driven by cognitive semantic communication. The present invention alleviates the problem of scarce computing resources for aerial equipment, improves the robustness of the system in a high compression rate and low signal-to-noise ratio wireless communication environment, introduces the prior information of the knowledge graph to assist the communication process and target detection, improves the system's global cognition of the detected image, and improves the target detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and specifically to an air-to-ground collaborative target detection method and system driven by cognitive semantic communication. The present invention uses a cognitive semantic communication paradigm based on a knowledge graph to coordinate aerial equipment and ground terminals to jointly perform target detection tasks, thereby reducing the computing resource requirements of aerial equipment, improving communication robustness and spectrum efficiency, and improving the accuracy of aerial image target detection tasks. Background Art

[0002] Thanks to their wide applicability and strong flexibility, aerial devices such as drones and FPVs have recently become a hot topic in the field of computer vision and remote sensing, and are used to perform a variety of tasks. Target detection based on aerial devices has great application value and prospects in various military and civilian fields, such as military reconnaissance, environmental monitoring, urban planning, etc., and has gradually attracted widespread attention from academia and industry. However, due to the limitation of computing power, target detection based on aerial devices often faces a trade-off between computing resources and detection accuracy, which has become one of its application bottlenecks. Therefore, it is urgent to develop new architectures to improve computing efficiency and detection accuracy, and to achieve technological breakthroughs to support its widespread application.

[0003] There are currently two architectures for target detection based on aerial devices. Architecture 1 deploys the target detection system on the aerial device and uses the onboard computing resources to perform the target detection task. Early work based on Architecture 1 did not consider the limitations of aerial device computing load and battery energy consumption, and focused on improving real-time performance and detection accuracy of aerial images. However, it soon encountered the upper limit of real-time performance and detection accuracy caused by computing resource limitations. Subsequent research considers breakthroughs through lightweight networks. Zhang, Pengyi and Zhong, Yunxin and Li, Xiaoqiong proposed an airborne target detection system based on the YOLOv3 algorithm in their paper "SlimYOLOv3: Narrower, faster and better for real-time UAV applications" (Proceedings of the IEEE / CVFInternational Conference on Computer Vision Workshops). By balancing the number of parameters, memory usage and inference time, it reduces the demand for computing resources while improving detection accuracy. However, the number of parameters is a major factor in determining the detection accuracy of target detection algorithms based on deep learning. Therefore, algorithms based on the trade-off between computing resources and parameter numbers are still limited by computing resource limitations and are difficult to fully exert the performance of the network. The main idea of ​​architecture 2 is to offload the detection task to the computing server. In architecture 2, the aerial device captures the aerial image signal and transmits it to the computing server through wireless communication, and the computing server uses additional abundant computing resources to process the detection task. Ozer, Sedat and Ilhan, Huseyin Enes and Ozkanoglu, Mehmet Akif and Cirpan, Hakan Ali designed a system to offload deep learning-driven vision tasks from aerial devices to remote servers in their paper "Offloading Deep Learning Powered Vision Tasks From UAV to 5G Edge Server With Denoising" (IEEE Transactions on Vehicular Technology) to alleviate the problem of scarce computing resources for aerial devices. Although this method reduces the computational overhead of aerial devices, the task accuracy is still reduced due to compression loss caused by wireless communication bandwidth limitations and noise interference caused by wireless communication channels. As a new wireless communication paradigm, semantic communication has extremely strong robustness and anti-interference, and is expected to solve the problems faced by the above two aerial device target detection architectures.The patent application "A method for joint allocation of communication and computing resources driven by video semantics in the Internet of Vehicles" (application number: CN202011600358.X, application publication number: CN112839382B) filed by Beijing University of Posts and Telecommunications proposes a joint resource allocation scheme based on QoC, which is used to maximize the video content understanding performance when communication and computing resources are limited. It first builds a vehicle network system model for video semantic communication between the vehicle side and the edge server side, and then builds a video semantic-driven joint allocation optimization target for communication resources and computing resources based on the target detection accuracy model under spectrum and computing power constraints. This method is a good attempt to jointly perform computationally intensive tasks with resource-constrained and resource-rich ends based on semantic communication, but it does not consider the semantic distortion caused by the interference of wireless channels on transmission semantics, which reduces the semantic interpretation ability of the receiving end. In addition, the detection algorithm based on deep learning has difficulty in capturing the global semantic association of images and has weak interpretability, which also curbs the accuracy of target detection.

[0004] The invention with patent publication number CN116844123A discloses a vehicle-road cooperative image information transmission method and device based on semantic communication, which implements corresponding wireless coding, transmission and decoding according to different semantic tasks, thereby improving transmission efficiency. However, the invention is based on the joint coding and decoding of semantics and source channels, and implements corresponding wireless coding, transmission and decoding methods according to the computing resources and communication resources available when performing different tasks. It cannot solve the defects of low communication robustness and spectrum efficiency, as well as insufficient accuracy of aerial image target detection tasks.

[0005] The above-mentioned target detection methods based on aerial devices mainly focus on lightweight detection networks, computing task offloading, semantic level communication, etc., but still fail to integrate the lack of computing resources of aerial devices, low spectrum efficiency of communication between aerial devices and ground terminals, wireless communication noise interference, and the lack of global understanding of images by traditional target detection methods to make an overall trade-off design for the aerial device target detection system. Therefore, it is urgent to develop a target detection solution for aerial devices. Summary of the invention

[0006] In view of the shortcomings of existing aerial equipment target detection scenarios, such as the lack of aerial equipment computing resources, low communication spectrum efficiency between aerial equipment and ground terminals, wireless communication noise interference, and the lack of global cognition of images in traditional target detection methods, the present invention proposes a cognitive semantic communication-driven air-ground collaborative target detection method and system, which coordinates aerial equipment and the ground terminal to jointly perform aerial target detection tasks. By introducing the cognitive semantic communication paradigm, some tasks are offloaded to the ground terminal, alleviating the problem of lack of aerial equipment computing resources; by designing a multi-scale codec, the robustness of the system in a high compression rate and low signal-to-noise ratio wireless communication environment is improved; by introducing the prior information of the knowledge graph to assist the communication process and target detection, the system's global cognition of the detection image is improved, thereby improving the target detection accuracy.

[0007] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0008] A method for detecting an air-ground collaborative target driven by cognitive semantic communication, the method comprising the following steps:

[0009] Step A: import the aerial image into a deep convolutional neural network, use multiple cascaded convolutional layers to extract image features from the aerial image, and extract multi-scale semantic features of the image based on a feature pyramid network;

[0010] Step B, using a plurality of parallel residual compression modules to respectively compress the extracted multi-scale semantic features, transforming the obtained compressed multi-scale semantic features into a one-dimensional vector and converting it into a first complex signal vector, performing a standardization operation on the first complex signal vector, and obtaining a message for wireless transmission;

[0011] Step C, transmitting the message to the ground through an additive white Gaussian noise channel or a fading channel to obtain a second complex signal vector interfered by wireless channel noise;

[0012] Step D, converting the second complex signal vector into a real vector, and transforming its dimension to be consistent with the dimension of the multi-scale semantic feature, to obtain a noisy multi-scale semantic feature, and reconstructing the semantic features of the corresponding scales through multiple residual decompression modules in a parallel structure;

[0013] Step E: Generate region proposal semantic features for the reconstructed semantic features of the corresponding scale through the region proposal network, use the metapath2vec method to embed nodes in the pre-built knowledge graph containing target detection category nodes, generate output classification confidence and regression results of the region proposal box through the first fully connected network and the second fully connected network, construct a weight graph according to the output classification result confidence of the region proposal box, the semantic similarity between the region proposal boxes, and the embedding distance between the target detection category nodes of the knowledge graph, input the weight graph into the relationship graph neural network for node aggregation, generate enhanced node representation, input the enhanced node representation into the third fully connected network to generate the final classification result, and constitute the target detection result with the regression result;

[0014] Step F, inputting the aerial image target detection training set, and training the aerial equipment target detection model based on cognitive semantic communication;

[0015] Step G: Input the aerial image target detection test set to verify the task accuracy and performance of the aerial equipment target detection model based on cognitive semantic communication.

[0016] Furthermore, in step A, the aerial image is introduced into a deep convolutional neural network, and image features are extracted from the aerial image using multiple cascaded convolutional layers. The process of extracting multi-scale semantic features of the image based on a feature pyramid network includes the following steps:

[0017] Step A1: import the aerial image into a deep convolutional neural network. The deep convolutional neural network consists of five cascaded convolutional layers. The semantic features P2, P3, P4, and P5 output by the last four convolutional layers are extracted to form a multi-scale semantic feature. r The length and width of the scale are P r+1 twice, r = 2, 3, 4;

[0018] Step A2: input semantic features P2, P3, P4, and P5 into the feature pyramid network. r+1 After 2 times upsampling, the dimension is the same as P r Consistent with P r Through additive fusion; the semantic features P2, P3, P4, and P5 after upsampling and fusion are all passed through a 3×3 convolution layer, and P5 is downsampled by 0.5 times to generate P6, forming a multi-scale semantic feature P I ={P2, P3, P4, P5, P6}.

[0019] Furthermore, in step B, the process of obtaining a message for wireless transmission includes the following steps:

[0020] Step B1, deploying a multi-scale encoder module in an aerial device, where the multi-scale encoder module includes five residual compression modules in a parallel structure;

[0021] Step B2: Input the multi-scale semantic features into the multi-scale encoder module, and the residual compression module compresses the multi-scale semantic features P through scale reduction and channel reduction strategies. I , the scale reduction factor is 0.5, and the channel reduction is determined by the bandwidth compression ratio R:

[0022] R=k / n

[0023] Where k is the channel input size, n is the size of the aerial image; the multi-scale encoder module performs encoding compression to obtain the compressed multi-scale semantic feature X I :

[0024]

[0025] Where P I represents multi-scale semantic features, α I represents the trainable parameters of the multi-scale encoder, represents the compression function;

[0026] Step B3: Compress the compressed multi-scale semantic features X I ={X2, X3, X4, X5, X6} is transformed into a one-dimensional vector and converted into a first complex signal vector

[0027] Step B4: for the first complex signal vector Perform regularization and obtain the message zi for wireless transmission:

[0028]

[0029] In the formula, k i represents the vector length, p represents the average transmission power limit, express The transpose of .

[0030] Furthermore, in step C, message z i The channel output is obtained by transmitting through the additive white Gaussian noise channel

[0031]

[0032] In the formula, the vector w is distributed by The independent and identically distributed samples under σ 2 represents the noise power, represents a cyclically symmetric complex Gaussian distribution;

[0033] Message i The channel output is obtained by transmitting through a fading channel

[0034]

[0035] Where h represents the channel gain.

[0036] Furthermore, in step D, the process of reconstructing the semantic features of the corresponding scales through multiple parallel residual decompression modules includes the following steps:

[0037] Step D1, deploying a multi-scale decoder module at the ground end, the multi-scale decoder module including five residual decompression modules in a parallel structure;

[0038] Step D2, using a multi-scale decoder module to convert the second complex signal vector into a real vector, and perform dimensional transformation to convert its dimension into the dimension of the multi-scale semantic feature compressed by the multi-scale encoder. The noisy multi-scale semantic feature after dimensional transformation is represented as Z T ;

[0039] Step D3: transform the noisy multi-scale semantic feature Z T Input the multi-scale decoder, and reconstruct the semantic features P of the corresponding scale through five row residual decompression modules T :

[0040]

[0041] In the formula, α T represents the trainable parameters of the multi-scale decoder, Represents the reconstruction function.

[0042] Furthermore, in step E, the process of forming the target detection result includes the following steps:

[0043] Step E1, through the region proposal network, the reconstructed multi-scale semantic features P T Generate region proposal semantic features S T :

[0044]

[0045] In the formula, β T represents the trainable parameters of the multi-scale decoder, represents the region proposal generation function;

[0046] Step E2, use the metapath2vec method based on random walk to embed the knowledge graph and obtain the relationship information between the target detection category nodes. The knowledge graph node embedding set is represented as N T ;

[0047] Step E3: Generate the output classification confidence r of the region proposal box through the first fully connected network and the second fully connected networkcls And the regression results r reg :

[0048] r cls =fc cls (S T , β cls )

[0049] r reg =fc reg (S T , β reg )

[0050] In the formula, β cls and β reg The trainable parameters of the classification network and regression network, fc cls (·) and fc reg (·) represents the classification network and regression network; the regression result is the final regression result, and the classification confidence exists as the weight edge between the recommended node and the category node when the weight graph is constructed;

[0051] Step E4, build a weight graph based on the semantic features of region proposals and the embedding of knowledge graph object detection category nodes:

[0052]

[0053] In the formula, Represents the weight graph construction function; where the semantic similarity of the proposed node features and the weights between category nodes are obtained by calculating the embedding distance through cosine similarity:

[0054]

[0055] Where u and v represent node feature vectors, respectively, and ||·||| represents the modulus of the vector;

[0056] Step E5, inputting the weight graph into the relationship graph neural network for node aggregation to generate enhanced node representation;

[0057] Step E6: Input the enhanced node representation into the third fully connected network to generate the final classification result, and form the target detection result with the regression result.

[0058] Furthermore, in step G, during the training phase, stochastic gradient descent is applied to minimize the loss function of the model, and the target detection loss function is defined as:

[0059]

[0060] Among them, k represents the subscript of the detection box in a training batch, p k Indicates the probability that there is a target in the kth detection box; if the current detection box is a positive example, the real label is 1, otherwise it is 0; t k is a four-dimensional vector representing the four parameterized coordinates of the predicted detection box. Represents the true detection box vector; classification loss L cls is logarithmic loss; regression loss R(·) is a smooth L1 function.

[0061] Furthermore, in step H, during the test phase, the average precision (mAP) of all categories is used to evaluate the target detection effect of the model.

[0062] The present invention also discloses an air-ground collaborative target detection system driven by cognitive semantic communication, the air-ground collaborative target detection system comprising:

[0063] A feature extraction module is deployed in the aerial device, and the feature extraction module includes a deep convolutional neural network and a feature pyramid network; the deep convolutional neural network includes a plurality of cascaded convolutional layers, and is used to extract image features from the aerial image; the feature pyramid network is used to extract multi-scale semantic features of the image;

[0064] A multi-scale encoder module is deployed in the air device, and includes a plurality of residual compression modules in a parallel structure, which are used to compress the extracted multi-scale semantic features respectively, transform the obtained compressed multi-scale semantic features into a one-dimensional vector and convert them into a first complex signal vector, perform a standardization operation on the first complex signal vector, and obtain a message for wireless transmission;

[0065] A wireless channel, used to transmit the message to the ground through an additive white Gaussian noise channel or a fading channel, and obtain a second complex signal vector interfered by the wireless channel noise;

[0066] The multi-scale decoder module is deployed on the ground end and is used to convert the second complex signal vector into a real vector and transform its dimension to be consistent with the multi-scale semantic feature dimension to obtain a noisy multi-scale semantic feature, and reconstruct the semantic features of the corresponding scale through multiple residual decompression modules in a parallel structure;

[0067] The target detection module is deployed on the ground side and driven by the knowledge graph. The target detection module includes a region proposal network, a knowledge graph module, a relationship graph neural network module and a result generation module. The region proposal network generates region proposal semantic features for the reconstructed semantic features of the corresponding scale. The knowledge graph module uses the metapath2vec method to embed nodes in the pre-built knowledge graph containing target detection category nodes, and generates the output classification confidence and regression results of the region proposal box through two fully connected networks. The weight graph is constructed according to the output classification result confidence of the region proposal box, the semantic similarity between the region proposal boxes, and the embedding distance between the knowledge graph target detection category nodes. The relationship graph neural network module inputs the weight graph into the relationship graph neural network for node aggregation to generate enhanced node representation. The result generation module inputs the enhanced node representation into the fully connected network to generate the final classification result, and constitutes the target detection result with the regression result.

[0068] A training module, used to input an aerial image target detection training set and train an aerial equipment target detection model based on cognitive semantic communication;

[0069] The test module is used to input the aerial image target detection test set to verify the task accuracy and performance of the aerial equipment target detection model based on cognitive semantic communication.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] First, the cognitive semantic communication-driven air-ground collaborative target detection method and system of the present invention, through the cognitive semantic communication method, offloads part of the aerial image target detection task from the aerial equipment to the ground end, reduces the computing load of the aerial equipment, and alleviates the problem of scarce computing resources of the aerial equipment.

[0072] Second, the cognitive semantic communication-driven air-ground collaborative target detection method and system of the present invention proposes a dual multi-scale semantic codec network structure. On the one hand, based on multi-scale semantic features, it solves the problems of small and difficult to detect targets and inconsistent scales in aerial target detection tasks. On the other hand, based on semantic feature coding and decoding, it improves the spectrum utilization efficiency and robustness in the wireless communication process.

[0073] Third, the cognitive semantic communication-driven air-ground collaborative target detection method and system of the present invention proposes a knowledge graph-driven target detection algorithm, which constructs a weight graph by introducing prior knowledge in the knowledge graph and the target detection area suggestion box, and then generates enhanced features by aggregating the weight graph through a relational graph neural network, so as to enhance the system's global cognition and understanding of aerial images, reduce the impact of communication transmission losses, and improve task accuracy and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1A flow chart of an air-to-ground collaborative target detection method driven by cognitive semantic communication according to an embodiment of the present invention;

[0075] Figure 2 This is a framework diagram of an air-to-ground collaborative target detection system driven by cognitive semantic communication according to an embodiment of the present invention;

[0076] Figure 3 1 is a comparison diagram of target detection accuracy under different channel conditions between the embodiments of the present invention and other existing technologies; wherein (a) represents the detection accuracy performance diagram when the bandwidth compression ratio is 1 / 6, and (b) represents the detection accuracy performance diagram when the bandwidth compression ratio is 1 / 12;

[0077] Figure 4 It is a comparison chart of the computational complexity of the embodiment of the present invention and other existing technologies. DETAILED DESCRIPTION

[0078] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0079] The present invention discloses an air-ground collaborative target detection method driven by cognitive semantic communication, and the air-ground collaborative target detection method comprises the following steps:

[0080] Step A: import the aerial image into a deep convolutional neural network, use multiple cascaded convolutional layers to extract image features from the aerial image, and extract multi-scale semantic features of the image based on a feature pyramid network;

[0081] Step B, using a plurality of parallel residual compression modules to respectively compress the extracted multi-scale semantic features, transforming the obtained compressed multi-scale semantic features into a one-dimensional vector and converting it into a first complex signal vector, performing a standardization operation on the first complex signal vector, and obtaining a message for wireless transmission;

[0082] Step C, transmitting the message to the ground through an additive white Gaussian noise channel or a fading channel to obtain a second complex signal vector interfered by wireless channel noise;

[0083] Step D, converting the second complex signal vector into a real vector, and transforming its dimension to be consistent with the dimension of the multi-scale semantic feature, to obtain a noisy multi-scale semantic feature, and reconstructing the semantic features of the corresponding scales through multiple residual decompression modules in a parallel structure;

[0084] Step E: Generate region proposal semantic features for the reconstructed semantic features of the corresponding scale through the region proposal network, use the metapath2vec method to embed nodes in the pre-built knowledge graph containing target detection category nodes, generate output classification confidence and regression results of the region proposal box through the first fully connected network and the second fully connected network, construct a weight graph according to the output classification result confidence of the region proposal box, the semantic similarity between the region proposal boxes, and the embedding distance between the target detection category nodes of the knowledge graph, input the weight graph into the relationship graph neural network for node aggregation, generate enhanced node representation, input the enhanced node representation into the third fully connected network to generate the final classification result, and constitute the target detection result with the regression result;

[0085] Step F, inputting the aerial image target detection training set, and training the aerial equipment target detection model based on cognitive semantic communication;

[0086] Step G: Input the aerial image target detection test set to verify the task accuracy and performance of the aerial equipment target detection model based on cognitive semantic communication.

[0087] Combined with Figure 1 The specific steps of the method of the present invention are described as follows.

[0088] Step 1: Extract multi-scale semantic features of images.

[0089] The feature extraction module is deployed on the aerial device and includes a deep convolutional neural network and a feature pyramid network. The deep convolutional neural network extracts image features through cascaded convolutional layers. The function of the feature pyramid network is to collect semantic features of different scales output by different convolutional layers.

[0090] The aerial images are fed into the deep convolutional neural network in the feature extraction module. The deep convolutional neural network consists of five cascaded convolutional layer groups. The semantic features P2, P3, P4, and P5 output by the last four convolutional layers are taken to form multi-scale semantic features, where the semantic feature P r The length and width of the scale are P r+1 twice as much;

[0091] Semantic features P2, P3, P4, and P5 are input into the feature pyramid network. r+1 After 2 times upsampling, the dimension and semantic feature P r Consistent with the semantic feature P r Through addition fusion. The up-sampled and fused semantic features P2, P3, P4, and P5 are all passed through a 3×3 convolution layer, and the semantic feature P5 is down-sampled by 0.5 times to generate the semantic feature P6. So far, P2, P3, P4, P5, and P6 constitute the multi-scale semantic feature P I , is sent to the multi-scale encoder module.

[0092] Step 2: Multi-scale semantic feature compression encoding.

[0093] The multi-scale encoder module is deployed on the aerial device, and five residual compression modules are connected in parallel. It is a parallel structure. The input is multi-scale semantic features, and the output is semantic features of five different scales after encoding and compression.

[0094] The multi-scale encoder module receives the multi-scale semantic features P I , and its five scale corresponding inputs are parallelized to form the residual compression module of the multi-scale encoder module. The residual compression module compresses the multi-scale semantic features P through scale reduction and channel reduction strategies. I , the scale reduction factor is 0.5, and the channel reduction is determined by the specific bandwidth compression ratio R:

[0095] R=k / n

[0096] Where k is the channel input size and n is the aerial image size. The multi-scale encoder module performs encoding compression:

[0097]

[0098] Where P I represents multi-scale semantic features, α I represents the trainable parameters of the multi-scale encoder, represents the compression function, X I Represents the compressed multi-scale semantic features.

[0099] Then the compressed multi-scale semantic feature X I = {X2, X3, X4, X5, X6} dimensions are transformed into a one-dimensional vector and converted into a complex signal vector

[0100] Finally, to meet the average transmission power constraint, regularization is performed on the complex signal vector:

[0101]

[0102] where k i represents the vector length, p represents the average transmission power limit, express The transpose of z i is the message obtained for wireless transmission.

[0103] Step 3: Channel semantics transmission.

[0104] Message i The channel output is obtained by transmitting through the additive white Gaussian noise channel

[0105]

[0106] where the vector w is distributed by The independent and identically distributed samples under σ 2 represents the noise power, represents a cyclically symmetric complex Gaussian distribution.

[0107] Message i The channel output is obtained by transmitting through a fading channel

[0108]

[0109] Where h represents the channel gain.

[0110] Step 4: Multi-scale semantic feature decoding and reconstruction.

[0111] The multi-scale decoder module is deployed on the ground side. It connects five residual decompression modules that are dual to the residual compression module in parallel. It is a parallel structure. The input is a complex signal vector interfered by wireless channel noise, and then it decodes and reconstructs semantic features of five different scales.

[0112] The multi-scale decoder module receives the complex signal vector interfered by the wireless channel noise, converts the complex signal vector into a real vector, and performs a dimension transformation to convert its dimension into the dimension of the multi-scale semantic feature compressed by the multi-scale encoder. The noisy multi-scale semantic feature after dimension transformation is represented as Z T .

[0113] The noisy multi-scale semantic feature Z T Input the multi-scale decoder, and reconstruct the semantic features P of the corresponding scale through five row residual decompression modules T :

[0114]

[0115] where α T represents the trainable parameters of the multi-scale decoder, Represents the reconstruction function.

[0116] Step 5: Knowledge graph-driven object detection.

[0117] The knowledge graph-driven target detection module is deployed on the ground, including the region proposal network, knowledge graph, relationship graph neural network module, and result generation module. Its main function is to combine the knowledge graph prior information, improve the global cognition of the image, alleviate the accuracy loss caused by compression and transmission loss, and ensure detection accuracy in harsh environments.

[0118] Through the region proposal network, the reconstructed multi-scale semantic features P TGenerate region proposal semantic features S T :

[0119]

[0120] where β T represents the trainable parameters of the multi-scale decoder, Represents the region proposal generation function.

[0121] The metapath2vec method based on random walk is used to embed the knowledge graph to obtain the relationship information between the target detection category nodes. The knowledge graph node embedding set is represented as N T .

[0122] The output classification confidence and regression results of the region proposal box are generated through two fully connected networks:

[0123] r cls =fc cls (S T , β cls )

[0124] r reg =fc reg (s T , β reg )

[0125] where β cls and β reg The trainable parameters of the classification network and regression network, fc cls (·) and fc reg (·) represents the classification and regression network. The regression result is the final regression result, and the classification confidence exists as the weight edge between the recommended node and the category node when the weight graph is constructed.

[0126] Construct a weight graph based on the semantic features of region proposals and the embedding of knowledge graph object detection category nodes:

[0127]

[0128] in Represents the weight graph construction function.

[0129] The weight graph integrates global image semantics and knowledge graph prior information to support the global cognition of the model and obtain more accurate target detection results. It is recommended that the semantic similarity of node features and the weights between category nodes be obtained by calculating the embedding distance through cosine similarity:

[0130]

[0131] Where u and v represent node feature vectors, respectively, and ||·|| represents the modulus of the vector.

[0132] The weight graph is then input into the graph neural network for node aggregation to generate enhanced node representation.

[0133] Finally, the enhanced node representation is input into a fully connected network to generate the final classification result, and together with the above regression result, it constitutes the target detection result.

[0134] Step 6: Input the aerial image target detection training set to train the air-ground collaborative target detection model driven by cognitive semantic communication.

[0135] During the training phase, stochastic gradient descent is applied to minimize the loss function of the model, and the target detection loss function is defined as:

[0136]

[0137] Among them, k represents the subscript of the detection box in a training batch, p k Indicates the probability that the target exists in the kth detection box. If the current detection box is a positive example, the true label is 1, otherwise it is 0. k is a four-dimensional vector representing the four parameterized coordinates of the predicted detection box. Represents the true detection box vector. Classification loss L cls is logarithmic loss; regression loss R(·) is a smooth L1 function. Through this loss function, all trainable parameters of the system are optimized.

[0138] When all training data are back-propagated, a training cycle ends.

[0139] Step 7: Input the aerial image target detection test set to verify the task accuracy and performance of the air-ground collaborative target detection model driven by cognitive semantic communication.

[0140] During the testing phase, the mean Average Precision (mAP) of all categories is used to evaluate the target detection effect of the model.

[0141] The present invention also discloses an air-ground collaborative target detection system driven by cognitive semantic communication, the air-ground collaborative target detection system comprising:

[0142] A feature extraction module is deployed in the aerial device, and the feature extraction module includes a deep convolutional neural network and a feature pyramid network; the deep convolutional neural network includes a plurality of cascaded convolutional layers, and is used to extract image features from the aerial image; the feature pyramid network is used to extract multi-scale semantic features of the image;

[0143] A multi-scale encoder module is deployed in the air device, and includes a plurality of residual compression modules in a parallel structure, which are used to compress the extracted multi-scale semantic features respectively, transform the obtained compressed multi-scale semantic features into a one-dimensional vector and convert them into a first complex signal vector, perform a standardization operation on the first complex signal vector, and obtain a message for wireless transmission;

[0144] A wireless channel, used to transmit the message to the ground through an additive white Gaussian noise channel or a fading channel, and obtain a second complex signal vector interfered by the wireless channel noise;

[0145] The multi-scale decoder module is deployed on the ground end and is used to convert the second complex signal vector into a real vector and transform its dimension to be consistent with the multi-scale semantic feature dimension to obtain a noisy multi-scale semantic feature, and reconstruct the semantic features of the corresponding scale through multiple residual decompression modules in a parallel structure;

[0146] The target detection module is deployed on the ground side and driven by the knowledge graph. The target detection module includes a region proposal network, a knowledge graph module, a relationship graph neural network module and a result generation module. The region proposal network generates region proposal semantic features for the reconstructed semantic features of the corresponding scale. The knowledge graph module uses the metapath2vec method to embed nodes in the pre-built knowledge graph containing target detection category nodes, and generates the output classification confidence and regression results of the region proposal box through two fully connected networks. The weight graph is constructed according to the output classification result confidence of the region proposal box, the semantic similarity between the region proposal boxes, and the embedding distance between the knowledge graph target detection category nodes. The relationship graph neural network module inputs the weight graph into the relationship graph neural network for node aggregation to generate enhanced node representation. The result generation module inputs the enhanced node representation into the fully connected network to generate the final classification result, and constitutes the target detection result with the regression result.

[0147] A training module, used to input an aerial image target detection training set and train an aerial equipment target detection model based on cognitive semantic communication;

[0148] The test module is used to input the aerial image target detection test set to verify the task accuracy and performance of the aerial equipment target detection model based on cognitive semantic communication.

[0149] The effect of the present invention is further described below in conjunction with simulation experiments.

[0150] 1. Simulation conditions and parameter settings:

[0151] The simulation experiment of the present invention is carried out on the simulation platform of Python3.8 and Pytorch1.10.0. The CPU model of the simulation platform is AMD EPYC 7302, and the graphics card model used for training is NVIDIA GeForce RTX 3090. The learning rate of network training is 0.005, and the Adam optimization algorithm is used as the optimizer for network training. The deep convolutional neural network in the feature extraction module adopts ResNet-34. In the target detection module driven by the knowledge graph, the knowledge graph stored in the knowledge graph library is pre-built through the open API of ConceptNet, and the relational graph neural network R-GAT used adopts a three-layer model and a multi-head attention mechanism.

[0152] In the embodiment of the present invention, the recognized benchmark aerial target detection dataset DOTA (large-scale Dataset for Object detection in Aerial images) is used to evaluate the results. The DOTA dataset contains 15 categories, each image is about 4000×4000 pixels in size, and contains targets to be detected of various scales.

[0153] The present invention selects two baseline methods for performance comparison. The deep learning baseline model uses an adaptive joint source-channel codec method based on deep learning as a communication solution and FasterRCNN as a target detection solution. The traditional method uses JPEG as the source codec and LDPC as the channel codec. The JPEG compression rate is fixed at 75%.

[0154] 2. Simulation content:

[0155] Attached Figure 3 The present invention is used together with the deep learning baseline model and the traditional method baseline model under the condition of additive white Gaussian noise channel. Figure 3 (a) and (b) in the figure correspond to the detection accuracy under different signal-to-noise ratios when the bandwidth compression ratio is 1 / 6 and 1 / 12 respectively. It can be seen that under two bandwidth compression ratios and all tested signal-to-noise ratios, the present invention shows better performance than the two baselines. When the present invention is not integrated with the knowledge graph, the performance under low signal-to-noise ratio is significantly better than the deep learning baseline and the traditional method baseline, which can prove the effectiveness of the multi-scale codec and the overall architecture. After integrating the knowledge graph, the performance of the present invention is further improved. At the same time, it can be seen from the trend of the curve change that it is less affected by the signal-to-noise ratio, which proves its robustness. Comparing the performance of each method at the two bandwidth compression ratios, when the bandwidth compression ratio changes from 1 / 6 to 1 / 12, the performance gap between the methods becomes larger, and the present invention is less affected by the bandwidth compression ratio than the comparison method, which proves the excellence of the present invention at high compression rate.

[0156] Attached Figure 4 This is a comparison chart of the computational complexity of aerial equipment of the present invention and other existing technologies. It can be seen that in terms of the three indicators of aerial equipment parameter quantity, number of additions and number of multiplications, the present invention is lower than the two baseline methods, has lower computational complexity, and alleviates the problem of scarce computing resources for aerial equipment.

[0157] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiment, complete software embodiment, or the embodiment in combination with software and hardware. Moreover, the present application can adopt the form of the computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can adopt various computer languages ​​to realize, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0158] The present application is a flowchart and a computer program product with reference to the method, device (system), and computer program product according to the embodiments of the present application.

[0159] It should be understood that each process and each flow chart and / or block diagram can be implemented by computer program instructions.

[0160] These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the flow chart and / or the block chart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0163] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0164] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for air-ground collaborative target detection driven by cognitive semantic communication, characterized in that: The air-ground collaborative target detection method comprises the following steps: Step A: import the aerial image into a deep convolutional neural network, use multiple cascaded convolutional layers to extract image features from the aerial image, and extract multi-scale semantic features of the image based on a feature pyramid network; Step B, using a plurality of parallel residual compression modules to respectively compress the extracted multi-scale semantic features, transforming the obtained compressed multi-scale semantic features into a one-dimensional vector and converting it into a channel input complex vector, performing a standardization operation on the channel input complex vector, and obtaining a message for wireless transmission; Step C, transmitting the message to the ground through an additive white Gaussian noise channel or a fading channel to obtain a channel output complex vector interfered by wireless channel noise; Step D, converting the channel output complex vector into a real vector, and transforming its dimension to be consistent with the multi-scale semantic feature dimension, to obtain a noisy multi-scale semantic feature, and reconstructing the semantic features of the corresponding scale through multiple parallel residual decompression modules; Step E: Generate region proposal semantic features for the reconstructed semantic features of the corresponding scale through the region proposal network, use the metapath2vec method to embed nodes in the pre-built knowledge graph containing target detection category nodes, generate output classification confidence and regression results of the region proposal box through the first fully connected network and the second fully connected network, construct a weight graph according to the output classification result confidence of the region proposal box, the semantic similarity between the region proposal boxes, and the embedding distance between the target detection category nodes of the knowledge graph, input the weight graph into the relationship graph neural network for node aggregation, generate enhanced node representation, input the enhanced node representation into the third fully connected network to generate the final classification result, and constitute the target detection result with the regression result; Step F, inputting the aerial image target detection training set, and training the aerial equipment target detection model based on cognitive semantic communication; Step G, inputting the aerial image target detection test set to verify the task accuracy and performance of the aerial equipment target detection model based on cognitive semantic communication; Step A, importing the aerial image into a deep convolutional neural network, using multiple cascaded convolutional layers to extract image features from the aerial image, and the process of extracting multi-scale semantic features of the image based on the feature pyramid network includes the following steps: Step A1: import the aerial image into a deep convolutional neural network. The deep convolutional neural network consists of five cascaded convolutional layers. The semantic features P2, P3, P4, and P5 output by the last four convolutional layers are extracted to form a multi-scale semantic feature. r The length and width of the scale are P r+1 twice, r = 2, 3, 4; Step A2: input semantic features P2, P3, P4, and P5 into the feature pyramid network. r+1 After 2 times upsampling, the dimension is the same as P r Consistent with P r Through additive fusion; the semantic features P2, P3, P4, and P5 after upsampling and fusion are all passed through a 3×3 convolution layer, and P5 is downsampled by 0.5 times to generate P6, forming a multi-scale semantic feature P I ={P2,P3,P4,P5,P6}.

2. The method for air-ground collaborative target detection driven by cognitive semantic communication according to claim 1 is characterized in that: In step B, the process of obtaining a message for wireless transmission includes the following steps: Step B1, deploying a multi-scale encoder module in an aerial device, where the multi-scale encoder module includes five residual compression modules in a parallel structure; Step B2: Input the multi-scale semantic features into the multi-scale encoder module, and the residual compression module compresses the multi-scale semantic features P through scale reduction and channel reduction strategies. I , the scale reduction factor is 0.5, and the channel reduction is determined by the bandwidth compression ratio R: R=k / n Where k is the channel input size, n is the size of the aerial image; the multi-scale encoder module performs encoding compression to obtain the compressed multi-scale semantic feature X I : Where P I represents multi-scale semantic features, α I represents the trainable parameters of the multi-scale encoder, represents the compression function; Step B3: Compress the compressed multi-scale semantic features X I = {X2, X3, X4, X5, X6} dimension is transformed into a one-dimensional vector and converted into a channel input complex vector Step B4: input a complex vector to the channel Regularization is performed and the message z obtained for wireless transmission is i : In the formula, k i represents the vector length, p represents the average transmission power limit, express The transpose of .

3. The method for air-ground collaborative target detection driven by cognitive semantic communication according to claim 1, characterized in that: In step C, if the message z i Through the additive white Gaussian noise channel transmission, the channel output is obtained for: In the formula, the vector w is distributed by The independent and identically distributed samples under σ 2 represents the noise power, represents a cyclically symmetric complex Gaussian distribution; If the message z i Through the fading channel, the transmission gets the channel output for: Where h represents the channel gain.

4. The method for air-ground collaborative target detection driven by cognitive semantic communication according to claim 1, characterized in that: In step D, the process of reconstructing the semantic features of the corresponding scales through multiple parallel residual decompression modules includes the following steps: Step D1, deploying a multi-scale decoder module at the ground end, the multi-scale decoder module including five residual decompression modules in a parallel structure; Step D2, using a multi-scale decoder module to convert the channel output complex vector into a real vector, and perform dimensionality transformation to convert its dimension into the dimension of the multi-scale semantic feature compressed by the multi-scale encoder. The noisy multi-scale semantic feature after dimensionality transformation is represented as Z T ; Step D3: transform the noisy multi-scale semantic feature Z T Input the multi-scale decoder, and reconstruct the semantic features P of the corresponding scale through five row residual decompression modules T : In the formula, α T represents the trainable parameters of the multi-scale decoder, Represents the reconstruction function.

5. The method for air-ground collaborative target detection driven by cognitive semantic communication according to claim 1, characterized in that: In step E, the process of forming the target detection result includes the following steps: Step E1, through the region proposal network, the reconstructed multi-scale semantic features P T Generate region proposal semantic features S T : In the formula, β T represents the trainable parameters of the multi-scale decoder, represents the region proposal generation function; Step E2, use the metapath2vec method based on random walk to embed the knowledge graph and obtain the relationship information between the target detection category nodes. The knowledge graph node embedding set is represented as N T ; Step E3: Generate the output classification confidence r of the region proposal box through the first fully connected network and the second fully connected network cls And the regression results r reg : r cls =fc cls (With T ,β cls ) r reg =fc reg (With T ,β reg ) In the formula, β cls and β reg Represent the trainable parameters of the classification network and regression network, respectively, fc cls (·) and fc reg (·) represents the classification network and regression network; the regression result is the final regression result, and the classification confidence exists as the weight edge between the recommended node and the category node when the weight graph is constructed; Step E4, build a weight graph based on the semantic features of region proposals and the embedding of knowledge graph object detection category nodes: In the formula, Represents the weight graph construction function; where the semantic similarity of the proposed node features and the weights between category nodes are obtained by calculating the embedding distance through cosine similarity: Where u and v represent node feature vectors, respectively, and ||·|| represents the modulus of the vector; Step E5, inputting the weight graph into the relationship graph neural network for node aggregation to generate enhanced node representation; Step E6: Input the enhanced node representation into the third fully connected network to generate the final classification result, and form the target detection result with the regression result.

6. The method for air-ground collaborative target detection driven by cognitive semantic communication according to claim 1, characterized in that: In step G, during the training phase, stochastic gradient descent is applied to minimize the loss function of the model, and the target detection loss function is defined as: Among them, k represents the subscript of the detection box in a training batch, p k Indicates the probability that there is a target in the kth detection box; if the current detection box is a positive example, the real label is 1, otherwise it is 0; t k is a four-dimensional vector, representing the four parameterized coordinates of the predicted detection box, Represents the true detection box vector; classification loss L cls is logarithmic loss; regression loss R(·) is a smooth L1 function.

7. The method for air-ground collaborative target detection driven by cognitive semantic communication according to claim 1, characterized in that: In step H, during the test phase, the average precision (mAP) of all categories is used to evaluate the target detection effect of the model.

8. A cognitive semantic communication driven air-ground collaborative target detection system based on the method of claim 1, characterized in that: The air-to-ground collaborative target detection system comprises: A feature extraction module is deployed in the aerial device, and the feature extraction module includes a deep convolutional neural network and a feature pyramid network; the deep convolutional neural network includes a plurality of cascaded convolutional layers, and is used to extract image features from the aerial image; the feature pyramid network is used to extract multi-scale semantic features of the image; A multi-scale encoder module is deployed in the air device, and includes a plurality of residual compression modules in a parallel structure, which are used to compress the extracted multi-scale semantic features respectively, transform the obtained compressed multi-scale semantic features into a one-dimensional vector and convert them into a channel input complex vector, perform a standardization operation on the channel input complex vector, and obtain a message for wireless transmission; A wireless channel is used to transmit the message to the ground through an additive white Gaussian noise channel or a fading channel to obtain a channel output complex vector interfered by the wireless channel noise; The multi-scale decoder module is deployed on the ground end and is used to convert the channel output complex vector into a real vector and transform its dimension to be consistent with the multi-scale semantic feature dimension to obtain the noisy multi-scale semantic feature. The semantic features of the corresponding scale are reconstructed through multiple parallel residual decompression modules. The target detection module is deployed on the ground side and driven by the knowledge graph. The target detection module includes a region proposal network, a knowledge graph module, a relationship graph neural network module and a result generation module. The region proposal network generates region proposal semantic features for the reconstructed semantic features of the corresponding scale. The knowledge graph module uses the metapath2vec method to embed nodes in the pre-built knowledge graph containing target detection category nodes, and generates the output classification confidence and regression results of the region proposal box through two fully connected networks. The weight graph is constructed according to the output classification result confidence of the region proposal box, the semantic similarity between the region proposal boxes, and the embedding distance between the knowledge graph target detection category nodes. The relationship graph neural network module inputs the weight graph into the relationship graph neural network for node aggregation to generate enhanced node representation. The result generation module inputs the enhanced node representation into the fully connected network to generate the final classification result, and constitutes the target detection result with the regression result. A training module, used to input an aerial image target detection training set and train an aerial equipment target detection model based on cognitive semantic communication; The test module is used to input the aerial image target detection test set to verify the task accuracy and performance of the aerial equipment target detection model based on cognitive semantic communication.

Images