UAV cross-modal semantic communication method and system for target detection tasks

Through the cross-modal semantic communication method, the multimodal information of RGB and infrared images is used for feature fusion and channel encoding and decoding, which solves the problems of insufficient utilization of multi-scale information and low single-modal detection accuracy in the prior art, and realizes the problem of inadequate use of multi-scale information and low single-modal detection accuracy in the prior art. Efficient object detection and accurate data transmission in complex channel environments.

CN118470581BActive Publication Date: 2024-11-15NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410941201.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-15
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The existing methods cannot effectively utilize multi-scale information, resulting in low classification accuracy at low signal-to-noise ratios, and single-modal object detection is susceptible to the environment, making it difficult to achieve accurate data transmission in complex channel environments.

Method used

Using cross-modal semantic communication method, the semantic encoder and decoder are constructed through the annotation and pre-training model of the training data set, the multimodal information of RGB and infrared images is used to perform feature fusion, the channel encoding and decoding is designed, and the training model is migrated to Adapt to the channel environment to achieve accurate target detection and low-latency transmission.

Benefits of technology

It improves the accuracy and accuracy of target detection, solves the problems of missed detection and missed detection in complex channel environments, and ensures efficient data transmission and visibility of processing results under low signal-to-noise ratio conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470581B_ABST
    Figure CN118470581B_ABST
Patent Text Reader

Abstract

The present invention discloses a cross-modal semantic communication method and system for unmanned aerial vehicles for target detection tasks, which relates to the field of communication technology, including collecting training data, constructing a data set, and marking the category and location information of the data set; constructing and pre-training a target detection model under channel-free conditions, and obtaining network parameters of the pre-trained model; constructing a channel environment, and designing semantic communication-related channel encoding and decoding, and constructing an overall model after adding the channel environment; migrating the network parameters of the pre-trained model, and performing migration training on the obtained parameters, and training the overall model after adding the channel environment; judging whether the network training is completed, inputting the test set data into the network, and outputting the detection results. The present invention utilizes semantic information of different scales, comprehensively utilizes multi-modal information for complementarity, and improves detection accuracy. It is also an effective method for Internet devices such as unmanned aerial vehicles to offload operations to edge servers when facing complex channel environments for target detection tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method and system for cross-modal semantic communication of unmanned aerial vehicles for target detection tasks. Background Art

[0002] In recent years, with the continuous development of target detection technology, drones have shown broad application prospects in many fields such as traffic monitoring due to their advantages such as fast speed, high degree of freedom and wide field of view. However, considering the limited computing power of drones, they usually transmit the captured data directly to the server for subsequent processing. When facing real-time target detection tasks, the transmitted data is extremely time-sensitive, so it is necessary to achieve low-latency and high-accuracy data transmission in a complex channel environment. With the continuous increase in business volume, the massive amount of transmitted data has gradually approached the transmission limit of the current communication link. Traditional communication methods are no longer competent for this task. Therefore, there is an urgent need for a new communication method with low latency and high reliability to meet this demand.

[0003] With the continuous development of artificial intelligence, communication systems are realizing a paradigm shift from bits to semantics. On this basis, semantic communication is expected to greatly improve communication efficiency by integrating user needs for information and task semantics into the communication process, and has become one of the key technologies of 6G. Xuewen Luo, Hsiao-Hwa Chen, et al. described the application prospects of semantic communication in the Internet of Things in their paper "Semantic Communications: Overview, Open Issues, and Future Research Directions" (IEEE Wireless Commun., vol. 29, no. 1, pp. 210-219, Feb. 2022). Semantic communication is a promising technology for achieving accurate and real-time data transmission in the Internet of Things network because it occupies less wireless resources and is relatively insensitive to channel noise. Therefore, drone communication methods based on semantic communication have become a research hotspot in academia.

[0004] Guangyi Zhang, Qiyu Hu, and others proposed a multi-task semantic communication architecture in their paper "A Unified Multi-Task Semantic Communication System with Domain Adaptation" (IEEE Global Commun. Conf., pp.3971-3976, Dec. 2022), which can significantly reduce transmission overhead and model parameter volume, but it does not take into account the importance of multi-scale features for target detection tasks, which will lead to inaccurate target detection classification and positioning, and the impact on the accuracy of small target detection is more serious. Zhenguo Zhang, Qianqian Yang, and others proposed a multi-level semantic perception communication system for wireless image transmission in their paper "Wireless Transmission of Images with the Assistance of Multi-level Semantic Information" (2022 International Symposiumon Wireless Communication Systems (ISWCS), pp. 1-6, 2022). The system transmits images in a multi-scale manner, retains global information and local spatial details through semantic information of different scales, and can better complete target detection tasks on this basis, but it does not consider cross-modal semantic communication scenarios. Single-mode target detection methods are easily affected by the environment. For example, single-mode target detection based on RGB images is prone to missed detection due to dark environments or occlusion by objects such as trees, while target detection based on infrared images is prone to false detection due to the lack of details such as texture and color. At the same time, compared with the traditional method of directly transmitting images to the receiver for target detection, how to ensure that the processing results of semantic communication methods are visible and effective is also a difficult problem to be solved. Therefore, it is urgent to design new UAV semantic communication methods for target detection tasks.

[0005] The present invention effectively solves the complex channel problem encountered by IoT devices in the process of offloading computing tasks to edge servers by integrating semantic communication with target detection. Multi-task training ensures that the receiver can obtain the detection category, location results and original images at the same time. Nonlinear reconstruction of semantic information effectively utilizes information of different scales to improve detection accuracy. Cross-modal information is used to effectively solve missed detection and false detection in dark or occluded scenes. Summary of the invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the problem to be solved by the present invention is: how to solve the problems that the existing methods cannot effectively utilize multi-scale information and have low classification accuracy under low signal-to-noise ratio.

[0008] To solve the above technical problems, the present invention provides the following technical solutions: a cross-modal semantic communication method for unmanned aerial vehicles for target detection tasks, comprising: collecting training data, constructing a data set, and marking the category and location information of the data set; constructing and pre-training a target detection model under channel-free conditions, and obtaining network parameters of the pre-trained model; constructing a channel environment, and designing semantic communication-related channel encoding and decoding, and constructing an overall model after adding the channel environment; migrating the network parameters of the pre-trained model, performing migration training on the obtained parameters, and training the overall model after adding the channel environment; judging whether the network training is completed, inputting the test set data into the network, and outputting the detection results.

[0009] As a preferred solution of the UAV cross-modal semantic communication method for target detection tasks described in the present invention, the training data is obtained using a UAV equipped with an RGB camera and an infrared camera, the shooting environment includes day and night, RGB images and infrared images containing a variety of vehicles are obtained, and the collected training data are divided into a training set and a test set; the labeling of the category and location information of the data set includes using an irregular polygon labeling method.

[0010] As a preferred solution of the cross-modal semantic communication method for unmanned aerial vehicles for target detection tasks described in the present invention, the target detection model under the channelless condition includes a semantic encoder part and a semantic decoder part, using the collected training data as the model input, using the obtained annotations as the model training labels, first using the channelless model for pre-training, and then adding the channel environment for re-training; the semantic encoder part includes an RGB encoder, an infrared encoder and a fusion module, using the obtained data set as input, processed by the corresponding semantic encoder, and the obtained mid-term features are fused after processing, which is expressed as:

[0011] ;

[0012] in, is the input infrared image, is the input RGB image, For infrared encoder, For the RGB encoder, are the neural network parameters of the infrared encoder, are the neural network parameters of the RGB encoder, For the fusion module, is the neural network parameter of the fusion module; both the RGB encoder and the infrared encoder are composed of multiple layers with a kernel size of , a convolutional layer with a stride of 2, which extracts the mid-term features of the image for feature-level fusion; the fusion module uses C3CBAM to fuse features.

[0013] As a preferred solution of the cross-modal semantic communication method for drones for target detection tasks described in the present invention, the semantic decoder part includes the use of multi-task training, and the two task heads output the image restoration results and the target detection results respectively. For the design of the task head, the image restoration task head adopts a multi-layer deconvolution network to restore the pixel values ​​of the original image, and the target detection task head is composed of some convolutional and fully connected layers to predict the position and category of the target, which is expressed as:

[0014] ;

[0015] in, is the input of the semantic decoder, is the shared part of the semantic decoder, is the task header for image recovery. is the task head for target detection, and are the neural network parameters of each part respectively.

[0016] As a preferred solution of the cross-modal semantic communication method for unmanned aerial vehicles for target detection tasks described in the present invention, the channel environment constructed includes an AWGN channel and a Rayleigh channel, and the channel is represented as:

[0017] ;

[0018] in, is the input of the channel, is the output of the channel, are the coefficients of the linear channel, is independent and distributed Gaussian noise; the source-channel joint coding is designed to complete the channel coding and decoding task. The channel codec is added on the basis of the semantic encoder part, and the coding operation at the drone is replaced by:

[0019] ;

[0020] in, is the channel encoder in the source-channel joint coding, is the parameter of the deep neural network; the decoding operation of the semantic decoder at the edge server is replaced by:

[0021] ;

[0022] in, is the channel decoder in the source-channel joint coding, are the parameters of the deep neural network.

[0023] As a preferred solution of the cross-modal semantic communication method for unmanned aerial vehicles for target detection tasks described in the present invention, the overall model after the training is added to the channel environment includes fixing the network parameters of the pre-trained model, randomly initializing the trainable parameters of the network, initializing the number of network training epochs, the maximum number of iterations and the learning rate, and the Adam optimization algorithm is used as the network training optimizer. The training data is input into the network training in batches, and the training error of each batch is back-propagated to optimize the network parameters. The two task heads respectively output the results of image restoration and target detection, and respectively calculate the loss function, which includes the loss function of the image restoration task and the loss function of the target detection task; for the loss function of the image restoration task, the target image is set to , is the input infrared image and RGB images The result of pixel-level fusion is the mean square error calculated by the output of the image restoration task head and the target image as the loss function, which is expressed as:

[0024] ;

[0025] in, is the number of batch samples, is the loss function for the image restoration task, Output the result of the image recovery task header. is the target image, is the total number of pixels in the image; for the loss function of the target detection task, the target detection task head outputs the category and position of the target. The loss function of the target detection task is the weighted sum of different parts. The loss function is expressed as:

[0026] ;

[0027] in, is the classification loss, is the confidence loss, For the positioning loss, is the loss function of the target detection task; the loss functions of the two tasks are weighted and summed to As the overall training result of the deep neural network, it is expressed as:

[0028] ;

[0029] in, is the task weight, is the overall loss function of the deep neural network.

[0030] As a preferred solution of the cross-modal semantic communication method of unmanned aerial vehicles for target detection tasks described in the present invention, the determination of whether the network training is completed includes determining whether the current training period has reached the maximum number of training periods. If the maximum number of training periods has been reached, the test set data is input into the network and the detection results are output. If the maximum number of training periods has not been reached, the network training continues.

[0031] Another object of the present invention is to provide a system for a UAV cross-modal semantic communication method for target detection tasks, which can solve the problem of UAV cross-modal semantic communication for target detection tasks by constructing a UAV cross-modal semantic communication system.

[0032] To solve the above technical problems, the present invention provides the following technical solutions: a cross-modal semantic communication system for unmanned aerial vehicles for target detection tasks, comprising a data acquisition module, a channel-free target detection model construction module, an overall target detection model construction module, a model training module and a testing module; the data acquisition module is used to collect training data, construct a data set, and annotate the category and location information of the data set; the channel-free target detection model construction module is used to construct and pre-train a target detection model under channel-free conditions, and obtain network parameters of the pre-trained model; the overall target detection model construction module is used to construct a channel environment, design semantic communication-related channel encoding and decoding, and construct an overall model after adding the channel environment; the model training module is used to migrate the network parameters of the pre-trained model, perform migration training on the obtained parameters, and train the overall model after adding the channel environment; the testing module is used to determine whether the network training is completed, input the test set data into the network, and output the detection results.

[0033] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned cross-modal semantic communication method for unmanned aerial vehicles for target detection tasks are implemented.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned cross-modal semantic communication method for unmanned aerial vehicles for target detection tasks.

[0035] The beneficial effects of the present invention are as follows: the drone cross-modal semantic communication method for target detection tasks provided by the present invention can effectively utilize semantic information of different scales, comprehensively utilize multi-modal information for complementarity, and improve detection accuracy. It is also an effective method for drones and other Internet devices to offload calculations to edge servers when facing complex channel environments for target detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0037] Figure 1 A flowchart of a UAV cross-modal semantic communication method for target detection tasks provided by the first embodiment of the present invention.

[0038] Figure 2 A model framework diagram of a UAV cross-modal semantic communication method for target detection tasks provided in the first embodiment of the present invention.

[0039] Figure 3 A structural diagram of a UAV cross-modal semantic communication system for target detection tasks provided in the second embodiment of the present invention.

[0040] Figure 4 A comparison chart of target detection accuracy under different signal-to-noise ratio conditions of AWGN channels between the cross-modal semantic communication method for target detection tasks of a drone provided in the third embodiment of the present invention and other technologies.

[0041] Figure 5 A comparison chart of the image recovery accuracy of the UAV cross-modal semantic communication method for target detection tasks provided in the third embodiment of the present invention and other technologies under different signal-to-noise ratio conditions of the AWGN channel.

[0042] Figure 6 A rendering of a cross-modal semantic communication method for a drone for a target detection task provided in the third embodiment of the present invention.

[0043] Figure 7 A comparison diagram of confusion matrix effects of a YOLO-based semantic communication method for a drone cross-modal semantic communication method for a target detection task provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0046] Example 1

[0047] Reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a UAV cross-modal semantic communication method for target detection tasks, including: collecting training data, constructing a data set, and annotating the category and location information of the data set; constructing and pre-training a target detection model under channel-free conditions, and obtaining network parameters of the pre-trained model; constructing a channel environment, and designing semantic communication-related channel codecs, and constructing an overall model after adding the channel environment; migrating the network parameters of the pre-trained model, performing migration training on the obtained parameters, and training the overall model after adding the channel environment; judging whether the network training is completed, inputting the test set data into the network, and outputting the detection results.

[0048] S1. Collect training data, build a data set, and label the category and location information of the data set.

[0049] The training data was acquired using a drone equipped with an RGB camera and an infrared camera. The shooting environment included day and night, and RGB images and infrared images of five types of vehicles, including cars, trucks, buses, vans, and trucks, were obtained. The collected training data was divided into a training set and a test set in a ratio of 7:3.

[0050] In this invention, a large-scale drone aerial vehicle dataset DroneVehicle collected and annotated by Yiming Sun, Bing Cao and others from Tianjin University is used as the model input. This dataset was published in the paper "Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning".

[0051] Commonly used location annotation methods for target detection include rectangular annotation bndbox, rotated rectangular annotation robndbox, and irregular polygon annotation poly. Among them, rectangular annotation only annotates the upper left and lower right points of the rectangle (xmax, ymax, xmin and ymin); rotated rectangular annotation requires annotating the center point (cx, cy), width and height (w, h) and rotation angle (angle) of the rectangle; irregular polygon annotation requires annotating each point of the polygon, taking an irregular quadrilateral as an example (x1, y1, x2, y2, x3, y3, x4, y4). Considering that the vehicle will rotate from a bird's-eye view angle, and the definition of the rotation angle range for the rotated rectangular annotation is easily confused, and the calculation of the loss function is relatively complicated, the present invention uses the Polygon irregular polygon annotation method, and the annotation boxes are all quadrilaterals. The data set includes five categories: cars, trucks, buses, vans, and trucks, represented by numbers 0 to 4 respectively.

[0052] The annotation results are saved in a txt file.

[0053] S2. Build and pre-train a target detection model under channel-free conditions to obtain the network parameters of the pre-trained model.

[0054] like Figure 2 As shown, in order to avoid the influence of channel noise on the target detection training results, the present invention first uses a channel-free model for pre-training, and then adds a channel environment for re-training to facilitate faster convergence.

[0055] The collected training data is used as the model input, and the obtained annotations are used as the model training labels.

[0056] The semantic encoder part consists of RGB encoder, infrared encoder and fusion module. The obtained data set is taken as input and first processed by the corresponding semantic encoder. After processing, the obtained mid-term features are fused. This part can be expressed by the following formula:

[0057] ;

[0058] in, is the input infrared image, is the input RGB image, For infrared encoder, For the RGB encoder, and are the neural network parameters of the two encoders respectively, is the fusion module, and its neural network parameters are .

[0059] Both encoders consist of multiple layers with kernel size , a convolutional layer with a stride of 2, in order to extract the mid-term features of the image for feature-level fusion.

[0060] The fusion of cross-modal information is an important task, and CNN or attention layer can be selected to be responsible for this part. The former shares weights in feature maps, while the latter can capture features of different regions and obtain more features by dynamically calculating similarity functions between pixels. The present invention uses C3CBAM to fuse features, which can improve the ability of feature fusion calculation expression.

[0061] For the semantic decoder part, in order to obtain the image and detection results at the same time, multi-task training is used, and the two task heads output the image restoration results and the target detection results respectively. By using multi-task learning, not only can the total number of system parameters be reduced by uniformly sharing the underlying parameters, but also the overfitting of the model can be alleviated to a certain extent and the generalization ability of the model can be improved. For the design of the task head, the image restoration task head uses a multi-layer deconvolution network to restore the pixel values ​​of the original image. The target detection task head consists of some convolutional and fully connected layers to predict the location and category of the target. This part can be expressed by the following formula:

[0062] ;

[0063] in, is the input of the semantic decoder, is the shared part of the semantic decoder, is the task header for image recovery. is the task head for target detection, and are the neural network parameters of each part respectively.

[0064] S3. Build the channel environment, design the channel codec related to semantic communication, and build the overall model after adding the channel environment.

[0065] In the training process, AWGN channel and Rayleigh channel are used to show the complexity of the channel environment. In order to achieve the optimal end-to-end performance of the communication system, the source channel joint coding is designed to complete the channel coding and decoding tasks. The channel codec is added on the basis of S2. The source symbol sequence will be directly mapped to the channel symbol by the channel encoder. After the channel symbol passes through the physical channel, it will be reconstructed by the channel decoding module at the destination. The channel is represented as:

[0066] ;

[0067] The input and output of the channel are respectively and express, represents the coefficients of the linear channel, represents independent and distributed Gaussian noise. Both represent the output of the encoder part.

[0068] In S2, the semantic coding and semantic decoding parts are pre-trained to avoid the influence of the channel; in S3, the channel coding and decoding and the physical channel are added to train the joint source channel coding (JSCC). The meaning is the same, except that the decoder input in S2 is x, while the decoder input in S3 is the perturbed .

[0069] In addition, in order to achieve the optimal end-to-end performance of the communication system, the source-channel joint coding is designed to complete the channel coding and decoding tasks. A channel codec is added on the basis of S2, and the coding operation at the drone can be replaced by:

[0070] ;

[0071] in, is the channel encoder in the source-channel joint coding, are the parameters of the deep neural network.

[0072] Similarly, the decoding operation at the edge server can be replaced by:

[0073] ;

[0074] in, is the channel decoder in the source-channel joint coding, are the parameters of the deep neural network.

[0075] S4. Migrate the network parameters of the pre-trained model, perform migration training on the obtained parameters, and train the overall model after adding the channel environment.

[0076] Pre-trained model parameter migration, retraining conversion model.

[0077] S4.1. Fix the network parameters of the pre-trained model.

[0078] S4.2. Randomly initialize the trainable parameters of the network, initialize the number of network training epochs to 1, the maximum number of iterations to 100, the learning rate to 0.0005, and use the Adam optimization algorithm as the network training optimizer.

[0079] S4.3. Input the training data into the network training in batches. The batch size can be adjusted. The training error of each batch is back-propagated to optimize the network parameters. The two task heads output the results of image restoration and target detection respectively, and calculate the loss function respectively.

[0080] The parameters obtained during the pre-training process are fixed, while the random trainable parameters belong to the channel encoding and decoding part (parameters not included in the pre-training process). The parameters of the semantic encoder neural network are frozen, that is, the parameters of this part are no longer learned through gradient descent, because the purpose of this part is to better extract feature information (semantic information) and has little effect on the channel environment. As the channel changes, the subsequent part (semantic decoder) needs to be adjusted appropriately according to the severity of the channel interference, so it is initialized to the corresponding parameters of the pre-training stage, but it is still updated. Using the pre-training method can reduce the risk of model overfitting and improve the effect of downstream tasks.

[0081] For the loss function of the image restoration task, the target image is set to , is the input infrared image and RGB images The result of pixel-level fusion. Output result of image restoration task header With target picture Calculate the mean square error as the loss function, and the process can be expressed by the following formula:

[0082] ;

[0083] in, is the mean square error distribution, represents the number of samples in a batch, is the total number of pixels in the image, Loss function for the image restoration task.

[0084] For the loss function of the target detection task, the task head outputs the category and location of the target, so the loss function of this task is the weighted sum of different parts, and the calculation formula of the loss function is:

[0085] ;

[0086] in, is the classification loss, which is used to determine whether the model can classify the object in the target box into the correct category. is the confidence loss, which is used to determine whether the target is detected, and the accuracy loss of the detection box when the target is detected. is the positioning loss, which is used to calculate the accuracy of target size and position judgment. is the loss function for the target detection task. The coefficients in the formula are used to determine the weights of different losses, and the overall loss is the weighted sum of the three losses. Generally, CIOU loss is used to calculate the positioning loss, and BCE loss is used to calculate the confidence loss and classification loss. The coefficients used here are set according to the common coefficients of the YOLO target detection framework.

[0087] Finally, the loss functions of the two tasks are weighted and summed to As the overall training result of the deep neural network, it can be expressed as:

[0088] ;

[0089] in, is the task weight, and in actual training , is the overall loss function of the deep neural network.

[0090] S5. Determine whether the network training is completed, input the test set data into the network, and output the detection results.

[0091] Determine whether the current training period has reached the maximum number of training periods. If so, input the test set data into the network and output the test results. If not, continue to train the network in S4.

[0092] Example 2

[0093] Reference Figure 3 , which is the second embodiment of the present invention, is different from the previous embodiment in that it provides a UAV cross-modal semantic communication system for target detection tasks, including: a data acquisition module, a channel-free target detection model construction module, an overall target detection model construction module, a model training module and a testing module.

[0094] The data collection module is used to collect training data, build data sets, and label the categories and location information of the data sets.

[0095] The channelless target detection model construction module is used to construct and pre-train the target detection model under channelless conditions and obtain the network parameters of the pre-trained model.

[0096] The overall target detection model construction module is used to construct the channel environment, design the semantic communication related channel codec, and construct the overall model after adding the channel environment.

[0097] The model training module is used to migrate the network parameters of the pre-trained model, perform migration training on the obtained parameters, and train the overall model after adding the channel environment.

[0098] The test module is used to determine whether the network training is completed, input the test set data into the network, and output the detection results.

[0099] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0100] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0101] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0102] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0103] Example 3

[0104] Reference Figure 4~Figure 7 , which is the third embodiment of the present invention, and which is different from the first two embodiments in that it is used to verify and illustrate the technical effects adopted in the present invention, so as to verify the real effects of this method.

[0105] Simulation conditions and parameter settings:

[0106] The simulation experiment of the present invention is carried out on the simulation platform of Python3.8, Pytorch2.0.1+cu118. The computer CPU model is 12th Gen Intel Core i5-12600kf, equipped with an independent graphics card model NVIDIA GeForceRTX4060ti. In terms of data sets, the present invention adopts the public data set DroneVehicle, which consists of 56,878 images collected by drones, half of which are RGB images and the rest are infrared images. The data set contains five categories, namely cars, trucks, buses, vans and freightcars. The comparison scheme of the present invention uses three schemes, namely the traditional scheme of JPEG+LDPC+BPSK, the single-modal semantic communication scheme, and the simple fusion scheme of YOLO architecture and semantic communication, to compare the improvement of semantic communication on system performance, the improvement of cross-modal information on system performance and the rationality of the framework designed by the present invention.

[0107] PSNR is selected as the control metric for image restoration, and mAP@0.5 is selected as the control metric for target detection tasks. The calculation method of mAP can be expressed by the formula:

[0108] ;

[0109] in, is the accuracy, is the recall rate, is the sample size.

[0110] To verify the performance of the proposed method, different methods are evaluated at SNR levels ranging from -3 to 15 dB with a step size of 3 dB.

[0111] Simulation content:

[0112] like Figure 4The figure shows the mAP comparison of the target detection under different signal-to-noise ratio conditions of the AWGN channel between the present invention and the control scheme. In the comparison of different schemes, it can be observed that due to the complementarity of cross-modal information and the effective processing of multi-level features, the mAP of the scheme proposed by the present invention reaches 0.64, which has good performance and robustness. For several control schemes, if the commonly used YOLO model for target detection is simply integrated with the semantic communication architecture, the mAP of the detection result will be reduced by about 10%, which shows that the present invention has significant advantages in processing multi-level features; traditional methods are extremely sensitive to noise. When the SNR in the channel is about 3, the performance of the traditional target detection system will drop significantly, and when the SNR is further reduced, the mAP will gradually approach zero; the single-modal target detection method is not as good as the method proposed by the present invention because it cannot achieve complementarity of information between modalities, and is more affected by the channel.

[0113] like Figure 5 Shown is a PSNR comparison diagram of image restoration under different signal-to-noise ratio conditions of the AWGN channel between the present invention and the control scheme, where the YOLO model only contains the output of target detection and does not involve image restoration, so it is not included in this group of comparisons. From the comparison between multiple schemes, it can be observed that if the image is sent directly using the traditional method, the PSNR of the image at the receiver will be at a low level, and when the SNR is at a low level, a cliff effect will occur, and the received image will be full of noise and completely unrecognizable. In contrast to the traditional method of transmitting different modal results separately, the method of the present invention can ensure that the image will not be severely distorted due to the reduction of the signal-to-noise ratio, and the fusion result of the infrared image and the RGB image can be directly used as the output to ensure that the information contained in the image is most easily accepted.

[0114] like Figure 6 and Figure 7 The figure shows the confusion matrix comparison between this method and the semantic communication method based on YOLO, where: Figure 6 This is a diagram showing the effect of using the method of the present invention. Figure 7 This is the effect diagram of the confusion matrix of the semantic communication method based on YOLO. The horizontal axis represents the real label, the vertical axis represents the network output result, and each grid number represents the corresponding classification accuracy. It can be seen that the method of the present invention has a significant improvement in the accuracy of target detection tasks compared with the semantic communication method based on YOLO, and the confusion between different categories has been significantly improved.

[0115] Based on the above simulation results and analysis, the drone cross-modal semantic communication method for target detection tasks proposed in the present invention can effectively utilize semantic information of different scales, comprehensively utilize multi-modal information for complementarity, and improve detection accuracy. It is also an effective method for Internet devices such as drones to offload calculations to edge servers when facing complex channel environments for target detection tasks.

[0116] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A cross-modal semantic communication method for UAVs for target detection tasks, characterized by: include, Collect training data, build a data set, and label the category and location information of the data set; Construct and pre-train a target detection model under channel-free conditions and obtain the network parameters of the pre-trained model; Construct the channel environment, design the channel codec related to semantic communication, and construct the overall model after adding the channel environment; Migrate the network parameters of the pre-trained model, perform migration training on the obtained parameters, and train the overall model after adding the channel environment; Determine whether the network training is finished, input the test set data into the network, and output the test results; The target detection model under the channelless condition includes a semantic encoder part and a semantic decoder part, uses the collected training data as the model input, uses the obtained annotations as the model training labels, first uses the channelless model for pre-training, and then adds the channel environment for re-training; The semantic encoder part includes an RGB encoder, an infrared encoder and a fusion module. The obtained data set is taken as input and processed by the corresponding semantic encoder. After processing, the obtained mid-term features are fused and expressed as: x=T f (T si (s1;α),T sr (s2;β);γ) Among them, s1 is the input infrared image, s2 is the input RGB image, T si (·; α) is the infrared encoder, T sr (·; β) is the RGB encoder, α is the neural network parameter of the infrared encoder, β is the neural network parameter of the RGB encoder, T f (·; γ) is the fusion module, γ is the neural network parameter of the fusion module; Both the RGB encoder and the infrared encoder consist of multiple convolutional layers with a kernel size of 3×3 and a stride of 2, which extract the mid-term features of the image for feature-level fusion; The fusion module uses C3CBAM to fuse features; The semantic decoder part includes the use of multi-task training, and the two task heads output the image restoration results and the target detection results respectively. For the design of the task head, the image restoration task head uses a multi-layer deconvolution network to restore the pixel values ​​of the original image. The target detection task head is composed of convolution and fully connected layers to predict the location and category of the target, which is expressed as: p=R sd (R s (x;h);l) Among them, x is the input of the semantic decoder, R s (·; η) is the shared part of the semantic decoder, R sc (·; θ) is the task head for image restoration, R sd (·; λ) is the task head for target detection, η, θ and λ are the neural network parameters of each part respectively; The channel environment includes AWGN channel and Rayleigh channel, and the channel is represented as: y=Hx+w Where x is the input of the channel, y is the output of the channel, H is the coefficient of the linear channel, and w is independent and distributed Gaussian noise; The source-channel joint coding is designed to complete the channel coding and decoding tasks. The channel codec is added on the basis of the semantic encoder part. The coding operation at the drone is replaced by: x=T c (T f (T si (s1;α),T sr (s2;β);γ);δ) Among them, T c (·; δ) is the channel encoder in the source-channel joint coding, δ is the parameter of the deep neural network; The semantic decoder at the edge server partially decodes the operation and replaces it with, p=R sd (R s (R c (y;e);h);l) Among them, R c (·; ε) is the channel decoder in the source-channel joint coding, and ε is the parameter of the deep neural network.

2. The cross-modal semantic communication method for unmanned aerial vehicle for target detection tasks as claimed in claim 1, characterized in that: The training data is acquired using a drone equipped with an RGB camera and an infrared camera. The shooting environment includes day and night, and RGB images and infrared images containing a variety of vehicles are obtained. The collected training data is divided into a training set and a test set. The labeling of the category and location information of the data set includes using an irregular polygon labeling method.

3. The cross-modal semantic communication method for unmanned aerial vehicle for target detection tasks as claimed in claim 1, characterized in that: The overall model after the training is added to the channel environment includes fixing the network parameters of the pre-trained model, randomly initializing the trainable parameters of the network, initializing the number of network training epochs, the maximum number of iterations and the learning rate, using the Adam optimization algorithm as a network training optimizer, inputting the training data into the network training in batches, back-propagating the training error of each batch to optimize the network parameters, and the two task heads output the results of image restoration and target detection respectively, and respectively calculate the loss functions, and the loss functions include the loss function of the image restoration task and the loss function of the target detection task; For the loss function of the image restoration task, the target image is set to s, which is the result of pixel-level fusion of the input infrared image s1 and the RGB image s2. The mean square error between the output result of the image restoration task head and the target image is calculated as the loss function, which is expressed as: Where N is the number of samples in a batch, is the loss function for the image restoration task, Output result of the image restoration task head, s is the target image, n is the total number of pixels of the image; For the loss function of the target detection task, the target detection task head outputs the category and position of the target. The loss function of the target detection task is the weighted sum of different parts. The loss function is expressed as, in, is the classification loss, is the confidence loss, is the positioning loss, is the loss function for the target detection task; The loss functions of the two tasks are weighted and calculated as As the overall training result of the deep neural network, it is expressed as, Among them, ρ is the task weight, is the overall loss function of the deep neural network.

4. The cross-modal semantic communication method for unmanned aerial vehicle for target detection tasks as claimed in claim 3, characterized in that: The determination of whether the network training is finished includes determining whether the current training period has reached the maximum number of training periods. If the maximum number of training periods has been reached, the test set data is input into the network and the detection result is output. If the maximum number of training periods has not been reached, the network training continues.

5. A system using the cross-modal semantic communication method for unmanned aerial vehicles for target detection tasks as described in any one of claims 1 to 4, characterized in that: Including data acquisition module, channelless target detection model building module, overall target detection model building module, model training module and testing module; The data acquisition module is used to collect training data, construct a data set, and annotate the category and location information of the data set; The channelless target detection model construction module is used to construct and pre-train the target detection model under the channelless condition to obtain the network parameters of the pre-trained model; The overall target detection model construction module is used to construct a channel environment, design semantic communication related channel codecs, and construct an overall model after adding the channel environment; The model training module is used to migrate the network parameters of the pre-trained model, perform migration training on the obtained parameters, and train the overall model after adding the channel environment; The test module is used to determine whether the network training is completed, input the test set data into the network, and output the test results.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the drone cross-modal semantic communication method for target detection tasks described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cross-modal semantic communication method for a drone for a target detection task as described in any one of claims 1 to 4 are implemented.