Intelligent communication data link system and method based on online semantic information extraction

Abstract

The invention relates to an intelligent communication data link system and method based on online semantic information extraction. The system comprises communication unmanned aerial vehicle equipment used for executing reconnaissance and monitoring tasks and a ground data control center used for analyzing data returned by the communication unmanned aerial vehicle equipment and controlling intelligent adjustment of a data link. The communication unmanned aerial vehicle equipment comprises a high-definition image sensing module, a target tracking module and a semantic coding module, the target tracking module is used for identifying and tracking a motion track of an interested target and generating frame coordinates containing the interested target, and the semantic coding module is used for performing multi-modal coding on original video data; the ground data control center comprises a user interest configuration module, a communication link quality analysis module, a signal synchronization module and a video decoding module. Compared with the prior art, the method has the advantages that the problem of poor video image data transmission quality in a severe channel environment is effectively solved, and the like.

Description

Technical Field

[0001] The present invention relates to an intelligent communication data link system and method, and in particular to an intelligent communication data link system and method based on online semantic information extraction. Background Art

[0002] UAV equipment has the advantages of wide field of vision, free operation, good equipment carrying capacity, etc. It can enter remote or dangerous areas and complete the target tasks set by users while ensuring the safety of personnel. UAV equipment equipped with communication modules has stronger information transmission capabilities, which helps to expand the coverage of wireless communications and can be applied to civil fields such as urban air traffic, emergency rescue, intelligent logistics, and military fields such as national defense reconnaissance and remote monitoring.

[0003] The technological progress and expansion of application scenarios of drone equipment have also brought many challenges to supporting communication solutions. Drone equipment usually transmits communication information back in the form of video data streams. For long-distance drone equipment, the transmission path loss is serious, resulting in a decrease in the signal quality received by the ground receiving end. In cities or complex environments, multipath propagation of signals will affect the fidelity of signals and communication quality. Extreme weather conditions, such as thunderstorms, hurricanes and other meteorological disasters, seriously affect the transmission quality of data information.

[0004] After searching, Chinese patent publication number CN113949826A discloses a method and system for cooperative reconnaissance of drone clusters under limited communication bandwidth conditions, which specifically discloses the following: S1, the first drone collects target images and analyzes them onboard to obtain target information from a single-machine perspective and share them within the cluster; while continuing to collect images, the obtained target information is sent to the comprehensive judgment module; S2, other drones each determine whether to conduct cooperative reconnaissance. If cooperative reconnaissance is conducted, images of the target are collected and analyzed onboard according to the received target information to obtain target information from their respective perspectives, and the obtained target information is sent to the comprehensive judgment module at the same time; S3, the comprehensive judgment module analyzes the received target information to obtain the final reconnaissance result. This patent solves the problems of limited communication bandwidth, large target detection delay, and low accuracy in drone cluster reconnaissance, but does not solve the transmission problem of video data streams of drone equipment in complex environments and extreme weather conditions. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an intelligent communication data link system and method based on online semantic information extraction.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to one aspect of the present invention, there is provided an intelligent communication data link system based on online semantic information extraction, the system comprising a communication drone device for performing reconnaissance and surveillance tasks and a ground data control center for analyzing data returned by the communication drone device and controlling intelligent adjustment of the data link;

[0008] The communication drone device includes a high-definition image sensing module, a target tracking module and a semantic coding module, wherein the target tracking module is used to identify and track the motion trajectory of the target of interest and generate the coordinates of the frame containing the target of interest, and the semantic coding module is used to perform multimodal coding on the original video data;

[0009] The ground data control center includes a user interest configuration module, a communication link quality analysis module, a signal synchronization module and a video decoding module, and the video decoding module is used to restore original video information from a semantically coded signal.

[0010] As a preferred technical solution, the target tracking module includes a single-stage target recognition algorithm and a discriminant-based target tracking algorithm. The target tracking module determines the target's movement trajectory by training a binary classifier of the target and the background, and outputs the target frame coordinates, thereby separating the key target image from the complex background environment and obtaining the video data within the target frame.

[0011] As a preferred technical solution, the high-definition image sensor module is used to capture video image information within the target area.

[0012] As a preferred technical solution, the user interest configuration module is used to access the user's interest task configuration.

[0013] As a preferred technical solution, the communication link quality analysis module is used to analyze and evaluate the quantitative communication channel quality.

[0014] As a preferred technical solution, the signal synchronization module is used to align the time axes of multiple data streams.

[0015] According to another aspect of the present invention, a method for using the intelligent communication data link system based on online semantic information extraction is provided, the method comprising the following steps:

[0016] Step S1, the ground data control center analyzes the communication target related information according to the user's interest task configuration and sends it to the UAV device;

[0017] Step S2, the communication drone device accepts the communication task, uses the high-definition image sensor module to capture video images, obtains raw video data, and feeds back to the ground data control center after preprocessing; and detects potential targets of interest in the environment, tracks the targets of interest, and obtains target-related video blocks;

[0018] Step S3, the communication link quality analysis module of the ground data control center analyzes and evaluates the communication channel quality based on the returned video data, estimates and quantifies the current communication channel capacity, and feeds back the link quality information to the UAV equipment platform;

[0019] Step S4: According to the communication quality, the semantic coding module carried by the UAV device adopts an intelligent adjustment coding scheme to extract and integrate the multimodal semantic information of the video, image and text in the original data, complete the semantic coding, and return the semantic information of different modes to the ground data control center through multi-channel push streaming;

[0020] Step S5: The ground data control center decodes the video data as reference information for subsequent communication link quality analysis, synchronously processes multiple semantic information streams, and realizes the fusion display of multimodal semantic data.

[0021] As a preferred technical solution, in step S2, the original video data is directly captured by a high-definition image sensor, and is transmitted back to the ground data control center after being subjected to an image defogging and enhancement algorithm.

[0022] As a preferred technical solution, in step S3, the communication link quality analysis module comprehensively analyzes the communication environment according to the packet loss rate, bit error rate and distortion degree of the received video data, and quantifies the channel quality, which is fed back to the UAV device as a basis for adjusting parameters in the semantic coding module.

[0023] As a preferred technical solution, in step S4, the semantic coding module selects a suitable bit rate to encode the original video data and the video data in the target frame according to the quality of the communication channel, and transmits the original video with a lower bit rate and the video data in the target frame with a higher bit rate back to the ground data control center.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] 1) The present invention effectively solves the transmission problem of video data streams of UAV equipment in complex communication environments. By extracting and transmitting the information in the target frame most relevant to the communication task, the bit rate of the two-way video data stream is dynamically adjusted to achieve high-quality data transmission to meet the mission requirements of UAV equipment such as reconnaissance and surveillance;

[0026] 2) The UAV equipment platform corresponding to the present invention is equipped with a target recognition and tracking algorithm to achieve continuous monitoring and data feedback of the target of interest, further meeting the requirements of real-time, safety and comprehensiveness in the reconnaissance mission;

[0027] 3) The present invention designs a corresponding target tracking module containing an image defogging and denoising algorithm for complex UAV communication environments, thereby achieving accurate recognition and real-time tracking of the target of interest;

[0028] 4) The present invention adopts a multimodal semantic coding scheme to encode low-bitrate original video data and high-bitrate target frame video or image data separately and push them in multiple channels to achieve better video communication effects;

[0029] 5) The present invention adds channel state feedback signaling, and the UAV equipment selects the semantic information most relevant to the communication task for transmission to adapt to limited communication resources and improve communication quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the system and method of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] like Figure 1 As shown, the data link based on intelligent semantic extraction of the present invention can analyze the current channel capacity according to real-time channel state feedback information, compress redundancy based on communication tasks, and transmit the semantic information with the highest relevance to the current task, thereby effectively improving the quality of information transmission.

[0033] The data link of the present invention uses the communication drone equipment and the ground data control center as the communication parties, and the intelligent communication data link includes four signaling interaction processes: the ground data center sends target information and issues target tracking tasks; the drone equipment tracks the target of interest and feeds back the monitoring video source data; based on the original video, the link quality analysis module feeds back the communication link quality information to the drone equipment; the drone equipment adjusts the parameters of the semantic coding module and transmits multimodal semantic information. Then, the ground data control center performs signal synchronization and video decoding on the multimodal data to complete the communication task. The present invention effectively solves the problem of poor video image data transmission quality in harsh channel environments. Based on the feedback information of the communication link quality, the mode and transmission bit rate of the transmitted data are intelligently adjusted to retain the information with the highest relevance to the task of interest, achieve efficient matching of the semantic coding scheme with the channel environment, and ensure the utilization rate of communication resources.

[0034] Among them, the communication drone equipment is used to perform reconnaissance and surveillance tasks, and is equipped with a high-definition image sensing module, a target tracking module, and a semantic coding module.

[0035] High-definition image sensor module, used to capture video image information within the target area.

[0036] The target tracking module is used to identify and track the motion trajectory of the target of interest and generate the coordinates of the bounding box containing the target of interest.

[0037] The semantic encoding module is used to perform multimodal encoding on the original video data to generate a data stream containing multimodal semantic information.

[0038] The ground data control center is used to analyze the data returned by the UAV equipment and control the intelligent adjustment of the data link, which includes a user interest configuration module, a communication link quality analysis module, a signal synchronization module and a video decoding module.

[0039] The user interest configuration module is used to access the user's interest task configuration.

[0040] The communication link quality analysis module is used to analyze and evaluate the quantified communication channel quality.

[0041] Signal synchronization module, used to align the time axis of multiple data streams.

[0042] The semantic decoding module is used to recover the original video information from the semantically encoded signal.

[0043] like Figure 1 As shown, the present invention discloses an intelligent communication data link based on online semantic information extraction, comprising the following steps:

[0044] Step 1: The ground data control center analyzes the communication target related information and sends it to the UAV device according to the user's interest task configuration.

[0045] Step 2: The communication unmanned equipment accepts the communication task, uses high-definition image sensors to capture video images, obtains raw video data, and feeds it back to the ground data control center after preprocessing; detects potential targets of interest in the environment, tracks targets of interest, and obtains target-related video blocks.

[0046] Step 3: The communication link quality analysis module of the ground data control center analyzes and evaluates the communication channel quality based on the returned video data, estimates and quantifies the current communication channel capacity, and feeds back the link quality information to the UAV equipment platform;

[0047] Step 4: Based on the communication quality, the semantic coding module onboard the drone device intelligently adjusts the coding scheme, extracts and integrates multimodal semantic information such as video, image, and text in the original video data, completes semantic coding, and returns the semantic information of different modalities to the ground data control center through multi-channel streaming.

[0048] Step 5: The ground data control center decodes the video data as reference information for subsequent communication link quality analysis, synchronously processes multiple semantic information streams, and realizes the fusion display of multi-modal semantic data.

[0049] In step 1, the ground data control center dispatches unmanned equipment to the target area for reconnaissance and patrol. The user selects specific surveillance targets through the interest configuration interface, including but not limited to vehicles, ships, and animals. The data control center analyzes the specific requirements of the reconnaissance and surveillance mission and the specific characteristics of the surveillance target, and encodes and sends it to the unmanned reconnaissance equipment platform to issue the surveillance mission.

[0050] In step 2, the visual sensing module carried by the drone equipment cluster perceives the environmental background based on the high-definition binocular image sensor and captures the original video data v containing the target of interest. After the necessary defogging and denoising preprocessing, the original video data obtains clearer video data F(v) and is transmitted back to the ground data control center as a reference for communication channel quality assessment. Based on the clear video data, the drone equipment cluster uses a single-stage target recognition algorithm such as the YOLOV5 algorithm to achieve target recognition in complex environments, and uses a discriminant-based target tracking algorithm, such as the anti-occlusion target tracking algorithm that integrates Kalman filtering and particle filtering, to train a binary classifier of the target and background to determine the target's movement trajectory.

[0051] In step 3, the ground data control center receives the original video data C(v) sent back by the UAV equipment platform. The communication link quality analysis module comprehensively evaluates and quantifies the current channel quality Q according to the packet loss rate, bit error rate and image distortion of the received video data C(v). channel =F assess (C(v)), the channel status is used as a reference condition for the subsequent transmission code rate adaptive adjustment and fed back to the UAV device end.

[0052] In step 4, the semantic coding module in the unmanned reconnaissance device calculates the channel quality Q channel , dynamically adjust the transmission bit rate of the original video to R whole , using H.264 or H.265 video encoding scheme to generate panoramic image video data stream 1. Then the semantic coding module obtains the target frame coordinates from the target tracking module, clears the background image outside the target frame, retains only the target image content most relevant to the communication task, and encodes the video data in the target frame using H.264 or H.265 technology to generate the target image video data stream 2, with a transmission bit rate of R aim Since the amount of information outside the target frame is 0, the H.264 or H.265 video coding scheme compresses redundant information, and the transmission bit rate R aim >R wholeThe two-way video data stream signals with different transmission bit rates are transmitted back to the ground data control center respectively.

[0053] In step 5, the ground data control center receives two video data streams with different communication delays, and uses a signal synchronization solution to align the time axes of the two video signals to achieve subsequent video decoding and video reorganization. According to the coordinates of the target frame, the two video data signals are spliced ​​and fused. Since the video data stream in the target frame has a higher bit rate, the target image most relevant to the communication task in the fused image is clearer under the same channel environment, achieving high-quality data transmission. The panoramic image video obtained by semantic decoding will be used as the input signal of the communication link quality analysis module, and the operations of steps 3 to 5 will be repeated to achieve continuous intelligent semantic communication.

[0054] This intelligent communication data link based on online semantic information extraction uses the UAV equipment platform to obtain the original video data of the target area, and combines the target recognition and tracking algorithm to achieve real-time and continuous inspection of the target area. The ground data platform estimates the channel communication quality based on the transmitted video data and feeds back to the UAV equipment platform. The UAV equipment platform is equipped with an intelligent semantic coding module, which intelligently extracts and integrates multimodal semantic communication data based on user task requirements and actual channel status and transmits it back to the data control center, effectively improving data transmission efficiency and adapting to harsh communication environments.

[0055] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An intelligent communication data link system based on online semantic information extraction, characterized in that: The system includes a communication UAV device for performing reconnaissance and surveillance missions and a ground data control center for analyzing data returned by the communication UAV device and controlling the intelligent adjustment of the data link; The communication drone device includes a high-definition image sensing module, a target tracking module and a semantic coding module, wherein the target tracking module is used to identify and track the motion trajectory of the target of interest and generate the coordinates of the frame containing the target of interest, and the semantic coding module is used to perform multimodal coding on the original video data; The ground data control center includes a user interest configuration module, a communication link quality analysis module, a signal synchronization module and a video decoding module, and the video decoding module is used to restore original video information from a semantically coded signal.

2. The intelligent communication data link system based on online semantic information extraction according to claim 1 is characterized in that: The target tracking module includes a single-stage target recognition algorithm and a discriminant-based target tracking algorithm. The target tracking module determines the target's movement trajectory by training a binary classifier of the target and the background, and outputs the target frame coordinates, thereby separating the key target image from the complex background environment and obtaining the video data within the target frame.

3. The intelligent communication data link system based on online semantic information extraction according to claim 1 is characterized in that: The high-definition image sensor module is used to capture video image information within the target area.

4. The intelligent communication data link system based on online semantic information extraction according to claim 1 is characterized in that: The user interest configuration module is used to access the user's interest task configuration.

5. The intelligent communication data link system based on online semantic information extraction according to claim 1 is characterized in that: The communication link quality analysis module is used to analyze and evaluate the quantified communication channel quality.

6. The intelligent communication data link system based on online semantic information extraction according to claim 1 is characterized in that: The signal synchronization module is used to align the time axes of multiple data streams.

7. A method using the intelligent communication data link system based on online semantic information extraction as claimed in claim 1, characterized in that: The method comprises the following steps: Step S1, the ground data control center analyzes the communication target related information according to the user's interest task configuration and sends it to the UAV device; Step S2, the communication drone device accepts the communication task, uses the high-definition image sensor module to capture video images, obtains raw video data, and feeds back to the ground data control center after preprocessing; and detects potential targets of interest in the environment, tracks the targets of interest, and obtains target-related video blocks; Step S3, the communication link quality analysis module of the ground data control center analyzes and evaluates the communication channel quality based on the returned video data, estimates and quantifies the current communication channel capacity, and feeds back the link quality information to the UAV equipment platform; Step S4: According to the communication quality, the semantic coding module carried by the UAV device adopts an intelligent adjustment coding scheme to extract and integrate the multimodal semantic information of the video, image and text in the original data, complete the semantic coding, and return the semantic information of different modes to the ground data control center through multi-channel push streaming; Step S5: The ground data control center decodes the video data as reference information for subsequent communication link quality analysis, synchronously processes multiple semantic information streams, and realizes the fusion display of multimodal semantic data.

8. The method according to claim 7, characterized in that In step S2, the original video data is directly captured by a high-definition image sensor, and is transmitted back to the ground data control center after being subjected to an image defogging and enhancement algorithm.

9. The method according to claim 7, characterized in that: In step S3, the communication link quality analysis module comprehensively analyzes the communication environment according to the packet loss rate, bit error rate and distortion degree of the received video data, and quantifies the channel quality, which is fed back to the UAV device as a basis for adjusting the parameters in the semantic coding module.

10. The method according to claim 7, characterized in that In step S4, the semantic coding module selects a suitable bit rate to encode the original video data and the video data in the target frame according to the quality of the communication channel, and transmits the original video with a lower bit rate and the video data in the target frame with a higher bit rate back to the ground data control center.

Images