Flight equipment graph number synchronization method and device under time window verification mechanism

Abstract

The invention relates to the technical field of graph number synchronization, and discloses a flight equipment graph number synchronization method and device under a time window verification mechanism, and the method comprises the steps: carrying out the demultiplexing operation of the video stream data, obtained in real time, of flight equipment, and obtaining a plurality of video frames corresponding to the video stream data and a frame timestamp corresponding to each video frame; performing target processing operation on all the video frames to obtain a metadata set corresponding to all the video frames, and determining a timestamp corresponding to each piece of metadata in the metadata set; and according to the frame timestamps corresponding to all the video frames and the timestamps corresponding to all the metadata, performing time window verification operation on all the video frames and all the metadata to obtain synchronous metadata corresponding to all the video frames. Therefore, by implementing the method and the device, time window verification can be performed on the timestamps between the video frames and the metadata, so that the synchronous metadata can be matched for the video frames, and improvement of image and data synchronization precision of flight equipment and reduction of image and data transmission delay are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of figure-data synchronization, and particularly to a figure-data synchronization method and device for a flight device under a time window verification mechanism. Background Art

[0002] Today, with the rapid development of flight device technology, real-time monitoring on the Web side has become a key requirement in the industry, which requires convenient access and control across systems and hardware platforms. Currently, users expect to obtain richer visual data (such as AI detection, AR overlay, etc.) through the Web front-end monitoring interface to ensure flight safety and optimize the operation process. Therefore, the flight device service system needs to have two core features: high-precision figure-data synchronization and low-latency front-end video playback. High-precision figure-data synchronization can ensure that the metadata and video images are highly consistent when displayed on the front end, guaranteeing the actual application effect; while low-latency front-end video playback can ensure real-time control of the flight device by users, improving the user experience.

[0003] However, in existing video data and telemetry data synchronization technologies, standard streaming media protocols (such as RTMP, HLS) and independent transmission protocols (such as TCP / UDP) are usually used to transmit video data and telemetry data in parallel. Since the two types of data are transmitted using different protocols, it may cause the data to be unable to be accurately synchronized at the receiving end, and even result in figure-data transmission delay, affecting the user experience and the practicality of the system. Therefore, it is particularly important to provide a method that can improve the figure-data synchronization accuracy of flight devices. Summary of the Invention

[0004] The present invention provides a figure-data synchronization method and device for a flight device under a time window verification mechanism, which can perform time window verification on the timestamps between each video frame and metadata, so as to match synchronous metadata for each video frame, which is beneficial to improving the figure-data synchronization accuracy of the flight device and reducing figure-data transmission delay.

[0005] To solve the above technical problems, a first aspect of the present invention discloses a figure-data synchronization method for a flight device under a time window verification mechanism, and the method includes:

[0006] Demultiplex the real-time obtained video stream data of the flight device through a preset demultiplexing module to obtain the demultiplexed data corresponding to the video stream data; the demultiplexed data includes multiple video frames and the frame timestamp corresponding to each video frame;

[0007] Perform target processing operations on all the video frames to obtain a metadata set corresponding to all the video frames, and determine the timestamp corresponding to each metadata in the metadata set; the target processing operations include target detection operations and / or image enhancement calculation operations;

[0008] Perform a time window verification operation on all the video frames and all the metadata according to the frame timestamps corresponding to all the video frames and the timestamps corresponding to all the metadata, to obtain the synchronization metadata corresponding to all the video frames.

[0009] As an optional implementation manner, in the first aspect of the present invention, the performing a time window verification operation on all the video frames and all the metadata according to the frame timestamps corresponding to all the video frames and the timestamps corresponding to all the metadata, to obtain the synchronization metadata corresponding to all the video frames, includes:

[0010] For the current video frame among all the video frames, determine a target time window size parameter that matches the current video frame, and determine a target time window that matches the current video frame according to the target time window size parameter;

[0011] Through the target time window, determine all the metadata to be matched corresponding to the current video frame from all the metadata;

[0012] According to the frame timestamp corresponding to the current video frame and the timestamps corresponding to all the metadata to be matched, calculate the absolute timestamp difference between the current video frame and each piece of the metadata to be matched;

[0013] According to the absolute timestamp difference between the current video frame and all the metadata to be matched, determine the target metadata with the smallest absolute timestamp difference from all the metadata to be matched;

[0014] Judge whether the absolute timestamp difference between the current video frame and the target metadata is less than or equal to a preset absolute difference threshold;

[0015] When the judgment result is yes, determine the target metadata as the synchronization metadata corresponding to the current video frame;

[0016] When the judgment result is no, determine the synchronization metadata corresponding to the previous video frame of the current video frame as the synchronization metadata corresponding to the current video frame.

[0017] As an optional implementation manner, in the first aspect of the present invention, the determining a target time window size parameter that matches the current video frame includes:

[0018] Calculate the network jitter parameters during transmission corresponding to all the video frames;

[0019] Calculate the historical video frame - metadata matching success rates corresponding to all the historical time windows among all the video frames;

[0020] Determine a target time window size parameter that matches the current video frame according to the network jitter parameter during transmission, the historical video frame-metadata matching success rate, a preset minimum window size parameter, a maximum window size parameter, an initial window size parameter, a first weight parameter corresponding to the network jitter parameter during transmission, a first reference value corresponding to the network jitter parameter during transmission, a second weight parameter corresponding to the historical video frame-metadata matching success rate, and a second reference value corresponding to the historical video frame-metadata matching success rate;

[0021] Wherein, the first weight parameter is determined by the following method:

[0022] Determine the transmission network conditions corresponding to all the video frames, and determine the first weight parameter corresponding to the network jitter parameter during transmission according to the transmission network conditions corresponding to all the video frames; the transmission network conditions include the transmission network speed condition and / or the transmission network stability condition.

[0023] As an optional implementation manner, in the first aspect of the present invention, calculating the network jitter parameter during transmission corresponding to all the video frames includes:

[0024] Determine an acquisition time parameter corresponding to when the demultiplexing module acquires the video stream data, and determine an arrival time parameter corresponding to each video frame according to the acquisition time parameter;

[0025] According to the arrival time parameters corresponding to all the video frames, calculate the arrival time interval parameters corresponding to each group of adjacent video frames among all the video frames;

[0026] According to the sum of the arrival time interval parameters corresponding to all the adjacent video frames and the total number of groups corresponding to all the adjacent video frames, calculate the average arrival time interval parameter corresponding to all the adjacent video frames;

[0027] According to the arrival time interval parameters corresponding to all the adjacent video frames and the average arrival time interval parameter, calculate the sum of squared time interval deviations corresponding to all the adjacent video frames;

[0028] According to the sum of squared time interval deviations and the total number of groups, calculate a target average value corresponding to the sum of squared time interval deviations, and perform a square root calculation on the target average value to obtain the time interval standard deviation corresponding to all the adjacent video frames, which is used as the network jitter parameter during transmission corresponding to all the video frames.

[0029] As an alternative implementation, in the first aspect of the present invention, calculating the historical video frame-metadata matching success rate corresponding to all historical time windows in all the video frames includes:

[0030] Determine the historical video frame-metadata sub-matching success rate corresponding to each historical time window in all the video frames; the historical video frame-metadata sub-matching success rate corresponding to each historical time window is used to indicate the probability of matching the corresponding synchronization metadata for all historical video frames in this historical time window;

[0031] According to the historical video frame-metadata sub-matching success rates corresponding to all the historical time windows, determine all target historical time windows in all the historical time windows where the historical video frame-metadata sub-matching success rate is greater than or equal to a preset sub-matching success rate threshold;

[0032] Determine the historical video frame-metadata matching success rate corresponding to all the historical time windows in all the video frames according to the ratio between the number of all the target historical time windows and the number of all the historical time windows.

[0033] As an alternative implementation, in the first aspect of the present invention, the target time window size parameter matching the current video frame is determined by the following formula:

[0034] W = max(W min , min(W max , W0 + K J *(J - J0) + K R *(R - R0)));

[0035] where W is the target time window size parameter, W min is the minimum window size parameter, W max is the maximum window size parameter, W0 is the initial window size parameter, K J is the first weight parameter, J is the network jitter parameter during transmission, J0 is the first reference value, K R is the second weight parameter, R is the historical video frame-metadata matching success rate, and R0 is the second reference value.

[0036] As an alternative implementation, in the first aspect of the present invention, the method further includes:

[0037] All of the video frames are transmitted to a preset rendering module through a media track in a preset WebRTC transmission channel, and at the same time, synchronization metadata corresponding to all of the video frames is transmitted to the rendering module through a data channel in the WebRTC transmission channel, so that the rendering module synchronously renders all of the video frames and the synchronization metadata corresponding to all of the video frames; the media track and the data channel share a target communication component of the same WebRTC transmission channel.

[0038] A second aspect of the present invention discloses a flight device map-data synchronization device under a time window verification mechanism, the device comprising:

[0039] A demultiplexing module, configured to perform a demultiplexing operation on video stream data of a flight device obtained in real time, to obtain demultiplexed data corresponding to the video stream data; the demultiplexed data includes a plurality of video frames and a frame timestamp corresponding to each of the video frames;

[0040] A processing module, configured to perform a target processing operation on all of the video frames, to obtain a metadata set corresponding to all of the video frames;

[0041] A determining module, configured to determine a timestamp corresponding to each metadata in the metadata set; the target processing operation includes a target detection operation and / or an image enhancement calculation operation;

[0042] A time window verification module, configured to perform a time window verification operation on all of the video frames and all of the metadata according to the frame timestamps corresponding to all of the video frames and the timestamps corresponding to all of the metadata, to obtain synchronization metadata corresponding to all of the video frames.

[0043] As an optional implementation manner, in the second aspect of the present invention, the manner in which the time window verification module performs a time window verification operation on all of the video frames and all of the metadata according to the frame timestamps corresponding to all of the video frames and the timestamps corresponding to all of the metadata, to obtain synchronization metadata corresponding to all of the video frames specifically includes:

[0044] For a current video frame among all of the video frames, determine a target time window size parameter matching the current video frame, and determine a target time window matching the current video frame according to the target time window size parameter;

[0045] Through the target time window, determine all of the metadata to be matched corresponding to the current video frame from all of the metadata;

[0046] Calculate the absolute timestamp difference between the current video frame and each of the metadata to be matched according to the frame timestamp corresponding to the current video frame and the timestamps corresponding to all the metadata to be matched.

[0047] Determine the target metadata with the smallest absolute timestamp difference from all the metadata to be matched according to the absolute timestamp differences between the current video frame and all the metadata to be matched.

[0048] Judge whether the absolute timestamp difference between the current video frame and the target metadata is less than or equal to a preset absolute difference threshold.

[0049] When the judgment result is yes, determine the target metadata as the synchronization metadata corresponding to the current video frame.

[0050] When the judgment result is no, determine the synchronization metadata corresponding to the previous video frame of the current video frame as the synchronization metadata corresponding to the current video frame.

[0051] As an optional implementation manner, in the second aspect of the present invention, the manner in which the time window verification module determines the target time window size parameter matching the current video frame specifically includes:

[0052] Calculate the network jitter parameters during transmission corresponding to all the video frames.

[0053] Calculate the historical video frame-metadata matching success rates corresponding to all the historical time windows in all the video frames.

[0054] Determine the target time window size parameter matching the current video frame according to the network jitter parameters during transmission, the historical video frame-metadata matching success rates, a preset minimum window size parameter, a maximum window size parameter, an initial window size parameter, a first weight parameter corresponding to the network jitter parameters during transmission, a first reference value corresponding to the network jitter parameters during transmission, a second weight parameter corresponding to the historical video frame-metadata matching success rates, and a second reference value corresponding to the historical video frame-metadata matching success rates.

[0055] Among them, the first weight parameter is determined by the following method:

[0056] Determine the transmission network conditions corresponding to all the video frames, and determine the first weight parameter corresponding to the network jitter parameters during transmission according to the transmission network conditions corresponding to all the video frames; the transmission network conditions include the speed situation of the transmission network and / or the stability situation of the transmission network.

[0057] As an alternative implementation, in the second aspect of the present invention, the manner in which the time window verification module calculates the network jitter parameters during transmission corresponding to all the video frames specifically includes:

[0058] Determine the acquisition time parameter corresponding to when the demultiplexing module acquires the video stream data, and based on the acquisition time parameter, determine the arrival time parameter corresponding to each video frame;

[0059] Based on the arrival time parameters corresponding to all the video frames, calculate the arrival time interval parameters corresponding to each group of adjacent video frames among all the video frames;

[0060] Based on the sum of the arrival time interval parameters corresponding to all the adjacent video frames and the total number of groups corresponding to all the adjacent video frames, calculate the average arrival time interval parameter corresponding to all the adjacent video frames;

[0061] Based on the arrival time interval parameters corresponding to all the adjacent video frames and the average arrival time interval parameter, calculate the sum of the squared time interval deviations corresponding to all the adjacent video frames;

[0062] Based on the sum of the squared time interval deviations and the total number of groups, calculate the target average value corresponding to the sum of the squared time interval deviations, and perform a square root calculation on the target average value to obtain the time interval standard deviation corresponding to all the adjacent video frames, which is used as the network jitter parameter during transmission corresponding to all the video frames.

[0063] As an alternative implementation, in the second aspect of the present invention, the manner in which the time window verification module calculates the historical video frame - metadata matching success rate corresponding to all historical time windows among all the video frames specifically includes:

[0064] Determine the historical video frame - metadata sub - matching success rate corresponding to each historical time window among all the video frames; the historical video frame - metadata sub - matching success rate corresponding to each historical time window is used to indicate the probability of matching corresponding synchronization metadata for all historical video frames in this historical time window;

[0065] Based on the historical video frame - metadata sub - matching success rates corresponding to all the historical time windows, determine all target historical time windows among all the historical time windows whose historical video frame - metadata sub - matching success rate is greater than or equal to a preset sub - matching success rate threshold;

[0066] Based on the ratio between the number of all the target historical time windows and the number of all the historical time windows, determine the historical video frame - metadata matching success rate corresponding to all the historical time windows among all the video frames.

[0067] As an alternative implementation, in the second aspect of the present invention, the target time window size parameter matching the current video frame is determined by the following formula:

[0068] W = max(W min , min(W max , W0 + K J *(J - J0) + K R *(R - R0)));

[0069] Wherein, W is the target time window size parameter, W min is the minimum window size parameter, W max is the maximum window size parameter, W0 is the initial window size parameter, K J is the first weight parameter, J is the network jitter parameter during transmission, J0 is the first reference value, K R is the second weight parameter, R is the matching success rate of the historical video frame - metadata, and R0 is the second reference value.

[0070] As an alternative implementation, in the second aspect of the present invention, the device further includes:

[0071] A transmission module, configured to transmit all the video frames to a preset rendering module through a media track in a preset WebRTC transmission channel, and at the same time transmit the synchronization metadata corresponding to all the video frames to the rendering module through a data channel in the WebRTC transmission channel, so that the rendering module performs synchronous rendering on all the video frames and the synchronization metadata corresponding to all the video frames; the media track and the data channel share the same target communication component of the WebRTC transmission channel.

[0072] The third aspect of the present invention discloses another flight device map - data synchronization device under a time window verification mechanism, and the device includes:

[0073] A memory storing executable program code;

[0074] A processor coupled to the memory;

[0075] The processor calls the executable program code stored in the memory and executes the flight device map - data synchronization method under the time window verification mechanism disclosed in the first aspect of the present invention.

[0076] A fourth aspect of the present invention discloses a computer storage medium storing computer instructions which, when called, are used to execute the method for synchronizing map data of a flight device under the time window verification mechanism disclosed in the first aspect of the present invention.

[0077] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0078] In the embodiments of the present invention, the demultiplexing operation is performed on the video stream data of the flight device obtained in real time to obtain a plurality of video frames corresponding to the video stream data and the frame timestamp corresponding to each video frame; the target processing operation is performed on all the video frames to obtain the metadata set corresponding to all the video frames, and the timestamp corresponding to each metadata in the metadata set is determined; according to the frame timestamps corresponding to all the video frames and the timestamps corresponding to all the metadata, the time window verification operation is performed on all the video frames and all the metadata to obtain the synchronized metadata corresponding to all the video frames. It can be seen that implementing the present invention can perform time window verification on the timestamps between each video frame and the metadata, so as to match synchronized metadata for each video frame, which is beneficial to improving the map data synchronization accuracy of the flight device and reducing the map data transmission delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings according to these drawings without creative efforts.

[0080] Figure 1 is a schematic flowchart of a method for synchronizing map data of a flight device under a time window verification mechanism disclosed in an embodiment of the present invention;

[0081] Figure 2 is a schematic flowchart of another method for synchronizing map data of a flight device under a time window verification mechanism disclosed in an embodiment of the present invention;

[0082] Figure 3 is a schematic structural diagram of a device for synchronizing map data of a flight device under a time window verification mechanism disclosed in an embodiment of the present invention;

[0083] Figure 4 is a schematic structural diagram of another device for synchronizing map data of a flight device under a time window verification mechanism disclosed in an embodiment of the present invention;

[0084] Figure 5 is a schematic structural diagram of yet another device for synchronizing map data of a flight device under a time window verification mechanism disclosed in an embodiment of the present invention;

[0085] Figure 6 It is a schematic diagram of the figure-number synchronization processing flow of a flight device disclosed in an embodiment of the present invention. Detailed implementation manners

[0086] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

[0087] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or terminal comprising a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or terminals.

[0088] Referring to "embodiment" herein means that a specific feature, structure or characteristic described in connection with the embodiment can be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0089] The present invention discloses a method and device for figure-number synchronization of a flight device under a time window verification mechanism, which can perform time window verification on the timestamps between each video frame and metadata, so as to match synchronous metadata for each video frame, which is beneficial to improving the figure-number synchronization accuracy of the flight device and reducing the figure-number transmission delay.

[0090] Embodiment 1

[0091] Please refer to Figure 1 , Figure 1 It is a schematic diagram of a method for figure-number synchronization of a flight device under a time window verification mechanism disclosed in an embodiment of the present invention. Among them, Figure 1The method for synchronizing the number of frames of a flying device under the described time window verification mechanism can be applied to synchronize video frames and metadata of flying devices such as drones, helicopters, and airplanes. Optionally, this method can be implemented by a frame-number synchronization device, which can be integrated in a frame-number synchronization device (such as a smart computer, tablet, smartphone, etc.), or can be a local server or cloud server for processing the frame-number synchronization process of a flying device under the time window verification mechanism, etc. The embodiments of the present invention do not make any limitations. As Figure 1 shown, the method for synchronizing the number of frames of a flying device under the time window verification mechanism may include the following operations:

[0092] 101. Through a preset demultiplexing module, perform demultiplexing operations on the real-time acquired video stream data of the flying device to obtain the demultiplexed data corresponding to the video stream data.

[0093] In the embodiments of the present invention, the demultiplexed data includes multiple video frames and the frame timestamp corresponding to each video frame. Among them, all video frames may be video frames that are not encapsulated in containers such as MP4, MKV, AVI, etc.

[0094] Further, as Figure 6 shown, Figure 6 is a schematic diagram of the frame-number synchronization processing flow of a flying device disclosed in the embodiments of the present invention. Among them, after obtaining multiple video frames (i.e., video raw stream) corresponding to the video stream data, all video frames can be separately stored in the push stream cache queue. At the same time, all video frames and the frame timestamps corresponding to all video frames are stored in the decoding cache queue in the form of key-value for subsequent asynchronous task threads to process (such as AI detection, AR calculation, etc.).

[0095] 102. Perform target processing operations on all video frames to obtain the metadata set corresponding to all video frames, and determine the timestamp corresponding to each metadata in the metadata set.

[0096] In the embodiments of the present invention, optionally, the target processing operations include target detection operations and / or image enhancement calculation (AR calculation) operations, and the metadata set corresponding to all video frames includes a target detection result set and / or an image enhancement calculation result set, such as detected object results, calculated object position coordinates, object movement parameters, etc.

[0097] For example, as Figure 6 shown, that is, start an asynchronous task thread, sequentially retrieve all video frames and corresponding frame timestamps from the decoding cache queue, send all video frames for decoding and then perform AI detection and / or AR calculation, and convert the corresponding result set into the json format for storage in the metadata cache queue.

[0098] Further, the timestamp corresponding to each piece of metadata in the metadata set can be determined based on the frame timestamp corresponding to the video frame, combined with the decoding, detection / calculation duration of the video frame (i.e., the target processing duration), the duration required for data transmission to the corresponding module, etc.

[0099] 103. Perform a time window verification operation on all video frames and all metadata according to the frame timestamps corresponding to all video frames and the timestamps corresponding to all metadata, to obtain the synchronized metadata corresponding to all video frames.

[0100] In the embodiment of the present invention, for example, in the main thread, all video frames are sequentially taken out from the push stream buffer queue, and timestamp verification is performed with the AI detection results / AR calculation results in the metadata buffer queue. Whenever a new video frame arrives, the system will search for the metadata item with the closest timestamp in the metadata buffer within the current time window (where the time window is used to collect all video frames and corresponding metadata within a period of time, and the size of the time window can be dynamically adjusted according to the actual network condition and task requirements: for example, a smaller time window is selected when the network is good to reduce latency; the window is appropriately increased when the network is unstable to ensure data integrity) for matching. As new data continuously flows in, the time window gradually slides forward, and each time it slides, only the newly added data is processed and the old data is removed to ensure processing efficiency. To cope with slight time deviations, a certain tolerance range can be set, that is, if the timestamp of the closest metadata item falls within this range, it is considered to match the video frame.

[0101] It can be seen that implementing the embodiment of the present invention can perform time window verification on the timestamps between each video frame and metadata, to match synchronized metadata for each video frame. In this way, it can ensure the precise matching of timestamps between the video image and metadata of the flying device, which is beneficial to improving the synchronization accuracy of the video and data of the flying device and reducing the video and data transmission delay, thereby being beneficial to improving the subsequent synchronization processing (such as rendering) efficiency and accuracy of the video frame and metadata, so as to accurately display the picture and improve the display effect at the web front end.

[0102] In an optional embodiment, the step 103 of performing a time window verification operation on all video frames and all metadata according to the frame timestamps corresponding to all video frames and the timestamps corresponding to all metadata, to obtain the synchronized metadata corresponding to all video frames, includes:

[0103] For the current video frame among all video frames, determine the target time window size parameter that matches the current video frame, and determine the target time window that matches the current video frame according to the target time window size parameter;

[0104] Determine all metadata to be matched corresponding to the current video frame from all metadata through the target time window;

[0105] Calculate the absolute timestamp difference between the current video frame and each metadata to be matched according to the frame timestamp corresponding to the current video frame and the timestamps corresponding to all metadata to be matched;

[0106] Determine the target metadata with the smallest absolute timestamp difference from all metadata to be matched according to the absolute timestamp difference between the current video frame and all metadata to be matched;

[0107] Judge whether the absolute timestamp difference between the current video frame and the target metadata is less than or equal to the preset absolute difference threshold;

[0108] When the judgment result is yes, determine the target metadata as the synchronization metadata corresponding to the current video frame;

[0109] When the judgment result is no, determine the synchronization metadata corresponding to the previous video frame corresponding to the current video frame as the synchronization metadata corresponding to the current video frame.

[0110] In this optional embodiment, for example, in the main thread, take the raw stream from the push stream cache queue in turn and perform timestamp verification with the AI detection result / AR calculation result in the metadata cache queue. Whenever a new video frame arrives, the system will find the metadata item with the closest timestamp in the metadata buffer within the current time window to match the new video frame. Poll the metadata in the metadata storage buffer in turn, query the timestamp corresponding to the metadata, and subtract the timestamp of the polled metadata from the frame timestamp of the new video frame, take the absolute value, and assign it to a temporary variable temp for storage. Then, compare the absolute value of the difference result of each poll with the previous result each time. If it is smaller, assign it to temp to replace the previous difference result, and record the metadata of this time. If the difference result of a certain poll is 0, it means that the timestamp of the metadata is the same as the frame timestamp of the new video frame, indicating a successful match; if the poll is completed and no metadata with the same timestamp is found, the metadata recorded with the smallest difference result can be taken, and further judge whether the absolute value of the timestamp difference between it and the new video frame is less than or equal to the preset absolute difference threshold. If so, it is considered a successful match, otherwise, take the synchronization metadata corresponding to the previous video frame corresponding to the new video frame as a substitute.

[0111] It can be seen that this optional embodiment can automatically perform a timestamp verification operation on the current video frame and the metadata to be matched within the target time window that matches the current video frame, so as to determine the corresponding synchronization metadata for the current video frame. In this way, the reliability and accuracy of determining the synchronization metadata of the current video frame are improved, which is conducive to improving the synchronization accuracy between the video frame and the metadata, and thus conducive to subsequent accurate display of the picture and rendering effect.

[0112] In another optional embodiment, determining the target time window size parameter that matches the current video frame in the above steps includes:

[0113] Calculate the network jitter parameters during transmission corresponding to all video frames;

[0114] Calculate the historical video frame-metadata matching success rate corresponding to all historical time windows in all video frames;

[0115] According to the network jitter parameters during transmission, the historical video frame-metadata matching success rate, the preset minimum window size parameter, the maximum window size parameter, the initial window size parameter, the first weight parameter corresponding to the network jitter parameters during transmission, the first reference value corresponding to the network jitter parameters during transmission, the second weight parameter corresponding to the historical video frame-metadata matching success rate, and the second reference value corresponding to the historical video frame-metadata matching success rate, determine the target time window size parameter that matches the current video frame.

[0116] In this optional embodiment, further, the first weight parameter is determined in the following manner:

[0117] Determine the transmission network conditions corresponding to all video frames, and determine the first weight parameter corresponding to the network jitter parameters during transmission according to the transmission network conditions corresponding to all video frames.

[0118] In this optional embodiment, optionally, the transmission network conditions include the transmission network speed condition and / or the transmission network stability condition. Among them, when the transmission network speed condition is faster and / or the transmission network stability condition is more stable, the first weight parameter corresponding to the network jitter parameters during transmission is smaller, so as to determine a smaller target time window to reduce latency; when the transmission network speed condition is slower and / or the transmission network stability condition is more fluctuating, the first weight parameter corresponding to the network jitter parameters during transmission is larger, so as to determine a larger target time window to ensure data integrity.

[0119] Furthermore, the target time window size parameter that matches the current video frame is determined by the following formula:

[0120] W = max(W min ,min(Wmax , W0 + K J *(J - J0) + K R *(R - R0)));

[0121] Wherein, W is the target time window size parameter, W min is the minimum window size parameter, W max is the maximum window size parameter, W0 is the initial window size parameter, K J is the first weight parameter, J is the network jitter parameter during transmission, J0 is the first reference value, K R is the second weight parameter, R is the matching success rate of historical video frames - metadata, R0 is the second reference value.

[0122] Optionally, J0 and R0 can take the initial configuration values or historical average values of the system.

[0123] It can be seen that this optional embodiment can dynamically adjust the size of the target time window according to the network jitter parameter during transmission and the matching success rate of historical video frames - metadata corresponding to all video frames, and in combination with the transmission network situation, so as to match synchronous metadata for the current video frame through the target time window. In this way, the adaptability of the time window to the network environment is improved, and further, the transmission delay between the video frames and the metadata can be reduced as much as possible while ensuring the integrity of data transmission, which is beneficial to enhancing the subsequent viewing experience of users for the video.

[0124] In another optional embodiment, calculating the network jitter parameter during transmission corresponding to all video frames includes:

[0125] Determine the acquisition time parameter corresponding to when the demultiplexing module obtains the video stream data, and determine the arrival time parameter corresponding to each video frame according to the acquisition time parameter;

[0126] Calculate the arrival time interval parameter corresponding to each group of adjacent video frames among all video frames according to the arrival time parameters corresponding to all video frames;

[0127] Calculate the average arrival time interval parameter corresponding to all adjacent video frames according to the sum of the arrival time interval parameters corresponding to all adjacent video frames and the total number of groups corresponding to all adjacent video frames;

[0128] Calculate the sum of the squares of the time interval deviations corresponding to all adjacent video frames according to the arrival time interval parameters corresponding to all adjacent video frames and the average arrival time interval parameter;

[0129] Calculate the target average value corresponding to the sum of squared time interval deviations and the total number of groups, and perform a square root calculation on the target average value to obtain the standard deviation of the time intervals corresponding to all adjacent video frames, which is used as the network jitter parameter during transmission corresponding to all video frames.

[0130] In this alternative embodiment, by calculating the network jitter parameters during transmission corresponding to all video frames, it helps to evaluate network stability and predict possible latency changes.

[0131] It can be seen that this alternative embodiment can determine the network jitter parameters during transmission corresponding to all video frames by calculating the standard deviation of the time intervals corresponding to all adjacent video frames, which is beneficial to improving the reliability and accuracy of the current network stability evaluation of the system, and further beneficial to improving the reliability and accuracy of determining the target time window, thereby being beneficial to enhancing the timestamp verification accuracy between video frames and metadata.

[0132] In yet another alternative embodiment, calculate the historical video frame - metadata matching success rate corresponding to all historical time windows in all video frames, including:

[0133] Determine the historical video frame - metadata sub - matching success rate corresponding to each historical time window in all video frames; the historical video frame - metadata sub - matching success rate corresponding to each historical time window is used to indicate the probability of matching corresponding synchronous metadata for all historical video frames in this historical time window;

[0134] According to the historical video frame - metadata sub - matching success rates corresponding to all historical time windows, determine all target historical time windows whose historical video frame - metadata sub - matching success rates are greater than or equal to a preset sub - matching success rate threshold from all historical time windows;

[0135] Determine the historical video frame - metadata matching success rate corresponding to all historical time windows in all video frames according to the ratio between the number of all target historical time windows and the number of all historical time windows.

[0136] In this alternative embodiment, for example, among the last 5 historical time windows, there are 3 historical time windows that can respectively match 90%, 93%, and 95% of the historical video frames to the corresponding synchronous metadata (90%, 93%, and 95% are respectively the historical video frame - metadata sub - matching success rates corresponding to these 3 historical time windows, and the sub - matching success rate threshold can be set to 90%), and then it can be determined that the historical video frame - metadata matching success rate corresponding to all historical time windows in all these video frames is 3 / 5.

[0137] It can be seen that this optional embodiment can determine the corresponding historical video frame - metadata matching success rate through the historical video frame - metadata sub - matching success rate corresponding to each historical time window in all video frames. In this way, the calculation reliability and accuracy of the historical video frame - metadata matching success rate are improved, which is conducive to comprehensively improving the determination reliability and accuracy of the target time window subsequently, and thus conducive to improving the timestamp verification effectiveness between the video frame and the metadata.

[0138] Embodiment 2

[0139] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a method for synchronizing flight device map - data under another time - window verification mechanism disclosed in an embodiment of the present invention. Among them, Figure 2 the method for synchronizing flight device map - data under the described time - window verification mechanism can be applied to synchronize video frames and metadata of flight devices such as unmanned aerial vehicles, helicopters, airplanes, etc. Optionally, this method can be implemented by a map - data synchronization device. The map - data synchronization device can be integrated in a map - data synchronization device (such as a smart computer, tablet, smartphone, etc.), or can be a local server or a cloud server for processing the flight device map - data synchronization process under the time - window verification mechanism. The embodiments of the present invention do not make limitations. As Figure 2 shown, the method for synchronizing flight device map - data under this time - window verification mechanism can include the following operations:

[0140] 201. Through a preset demultiplexing module, perform a demultiplexing operation on the real - time acquired video stream data of the flight device to obtain the demultiplexed data corresponding to the video stream data.

[0141] 202. Perform a target processing operation on all video frames to obtain a metadata set corresponding to all video frames, and determine the timestamp corresponding to each metadata in the metadata set.

[0142] 203. According to the frame timestamps corresponding to all video frames and the timestamps corresponding to all metadata, perform a time - window verification operation on all video frames and all metadata to obtain the synchronized metadata corresponding to all video frames.

[0143] 204. Through the media track in a preset WebRTC transmission channel, transmit all video frames to a preset rendering module, and at the same time, through the data channel in the WebRTC transmission channel, transmit the synchronized metadata corresponding to all video frames to the rendering module, so that the rendering module performs synchronous rendering on all video frames and the synchronized metadata corresponding to all video frames.

[0144] In the embodiment of the present invention, among them, the media track and the data channel share the target communication component of the same WebRTC transmission channel.

[0145] It should be noted that in the existing video frame synchronization and display technologies, generally, the video stream is demultiplexed by the web backend, and then manually decoded and rendered at the web front end or transmitted through a two-channel protocol (the image and metadata are divided into two channels), pushing the raw stream and metadata to the front end. In this way, it will increase the data transmission delay and affect the real-time performance of the data. In the embodiments of the present invention, by using the dedicated encoding and flow control mechanisms of WebRTC, the video and metadata are respectively transmitted through the Media Track and DataChannel of WebRTC. They do not interfere with each other, but share the same target communication component (such as an RTPPeerConnection instance), which can ensure that the video and metadata can reach the receiving end almost simultaneously, and also ensure the sending order and integrity of the video frames and metadata. In this way, even in the case of large network fluctuations, high-precision synchronization can be maintained.

[0146] In the embodiments of the present invention, for other descriptions of steps 201 - 203, please refer to the detailed descriptions of steps 101 - 103 in Embodiment 1, and the embodiments of the present invention will not be elaborated here.

[0147] It can be seen that implementing the embodiments of the present invention can transmit the video and metadata respectively through the MediaTrack and DataChannel in the WebRTC transmission channel, optimize the transmission paths of the video and metadata, is conducive to reducing the transmission delay of the video and metadata, and further conducive to improving the rendering synchronization degree between the video frames and metadata, thus conducive to improving the picture display effect of the web front end.

[0148] Embodiment 3

[0149] Please refer to Figure 3 , Figure 3 which is a schematic structural diagram of a flight device video frame synchronization device under a time window verification mechanism disclosed in the embodiments of the present invention. As Figure 3 shown, the flight device video frame synchronization device under the time window verification mechanism may include:

[0150] A demultiplexing module 301, configured to perform a demultiplexing operation on the video stream data of the flight device obtained in real time, and obtain the demultiplexed data corresponding to the video stream data; the demultiplexed data includes multiple video frames and the frame timestamp corresponding to each video frame;

[0151] A processing module 302, configured to perform a target processing operation on all the video frames to obtain a metadata set corresponding to all the video frames;

[0152] A determination module 303, configured to determine the timestamp corresponding to each metadata in the metadata set;

[0153] The time window verification module 304 is configured to perform a time window verification operation on all video frames and all metadata according to the frame timestamps corresponding to all video frames and the timestamps corresponding to all metadata, so as to obtain the synchronized metadata corresponding to all video frames.

[0154] In an embodiment of the present invention, the target processing operation includes a target detection operation and / or an image enhancement calculation operation.

[0155] It can be seen that Figure 3 the flight device map data synchronization device under the described time window verification mechanism can perform a time window verification on the timestamps between each video frame and the metadata, so as to match the synchronized metadata for each video frame. In this way, it can ensure the precise matching of the timestamps between the video images and the metadata of the flight device, which is beneficial to improving the map data synchronization accuracy of the flight device and reducing the map data transmission delay, thereby being beneficial to improving the efficiency and accuracy of subsequent synchronous processing (such as rendering) of the video frames and the metadata, so as to accurately display the picture and improve the display effect at the web front end.

[0156] In an optional embodiment, the manner in which the time window verification module 304 performs a time window verification operation on all video frames and all metadata according to the frame timestamps corresponding to all video frames and the timestamps corresponding to all metadata, and obtains the synchronized metadata corresponding to all video frames specifically includes:

[0157] For the current video frame among all video frames, determine a target time window size parameter that matches the current video frame, and determine a target time window that matches the current video frame according to the target time window size parameter;

[0158] Through the target time window, determine all the metadata to be matched corresponding to the current video frame from all the metadata;

[0159] According to the frame timestamp corresponding to the current video frame and the timestamps corresponding to all the metadata to be matched, calculate the absolute timestamp difference between the current video frame and each metadata to be matched;

[0160] According to the absolute timestamp differences between the current video frame and all the metadata to be matched, determine the target metadata with the smallest absolute timestamp difference from all the metadata to be matched;

[0161] Judge whether the absolute timestamp difference between the current video frame and the target metadata is less than or equal to a preset absolute difference threshold;

[0162] When the judgment result is yes, determine the target metadata as the synchronized metadata corresponding to the current video frame;

[0163] When the judgment result is negative, the synchronization metadata corresponding to the previous video frame corresponding to the current video frame is determined as the synchronization metadata corresponding to the current video frame.

[0164] It can be seen that implementing Figure 4 the flight device map number synchronization device under the described time window verification mechanism can automatically perform a timestamp verification operation on the current video frame and the metadata to be matched within the target time window that matches the current video frame, so as to determine the corresponding synchronization metadata for the current video frame. In this way, the reliability and accuracy of determining the synchronization metadata of the current video frame are improved, which is conducive to improving the synchronization accuracy between the video frame and the metadata, and thus conducive to subsequent precise display of the picture and rendering effect.

[0165] In another optional embodiment, the manner in which the time window verification module 304 determines the target time window size parameter that matches the current video frame specifically includes:

[0166] Calculate the network jitter parameters during transmission corresponding to all video frames;

[0167] Calculate the historical video frame - metadata matching success rate corresponding to all historical time windows in all video frames;

[0168] According to the network jitter parameters during transmission, the historical video frame - metadata matching success rate, the preset minimum window size parameter, the maximum window size parameter, the initial window size parameter, the first weight parameter corresponding to the network jitter parameters during transmission, the first reference value corresponding to the network jitter parameters during transmission, the second weight parameter corresponding to the historical video frame - metadata matching success rate, and the second reference value corresponding to the historical video frame - metadata matching success rate, determine the target time window size parameter that matches the current video frame

[0169] In this optional embodiment, further, the first weight parameter is determined by the following method:

[0170] Determine the transmission network conditions corresponding to all video frames, and determine the first weight parameter corresponding to the network jitter parameters during transmission according to the transmission network conditions corresponding to all video frames.

[0171] In this optional embodiment, optionally, the transmission network conditions include the transmission network speed condition and / or the transmission network stability condition.

[0172] Furthermore, the target time window size parameter that matches the current video frame is determined by the following formula:

[0173] W = max(W min , min(W max , W0 + K J*(J - J0)+K R *(R - R0)));

[0174] Wherein, W is the target time window size parameter, W min is the minimum window size parameter, W max is the maximum window size parameter, W0 is the initial window size parameter, K J is the first weight parameter, J is the network jitter parameter during transmission, J0 is the first reference value, K R is the second weight parameter, R is the matching success rate of historical video frame - metadata, and R0 is the second reference value.

[0175] It can be seen that implementing Figure 4 the flight device map number synchronization device under the described time window verification mechanism can, according to the network jitter parameter during transmission and the matching success rate of historical video frame - metadata corresponding to all video frames, and in combination with the transmission network situation, dynamically adjust the size of the target time window to match synchronization metadata for the current video frame through the target time window. In this way, the adaptability of the time window to the network environment is improved, and then, while ensuring the integrity of data transmission, the transmission delay between video frames and metadata can be reduced as much as possible, which is conducive to enhancing the subsequent viewing experience of users for the video.

[0176] In another optional embodiment, the specific method for the time window verification module 304 to calculate the network jitter parameter during transmission corresponding to all video frames includes:

[0177] Determine the acquisition time parameter corresponding to when the demultiplexing module 301 acquires the video stream data, and determine the arrival time parameter corresponding to each video frame according to the acquisition time parameter;

[0178] Calculate the arrival time interval parameter corresponding to each group of adjacent video frames among all video frames according to the arrival time parameters corresponding to all video frames;

[0179] Calculate the average arrival time interval parameter corresponding to all adjacent video frames according to the sum of the arrival time interval parameters corresponding to all adjacent video frames and the total number of groups corresponding to all adjacent video frames;

[0180] Calculate the sum of the squares of the time interval deviations corresponding to all adjacent video frames according to the arrival time interval parameters corresponding to all adjacent video frames and the average arrival time interval parameter;

[0181] Calculate the target average value corresponding to the sum of the squares of the time interval deviations according to the sum of the squares of the time interval deviations and the total number of groups, and perform a square root calculation on the target average value to obtain the standard deviation of the time interval corresponding to all adjacent video frames, which is used as the network jitter parameter during transmission corresponding to all video frames.

[0182] It can be seen that implementing Figure 4 the flight equipment map-data synchronization device under the described time window verification mechanism can determine the network jitter parameters during transmission corresponding to all video frames by calculating the standard deviation of the time intervals corresponding to all adjacent video frames, which is beneficial to improving the reliability and accuracy of the current network stability evaluation of the system, and further beneficial to improving the reliability and accuracy of determining the target time window, thereby being beneficial to improving the timestamp verification accuracy between video frames and metadata.

[0183] In another optional embodiment, the manner in which the time window verification module 304 calculates the historical video frame-metadata matching success rate corresponding to all historical time windows in all video frames specifically includes:

[0184] Determine the historical video frame-metadata sub-matching success rate corresponding to each historical time window in all video frames; the historical video frame-metadata sub-matching success rate corresponding to each historical time window is used to indicate the probability of matching the corresponding synchronization metadata for all historical video frames in this historical time window;

[0185] According to the historical video frame-metadata sub-matching success rates corresponding to all historical time windows, determine all target historical time windows whose historical video frame-metadata sub-matching success rates are greater than or equal to a preset sub-matching success rate threshold from all historical time windows;

[0186] According to the ratio between the number of all target historical time windows and the number of all historical time windows, determine the historical video frame-metadata matching success rate corresponding to all historical time windows in all video frames.

[0187] It can be seen that implementing Figure 4 the flight equipment map-data synchronization device under the described time window verification mechanism can determine the corresponding historical video frame-metadata matching success rate through the historical video frame-metadata sub-matching success rate corresponding to each historical time window in all video frames. In this way, the calculation reliability and accuracy of the historical video frame-metadata matching success rate are improved, which is beneficial to comprehensively improving the reliability and precision of subsequent determination of the target time window, and thus beneficial to improving the effectiveness of timestamp verification between video frames and metadata.

[0188] In another optional embodiment, the device further includes:

[0189] A transmission module 305 is configured to transmit all video frames to a preset rendering module through a media track in a preset WebRTC transmission channel, and simultaneously transmit synchronization metadata corresponding to all video frames to the rendering module through a data channel in the WebRTC transmission channel, so that the rendering module performs synchronous rendering on all video frames and the synchronization metadata corresponding to all video frames.

[0190] In this optional embodiment, the media track and the data channel share the same target communication component of the WebRTC transmission channel.

[0191] It can be seen that Figure 4 the flight device map data synchronization device under the described time window verification mechanism can transmit videos and metadata respectively through the media track (Media Track) and the data channel (DataChannel) in the WebRTC transmission channel, optimize the transmission paths of videos and metadata, help reduce the transmission latency of videos and metadata, and thus help improve the rendering synchronization degree between video frames and metadata, and further help improve the screen display effect of the web front end.

[0192] Embodiment 4

[0193] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of another flight device map data synchronization device under the time window verification mechanism disclosed in the embodiments of the present invention. As Figure 5 shown, the flight device map data synchronization device under the time window verification mechanism may include:

[0194] A memory 401 storing executable program code;

[0195] A processor 402 coupled to the memory 401;

[0196] The processor 402 calls the executable program code stored in the memory 401 and executes the steps in the flight device map data synchronization method under the time window verification mechanism described in Embodiment 1 or Embodiment 2 of the present invention.

[0197] Embodiment 5

[0198] The embodiments of the present invention disclose a computer storage medium storing computer instructions, which are used to execute the steps in the flight device map data synchronization method under the time window verification mechanism described in Embodiment 1 or Embodiment 2 of the present invention when the computer instructions are called.

[0199] Embodiment 6

[0200] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the steps in the flight equipment map data synchronization method under the time window verification mechanism described in Embodiment 1 or Embodiment 2.

[0201] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0202] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium. The storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc memories, a magnetic disk memory, a tape memory, or any other computer-readable medium capable of carrying or storing data.

[0203] Finally, it should be noted that: The time window verification mechanism-based flight device map data synchronization method and device disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than limiting it; Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synchronizing flight device map data under a time window verification mechanism, characterized in that The method includes: Performing a demultiplexing operation on the video stream data of the flying device obtained in real time through a preset demultiplexing module to obtain the demultiplexed data corresponding to the video stream data; the demultiplexed data includes a plurality of video frames and a frame timestamp corresponding to each video frame; Performing a target processing operation on all the video frames to obtain a set of metadata corresponding to all the video frames, and determining the timestamp corresponding to each piece of metadata in the metadata set; the target processing operation includes a target detection operation and / or an image enhancement calculation operation; Performing a time window verification operation on all the video frames and all the metadata according to the frame timestamps corresponding to all the video frames and the timestamps corresponding to all the metadata to obtain the synchronized metadata corresponding to all the video frames.

2. The method for synchronizing flight device map data under the time window verification mechanism according to claim 1, wherein The performing a time window verification operation on all the video frames and all the metadata according to the frame timestamps corresponding to all the video frames and the timestamps corresponding to all the metadata to obtain the synchronized metadata corresponding to all the video frames includes: For the current video frame among all the video frames, determining a target time window size parameter matching the current video frame, and determining a target time window matching the current video frame according to the target time window size parameter; Determining all the metadata to be matched corresponding to the current video frame from all the metadata through the target time window; Calculating the absolute timestamp difference between the current video frame and each piece of the metadata to be matched according to the frame timestamp corresponding to the current video frame and the timestamps corresponding to all the metadata to be matched; Determining the target metadata with the smallest absolute timestamp difference from all the metadata to be matched according to the absolute timestamp difference between the current video frame and all the metadata to be matched; Judging whether the absolute timestamp difference between the current video frame and the target metadata is less than or equal to a preset absolute difference threshold; When the judgment result is yes, determining the target metadata as the synchronized metadata corresponding to the current video frame; When the judgment result is no, determining the synchronized metadata corresponding to the previous video frame of the current video frame as the synchronized metadata corresponding to the current video frame.

3. The method for synchronizing flight equipment map data under the time window verification mechanism according to claim 2, characterized in that, The determining a target time window size parameter matching the current video frame includes: Calculating the network jitter parameter during transmission corresponding to all the video frames; Calculating the historical video frame-metadata matching success rate corresponding to all the historical time windows among all the video frames; Determining a target time window size parameter matching the current video frame according to the network jitter parameter during transmission, the historical video frame-metadata matching success rate, a preset minimum window size parameter, a maximum window size parameter, an initial window size parameter, a first weight parameter corresponding to the network jitter parameter during transmission, a first reference value corresponding to the network jitter parameter during transmission, a second weight parameter corresponding to the historical video frame-metadata matching success rate, and a second reference value corresponding to the historical video frame-metadata matching success rate; Among them, the first weight parameter is determined by the following method: Determine the transmission network conditions corresponding to all the video frames, and determine the first weight parameter corresponding to the network jitter parameter during transmission according to the transmission network conditions corresponding to all the video frames; the transmission network conditions include the speed of the transmission network and / or the stability of the transmission network.

4. The method for synchronizing flight equipment map data under the time window verification mechanism according to claim 3, wherein The calculating the network jitter parameters during transmission corresponding to all the video frames includes: Determine the acquisition time parameter corresponding to when the demultiplexing module acquires the video stream data, and determine the arrival time parameter corresponding to each video frame according to the acquisition time parameter; According to the arrival time parameters corresponding to all the video frames, calculate the arrival time interval parameters corresponding to each group of adjacent video frames among all the video frames; According to the sum of the arrival time interval parameters corresponding to all the adjacent video frames and the total number of groups corresponding to all the adjacent video frames, calculate the average arrival time interval parameter corresponding to all the adjacent video frames; According to the arrival time interval parameters corresponding to all the adjacent video frames and the average arrival time interval parameter, calculate the sum of the squared time interval deviations corresponding to all the adjacent video frames; According to the sum of the squared time interval deviations and the total number of groups, calculate the target average value corresponding to the sum of the squared time interval deviations, and perform a square root calculation on the target average value to obtain the time interval standard deviation corresponding to all the adjacent video frames, which is used as the network jitter parameter during transmission corresponding to all the video frames.

5. The method for synchronizing flight device map data under the time window verification mechanism according to claim 3, characterized in that, The calculating the historical video frame-metadata matching success rate corresponding to all historical time windows among all the video frames includes: Determine the historical video frame-metadata sub-matching success rate corresponding to each historical time window among all the video frames; the historical video frame-metadata sub-matching success rate corresponding to each historical time window is used to indicate the probability of matching the corresponding synchronization metadata for all historical video frames in this historical time window; According to the historical video frame-metadata sub-matching success rates corresponding to all the historical time windows, determine all the target historical time windows among all the historical time windows where the historical video frame-metadata sub-matching success rate is greater than or equal to a preset sub-matching success rate threshold; According to the ratio between the number of all the target historical time windows and the number of all the historical time windows, determine the historical video frame-metadata matching success rate corresponding to all the historical time windows among all the video frames.

6. The method for synchronizing flight device diagrams under the time window verification mechanism according to any one of claims 3-5, characterized in that The target time window size parameter matching the current video frame is determined by the following formula: W = max(W min , min(W max , W0 + K J *(J - J0) + K R *(R - R0))); Among them, W is the target time window size parameter, W min is the minimum window size parameter, W max is the maximum window size parameter, W0 is the initial window size parameter, K J is the first weight parameter, J is the network jitter parameter during transmission, J0 is the first reference value, K R is the second weight parameter, R is the historical video frame-metadata matching success rate, R0 is the second reference value.

7. The method for synchronizing flight equipment map numbers under the time window verification mechanism according to any one of claims 1-5, characterized in that The method further includes: All the video frames are transmitted to a preset rendering module through media tracks in a preset WebRTC transmission channel, and at the same time, synchronization metadata corresponding to all the video frames is transmitted to the rendering module through a data channel in the WebRTC transmission channel, so that the rendering module performs synchronous rendering on all the video frames and the synchronization metadata corresponding to all the video frames; the media track and the data channel share the same target communication component of the WebRTC transmission channel.

8. A flight equipment map number synchronization device under a time window verification mechanism, characterized in that The device includes: A demultiplexing module, configured to perform a demultiplexing operation on the video stream data of the flying device obtained in real time to obtain the demultiplexed data corresponding to the video stream data; the demultiplexed data includes a plurality of video frames and a frame timestamp corresponding to each video frame; A processing module, configured to perform a target processing operation on all the video frames to obtain a metadata set corresponding to all the video frames; A determination module, configured to determine the timestamp corresponding to each metadata in the metadata set; the target processing operation includes a target detection operation and / or an image enhancement calculation operation; A time window verification module, configured to perform a time window verification operation on all the video frames and all the metadata according to the frame timestamps corresponding to all the video frames and the timestamps corresponding to all the metadata, to obtain the synchronization metadata corresponding to all the video frames.

9. A flight device map number synchronization device under a time window verification mechanism, characterized in that The device includes: A memory storing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the flying device map data synchronization method under the time window verification mechanism according to any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which are used to execute the flying device map data synchronization method under the time window verification mechanism according to any one of claims 1-7 when the computer instructions are called.

Images