A multi-channel satellite frequency point file processing method and system based on threshold optimization

Abstract

The application discloses a multi-channel satellite frequency point file processing method and system based on threshold optimization, relates to the technical field of satellite communication data processing, and comprises the following steps: calculating the burst position and burst length of a main channel file; counting the burst quantity in a historical main channel file, and calculating a threshold according to the burst quantity in the historical main channel file; if the burst quantity of the main channel file is smaller than the threshold, a low-burst processing mode is selected, otherwise, a high-burst processing mode is selected; and according to the selected mode, the data corresponding to the burst is read from a main auxiliary channel file and processed. The dynamic threshold is estimated, and the reading mode is selected according to the dynamic threshold, which is suitable for efficiently processing the burst data in the multi-channel satellite frequency point file, and solves the problems of burstiness, low-delay requirement of data access and accurate reading according to position in the satellite positioning scene.

Description

Technical Field

[0001] This invention relates to the field of satellite communication data processing technology, and in particular to a multi-channel satellite frequency point file processing method and system based on threshold optimization. Background Technology

[0002] In the fields of 3D GIS (Geographic Information System) terrain visualization, large-scale data synchronization, and file transfer technology, existing technologies mainly revolve around how to efficiently organize, schedule, and load massive data files. However, in scenarios requiring high real-time performance, such as satellite positioning, the core challenge lies in quickly locating and loading the necessary data, while overcoming issues such as network I / O latency, mismatch between file size and transmission bandwidth, and low efficiency in loading large files into memory.

[0003] Existing solutions include creating pre-indexed files to accelerate data location, dynamically adjusting file sizes based on network conditions to optimize transmission, and improving large file loading efficiency through file segmentation and message queuing technologies.

[0004] However, while pre-indexing schemes reduce database query latency, they rely on pre-created static index files. In satellite positioning scenarios, hotspots or sudden access areas of observation data are dynamic and unpredictable, making it impossible to fully cover them in advance with static indexes, resulting in insufficient real-time performance of this method.

[0005] While dynamically adjusting file sizes optimizes overall file transfer efficiency, its operation of merging small files to the optimal size is unsuitable for scenarios requiring immediate access to specific locations within a file. This is because the merging process requires time and memory for file integration, which conflicts with the instantaneous response requirements of satellite positioning scenarios; its optimization goal is transfer efficiency, not access accuracy.

[0006] While solutions employing file splitting and message queuing improve the throughput of loading large files, their splitting strategies are based solely on file size, neglecting the inherent semantics or spatial distribution of the data. This makes them unsuitable for accurately reading relevant portions of a file based on bursty locations. Furthermore, their design fails to adequately address the handling of numerous small files.

[0007] Therefore, existing solutions have significant limitations in dealing with the suddenness of data access, strict low-latency requirements, and the need to read data based on a specific location (rather than the entire file) that are unique to satellite positioning scenarios. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a multi-channel satellite frequency point file processing method and system based on threshold optimization. The method estimates a dynamic threshold and selects a reading mode based on the dynamic threshold, making it suitable for efficiently processing bursty data in multi-channel satellite frequency point files. This solves the problems of bursty data access, low latency requirements, and accurate location-based reading in satellite positioning scenarios.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a multi-channel satellite frequency point file processing method based on threshold optimization, comprising:

[0011] Calculate the burst position and burst length of the main channel file;

[0012] The number of bursts in the historical main channel file is counted, and a threshold is calculated based on the number of bursts in the historical main channel file. If the number of bursts in the main channel file is less than the threshold, the low burst processing mode is selected; otherwise, the high burst processing mode is selected.

[0013] Based on the selected mode, the corresponding burst data is read from the primary and secondary channel files and processed.

[0014] As an alternative implementation, the burst location includes a file offset, which is represented in bits; the burst length includes the number of bits.

[0015] As an alternative implementation method, the formula for calculating the threshold T is:

[0016]

[0017] in, This represents the historical average transmission cost. This represents the historical average number of sudden outbreaks; The network transmission cost for each burst of content; Cost per disk seek; This refers to network latency costs.

[0018] As an alternative implementation, when the threshold is less than 0, the high burst mode is selected.

[0019] As an alternative implementation method, the low burst mode is as follows: based on the file offset and burst length of each burst, the content of the corresponding burst is directly read from the primary and secondary channel files in the disk array each time and stored in memory.

[0020] As an alternative implementation method, the high burst mode is as follows: all burst locations and burst lengths are stored as burst information files; the primary and secondary channel files are read into memory at once, and a hash table of burst locations and burst content is established based on the burst information files to index the burst content.

[0021] Secondly, the present invention provides a multi-channel satellite frequency point file processing system based on threshold optimization, comprising:

[0022] The burst detection module is configured to calculate the burst position and burst length of the main channel file;

[0023] The mode selection module is configured to count the number of bursts in the historical main channel file and calculate a threshold based on the number of bursts in the historical main channel file; if the number of bursts in the main channel file is less than the threshold, the low burst processing mode is selected, otherwise the high burst processing mode is selected.

[0024] The positioning processing module is configured to read and process the corresponding burst data from the primary and secondary channel files according to the selected mode.

[0025] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0026] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0027] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] This invention proposes a multi-channel satellite frequency point file processing method and system based on threshold optimization. It breaks away from the idea of ​​simply optimizing transmission or establishing static indexes, and designs a novel data organization and scheduling method that can dynamically and efficiently support the rapid location and loading of relevant data segments in the file. It is suitable for efficiently processing bursty data in multi-channel satellite frequency point files, solving the problems of bursty data access, low latency requirements, and accurate location reading in satellite positioning scenarios, and filling the gap in the existing technology in terms of accurate data access capabilities.

[0030] The method of this invention estimates a dynamic threshold based on the number of historical bursts, and determines whether to use "whole file reading mode" (low burst mode) or "read by offset mode" (high burst mode) based on the dynamic threshold. The threshold is not a fixed value, but is calculated based on one or more system dynamic parameters, reflecting the system's adaptive capability.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0033] Figure 1 This is a flowchart of a multi-channel satellite frequency point file processing method based on threshold optimization provided in Embodiment 1 of the present invention;

[0034] Figure 2 This is a diagram of the multi-channel satellite frequency point file processing system architecture based on threshold optimization provided in Embodiment 2 of the present invention;

[0035] Figure 3 This is a schematic diagram of a multi-channel satellite frequency point file processing system based on threshold optimization provided in Embodiment 2 of the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0040] Example 1

[0041] This embodiment provides a multi-channel satellite frequency point file processing method based on threshold optimization, such as... Figure 1 As shown, it includes:

[0042] Calculate the burst position and burst length of the main channel file;

[0043] The number of bursts in the historical main channel file is counted, and a threshold is calculated based on the number of bursts in the historical main channel file. If the number of bursts in the main channel file is less than the threshold, the low burst processing mode is selected; otherwise, the high burst processing mode is selected.

[0044] Based on the selected mode, the corresponding burst data is read from the primary and secondary channel files and processed.

[0045] In this embodiment, after receiving the main channel file, a signal processing algorithm (such as time domain analysis or frequency domain analysis) is used to calculate burst information, which includes burst location and burst length.

[0046] The burst location includes the file offset, which is represented in bits; the burst length includes the number of bits.

[0047] This involves storing burst information as a data structure, such as a list of key-value pairs containing file offsets and burst lengths.

[0048] The primary and secondary channel files are the same size.

[0049] In this embodiment, the number of bursts (i.e., the number of bursts) in the historical main channel file is counted; based on the number of bursts in the historical main channel file, a threshold T is calculated:

[0050]

[0051] in, The historical average transfer cost for the entire file; This represents the historical average number of sudden outbreaks; The network transmission cost for each burst of content; Cost per disk seek; This refers to network latency costs (such as the latency of establishing a TCP connection).

[0052] Among them, the selection , , The purpose is to estimate the overall transmission cost by considering that each time a burst of content is read from the disk array, it involves one disk seek, one TCP connection establishment, and one network transfer.

[0053] The specific values ​​for these parameters need to be determined based on the performance of actual physical machines, such as the cost of a single disk seek. Traditional HDD (Hard Disk Drive): 3-5ms (not 10ms), Enterprise-grade SSD (Solid State Drive): 0.08-0.15ms, NVMe SSD (Non-Volatile Memory Express Solid State Drive): 0.01-0.05ms (almost negligible).

[0054] For network latency costs and the network transmission cost of each piece of breaking content It needs to be calculated based on the actual bandwidth and topology.

[0055] In this embodiment, if the number of bursts in the main channel file is less than the threshold T, a low burst processing mode is selected; otherwise, a high burst processing mode is selected.

[0056] Since the value of T may be less than 0, but the number of bursts is always greater than 0, the high burst mode is selected when T is less than 0.

[0057] Its physical meaning is: if the cost of transmitting unnecessary content is higher than the cost of transmitting only burst content, then only burst content should be transmitted, which is the low burst processing mode; conversely, the transmission of some unnecessary content should be tolerated, and the entire file should be transmitted, which is the high burst processing mode.

[0058] Specifically:

[0059] (1) Low-burst mode:

[0060] Based on the file offset and burst length of each burst, the content of the corresponding burst is read directly from the primary and secondary channel files in the disk array each time and stored in memory.

[0061] (2) High-suddenness mode:

[0062] Store all burst information (file offset and burst length) as a burst information file (e.g., JSON or binary format).

[0063] The primary and secondary channel files are read into memory at once, and a hash table of burst location and burst content is built based on the burst information file to facilitate the program's rapid indexing of burst content.

[0064] In this embodiment, the read burst data is finally analyzed, such as feature extraction and data decoding, to generate processing results, which are then transmitted to subsequent system modules or storage devices.

[0065] In this embodiment, the method estimates a dynamic threshold based on the number of historical bursts, and then determines whether to use "whole file read mode" (low burst mode) or "read by offset mode" (high burst mode) based on the dynamic threshold. The threshold is not a fixed value, but is calculated based on one or more system dynamic parameters, reflecting the system's adaptive capability.

[0066] Verification example.

[0067] Comparison of modern memory seek times:

[0068] Traditional HDD: 3-5ms (not 10ms);

[0069] Enterprise-grade SSDs: 0.08-0.15ms;

[0070] NVMe SSD: 0.01-0.05ms (almost negligible).

[0071] Assume the file size per satellite channel is 1GB;

[0072] Network transmission bandwidth: 5MB / s;

[0073] Read time: 204.8 seconds;

[0074] Cost quantification: Assume 1 second = 1000 units of cost;

[0075] =204.8 × 1000 = 204,800;

[0076] =4 (estimated based on enterprise-grade HDD array, approximately 3-4ms);

[0077] =0.5 (Network round-trip latency cost, approximately 0.5ms);

[0078] =1000 (The historical average number of bursts is 1000, and the size of each burst is 2kb).

[0079] =2 / 1024 / 5*1000=0.39 (The time required to transmit a burst of content is 0.39 milliseconds).

[0080] Thus, the estimated value of T is 4890, meaning that the cost of transmitting unnecessary content is 4890 higher than the cost of transmitting burst content. Clearly, the preferred option is to transmit only burst content, i.e., the high-burst mode.

[0081] When a file is only 10MB in size, T = -3227. This means that transmitting only bursty content would be too costly due to the large number of network connections, so it's better to read the entire file directly, which is the low-burst mode.

[0082] Experiments showed that existing technologies only support high-burst modes and lack dynamic adjustment capabilities. Compared to the method in this embodiment, when the satellite file size is 10Mb, the proposed method takes 2.62s, while the existing technology takes 5.89s. The proposed method takes only 45% of the time of the proposed method.

[0083] To address the need for low-latency, location-accurate access to burst data in frequency point files during satellite positioning scenarios, and to overcome the limitations of static indexing and fixed file processing modes, this embodiment proposes a multi-channel satellite frequency point file-level burst data reading and processing method. It calculates dynamic thresholds using historical data and selects between low-burst (accurate reading by offset) or high-burst (whole file reading + hash indexing) modes based on burst length / quantity, adapting to burst data processing of multi-channel frequency point files. This method can be applied to the storage, reading, and data scheduling of satellite frequency point files, focusing on file-level data access optimization, and is suitable for scenarios with high real-time and accuracy requirements, such as satellite positioning.

[0084] Example 2

[0085] This embodiment provides a multi-channel satellite frequency point file processing system based on threshold optimization, such as... Figures 2-3 As shown, it includes:

[0086] The burst detection module is configured to calculate the burst position and burst length of the main channel file;

[0087] The mode selection module is configured to count the number of bursts in the historical main channel file and calculate a threshold based on the number of bursts in the historical main channel file; if the number of bursts in the main channel file is less than the threshold, the low burst processing mode is selected, otherwise the high burst processing mode is selected.

[0088] The positioning processing module is configured to read and process the corresponding burst data from the primary and secondary channel files according to the selected mode.

[0089] In this embodiment, a message middleware is also included, which is used to transmit burst information or file addresses between the burst detection module and the location processing module, and supports communication protocols such as RabbitMQ and Kafka.

[0090] Specifically:

[0091] In low burst processing mode, the burst detection module sends the file offset and burst length of each burst one by one through the message middleware. The positioning and processing module receives each message, reads the corresponding burst data from the main and auxiliary channel files, and processes it (such as demodulation, decoding, etc.) to generate the processing result. Finally, the result is output to the subsequent system module or storage device.

[0092] In high burst processing mode, the burst detection module stores all burst information (file offset and burst length) as a burst information file (e.g., JSON or binary format) and generates the file address;

[0093] The burst detection module sends the file address of the burst information file through the message middleware;

[0094] The positioning and processing module receives the file address, reads the burst information file into memory at once, and reads all burst data in the main and auxiliary channel files in batches according to the content of the burst information file. It then processes the read data, generates processing results, and finally outputs the results to subsequent system modules or storage devices.

[0095] This embodiment also includes a hard disk storage module for persistently storing files and intermediate results. Hard disk storage space and services are simultaneously provided to multiple servers or computers via a network (rather than internal computer data cables).

[0096] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0097] In further embodiments, the following is also provided:

[0098] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0099] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0100] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0101] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0102] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0103] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0104] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0105] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0106] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0107] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0108] It should be noted that all data acquisition is conducted in accordance with laws and regulations and with user consent, and the data is used legally.

[0109] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multi-channel satellite frequency point file processing method based on threshold optimization, characterized in that, include: Calculate the burst position and burst length of the main channel file; The number of bursts in the historical main channel file is counted, and a threshold is calculated based on the number of bursts in the historical main channel file. If the number of bursts in the main channel file is less than the threshold, the low burst processing mode is selected; otherwise, the high burst processing mode is selected. The formula for calculating the threshold T is as follows: in, This represents the historical average transmission cost. This represents the historical average number of sudden outbreaks; The network transmission cost for each burst of content; Cost per disk seek; Cost of network latency; Based on the selected mode, read the corresponding burst data from the primary and secondary channel files and process it; The low burst mode is as follows: based on the file offset and burst length of each burst, the content of the corresponding burst is directly read from the primary and secondary channel files in the disk array each time and stored in memory; The high-burst mode is as follows: all burst locations and burst lengths are stored as burst information files; the primary and secondary channel files are read into memory at once, and a hash table of burst locations and burst content is built based on the burst information files to index the burst content.

2. The multi-channel satellite frequency point file processing method based on threshold optimization as described in claim 1, characterized in that, The burst location includes a file offset, which is represented in bits; the burst length includes the number of bits.

3. The multi-channel satellite frequency point file processing method based on threshold optimization as described in claim 1, characterized in that, When the threshold is less than 0, select the high burst mode.

4. A multi-channel satellite frequency point file processing system based on threshold optimization, characterized in that, include: The burst detection module is configured to calculate the burst position and burst length of the main channel file; The mode selection module is configured to count the number of bursts in the historical main channel file and calculate a threshold based on the number of bursts in the historical main channel file; if the number of bursts in the main channel file is less than the threshold, the low burst processing mode is selected, otherwise the high burst processing mode is selected. The formula for calculating the threshold T is as follows: in, This represents the historical average transmission cost. This represents the historical average number of sudden outbreaks; The network transmission cost for each burst of content; Cost per disk seek; Cost of network latency; The positioning processing module is configured to read and process the corresponding burst data from the primary and secondary channel files according to the selected mode. The low burst mode is as follows: based on the file offset and burst length of each burst, the content of the corresponding burst is directly read from the primary and secondary channel files in the disk array each time and stored in memory; The high-burst mode is as follows: all burst locations and burst lengths are stored as burst information files; the primary and secondary channel files are read into memory at once, and a hash table of burst locations and burst content is built based on the burst information files to index the burst content.

5. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-3.

7. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-3.

Images