Large-data-volume and high-concurrency echo data playback method

Abstract

The invention relates to a large-data-volume high-concurrency echo data playback method, and belongs to the technical field of radar signal processing. Comprising the following two steps of: 1, controlling communication logic between a thread and an optical fiber analysis thread: realizing a synchronization mechanism among optical fiber data read by multiple threads through the control thread; the scheduling time of the control thread is controllable by analyzing high-level control words, namely message information and returning the message information to the control thread; step 2, a ping-pong pre-reading mechanism during data reading of the optical fiber analysis thread comprises pre-reading design of two cache regions; and data analysis and transmission between the two caches are realized through a ping-pong mechanism. Through multiple threads, the performance advantages of a hardware platform are fully exerted, cross-thread synchronization between optical fiber data analysis is ensured by adding a control thread, and the universality and the real-time performance of optical fiber analysis are improved through a ping-pong data pre-reading mechanism; and the authenticity of radar echo data playback and the solving efficiency of a reappearing problem are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a method for replaying echo data with a large amount of data and high concurrency, and belongs to the technical field of radar signal processing. Background Art

[0002] Nowadays, with the significant improvement in the degree of softwareization in radar development, the flexibility of signal processing in the radar backend system is also increasing; currently, when a CPU-based computing blade server performs backend signal processing through software, in addition to the main processing functions, data acquisition and data playback functions are also added. However, the current data playback function of backend signal processing cannot truly reproduce radar echo data because it receives multi-fiber data transmitted through optical fibers by DBF, requiring a high degree of parallelism between the data, that is, the high synchronization performance does not meet the standard. As a result, the collected DBF multi-beam data is played back through software, and it is difficult for the signal processing subsystem to accurately reproduce the historical processing scenario of the entire radar. Naturally, it is impossible to analyze and solve the problems that occurred before by replaying historical data. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for replaying echo data with a large amount of data and high concurrency in view of the above-mentioned deficiencies of the prior art. The echo data has the characteristics of large storage files, strong replay concurrency, and high real-time requirements. The ping-pong mechanism is used to read fiber data files to improve data fault tolerance; control instructions are issued by the working stage control thread to control the work of the fiber thread; the fiber thread responds to the control thread with the current fiber data status, thereby affecting the next control of the control thread to complete the closed loop; it solves the problem that in practical applications, due to the unsatisfactory synchronization performance of the existing backend signal processing, the radar echo data cannot be truly reproduced, and it is impossible to solve historical faults by repeatedly replaying historical data.

[0004] The present invention realizes the above-mentioned invention purpose through the following technical solutions: A method for replaying echo data with a large amount of data and high concurrency is characterized in that it is realized through the following steps: Step 1: The communication logic between the control thread and the fiber parsing thread; The designed communication logic between the control thread and the fiber parsing thread includes: 1.1 Implement the synchronization mechanism between multi-threaded reading of fiber data through the control thread; 1.2 Return the parsed high-level control word, that is, the message information, to the control thread to make the scheduling time of the control thread controllable and closer to simulating the real scenario; Step 2: The ping-pong pre-reading mechanism when the fiber parsing thread reads data; The designed ping-pong pre-reading mechanism when the fiber parsing thread reads data includes: 2.1 Two - buffer pre - reading design; 2.2 Data parsing and sending are realized between the two buffers through the ping - pong mechanism.

[0005] The data replay program flow is as follows: 1) Program initialization: According to the configuration file, start the corresponding number of optical fiber parsing threads and a control thread based on the number of optical fiber files to be replayed. 2) Execute the control thread to read the data file once and obtain all resident control information. 3) Synchronously execute multiple optical fiber parsing threads. Inside the thread, use the ping - pong mechanism to read the optical fiber data file to improve data fault tolerance. 4) Start data replay. The control thread issues control instructions to control the work of the optical fiber threads. The optical fiber threads respond to the control thread with the current optical fiber data status, thereby affecting the subsequent control of the control thread to complete the closed - loop. 5) After all files are replayed, choose to stop working or start over. Starting over means repeating step 4).

[0006] Furthermore, for a method of echo data replay with large data volume and high concurrency as described in claim 1, its feature is that: 1.1 in step one realizes the synchronization mechanism between multi - thread reading of optical fiber data through the control thread. The specific operation is to first initialize the program, obtain the number of optical fiber files to be replayed according to the configuration file, and start the corresponding number of optical fiber parsing threads and a control thread.

[0007] Furthermore, for a method of echo data replay with large data volume and high concurrency as described in claim 1, its feature is that: 1.2 in step one returns the parsed high - level control word, that is, the message information, to the control thread. The specific operation is to execute the control thread, read the data file once, and obtain all resident control information.

[0008] Furthermore, for a method of echo data replay with large data volume and high concurrency as described in claim 1, its feature is that: in step two, synchronously execute multiple optical fiber parsing threads, and inside the thread, use the ping - pong mechanism to read the optical fiber data file to improve data fault tolerance.

[0009] Furthermore, for a method of echo data replay with large data volume and high concurrency as described in claim 1, its feature is that the specific operation of realizing data parsing and sending between the two buffers in step two through the ping - pong mechanism is: use buffers of the same size to alternately obtain and parse the optical fiber data. When the data in one buffer is sent and the other buffer starts to be sent, the current buffer re - parses the optical fiber data. This mode alternates continuously to realize continuous data parsing and sending, improving the overall data processing efficiency.

[0010] The beneficial effects of the present invention compared with the prior art are as follows: A method for replaying echo data with a large amount of data and high concurrency. The fiber optic data is read in a ping-pong manner through a ping-pong pre-reading mechanism. On the one hand, the real-time performance of data playback is improved, and thus the real-time performance of the entire radar scenario is greatly improved. On the other hand, the dependence of the replay program on the data format is eliminated, and the generality is improved. By controlling the threads, a high degree of synchronization between multiple fiber optic parsing threads is achieved. When applied to a certain type of radar, compared with the existing data playback means, it is proved that by controlling the threads to control the playback between multiple fiber optic parsing threads, the synchronization consistency is good, and the processing state at the backend at the data acquisition moment can be better reproduced, and the radar echo data can be perfectly and truly reproduced.

[0011] Through the actual measurement of the replay data experiment, when synchronously replaying the echo data input by four fiber optics on the FT2000+ cpu computing blade, the data transmission bandwidth can reach 2.4 Gbit / s, which is basically the same as the echo data bandwidth of DBF backhaul in the entire machine environment. At the same time, the CPU load is about 75%, and the replay program can also run stably, proving that the quality of the replay program of the method for replaying echo data with a large amount of data and high concurrency of the present invention fully meets the actual application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is the flowchart of the replay program of the present invention; Figure 2 It is the schematic diagram of the signal processing interface of the present invention; Figure 3 It is the flowchart of the control thread of the present invention; Figure 4 It is the flowchart of the fiber optic thread of the present invention; Figure 5 It is the parameter diagram of the FT2000+ CPU; Figure 6 It is the bandwidth actual measurement result diagram of the replay program of the present invention; Figure 7 It is the usage situation diagram of the FT2000+ CPU of the actual measurement replay program of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following further describes in detail the specific implementation manners of the method for replaying echo data with a large amount of data and high concurrency of the present invention in conjunction with the drawings (see Figure 1-7 ): A method for replaying echo data with a large amount of data and high concurrency, characterized in that it includes the following steps: Step 1, the communication logic between the control thread and the fiber optic parsing thread; Step 2, the ping-pong pre-reading mechanism when the fiber optic parsing thread reads data; The communication logic between the control thread and the fiber optic parsing thread designed in Step 1 includes: 1.1 Implement a synchronization mechanism for multi-threaded reading of fiber optic data through the control thread; 1.2 Return the parsed advanced control word, i.e., the message information, to the control thread to make the scheduling time of the control thread controllable and closer to simulating the real scenario; The ping-pong pre-reading mechanism when the fiber optic parsing thread designed in Step 2 reads data includes: 2.1 Pre-reading design with two buffer areas; 2.2 Implement data parsing and sending between the two buffers through the ping-pong mechanism (See Figure 1 ) Figure 1 This is the flowchart of the replay program of the present invention.

[0014] (See Figure 2 ) Figure 2 This is the schematic diagram of the signal processing interface of the present invention; after passing through the timer and the TR component, the DBF will provide echo information and echo data to the signal processing as the original data; the main line provides control parameters for the signal processing to control the signal processing to output traces and data acquisition; the signal processing will send the processed trace information to the data processing; under the control of the main display, the signal processing will also send the original data sent by the DBF to the data acquisition for storage.

[0015] (See Figure 3 ) Figure 3 This is the flowchart of the control thread of the present invention, as follows: Step 1: Program initialization. Start the corresponding number of fiber optic parsing threads and a control thread according to the number of fiber optic files to be replayed obtained from the configuration file; in the initialization stage, first obtain the number of fiber optics to be replayed to confirm the number of fiber optic states to be analyzed; Step 2: Execute the control thread to read the data file once to obtain all the resident control information; read the advanced control word file to obtain the resident information and control information required for replay residency; Step 3: Synchronously execute multiple fiber optic parsing threads. The threads internally use the ping-pong mechanism to read the fiber optic data files to improve data fault tolerance; enter the working stage, first obtain various resident information of the current residency, including the residency number, working mode, residency time, and cycle time; Step 4: Start data replay. The control thread issues control instructions to control the operation of the optical fiber thread. The optical fiber thread responds to the control thread with the current optical fiber data status, thereby affecting the subsequent control of the control thread and completing the closed loop. After Step 3 is completed, utilize the high transmission efficiency feature of UDP (User Datagram Protocol) to access the status of all used optical fibers, and simultaneously issue an instruction on whether to send the previous dwell data. At this time, controlling the transmission of optical fiber data is to ensure that the dwell control information is sent before the optical fiber data. Step 5: After all file replays are completed, select to stop working or restart. Restarting means repeating Step 4. Judge based on the status information fed back by each optical fiber thread. If there is a situation where the packets of a dwell on a certain optical fiber are not complete, record this information and enter the discrimination of the next dwell. Otherwise, enter the working mode discrimination. In the working mode discrimination, if it does not belong to the required working mode, enter the discrimination of the next dwell. Otherwise, count the wave position sequence code of the corresponding working mode and perform a delay according to the corresponding dwell number. Finally, send out the advanced control word information after various discriminations and enter the discrimination of the next dwell.

[0016] (See Figure 4 ) Figure 4 The flowchart of the optical fiber thread of the present invention is as follows: Step 1: In the initialization stage, first read the corresponding optical fiber data file. The reading adopts the ping-pong mechanism and reads 100 megabytes of data each time. Store a large piece of data in the cache at different times in multiple times to reduce the memory consumption of the process. Each time the optical fiber data file is read, first offset the file pointer 40,000 bytes forward to prevent the data packet from being disassembled. Here, 40,000 bytes are determined according to the data packet with the largest number of bytes.

[0017] Step 2: The parsed data packets are stored in a list for convenient subsequent retrieval.

[0018] Step 3: Enter the working stage. Before receiving the control instruction, first check whether the ping-pong cache needs to be updated to prevent misjudgment due to overly old detection data.

[0019] Step 4: After receiving the control instruction, select whether to send the previous dwell data packet according to the instruction information, judge whether the current dwell data is complete on the current optical fiber and respond, and at the same time parse out the optical fiber content involved in the control information and send it to the backend for processing.

[0020] Step 5: Repeatedly execute Step 4 to complete the parsing and sending of the entire optical fiber data file, that is, complete the data replay process.

[0021] (See Figure 5 ) Figure 5 is the parameter diagram of FT2000+CPU; Architecture: aarch64 and CPU operating mode: 64-bit, indicating that the system is an ARM 64-bit processor; Byte order: Little Endian indicates that the system byte order is in little endian mode; CPU: 64 indicates that the system has 64 logical CPUs; Online CPU list: "0 - 63" indicates that the 64 CPUs are numbered from 0 to 63; Number of threads per core: 1 indicates that the system does not enable hyper-threading technology; Number of cores per socket: 4 indicates that each processor socket in the system has 4 physical cores; Sockets: 16 indicates that the system has a total of 16 processor sockets; NUMA nodes: 8 indicates that the system has 8 NUMA nodes; BogoMIPS: 100.00 indicates that the system CPU can execute approximately 100,000,000 instructions per second; NUMA node 0 CPU: 0 - 7 indicates that the first NUMA node contains CPU cores numbered from 0 to 7; NUMA node 1 CPU: 8 - 15 indicates that the second NUMA node contains CPU cores numbered from 8 to 15; NUMA node 2 CPU: 16 - 23 indicates that the third NUMA node contains CPU cores numbered from 16 to 23; NUMA node 3 CPU: 24 - 31 indicates that the fourth NUMA node contains CPU cores numbered from 24 to 31; NUMA node 4 CPU: 32 - 39 indicates that the fifth NUMA node contains CPU cores numbered from 32 to 39; NUMA node 5 CPU: 40 - 47 indicates that the sixth NUMA node contains CPU cores numbered from 40 to 47; NUMA node 6 CPU: 48 - 55 indicates that the seventh NUMA node contains CPU cores numbered from 48 to 55; NUMA node 7 CPU: 56 - 63 indicates that the eighth NUMA node contains CPU cores numbered from 56 to 63.

[0022] (See Figure 6 ) Figure 6 This is the measured bandwidth result graph of the replay program of the present invention. The replay bandwidth per fiber at the low wavelength is about 2650 MB / s, and the retransmission bandwidth per fiber at the high wavelength is about 340 MB / s. By adopting the parsing of the advanced control word, that is, the message information is returned to the control thread, the data replay rate control in different working modes is realized; the ping-pong mechanism is used to realize data parsing and sending, improving the stability of the replay rate.

[0023] (See Figure 7 ) Figure 7 This is a usage diagram of the FT2000+ CPU in the actual replay program of the present invention. As Figure 7 shown, it is the CPU occupancy when 4 optical fibers are replayed. By adopting a control thread to implement the synchronization mechanism between multi-threaded reading of optical fiber data, the occupancy of all CPU cores where the optical fibers are located is stabilized at 50% to 75%, improving the stability of resource scheduling during data replay of the program.

[0024] For this method of echo data replay with large data volume and high concurrency, to improve the efficiency of data file parsing, a multi-threaded design concept is adopted to give full play to the performance advantages of the hardware platform; a control thread is added to ensure cross-thread synchronization between optical fiber data parsing; a ping-pong data pre-reading mechanism is adopted to improve the versatility and real-time performance of optical fiber parsing; therefore, the authenticity of radar echo data replay is significantly improved, and the efficiency of solving past problems corresponding to the reproduction of radar echo data is greatly improved; this method of echo data replay with large data volume and high concurrency is applied to a certain warning radar. The actual installation test shows that the synchronization performance of echo data with large data volume and high concurrency is strong, the processing effect at the backend during data acquisition is completely reproduced, and the replay effect is good.

[0025] The above are only the preferred embodiments of the present invention. The above examples do not impose any formal restrictions on the essence of the present invention. Any simple modification or deformation made by those of ordinary skill in the art in the technical field after reading this specification based on the technical essence of the present invention, and any equivalent embodiments that may be modified or decorated with the above-disclosed technical content into equivalent variations still fall within the scope of the technical solution of the present invention without departing from the essence and scope of the present invention.

Claims

1. A method for replaying echo data with a large amount of data and high concurrency, characterized in that, It is achieved by including the following steps: Step 1, the communication logic between the control thread and the optical fiber parsing thread; The designed communication logic between the control thread and the optical fiber parsing thread includes: 1.1 Implement the synchronization mechanism between multi-threaded reading of optical fiber data through the control thread; 1.2 Return the parsed high-level control word, i.e., the message information, to the control thread to make the scheduling time of the control thread controllable and closer to simulating the real scenario; Step 2, the ping-pong pre-reading mechanism when the optical fiber parsing thread reads data; The designed ping-pong pre-reading mechanism when the optical fiber parsing thread reads data includes: 2.1 The pre-reading design of two buffer areas; 2.2 The data parsing and sending are realized between the two buffers through the ping-pong mechanism.

2. The echo data replay method with a large amount of data and high concurrency according to claim 1, characterized in that: For 1.1 in Step 1 to implement the synchronization mechanism between multi-threaded reading of optical fiber data through the control thread, the specific operation is as follows: First, initialize the program, obtain the number of optical fiber files to be replayed according to the configuration file, and start the corresponding number of optical fiber parsing threads and a control thread.

3. A method for replaying echo data with a large amount of data and high concurrency according to claim 1, characterized in that: For 1.2 in Step 1 to return the parsed high-level control word, i.e., the message information, to the control thread, the specific operation is as follows: Execute the control thread, read the data file once, and obtain all the resident control information.

4. A method for replaying echo data with a large amount of data and high concurrency according to claim 1, characterized in that: For Step 2, multiple optical fiber parsing threads are executed synchronously, and the ping-pong mechanism is used inside the thread to read the optical fiber data file to improve data fault tolerance.

5. A method for replaying echo data with a large amount of data and high concurrency according to claim 1, characterized in that: The specific operation of realizing data parsing and sending between the two buffers in Step 2 through the ping-pong mechanism is as follows: Use buffer areas of the same size to alternately obtain and parse the optical fiber data. When the data in one buffer area is sent and the data sending of the other buffer area starts, the current buffer area re-parses the optical fiber data; this mode alternates continuously to realize the continuous parsing and sending of data and improve the overall data processing efficiency.

Images