Data processing method, forwarding device and data transmission system

By using the file reader and write manager in the data forwarding device to create a file reader and write purpose in the write and read purpose, real-time data collection, storage and forwarding is realized, and the problem of inefficiency in traditional data forwarding methods is solved and the efficiency of data forwarding is improved. and real-time, meeting the needs of high timeliness.

CN118488095BActive Publication Date: 2024-11-19BEIJING RONGWEI TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410940001.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-19
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

Traditional data forwarding methods are inefficient and difficult to meet the demand for high timeliness. Especially in remote sensing satellite data transmission, due to the mismatch of the transmitter and receiver transmission rate and inconsistent communication protocols, the data forwarding process is inefficient.

Method used

Using forwarding equipment including data collector, file reader and write manager and data sender, the file reader and write reader are created through the file reader and write manager to realize real-time data collection, storage and forwarding, and data collection is carried out simultaneously. , storage and send, use file mapping technology to increase cache queue capacity, reduce storage pressure and forwarding delay.

Benefits of technology

It improves the efficiency and real-time nature of the data forwarding process, realizes the synchronization of data collection, storage and transmission, reduces forwarding delay, and meets the needs of high timeliness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118488095B_ABST
    Figure CN118488095B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of data transmission, and provides a data processing method, a forwarding device and a data transmission system. The method comprises: after establishing a data acquisition thread, creating a write destination file reader through a file read-write manager; collecting and parsing target data sent by a transmitter through a data collector to obtain parsed data; writing the parsed data into a data file through the write destination file reader until the data file is full or the data acquisition thread ends, and reclaiming the write destination file reader through the file read-write manager; after establishing a data forwarding thread, creating a read destination file reader through the file read-write manager; reading the parsed data in the data file through the read destination file reader, and sending the parsed data to a receiver after packaging through a data transmitter; until the parsed data is read or the data forwarding thread ends, the read destination file reader is reclaimed through the file read-write manager. The efficiency of the data forwarding process is effectively improved, and the real-time performance is stronger.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data communication and data transmission, and in particular to a data processing method, forwarding equipment and a data transmission system. Background Art

[0002] In the application layer data communication scenario, due to the mismatch of the sending and receiving bit rates between the transmitter and the receiver and the inconsistent communication protocols, a forwarding device is usually required to realize data transmission from the transmitter to the receiver through data forwarding. For example, in the field of remote sensing satellites, the ground station collects remote sensing data to the ground baseband device, and the single-channel acquisition bit rate of remote sensing data can reach 5Gbps. After the acquisition, the ground station needs to transmit the data to the remote sensing data receiving device in other areas through the Internet or a private network. However, due to the communication distance or other reasons, the commonly used Internet or private network communication bandwidth is relatively low, and the sending protocol of the ground station baseband device and the receiving protocol of the remote sensing data receiving device do not match, so data communication cannot be completed directly.

[0003] At present, the commonly used data forwarding method is that the forwarder stores the received data in the form of files to the disk according to the specified rules during collection, and starts the forwarding of this file after the storage of a single file is completed. In this way, the forwarding start time will be later than the collection start time, especially under the requirements of large file storage. The forwarding start time will lag far behind the collection start time, resulting in inefficient data forwarding process and difficulty in meeting high-time forwarding needs. Summary of the invention

[0004] The present invention provides a data processing method, a forwarding device and a data transmission system, which are used to solve the defects of the traditional data forwarding method that the forwarding process is inefficient and it is difficult to meet the forwarding requirements with high timeliness.

[0005] In one aspect, the present invention provides a data processing method, which is performed by a forwarding device including a data collector, a file read-write manager, and a data transmitter, and the method comprises:

[0006] After establishing the data collection thread, creating a file reader / writer for the writing purpose through the file reader / writer manager;

[0007] The target data sent by the transmitter is collected and analyzed by the data collector to obtain analyzed data;

[0008] The parsed data is written into a data file through the write destination file reader / writer until the data file is full or the data acquisition thread ends, and then the write destination file reader / writer is recycled through the file read / write manager;

[0009] After establishing the data forwarding thread, creating a file reader / writer for the read destination through the file reader / writer manager;

[0010] The parsed data in the data file is read by the read destination file reader / writer, and the parsed data read is packaged and sent to the receiver by the data transmitter;

[0011] Until the parsed data is read completely or the data forwarding thread ends, the file reader / writer of the read destination is recycled through the file reader / writer manager.

[0012] According to the data processing method provided by the present invention, the step of creating a file reader / writer for a writing destination through the file reader / writer manager includes:

[0013] When the file read / write manager receives the write destination reader / writer creation instruction initiated by the requester, it determines whether the target data file corresponding to the write destination reader / writer creation instruction already exists in the file reader / writer according to the file path corresponding to the write destination reader / writer creation instruction, and obtains a first judgment result;

[0014] If the first judgment result is that there is a file reader / writer, then judging whether there is a write operator in the file reader / writer to obtain a second judgment result;

[0015] If the second judgment result is that a write operator exists, an error prompt message is fed back to the user;

[0016] If the second judgment result is that there is no write operator, creating a write operator in the file reader;

[0017] If the first judgment result is that there is no file reader / writer, then creating a file reader / writer corresponding to the target data file, and creating a write operator in the file reader / writer;

[0018] Feeds the file reader / writer created with the write operator back to the requester.

[0019] According to the data processing method provided by the present invention, writing the parsed data into a data file through the write destination file reader / writer comprises:

[0020] After the write destination file reader receives the data write request initiated by the requester, determining whether the requester has the write permission;

[0021] After determining that the requester has write permission, determining a target cache space to be written;

[0022] The parsed data is written into a data file and stored in the target cache space.

[0023] According to the data processing method provided by the present invention, the determining of the target cache space to be written includes:

[0024] Determine whether the remaining space to be written in the current cache space mapped from the disk is sufficient;

[0025] If the space to be written is sufficient, the current cache space is used as the target cache space;

[0026] If the space to be written is insufficient, the data written into the current cache space is recorded to the disk through the file handle corresponding to the write destination file reader, and a new mapping cache space is established, and the new mapping cache space is used as the target cache space.

[0027] According to the data processing method provided by the present invention, after writing the parsed data into a data file and storing it in the target cache space, the method further includes:

[0028] When the size of the data file reaches the upper limit of the single file space, the file reader / writer of the current writing destination is recycled by the file reader / writer manager;

[0029] Identify the new data file and create a new file reader / writer for the write destination.

[0030] According to the data processing method provided by the present invention, the step of creating a read-destination file reader / writer through the file reader / writer manager includes:

[0031] When the file read / write manager receives a read destination reader / writer creation instruction initiated by the requester, it determines whether a file reader / writer already exists for the target data file corresponding to the read destination reader / writer creation instruction according to the file path corresponding to the read destination reader / writer creation instruction, and obtains a third determination result;

[0032] If the third judgment result is that a file reader / writer exists, creating a read operator in the current file reader / writer;

[0033] If the third judgment result is that there is no file reader / writer, then creating a file reader / writer corresponding to the target data file, and creating a read operator in the file reader / writer;

[0034] Feeds the file reader / writer that created the read operator back to the requester.

[0035] According to the data processing method provided by the present invention, the reading of the parsed data in the data file by the read destination file reader / writer comprises:

[0036] After the read destination file reader receives the data read instruction sent by the requester, determining whether the requester has the read permission;

[0037] After determining that the requester has the read permission, determining the read operator corresponding to the requester, and determining whether the parsed data in the current cache space mapped by the disk corresponding to the read operator has been read;

[0038] If it is determined that the parsed data in the current cache space has been read, a new mapping cache space is determined through the file handle corresponding to the read destination file reader, and the parsed data in the new mapping cache space is read until all the parsed data in the data file are read.

[0039] According to the data processing method provided by the present invention, the file reader / writer of the read destination is recycled by the file reader / writer manager, comprising:

[0040] Determine whether there is another user holding the file reader / writer of the reading target at the current moment;

[0041] If there is another user holding the file reader / writer of the read destination file at the current moment, only the read operator corresponding to the requester who initiated the data read instruction is deleted;

[0042] If no other user holds the target file reader / writer at the current moment, the target file reader / writer is deleted.

[0043] On the other hand, the present invention also provides a forwarding device, comprising: a data collector, a file read-write manager and a data transmitter;

[0044] The file read-write manager is used to create a file reader for writing after establishing a data acquisition thread;

[0045] The data collector is used to collect and analyze the target data sent by the transmitter to obtain analyzed data;

[0046] The file read / write manager is further used to write the parsed data into a data file at the write destination file reader / writer, and recycle the write destination file reader / writer when the data file is full or the data collection thread ends;

[0047] The file read / write manager is also used to create a read-destination file reader / writer after establishing a data forwarding thread; and to recycle the read-destination file reader / writer after the read-destination file reader / writer reads the parsed data in the data file and the parsed data is read or the data forwarding thread ends.

[0048] The data transmitter is used for packaging the read parsed data and sending it to the receiver.

[0049] On the other hand, the present invention further provides a data transmission system, comprising: a transmitter, a receiver and the above-mentioned forwarding device.

[0050] The data processing method, forwarding device and data transmission system provided by the present invention, after establishing a data acquisition thread, creates a write destination file reader / writer through a file read / write manager, collects and analyzes target data sent by a transmitter through the data collector to obtain analyzed data, and writes the analyzed data into a data file through the write destination file reader / writer until the data file is full or the data acquisition thread ends, and then reclaims the write destination file reader / writer through the file read / write manager;

[0051] After establishing the data forwarding thread, create the destination file reader / writer through the file reader / writer manager;

[0052] The parsed data in the data file is read by the destination file reader, and the parsed data is packaged and sent to the receiver by the data transmitter; until the parsed data is read or the data forwarding thread ends, the destination file reader is recycled by the file reader manager. The data collection thread and the data forwarding thread are synchronized during the entire data forwarding process, and the data collection, storage and sending processes can be realized synchronously, so that the efficiency of the data forwarding process is effectively improved and the forwarding process is more real-time. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a flow chart of a data processing method provided by an embodiment of the present invention;

[0055] Figure 2 It is a schematic diagram of the structure of the file read and write manager;

[0056] Figure 3 It is a flowchart of creating a file reader / writer for writing purpose;

[0057] Figure 4 It is a flowchart of writing parsed data into a data file through a destination file reader;

[0058] Figure 5 It is a flowchart of creating a reader / writer for a read destination file;

[0059] Figure 6 It is a schematic diagram of the process of reading the parsed data in the data file by reading the destination file reader;

[0060] Figure 7It is a schematic diagram of the principle of the entire process of data collection, synchronous storage and real-time forwarding by the file read-write manager;

[0061] Figure 8 is a schematic diagram of the structure of a forwarding device provided in an embodiment of the present invention;

[0062] Fig. 9 It is a structural diagram of a data transmission system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Combine the following Figures 1 to 9 Describe the details of the data processing method, forwarding device and data transmission system provided by the embodiments of the present invention.

[0065] Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present invention.

[0066] like Figure 1 As shown, the data processing method provided by the embodiment of the present invention can be executed by a forwarding device including a data collector, a file read-write manager, and a data transmitter. The method mainly includes the following steps:

[0067] Step 110: After establishing the data collection thread, create a file reader / writer for the writing destination through the file reader / writer manager.

[0068] In this embodiment, a file read / write manager can be established before the collection thread starts. The file read / write manager uses file mapping technology and large-capacity disk space to establish a huge cache queue. While caching, the collected target data is persisted to the disk and provides write and read operations to the outside. After the collection thread starts, the file reader / writer for the write destination can be created through the file read / write manager according to the current data collection task.

[0069] Step 120: The target data sent by the transmitter is collected and analyzed by the data collector to obtain analyzed data.

[0070] In practical applications, the data collector can continuously collect the target data sent by the transmitter from the network interface, and parse the target data according to the protocol to obtain parsed data.

[0071] Step 130: Write the parsed data into the data file through the write destination file reader until the data file is full or the data collection thread ends, and then recycle the write destination file reader through the file read-write manager.

[0072] In this embodiment, when it is detected that the data file is full or the data collection thread ends, the file reader / writer of the current writing destination can be recycled through the file reader / writer manager to reduce the data storage pressure.

[0073] It can be understood that the data acquisition part in this embodiment can be connected to a high-bitrate transmitter. It is an independent thread that can receive data sent by the transmitter in real time. The network layer can adopt TCP (Transmission Control Protocol), UDP (User Datagram Protocol) and other communication methods. At the same time, it is necessary to adapt the application layer communication protocol of the transmitter, complete the protocol analysis to achieve the purpose of data collection, and finally push the data to the file read and write manager for subsequent use.

[0074] Step 140: After the data forwarding thread is established, a destination file reader is created through the file reader manager.

[0075] In this embodiment, a destination file reader / writer may be established through a file reader / writer manager for this data forwarding task.

[0076] Step 150: Read the parsed data in the data file through the destination file reader, and package the parsed data and send it to the receiver through the data transmitter.

[0077] It is understandable that the destination file reader can continuously read and parse data from the data file, and package the read data according to the protocol through the data transmitter, and forward it to the receiver from the network interface.

[0078] Step 160: until the parsed data is read completely or the data forwarding thread ends, the destination file reader is recycled through the file reader manager.

[0079] In this embodiment, when it is detected that the data has been read or the data forwarding thread has ended, the created destination file reader / writer can be recycled through the file reader / writer manager, thereby reducing the data storage pressure.

[0080] It can be understood that the data forwarding part in this embodiment can be connected to a low-bit rate receiver and is an independent thread. The data transmitter reads data from the file read and write manager in real time, and completes data encapsulation and packaging according to the receiver application layer protocol. Finally, the data is sent to the receiver through a low-bit rate communication network environment using TCP, UDP and other communication methods.

[0081] See also Figure 2 The file read-write manager specifically includes a file management module 210 and a file read-write module 220 , wherein the file read-write module 220 includes a file handle submodule 2201 , an information cache submodule 2202 and a file operation submodule 2203 .

[0082] The file management module 210 is used to manage all file readers and writers. When a file needs to be written or read, it is necessary to apply to create a file reader and writer through the file management module 210. Subsequently, the actual file operation is completed by writing and reading the file reader and writer. After the file operation is completed, the file reader and writer must be recycled through the file management module 210 so that the file management module 210 can grasp the status of all file readers and writers.

[0083] Each data file can only have one file reader at the same time, and a file reader can be held by multiple users at the same time. When creating a file reader, the file management module 210 will determine whether a file reader for this data file already exists based on the file path. If it already exists, the corresponding file reader is directly returned. If it does not exist, the file reader is created and initialized, and each request is marked once.

[0084] When recycling a file reader / writer, the file management module 210 will modify the mark and determine whether this recycling is the last recycling and whether there are other users holding the file reader / writer of this file. If not, the file reader / writer will be deleted after modifying the mark.

[0085] The file reading and writing module 220 is an agent for file operations. Each data file corresponds to a file reading and writing module 220, that is, one data file corresponds to one file reader. Figure 2 The file reader 1, file reader 2 and file reader 3 corresponding to file 1, file 2 and file 3 are shown as examples, and the specific composition of the file reading and writing module 220 is explained by taking the file reader 1 as an example.

[0086] In actual applications, when a specified file needs to be read or written, a file reader / writer is created for this file. There can only be one file reader / writer in the case of multiple read or write scenarios at the same time. Only one file reader / writer is allowed to exist for the same file at the same time. After all read or write operations on the data file are completed, the file reader / writer needs to be recycled to release system resources until the data file needs to be read or written again, and then a new file reader / writer is created.

[0087] When creating a file reader / writer through the file management module 210, it is necessary to specify the purpose of obtaining the current file reader / writer. If it is used for data writing, a write-purpose file reader / writer will be established to serve this writing. If it is used for data reading, a read-purpose file reader / writer will be established to serve this reading.

[0088] In order to ensure data security, only one write operator is allowed to exist in the destination file reader at the same time. Data can be written to the data file through the write operator; multiple readers can exist at the same time, and data can be read from the data file through their own readers, that is, one writer and multiple readers can exist at the same time. When a file reader is created, a file handle is created synchronously and a file information cache is established.

[0089] In this embodiment, the file handle submodule 2201 is used to manage the open handle of the data file. When the file reader / writer is created, the corresponding file is opened synchronously with read and write permissions, and the file reader / writer holds the open handle of this file. All subsequent file operations need to rely on this handle. The user will not directly operate the file handle. All operations related to the file handle are completed by the file operation submodule 2203.

[0090] The information cache submodule 2202 is mainly used to record the file status information of data files. When creating a file reader / writer, a file information cache is synchronously opened and initialized with the current file status information. The file status information recorded in the cache includes: file write status and file effective size. For this cache operation, a synchronous lock is required to ensure atomicity to prevent synchronization problems in multi-threaded situations.

[0091] The file write status indicates whether the current file is in the process of being written. If the file reader is opened for writing, this status is marked as being written. If the file reader is opened for writing, this status is marked as not being written. The effective size of the file indicates the size of the data that has been recorded to the disk. Only the write operator can modify the effective size of the current file during the file writing process, and the read operator cannot modify the effective size of the file.

[0092] The file operation submodule 2203 is mainly used to complete the user's write operation and read operation, and is divided into a read operator and a write operator. Among them, the write operator is used to complete the user's write request. When creating a write destination file reader, a write operator will be created synchronously until the created write destination file reader is recycled and then the write operator is deleted. The same file reader can only have one write operator at the same time.

[0093] In one embodiment, see Figure 3 , create a file reader / writer for the write destination through the file reader / writer manager, including:

[0094] Step 310: When the file read / write manager receives the destination reader / writer creation instruction initiated by the requester, it determines whether the target data file corresponding to the destination reader / writer creation instruction already exists in the file reader / writer according to the file path corresponding to the destination reader / writer creation instruction, and obtains a first judgment result.

[0095] Step 320: If the first judgment result is that a file reader exists, determine whether a write operator exists in the file reader to obtain a second judgment result; if the second judgment result is that a write operator exists, feedback an error prompt information to the user.

[0096] Step 330: If the second judgment result is that there is no write operator, create a write operator in the file reader / writer.

[0097] Step 340: If the first judgment result is that there is no file reader / writer, create a file reader / writer corresponding to the target data file, and create a write operator in the file reader / writer.

[0098] Step 350: Feedback the file reader / writer created with the write operator to the requester.

[0099] In this step, the file write status of the data file corresponding to the current file reader / writer is also marked as file writing in progress, and the effective file size is set to zero, and the modified file status information is cached.

[0100] It is understandable that when the requester applies to the file reader manager to create a file reader, the file reader manager will determine whether the file reader for the corresponding file already exists. If not, a file reader will be created. If it already exists, it does not need to be created again. Since there can only be one write operator for the same data file at the same time, it is also necessary to determine whether the write operator already exists in this file reader. If it already exists, an error is returned. Otherwise, a write operator is created and this file reader is returned. Once the file reader is successfully returned, it means that the requester has the write permission for this file.

[0101] It should be noted that the write here is only for new files, that is, when creating a file reader, it will determine whether the file exists. If it already exists, an error will be returned. Otherwise, a new file will be created and reset to the specified size. At the same time, the file write status in the file information cache will be modified to file writing, and the effective file size will be set to zero. When reclaiming the write operator, the file needs to be reset to the effective file size specified in the file information cache to ensure that only valid data content is stored in the disk file.

[0102] In one embodiment, see Figure 4 , write the parsed data into the data file through the destination file reader, including:

[0103] Step 410: After the destination file reader receives the data write request initiated by the requester, it determines whether the requester has the write permission.

[0104] Step 420: After determining that the requester has write permission, determine the target cache space to be written.

[0105] like Figure 4 As shown, determining the target cache space to be written specifically includes:

[0106] First, determine whether the remaining space to be written in the current cache space mapped from the disk is sufficient.

[0107] On the one hand, if the space to be written is sufficient, the current cache space is used as the target cache space.

[0108] On the other hand, if the space to be written is insufficient, the data written to the current cache space is recorded to the disk through the file handle corresponding to the destination file reader / writer, and a new mapped cache space is established, that is, the received data is Unmap (released) through the file handle, the effective size of the file is modified, and the new mapped cache space is used as the target cache space.

[0109] Step 430: writing data, specifically writing the parsed data into a data file and storing it into a target cache space.

[0110] In one embodiment, see Figure 4 After writing the parsed data into the data file and storing it in the target cache space, the method may further include:

[0111] Step 440: determine whether the file size has reached the maximum limit, that is, whether the size of the data file has reached the upper limit of the single file space. When the size of the data file has reached the upper limit of the single file space, recycle the current file reader / writer of the writing destination through the file reader / writer manager;

[0112] Step 450: Re-create a new file reader / writer, specifically, determine a new data file and create a new write destination file reader / writer.

[0113] It can be understood that after obtaining the destination file reader, data can be written to the data file. The process of writing data is to write from the front to the back in sequence until the writing is completed, and the writing can be completed in multiple times.

[0114] During each write process, the requester directly initiates a write request to the file reader / writer. The file reader / writer will determine whether it currently has write permission. If it does not return, the write fails. If it has permission, it will find the write operator and determine whether the remaining space to be written in the cache currently mapped from the disk is sufficient. If it is insufficient, the data that has been written to the mapping cache needs to be recorded to the disk through the Unmap operation through the file handle, and a new mapping that is sufficient to write the requested data is established through the Map (allocation) operation, and the new mapping cache is obtained, and then the data is written. After writing the data, it is necessary to determine whether the maximum limit of a single file has been reached. If it has been reached, the file reader / writer needs to be recycled and a new file reader / writer needs to be recreated. Finally, the write success is returned after the operation is completed.

[0115] When reclaiming a file reader / writer, it is necessary to determine whether there are other users holding this file reader / writer. If there are, only the read operator in the file reader / writer needs to be deleted; otherwise, the entire file reader / writer needs to be deleted.

[0116] It is understandable that the read operator is mainly used to complete the user's read request. When creating a read destination file reader, a read operator will be created synchronously until the read destination file reader is recycled and then the read operator is deleted. The same file reader can have multiple read operators at the same time.

[0117] In one embodiment, see Figure 5 , create a file reader / writer for the read destination through the file reader / writer manager, including:

[0118] Step 510: When the file read / write manager receives the destination reader / writer creation instruction initiated by the requester, it determines whether the target data file corresponding to the destination reader / writer creation instruction already exists in the file reader / writer according to the file path corresponding to the destination reader / writer creation instruction, and obtains a third judgment result.

[0119] Step 520: If the third judgment result is that the file reader / writer exists, a read operator is created in the current file reader / writer.

[0120] Step 530: If the third judgment result is that there is no file reader / writer, create a file reader / writer corresponding to the target data file, and create a read operator in the file reader / writer.

[0121] Step 540: Feedback the file reader / writer that created the read operator to the requester. In this step, if the file reader / writer creates this file for the first time, the effective file size will be set according to the actual file size, and the file status information will be cached and returned to the requester.

[0122] It is understandable that when the requester applies to the file reader manager to create a file reader for the read destination, the file reader manager will determine whether a file reader for the corresponding file already exists. If not, a file reader will be created. If it already exists, it does not need to be created again. The file reader will create a read operator for this request and return this file reader. Once the file reader is successfully returned, it means that the requester has the read permission for this file. If the file reader is created for the first time, the file effective size value in the file information cache will be modified synchronously with the actual size of the file to be read during creation.

[0123] In one embodiment, see Figure 6 , read the parsed data in the data file through the target file reader, including:

[0124] Step 610: After receiving the data read instruction sent by the requester, the destination file reader determines whether the requester has the read permission.

[0125] Step 620: After determining that the requester has the read permission, determine the read operator corresponding to the requester, and determine whether the parsed data in the current cache space mapped by the disk corresponding to the read operator has been read.

[0126] Step 630: If it is determined that the parsed data in the current cache space has been read, a new mapping cache space is determined by reading the file handle corresponding to the destination file reader / writer, and the parsed data in the new mapping cache space is read.

[0127] Step 640: During the data reading process, it is necessary to determine whether the data has been read completely.

[0128] Step 650: After all the parsed data in the data file are read, the file reader is recycled and feedback information indicating successful reading is returned to the requester.

[0129] In one embodiment, reclaiming the destination file reader / writer through the file reader / writer manager specifically includes:

[0130] First, determine whether there is another user holding a reader / writer for the target file at the current moment.

[0131] On the one hand, if there are other users holding the target file reader at the current moment, only the read operator corresponding to the requester who initiated the data read instruction is deleted.

[0132] On the other hand, if no other user holds the destination file reader / writer at the current moment, the destination file reader / writer is deleted.

[0133] It can be understood that after obtaining the read destination file reader, data can be read from the data file. The reading process is to read from the front to the back in sequence until the reading is completed. The reading can be completed in multiple times. During each reading process, the requester directly initiates a read request to the read destination file reader. The read destination file reader will determine whether it currently has read permission. If not, it returns a read failure. If it has permission, it will find the read operator corresponding to the current user and determine whether the cache currently mapped from the disk has been read. If this cache has been read, it is necessary to remap the next set of data through the file handle, obtain a new mapping cache, and then read the data. After reading all the data in the data file, it is necessary to recycle this file reader. If no other user holds this file reader, it is necessary to delete the file reader. If there are other users holding it, just delete the read operator associated with this requester in the file reader, and keep the file reader for other users to use.

[0134] In practical applications, the implementation principle of the entire process of using file read and write managers to realize data collection, synchronous storage and real-time forwarding can be found in Figure 7 ,like Figure 7 As shown, the entire data processing process is as follows:

[0135] First, create a file read-write manager. All file readers and writers are managed by this file read-write manager.

[0136] On the one hand, create a data collection thread. After the data collection thread is created, it needs to be executed according to the following process:

[0137] The first step is to create a file reader / writer for the current collection task from the file reader / writer manager.

[0138] The second step is to continuously collect the target data sent by the transmitter from the network interface and parse the data according to the protocol.

[0139] The third step is to write the target data into the data file through the destination file reader / writer.

[0140] The fourth step is to determine whether to end this collection task. If it is ended, close the collection thread and recycle the file reader / writer of the writing destination.

[0141] The fifth step is to determine the writing status of the current writing destination file reader / writer. If the file is not full, continue writing; if the file is full, recycle the current writing destination file reader / writer.

[0142] On the other hand, create a data forwarding thread. After the data forwarding thread is created, it needs to be executed according to the following process:

[0143] The first step is to create a destination file reader / writer from the file reader / writer manager for this forwarding task.

[0144] The second step is to continue reading data from the data file.

[0145] The third step is to package and encapsulate the read data according to the protocol and forward it from the network interface to the receiver.

[0146] The fourth step is to determine whether to end the forwarding task. If it is ended, the data forwarding thread is closed, and the created destination file reader is recycled through the file reader / writer manager.

[0147] The fifth step is to determine the reading status of the current reading destination file reader / writer. If the reading is not completed, continue to read the data; if the data file has been read, recycle the current reading destination file reader / writer.

[0148] In summary, the data processing method provided by the embodiment of the present invention has at least the following advantages compared with the traditional data forwarding solution:

[0149] First, under normal circumstances, the computer memory disk is relatively small, and using the memory as a cache queue will cause insufficient memory resources after a certain period of time when the bit rate is unbalanced. This embodiment adopts file mapping technology and uses the large capacity space of the disk to establish a huge cache queue, which is no longer limited by the impact of insufficient memory.

[0150] Second, the solution provided in this embodiment has persisted the collected data on the disk during the data forwarding process, without the need to implement a separate save operation.

[0151] Third, the solution provided in this embodiment supports simultaneous writing and reading of the same data file. Collection, storage and reading and forwarding can be performed simultaneously. There is no need to wait for a file to be written before starting the forwarding of this file, which reduces the forwarding delay.

[0152] Fourth, the solution provided in this embodiment is that the file read-write manager supports single-write and multiple-read scenarios and can support multi-channel simultaneous forwarding. At the same time, it separates the three parts of data collection, file management, and data forwarding, reduces their respective coupling, and makes the data forwarding process more efficient and stable.

[0153] Based on the same general inventive concept, the present invention also protects a forwarding device and a data transmission system. The forwarding device and the data transmission system provided by the present invention are described below. The forwarding device and the data transmission system described below can be referenced to each other with the data processing method described above.

[0154] Figure 8 It is a structural diagram of a forwarding device provided in an embodiment of the present invention.

[0155] See also Figure 8The forwarding device provided in the embodiment of the present invention specifically includes: a data collector 710, a file read and write manager 720 and a data transmitter 730.

[0156] The file read / write manager 720 is used to create a file reader / writer for writing after establishing a data collection thread.

[0157] The data collector 710 is used to collect and analyze the target data sent by the transmitter to obtain analyzed data.

[0158] The file read / write manager 720 is also used to recycle the write-destination file reader / writer after the write-destination file reader / writer writes the parsed data into the data file and the data file is full or the data collection thread ends.

[0159] The file read / write manager 720 is also used to create a destination file reader / writer after establishing a data forwarding thread; and to recycle the destination file reader / writer after the destination file reader / writer reads parsed data in the data file and the parsed data is read or the data forwarding thread ends.

[0160] The data transmitter 730 is used to package the read parsed data and send it to the receiver.

[0161] The forwarding device provided in this embodiment can establish a file read-write manager based on disk file mapping technology. Through the file read-write manager, different files can be read and written separately, and the same file can be read and written simultaneously. The data collector of the forwarding device writes data at high speed through the file read-write manager, and then the data transmitter reads and forwards the file at low speed according to the forwarding rate through the file read-write manager, thereby achieving the purpose of high-speed collection, synchronous storage, and low-speed real-time forwarding.

[0162] Fig. 9 It is a structural diagram of a data transmission system provided by an embodiment of the present invention.

[0163] like Fig. 9 As shown, the data transmission system provided in the embodiment of the present invention specifically includes: a transmitter 810, a receiver 830 and the forwarding device 820 provided in the above embodiment.

[0164] In practical applications, the target data sent by the transmitter 810 can be collected by the forwarding device 820 and forwarded to the receiver 830 after being processed by the data processing methods provided in the above embodiments, thereby completing the data forwarding task.

[0165] Regarding the forwarding device in the above embodiment, the specific manner in which the file read and write manager performs operations has been described in detail in the embodiment of the relevant data processing method, and will not be elaborated again here.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, It is characterized in that The method is performed by a forwarding device including a data collector, a file read-write manager and a data transmitter, and includes: After establishing the data collection thread, creating a file reader / writer for the writing purpose through the file reader / writer manager; The target data sent by the transmitter is collected and analyzed by the data collector to obtain analyzed data; The parsed data is written into a data file through the write destination file reader until the data file is full or the data acquisition thread ends, and then the write destination file reader is recycled through the file read-write manager; the parsed data is written into a data file through the write destination file reader, comprising: after the write destination file reader receives a data write request initiated by a requester, determining whether the requester has write permission; after determining that the requester has write permission, determining a target cache space to be written; writing the parsed data into a data file, and storing it in the target cache space; The determining of the target cache space to be written includes: judging whether the remaining space to be written in the current cache space mapped from the disk is sufficient; if the space to be written is sufficient, using the current cache space as the target cache space; if the space to be written is insufficient, recording the data written into the current cache space to the disk through the file handle corresponding to the write destination file reader, and establishing a new mapping cache space, using the new mapping cache space as the target cache space; After establishing the data forwarding thread, creating a file reader / writer for the read destination through the file reader / writer manager; The parsed data in the data file is read by the read destination file reader / writer, and the parsed data read is packaged and sent to the receiver by the data transmitter; Until the parsed data is read completely or the data forwarding thread ends, the file reader / writer of the read destination is recycled through the file reader / writer manager.

2. The data processing method according to claim 1, It is characterized in that The step of creating a file reader / writer for a writing destination through the file reader / writer manager includes: When the file read / write manager receives the write destination reader / writer creation instruction initiated by the requester, it determines whether the target data file corresponding to the write destination reader / writer creation instruction already exists in the file reader / writer according to the file path corresponding to the write destination reader / writer creation instruction, and obtains a first judgment result; If the first judgment result is that there is a file reader / writer, then judging whether there is a write operator in the file reader / writer to obtain a second judgment result; If the second judgment result is that a write operator exists, an error prompt message is fed back to the user; If the second judgment result is that there is no write operator, creating a write operator in the file reader; If the first judgment result is that there is no file reader / writer, then creating a file reader / writer corresponding to the target data file, and creating a write operator in the file reader / writer; Feeds the file reader / writer created with the write operator back to the requester.

3. The data processing method according to claim 1, It is characterized in that After writing the parsed data into a data file and storing it in the target cache space, the method further includes: When the size of the data file reaches the upper limit of the single file space, the file reader / writer of the current writing destination is recycled by the file reader / writer manager; Identify the new data file and create a new file reader / writer for the write destination.

4. The data processing method according to claim 1, It is characterized in that The step of creating a destination file reader / writer through the file read / write manager includes: When the file read / write manager receives a read destination reader / writer creation instruction initiated by the requester, it determines whether a file reader / writer already exists for the target data file corresponding to the read destination reader / writer creation instruction according to the file path corresponding to the read destination reader / writer creation instruction, and obtains a third determination result; If the third judgment result is that a file reader / writer exists, creating a read operator in the current file reader / writer; If the third judgment result is that there is no file reader / writer, then creating a file reader / writer corresponding to the target data file, and creating a read operator in the file reader / writer; Feedback the file reader / writer that created the read operator to the requester.

5. The data processing method according to claim 1, It is characterized in that The step of reading the parsed data in the data file by the read destination file reader / writer comprises: After the read destination file reader receives the data read instruction sent by the requester, determining whether the requester has the read permission; After determining that the requester has the read permission, determining the read operator corresponding to the requester, and determining whether the parsed data in the current cache space mapped by the disk corresponding to the read operator has been read; If it is determined that the parsed data in the current cache space has been read, a new mapping cache space is determined through the file handle corresponding to the read destination file reader, and the parsed data in the new mapping cache space is read until all the parsed data in the data file are read.

6. The data processing method according to claim 5, It is characterized in that Reclaiming the destination file reader / writer through the file read / write manager includes: Determine whether there is another user holding the file reader / writer of the reading target at the current moment; If there is another user holding the file reader / writer of the read destination file at the current moment, only the read operator corresponding to the requester who initiated the data read instruction is deleted; If no other user holds the target file reader / writer at the current moment, the target file reader / writer is deleted.

7. A forwarding device, It is characterized in that include: Data collector, file read and write manager and data sender; The file read-write manager is used to create a file reader for writing after establishing a data acquisition thread; The data collector is used to collect and analyze the target data sent by the transmitter to obtain analyzed data; The file read / write manager is also used to write the parsed data into a data file at the write destination file reader / writer, and recycle the write destination file reader / writer when the data file is full or the data collection thread ends; specifically, it is used to: after the write destination file reader / writer receives a data write request initiated by a requester, determine whether the requester has write permission; After determining that the requester has write permission, determining a target cache space to be written; Writing the parsed data into a data file and storing it in the target cache space; The determining of the target cache space to be written includes: judging whether the remaining space to be written in the current cache space mapped from the disk is sufficient; if the space to be written is sufficient, using the current cache space as the target cache space; if the space to be written is insufficient, recording the data written into the current cache space to the disk through the file handle corresponding to the write destination file reader, and establishing a new mapping cache space, using the new mapping cache space as the target cache space; The file read / write manager is also used to create a read-destination file reader / writer after establishing a data forwarding thread; and to recycle the read-destination file reader / writer after the read-destination file reader / writer reads the parsed data in the data file and the parsed data is read or the data forwarding thread ends; The data transmitter is used for packaging the read parsed data and sending it to the receiver.

8. A data transmission system, It is characterized in that include: A transmitter, a receiver, and a forwarding device as claimed in claim 7.