Data processing method, satellite equipment, storage medium and computer program product

Abstract

The invention discloses a data processing method, satellite equipment, a storage medium and a computer program product. The method comprises the steps that a target data file is acquired, and the target data file is used for recording ship automatic identification system message data to be processed in the satellite communication process; based on the frame structure of the ship automatic identification system message data, performing multi-thread parallel analysis on the target data file to obtain an analysis result; and carrying out multi-dimensional error data classification statistics and / or data batch storage on an analysis result. According to the method and the device, the technical problems of low data processing efficiency and difficulty in error positioning during analysis of the message data of the automatic ship identification system in related technologies are solved.

Description

Technical Field

[0001] This invention relates to the intersection of satellite communication and automatic identification system (AIS) technologies, and more specifically, to a data processing method, satellite equipment, storage medium, and computer program product. Background Technology

[0002] With the increasing capabilities of satellite payloads, massive amounts of GB-level Automatic Identification System (AIS) message data need to be rapidly parsed, verified, and stored to enable real-time maritime monitoring, vessel tracking, and shipping safety management. Related technologies typically rely on single-threaded parsing methods or multi-threaded decoding systems to parse AIS message data. However, single-threaded parsing methods are limited by serial processing flows and fail to fully utilize the parallel computing capabilities of modern multi-core processors, resulting in low data processing efficiency and difficulty in meeting real-time requirements. While multi-threaded decoding systems introduce multiple threads to improve processing speed, they lack effective data retention and error classification and statistical mechanisms when facing verification failures or complex error types. This leads to compromised data integrity and makes it difficult to trace and analyze the source of data anomalies.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a data processing method, satellite equipment, storage medium, and computer program product to at least solve the technical problems of low data processing efficiency and difficulty in error location when parsing Automatic Identification System (AIS) message data in related technologies.

[0005] According to one aspect of the present invention, a data processing method is provided, comprising: acquiring a target data file, wherein the target data file is used to record Automatic Identification System (AIS) message data to be processed during satellite communication; performing multi-threaded parallel parsing on the target data file based on the frame structure of the AIS message data to obtain parsing results; and performing multi-dimensional error data classification statistics and / or batch storage on the parsing results.

[0006] Optionally, the frame structure of the Automatic Identification System (AIS) message data includes the following fields: frame control header field, start identifier field, message body field, start marker field for a single message, end marker field for a single message, and checksum field.

[0007] Optionally, obtaining the target data file includes: loading the original data file using a binary reading method; converting the binary data in the original data file into a hexadecimal string to obtain the target data file.

[0008] Optionally, based on the frame structure of the Automatic Identification System (AIS) message data, the target data file is parsed in a multi-threaded parallel manner to obtain the parsing results, including: extracting multiple valid data frames from the target data file based on the frame structure of the AIS message data; and parsing the multiple valid data frames in a multi-threaded parallel manner to obtain the parsing results.

[0009] Optionally, based on the frame structure of the Automatic Identification System (AIS) message data, extracting multiple valid data frames from the target data file includes: extracting multiple valid data frames from the target data file based on the start identifier field, the start marker field of a single message, and the end marker field of a single message, wherein multiple valid data frames simultaneously match the start identifier field, the start marker field of a single message, and the end marker field of a single message; wherein the data processing method further includes: detecting at least one erroneous data frame from the target data file based on the start identifier field, the start marker field of a single message, and the end marker field of a single message, and classifying and statistically analyzing at least one erroneous data frame, wherein at least one erroneous data frame matches the start identifier field, but at least one erroneous data frame does not match the start marker field of a single message or does not match the end marker field of a single message.

[0010] Optionally, multiple valid data frames are parsed in parallel using multiple threads to obtain the parsing results, including: extracting message body fields from multiple valid data frames; performing data verification on the message body fields based on the verification fields to obtain the verification results; parsing the Automatic Identification System (AIS) message type from the message body fields; classifying the multiple valid data frames into multiple task queues according to the verification results and the AIS message type; and processing the tasks assigned in the multiple task queues in parallel to obtain the parsing results.

[0011] Optionally, the data processing method in this embodiment of the invention further includes: configuring a thread pool based on the number of processor cores, wherein the thread pool includes: multiple producer threads and multiple consumer threads; classifying multiple valid data frames into multiple task queues according to the verification results and the Automatic Identification System (AIS) message type, including: using multiple producer threads to classify multiple valid data frames into multiple task queues according to the verification results and the AIS message type; and performing parallel processing on the tasks allocated in the multiple task queues to obtain parsing results, including: using multiple consumer threads to perform parallel processing on the tasks allocated in the multiple task queues to obtain parsing results.

[0012] Optionally, the data processing method in this embodiment of the invention further includes: in response to the message body field of some data frames in a plurality of valid data frames failing the verification, retaining the message body of some data frames and marking the verification status of some data frames.

[0013] Optionally, the data processing method in this embodiment of the invention further includes at least one of the following: extracting frame control header fields from multiple valid data frames respectively, and parsing the frame control header fields to obtain a structured dictionary, wherein the structured dictionary is used to record multiple data frame traceability fields; establishing an association mapping between the frame control header fields and the message body fields for each valid data frame among the multiple valid data frames, wherein the association mapping is used to record the traceability information of each valid data frame; and using a dedicated control header tracing log to perform full-link tracing of the data processing process of each valid data frame among the multiple valid data frames.

[0014] Optionally, performing multidimensional error data classification and statistics on the parsing results includes: determining the error type and error cause of multiple error data frames based on the parsing results; and performing multidimensional error data classification and statistics on multiple error data frames based on the error type and error cause.

[0015] Optionally, batch storage of the parsing results includes: selecting a database connection from a database connection pool, wherein the number of database connections in the database connection pool is determined based on the number of threads in the thread pool; dividing the parsing results into batches based on the message type of the Automatic Identification System (AIS) to obtain multiple batches of data of the same type; and using a database connection, batch storing multiple batches of data of the same type through transaction commit.

[0016] According to another aspect of the present invention, a data processing apparatus is also provided, comprising: an acquisition module for acquiring a target data file, wherein the target data file is used to record Automatic Identification System (AIS) message data to be processed during satellite communication; a parsing module for performing multi-threaded parallel parsing of the target data file based on the frame structure of the AIS message data to obtain parsing results; and a processing module for performing multi-dimensional error data classification statistics and / or batch storage of the parsing results.

[0017] According to another aspect of the present invention, a satellite device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the data processing method of the present invention during runtime.

[0018] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the storage medium is located to execute the data processing method of the present invention.

[0019] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the data processing method of the present invention.

[0020] According to another aspect of the present invention, a chip system is also provided, comprising: a processor for calling and running a computer program from a memory, causing a communication device equipped with the chip system to perform the data processing method of the present invention.

[0021] In this embodiment of the invention, by acquiring a target data file and then performing multi-threaded parallel parsing on the target data file based on the frame structure of the Automatic Identification System (AIS) message data, the parsing results are obtained. Finally, the parsing results are subjected to multi-dimensional error data classification and / or batch storage, thereby achieving the goal of improving the throughput and data integrity of AIS message data parsing. This achieves the technical effect of improving data processing efficiency and enhancing the traceability of error data, and solves the technical problems of low data processing efficiency and difficulty in error location in related technologies when parsing AIS message data. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0023] Figure 1 This is a flowchart of a data processing method according to one embodiment of the present invention;

[0024] Figure 2 This is a structural block diagram of a data processing apparatus according to one embodiment of the present invention; Detailed Implementation

[0025] For ease of understanding, some concepts related to embodiments of the present invention are illustrated below for reference.

[0026] Automatic Identification System (AIS): A key component of an advanced maritime safety and vessel traffic management system, AIS utilizes digital communication technology to enable automatic information exchange between vessels and between vessels and shore-based facilities globally. AIS was designed to improve maritime traffic safety, increase navigation efficiency, and contribute to environmental protection and maritime search and rescue operations. Specifically, AIS operates based on Very High Frequency (VHF) radio signals, broadcasting and receiving information through two different channels (AIS1 and AIS2). It automatically transmits information such as a vessel's position, speed, course, vessel identification (e.g., Maritime Mobile Service Identity (MMSI)), name, call sign, size, type, destination, and estimated time of arrival. This information is broadcast every few seconds, allowing nearby vessels or shore-based receiving stations to be aware of the real-time movement of vessels in the surrounding waters, thus preventing collisions and achieving safer and more efficient maritime navigation. Furthermore, AIS information exchange is bidirectional; it can not only send its own information but also receive broadcasts from other AIS-equipped vessels or shore stations.

[0027] ITU-R M.1371 standard: The technical specifications for AIS developed by the International Telecommunication Union (ITU). This standard details the types, data formats, encoding rules, and communication protocols of AIS messages. It is a fundamental document for the design, implementation, and application of AIS systems, ensuring the interoperability and standardization of AIS systems worldwide.

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] With the increasing capabilities of satellite payloads, massive amounts of GB-level Automatic Identification System (AIS) message data need to be rapidly parsed, verified, and stored to enable real-time maritime monitoring, vessel tracking, and shipping safety management. Related technologies typically rely on single-threaded parsing methods or multi-threaded decoding systems to parse AIS message data. However, single-threaded parsing methods are limited by serial processing flows and fail to fully utilize the parallel computing capabilities of modern multi-core processors, resulting in low data processing efficiency and difficulty in meeting real-time requirements. While multi-threaded decoding systems introduce multiple threads to improve processing speed, they lack effective data retention and error classification and statistical mechanisms when facing verification failures or complex error types. This leads to compromised data integrity and makes it difficult to trace and analyze the source of data anomalies.

[0031] Specifically, when parsing data from Automatic Identification System (AIS) messages, the inherent serial processing architecture of single-threaded parsing methods requires that the parsing, verification, and storage of each data frame be completed sequentially, preventing concurrent execution. Even in high-performance hardware environments, processing speed is limited by the processor's single working thread, hindering acceleration and efficiency improvements, thus reducing the real-time performance and response speed of maritime monitoring. On the other hand, while multi-threaded decoding systems can alleviate the bottleneck effect of single-threaded systems and improve data throughput to some extent by introducing parallel processing, they often employ a simple discard strategy when verification fails or other complex errors exist. This makes it impossible to determine whether the data contains potential usable information, and it is difficult to identify the characteristics and frequency of various error types, thus hindering the identification of the root cause of the problem. Furthermore, in the relevant technologies, when parsing the data of Automatic Identification System (AIS) messages, no customized satellite AIS data frame structure was designed. Instead, data was extracted by relying solely on general frame markers, resulting in low frame extraction accuracy, difficulty in identifying cross-frame error data, and a lack of a full-link log tracking mechanism. This makes it impossible to trace the entire process of data extraction, parsing, and storage, leading to low efficiency in troubleshooting.

[0032] According to embodiments of the present invention, a method embodiment for data processing is provided. The method embodiment provided in this invention can be executed in a terminal device (the terminal device may include, but is not limited to, a satellite terminal, a mobile terminal), or a similar network device. Taking operation on a satellite device as an example, the satellite device may include one or more processors (processors may include, but are not limited to, processing devices such as microprocessors (MCUs) or field-programmable gate arrays (FPGAs), etc.), a memory for storing data, and a transmission device for communication functions. Those skilled in the art will understand that the above structure is merely exemplary and does not limit the structure of the satellite device. For example, the satellite device may also include more or fewer components, or have different configurations.

[0033] The memory can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing method in this embodiment of the invention. The processor executes various functional applications and data processing by running the computer program stored in the memory, thereby implementing the above-described method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to satellite equipment via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] Transmission equipment is used to receive or send data via satellite networks. Specific examples of these networks may include satellite communication services provided by satellite equipment communication providers, enabling global data transmission through satellites deployed in Earth orbit. In one example, transmission equipment includes a radio frequency (RF) transceiver, an antenna, a signal processor, and a network interface controller (NIC). The RF transceiver is used for direct data communication with the satellite, while the NIC connects to ground stations or other network equipment, enabling the satellite equipment to access the Internet or dedicated networks, achieving global data transmission and information sharing.

[0035] Figure 1 This is a flowchart of a data processing method according to one embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0036] Step S11: Obtain the target data file, wherein the target data file is used to record the Automatic Identification System (AIS) message data to be processed during satellite communication;

[0037] The aforementioned target data file refers to the data file formed after preliminary conversion of AIS message data received from satellites. Specifically, the target data file records detailed navigation data of vessels within a wide sea area, including but not limited to key information such as the vessel's position coordinates, speed, course, and unique identifier, aiming to provide comprehensive data support for maritime monitoring, vessel tracking, and shipping safety management.

[0038] The aforementioned Automatic Identification System (AIS) message data refers to electronic data automatically broadcast by ships navigating at sea, captured via satellite communication systems. This data contains rich and specific nautical information, aiming to enhance the safety and efficiency of maritime navigation. Specifically, AIS message data includes, but is not limited to, static information of the ship, dynamic information of the ship, voyage-related data, safety-related data, and other auxiliary information.

[0039] Specifically, a ship's static information includes, but is not limited to, information such as the MMSI assigned by the International Maritime Organization (IMO), ship name, ship type, ship size, and call sign, which helps to identify each ship at sea. A ship's dynamic information includes, but is not limited to, the ship's position coordinates (latitude and longitude), instantaneous speed, heading, turning rate, roll angle, and pitch angle, used to understand the ship's real-time motion status. Voyage-related data includes, but is not limited to, the name of the destination port, the estimated time of arrival (ETA), and the ship's voyage plan, which helps maritime authorities understand the ship's navigational intentions in advance and is of great significance for route planning and scheduling. Safety-related data includes, but is not limited to, the ship's navigation status and any safety-related information, such as the loading status of dangerous goods or warning messages issued by the ship in case of an emergency. Other auxiliary information includes, but is not limited to, the ship's draft, cargo hold loading status, and data related to weather and sea conditions, used to assess the ship's stability and safety.

[0040] For example, a satellite AIS system, using satellites as relay nodes, can receive radio frequency signals transmitted from AIS terminals on ships at sea and transmit the data to a ground processing center, achieving full coverage monitoring of ships over a wide maritime area. Specifically, the radio frequency signals received by the satellite need to undergo down-conversion and analog-to-digital conversion by a radio frequency front-end module, converting the analog signal into a digital signal for subsequent processing. This process involves extracting useful information from the high-frequency carrier signal, down-converting the signal to a lower intermediate frequency (IF), and then using an analog-to-digital converter (ADC) to convert the IF analog signal into a binary digital signal. After demodulation and decoding, the digital signal forms a raw data file in .dat format. Furthermore, since the raw data file may contain a large amount of unprocessed data and potential errors, further processing is required to obtain the target data file.

[0041] Step S12: Based on the frame structure of the Automatic Identification System (AIS) message data, perform multi-threaded parallel parsing on the target data file to obtain the parsing results;

[0042] The frame structure described above refers to a data packet format designed for organizing and transmitting information in communication or data transmission. For AIS message data, the frame structure specifically refers to how the data format for organizing and representing each group of ship position reports or other types of AIS messages is organized within the .dat file.

[0043] The aforementioned multi-threaded parallel parsing refers to a technical strategy that leverages the advantages of multi-core processors in modern computer systems to accelerate data processing by decomposing tasks and executing them simultaneously on multiple threads when dealing with large amounts of data or performing complex computational tasks.

[0044] Step S13: Perform multi-dimensional error data classification statistics and / or batch storage of the parsing results.

[0045] The aforementioned multidimensional error data classification and statistics refer to the systematic classification and statistical analysis of failed AIS message parsing data according to different error types and sources. This process aims to gain a deeper understanding of abnormal patterns in data processing, helping to identify whether data parsing anomalies are caused by verification failures, missing fields, cross-frame errors, or other reasons. By recording the number of instances of each error type and the associated control header information (such as satellite number, timestamp, etc.), the root cause of the problem can be quickly located, thereby improving the transparency of data processing and the efficiency of problem troubleshooting. This allows maintenance personnel to quickly take measures to optimize data transmission and reception conditions, thereby improving the overall integrity and accuracy of data processing.

[0046] The aforementioned batch data storage refers to writing a certain amount of structured data into the database in a single transaction after successful data parsing, rather than inserting it row by row. Batch data storage optimizes database write performance, reduces the overhead caused by frequent database connection creation and destruction, and the I / O cost of each insert operation. Specifically, after parsing a certain amount of message data, this data can be converted into a batch SQL insert statement, which is then executed on a single database connection to insert all data at once. Furthermore, using a database connection pool allows for the pre-creation and maintenance of a certain number of database connections for use in multi-threaded parallel processing, thereby reducing connection creation latency and ensuring the atomicity, consistency, isolation, and durability of data writes. Even if problems occur during data writing, the transaction rollback mechanism can restore the complete state of the data, thus improving data processing efficiency and data security.

[0047] Based on steps S11 to S13 above, by acquiring the target data file, and then performing multi-threaded parallel parsing on the target data file based on the frame structure of the Automatic Identification System (AIS) message data, the parsing results are obtained. Finally, the parsing results are subjected to multi-dimensional error data classification statistics and / or batch data storage, thereby achieving the goal of improving the throughput and data integrity of AIS message data parsing. This achieves the technical effect of improving data processing efficiency and enhancing the traceability of error data, and solves the technical problems of low data processing efficiency and difficulty in error location in related technologies when parsing AIS message data.

[0048] Optionally, the frame structure of the Automatic Identification System (AIS) message data includes the following fields: frame control header field, start identifier field, message body field, start marker field for a single message, end marker field for a single message, and checksum field. For example, the frame structure of AIS message data in this embodiment of the invention is shown in Table 1.

[0049] Table 1

[0050]

[0051] For example, the above verification algorithm may include, but is not limited to, the CRC-16 / CCITT-FALSE algorithm.

[0052] Optionally, in step S11, obtaining the target data file includes:

[0053] Step S111: Load the original data file using binary reading method;

[0054] Step S112: Convert the binary data in the original data file into a hexadecimal string to obtain the target data file.

[0055] The aforementioned binary reading method refers to reading file content directly as a byte stream without any encoding conversion. Specifically, the binary reading method is suitable for processing non-text files, such as images, audio, video, or binary data files, and is also commonly used for directly processing binary format data.

[0056] The aforementioned raw data file refers to a .dat format file containing satellite AIS data. Specifically, the raw data file directly stores the digitized data of the raw radio frequency signals received from the satellite, typically saved in binary form without any conversion or parsing. When processing these files, using binary reading avoids unnecessary encoding conversions, maintains the originality and integrity of the data, and facilitates subsequent operations such as frame boundary identification and message body parsing.

[0057] For example, you can use Python's open() function to open a .dat file at a specified path in binary mode ('rb'), and use bin_file.read() to read the file contents into memory all at once. This avoids unnecessary conversion steps, reduces the complexity of data processing, and helps maintain the integrity and accuracy of the data.

[0058] For example, since certain identifiers (such as the start identifier field) in the AIS message data frame structure are in hexadecimal, the binary data in the original data file can be converted into a hexadecimal string to obtain the target data file, facilitating subsequent frame boundary identification and message body parsing. Specifically, the hexadecimal string format makes the data easier to recognize by regular expressions or other string processing tools. The conversion process can be achieved by traversing the read byte stream, using Python's format() function or similar methods to convert each byte to its hexadecimal form, and then concatenating them into a complete hexadecimal string.

[0059] For example, when loading raw data files using binary reading, reading the entire file at once can consume a significant amount of memory. Therefore, setting a reasonable buffer size for file reading can significantly improve reading efficiency and reduce memory pressure. For instance, a file reading buffer size of 100MB can be set, allowing only the portion of the file that needs processing to be read at a time, thereby improving file reading efficiency.

[0060] Based on steps S111 to S112 above, the original data file is loaded using a binary reading method, and then the binary data in the original data file is converted into a hexadecimal string to obtain the target data file. This improves the efficiency and accuracy of data processing, avoids data loss or the introduction of unnecessary characters during the conversion process, and ensures the accuracy and integrity of subsequent parsing processes. When processing large datasets, the above conversion method can effectively reduce the number of I / O operations and speed up data loading.

[0061] Optionally, in step S12, based on the frame structure of the Automatic Identification System (AIS) message data, the target data file is parsed in a multi-threaded parallel manner to obtain the parsing results, including:

[0062] Step S121: Based on the frame structure of the Automatic Identification System (AIS) message data, extract multiple valid data frames from the target data file;

[0063] Step S122: Perform multi-threaded parallel parsing on multiple valid data frames to obtain the parsing results.

[0064] The aforementioned multiple valid data frames refer to the set of AIS message data that conforms to the ITU-R M.1371 standard, is structurally complete, and can be correctly parsed, selected through specific frame boundary identifiers and regular expression matching during the AIS message data transmission process.

[0065] For example, based on the frame structure of AIS message data, multiple valid data frames are extracted from the target data file. That is, based on the frame structure of AIS message data, the target data file is split to remove empty and redundant frames. Specifically, empty frames are frames without valid data or with insufficient length, which can be filtered by checking the frame length or data content. Redundant frames are identical data frames that are transmitted repeatedly. This can be filtered by establishing a hash table of read frames or storing the frame control header information for comparison during subsequent reads. Furthermore, after obtaining multiple valid data frames based on the above steps, these multiple valid data frames can be parsed in parallel using multi-threading to obtain the parsing result.

[0066] For example, data segments output by AIS receiving devices are typically designed to be of fixed length, most commonly twice 1024 bytes, or 2048 bytes. This standard length is based on the device's baseband processing capabilities and data transmission protocols, designed to ensure data packet consistency and compatibility during transmission. When processing large .dat format files, any tail data shorter than the fixed length can be treated as invalid data segments.

[0067] Based on steps S121 to S122 above, and based on the frame structure of the Automatic Identification System (AIS) message data, multiple valid data frames are extracted from the target data file. Then, multiple valid data frames are parsed in parallel using multiple threads to obtain the parsing results. This effectively filters and manages empty and redundant frames, avoids invalid data from occupying processing resources, further optimizes the data processing flow, thereby reducing storage costs and improving data processing efficiency.

[0068] Optionally, in step S121, based on the frame structure of the Automatic Identification System (AIS) message data, extracting multiple valid data frames from the target data file includes:

[0069] Based on the start identifier field, the start tag field of a single message, and the end tag field of a single message, multiple valid data frames are extracted from the target data file, wherein multiple valid data frames simultaneously match the start identifier field, the start tag field of a single message, and the end tag field of a single message.

[0070] For example, as shown in Table 1, the boundaries of a data frame can be located by matching the start identifier field "a1b2c3d4", the start marker of a single message "005555557e", and the end marker of a single message "7e(?:[0-9a-fA-F]{2}){1,2}00". Furthermore, for all possible valid data frames, it can be checked whether they contain the start identifier field, the start marker field of a single message, and the end marker field of a single message. If all three marker fields are present, the data frame is considered a valid data frame.

[0071] The end marker of the aforementioned single message, "7e(?:[0-9a-fA-F]{2}){1,2}00", is a regular expression for the end marker of a single AIS message, used to identify the end boundary of the AIS message in the .dat file.

[0072] Specifically, "7e" represents the first two hexadecimal characters of the AIS message end marker.

[0073] "(?:[0-9a-fA-F]{2})" represents an arbitrary hexadecimal pair. "[0-9a-fA-F]" represents the range of hexadecimal digits, including the numbers 0 to 9 and the letters a to f (or A to F). "{2}" indicates that the characters within this range will appear twice, forming a complete hexadecimal pair.

[0074] "{1,2}" indicates that the first part (hexadecimal pair) can appear once or twice. In the end marker of an AIS message, it usually represents the signal-to-noise ratio (SNR) information, consisting of one or two hexadecimal pairs, used to indicate the signal quality when the message is received.

[0075] "00" is the last two hexadecimal characters of the AIS message end marker. In the AIS protocol, 00 is usually used to indicate the end boundary of a message and together with "7e" forms a standard termination sequence.

[0076] Therefore, the entire regular expression “7e(?:[0-9a-fA-F]{2}){1,2}00” is used to match the end of the AIS message, where “7e” and “00” are fixed end markers, and “(?:[0-9a-fA-F]{2}){1,2}” represents a variable length (1 or 2) SNR message.

[0077] The data processing methods also include:

[0078] Based on the start identifier field, the start tag field of a single message, and the end tag field of a single message, at least one erroneous data frame is detected from the target data file, and the at least one erroneous data frame is classified and statistically analyzed. Among them, at least one erroneous data frame matches the start identifier field, but at least one erroneous data frame does not match the start tag field of a single message or does not match the end tag field of a single message.

[0079] For example, for at least one erroneous data frame, characterized by matching the start identifier field "a1b2c3d4", but failing to match the start marker field "005555557e" of a single message or the end marker field "7e(?:[0-9a-fA-F]{2}){1,2}00" of a single message, it indicates that although the beginning of the data frame was correctly identified, the start or end of the message body failed to appear in the standard format of the AIS protocol, possibly due to noise interference, signal distortion or device error during transmission, resulting in damage to the data frame structure.

[0080] For example, during the reading and parsing process, data frames that fail to match the start marker field or the end marker field of a single message can be monitored and marked as error data frames. For the detected error data frames, they can be classified and statistically analyzed according to their specific error type (e.g., only the start marker field of a single message is not matched, only the end marker field of a single message is not matched, or neither is matched), and the number of each type of error and the corresponding error data sample can be recorded.

[0081] Based on the above optional embodiments, by locating the start identifier field "a1b2c3d4", the single message start marker "005555557e", and the end marker regular expression "7e(?:[0-9a-fA-F]{2}){1,2}00", the valid data frame boundaries in the .dat file can be effectively extracted and verified, ensuring that each frame of data fully complies with the AIS protocol standard. Simultaneously, for erroneous data frames that only match the start identifier field but lack the start marker field of a single message or the end marker field of a single message, monitoring and classification statistics are implemented to record the frequency and specific instances of different error patterns, thereby improving data processing accuracy and anomaly location efficiency.

[0082] Optionally, in step S122, multiple valid data frames are parsed in parallel using multiple threads to obtain the parsing results, including:

[0083] Step S1221: Extract message body fields from multiple valid data frames;

[0084] Step S1222: Perform data validation on the message body fields based on the validation fields to obtain the validation results;

[0085] Step S1223: Parse the Automatic Identification System (AIS) message type from the message body field;

[0086] Step S1224: Based on the verification results and the message type of the Automatic Identification System (AIS), classify multiple valid data frames into multiple task queues;

[0087] Step S1225: Perform parallel processing on the tasks allocated in multiple task queues to obtain the parsing results.

[0088] The aforementioned Automatic Identification System (AIS) message types refer to a classification of data messages defined in the AIS communication protocol. Each type corresponds to specific maritime communication needs, covering various information from ship identification to detailed navigation status. According to ITU-R Recommendation M.1371, AIS message types are subdivided into 0 to 27 categories, each carrying different types of maritime navigation information. Specifically, type 0 is a reserved type for future expansion. Types 1 to 4 carry ship dynamic information, including but not limited to position reports, fixed position reports, extended position reports, and ship operational status reports. Type 5 carries ship static information and navigation-related data reports, providing static information such as name, call sign, IMO number, and navigation plan. Type 6 is a standard type message used for standard communication between ships or between a ship and a shore station. Types 7 and 8 are auxiliary types messages used for acknowledgment of reception, testing, and calling. Type 9 is a search and rescue aircraft report used for the location and status reporting of search and rescue aircraft. Types 10 to 27 are extended message types, covering everything from AIS operations on geostationary satellites to various extended information reports, such as service layer status, enhancement group address, and enhanced single address calls.

[0089] Specifically, for each valid data frame, the start marker field ("005555557e") and the end marker field ("7e(?:[0-9a-fA-F]{2}){1,2}00") can be removed, retaining only the middle 882-byte message body field. Subsequently, the message body field is converted into a binary data stream for decoding according to ITU-R M.1371 recommendations. Simultaneously, a CRC-16 / CCITT-FALSE algorithm check can be performed on each message body field, comparing it with the check field recorded in the frame structure to obtain the check result (pass / fail) to verify data integrity. During message body field decoding, a dedicated AIS message decoding library (such as pyais) can be used to identify and parse the AIS message type (from class 0 to 27) to extract key information such as ship position, speed, heading, and ship identification. Furthermore, based on the verification results and AIS message type, multiple valid data frames can be classified into multiple task queues, and the tasks assigned in the multiple task queues can be processed in parallel to obtain the parsing results.

[0090] Based on steps S1221 to S1225 above, message body fields are extracted from multiple valid data frames, and data verification is performed on the message body fields based on the verification fields to obtain verification results. Then, the Automatic Identification System (AIS) message type is parsed from the message body fields. According to the verification results and the AIS message type, multiple valid data frames are classified into multiple task queues. Finally, the tasks assigned in the multiple task queues are processed in parallel to obtain the parsing results. This can improve data processing efficiency and data integrity, realize the rapid parsing and classification of massive amounts of data, and ensure the verification and tracking of each piece of data, providing more accurate and real-time data support for maritime monitoring, ship trajectory tracking, and shipping safety management.

[0091] Optionally, the data processing method in this embodiment of the invention further includes:

[0092] Configure a thread pool based on the number of processor cores. The thread pool includes multiple producer threads and multiple consumer threads.

[0093] The aforementioned producer thread is responsible for reading valid data frames from the pre-processed .dat file, and classifying the data frames according to the control header information and message body field verification results in the valid data frames, as well as the AIS message type, and pushing them to the corresponding task queue.

[0094] The aforementioned consumer thread is responsible for retrieving data frames from the task queue, parsing and processing them, and writing the parsing results to the database.

[0095] Specifically, configuring thread pools based on the number of processor cores can maximize processor utilization while avoiding context switching overhead and resource contention issues caused by too many threads, thereby improving program execution efficiency.

[0096] For example, the size of the thread pool can be configured to be twice the number of processor cores.

[0097] For example, the thread pool size can also be dynamically adjusted based on the actual processor load and task nature. For instance, the number of threads can be reduced to save resources when the processor load is low, and the number of threads can be increased to improve data processing capacity when the load is high.

[0098] Based on the above optional embodiments, configuring the thread pool based on the number of processor cores can optimize thread management and improve the program's concurrent processing capability and overall execution efficiency.

[0099] In step S1224, based on the verification results and the Automatic Identification System (AIS) message type, multiple valid data frames are classified into multiple task queues, including:

[0100] Using multiple producer threads, multiple valid data frames are classified into multiple task queues based on the verification results and the message type of the Automatic Identification System (AIS).

[0101] For example, multiple producer threads can be used to detect the CRC check result and AIS message type of each valid data frame, and then classify the multiple valid data frames into specific task queues based on the CRC check result and AIS message type. Specifically, the producer threads can classify data frames according to the CRC check result of the control header information and message body in the valid data frames. Furthermore, the producer threads can push the classified valid data frames into specific task queues according to the AIS message type to ensure the orderliness and efficiency of subsequent processing.

[0102] For example, message type 0 data frames that pass the CRC check can be placed in queue 0, message type 1 data frames that fail the CRC check can be placed in queue 1, and so on.

[0103] Based on the above optional embodiments, multiple producer threads are used to classify multiple valid data frames into multiple task queues according to the verification results and AIS message types. This can make full use of the advantages of multi-core processors, avoid the bottleneck of single-threaded processing, and thus improve data processing efficiency.

[0104] In step S1225, the tasks allocated in multiple task queues are processed in parallel, and the parsing results include:

[0105] Multiple consumer threads are used to process tasks allocated in multiple task queues in parallel to obtain the parsing results.

[0106] For example, consumer threads can retrieve valid data frames from different task queues in parallel for processing, making full use of CPU resources and improving data processing efficiency. Specifically, multiple consumer threads are used to process tasks allocated in multiple task queues in parallel. After obtaining the parsing results, the parsing results can be written to the database in batches, while recording various errors in the parsing process for later tracking and analysis.

[0107] Based on the above optional embodiments, using multiple consumer threads to process tasks allocated in multiple task queues in parallel can improve the efficiency and throughput of data parsing. Specifically, by reasonably allocating task load, each thread can focus on processing a specific type of data or a portion of the dataset, thereby reducing task waiting time and resource contention, and maximizing the concurrency of task processing.

[0108] Optionally, the data processing method in this embodiment of the invention further includes:

[0109] In response to the failure of some data frames' message body fields to pass validation among multiple valid data frames, the message body of some data frames is retained and the validation status of some data frames is marked.

[0110] Specifically, in processing satellite AIS data, when faced with a large number of data frames, some frames may fail the CRC checksum verification of their message body fields due to channel interference or other factors during transmission. Related data processing methods tend to discard these failed verification frames to maintain data integrity. However, these methods may miss some data that, although failing the verification, still carries some valuable information.

[0111] In this embodiment of the invention, during the data parsing stage, when the CRC-16 / CCITT-FALSE check result is found to be inconsistent with the check field carried in the data frame, i.e., the CRC check fails, the data will not be discarded immediately. Instead, the entire message body field along with its CRC check status will be saved, and the status of the data frame will be marked as "CRC check failed".

[0112] Based on the above optional embodiments, when the message body fields of some data frames among multiple valid data frames fail validation, retaining the message bodies of some data frames and marking their validation status ensures data integrity and avoids the loss of potentially useful data. Furthermore, retaining the data frames that failed validation also aids in subsequent error localization and problem investigation, ensuring the traceability of the entire data processing flow.

[0113] Optionally, the data processing method in this embodiment of the invention further includes at least one of the following:

[0114] The frame control header fields are extracted from multiple valid data frames, and the frame control header fields are parsed to obtain a structured dictionary. The structured dictionary is used to record the source fields of multiple data frames.

[0115] For each valid data frame among multiple valid data frames, an association mapping is established between the frame control header field and the message body field. The association mapping is used to record the traceability information of each valid data frame.

[0116] A dedicated control header is used to track the logs and perform full-link tracing of the data processing process for each valid data frame in multiple valid data frames.

[0117] Specifically, extracting and parsing the frame control header fields from multiple valid data frames is a crucial step in ensuring data integrity and traceability. The frame control header fields are typically located in the first 40 bytes of each valid data frame and contain a series of data frame tracing fields crucial to the data's origin and status. These data frame tracing fields include, but are not limited to, 18 core tracing fields such as baseband time, synchronization header, master header, satellite number, message reception timestamp, and virtual channel identifier, used to identify and trace the detailed origin and transmission time of each data frame.

[0118] For example, the start position of a data frame can be located based on the start identifier field (“a1b2c3d4”), and then the entire data frame can be extracted based on its total length. For a valid data frame, the first 40 bytes can be extracted as a frame control header field, and the frame control header field can be parsed to obtain a structured dictionary.

[0119] For example, the frame control header fields can be converted into a binary stream, and 18 key fields can be parsed one by one according to the predetermined field length and position, including but not limited to baseband time, synchronization header, master header, satellite number, message reception timestamp, etc. During the parsing process, each field can be converted into the corresponding data type (such as integer, string, or timestamp) according to the definition in the ITU-R M.1371 recommendation standard. After the field parsing is completed, the above information is organized into a structured dictionary format, where the keys are field names and the values ​​are the parsed field contents.

[0120] For example, an entry can be created for each valid data frame, associating the parsed structured dictionary with the message body fields. Specifically, a mapping table containing frame control header fields and message body field IDs can be created to record source information for each frame, such as the origin satellite, transmission time, and virtual channel, ensuring that the message body fields are closely linked to their frame control header fields. This facilitates finding the source and status of the data during subsequent data analysis.

[0121] For example, for each valid data frame, a dedicated control header can be used to track and log information such as frame index, parsing status, and key fields, enabling real-time monitoring and end-to-end traceability.

[0122] Based on the above optional embodiments, the frame control header field is extracted and parsed from multiple valid data frames to obtain a structured dictionary. This ensures accurate traceability of each data entry, enabling immediate identification of potential problems in data processing and rapid location of specific data sources or transmission stages, thus improving the efficiency and accuracy of troubleshooting. For each valid data frame among the multiple valid data frames, an association mapping is established between the frame control header field and the message body field. This integrates and structures the information within the data frame, ensuring that the correlation between data is not lost during large-scale data processing, thereby improving data integrity and consistency. A dedicated control header tracing log is used to perform full-link tracing of the data processing process for each valid data frame among the multiple valid data frames. This records every step from data reception and parsing to storage, including specific timestamps, processing status, error types, etc., providing a detailed operation history for system maintenance. This facilitates rapid backtracking and analysis of the root cause of problems when data processing anomalies occur. It also provides data support for optimizing processing flows and improving system performance, ensuring the transparency and auditability of data processing.

[0123] Optionally, in step S13, the multidimensional error data classification and statistics of the parsing results include:

[0124] Step S1311: Determine the error type and cause of multiple erroneous data frames based on the parsing results;

[0125] Step S1312: Based on the error type and error cause, perform multidimensional error data classification and statistics on multiple error data frames.

[0126] The aforementioned error types include, but are not limited to, CRC check failure, cross-frame data loss, missing fields, abnormal satellite number or timestamp, incorrect message type identification, and unknown errors. Specifically, a CRC check failure occurs when the calculated CRC checksum does not match the checksum field in the data frame. Cross-frame data loss refers to the detection of an incomplete message body in consecutive data frames, i.e., the previous frame has not ended while the next frame has begun; this is usually caused by interference during transmission or reception problems. Missing fields refer to the inability to correctly extract or parse key fields in the data frame (such as ship position, speed, and heading), possibly due to incorrect data format or data corruption during transmission. Abnormal satellite number or timestamp may be caused by data transmission errors or equipment malfunctions. Incorrect message type identification occurs when the message type field of the data frame is corrupted or incorrectly formatted. Unknown errors include, but are not limited to, hardware failures and software bugs.

[0127] For example, multiple counters can be created, each corresponding to a specific error type, such as a CRC failure counter, a field missing counter, a cross-frame error counter, etc. Whenever an erroneous data frame is parsed, the corresponding counter count can be updated based on its error type and cause. Furthermore, erroneous data frames are categorized based on error type and cause, and the number of errors in each category, along with sample information of the erroneous data frames, such as message ID and frame index, is recorded for easy traceability.

[0128] Based on steps S1311 to S1312 above, the error types and causes of multiple erroneous data frames are determined according to the parsing results. Then, based on the error types and causes, multi-dimensional error data classification and statistics are performed on the multiple erroneous data frames. This not only counts the frequency of CRC check failures, but also identifies multiple types of errors such as cross-frame data loss, missing specific fields, and non-standard formats. At the same time, satellite information, timestamps, and virtual channel identifiers related to the errors are recorded, which facilitates tracing the specific environment and conditions in which the errors occurred. This improves data integrity and reliability and shortens the fault location time.

[0129] Optionally, in step S13, batch storage of the parsing results includes:

[0130] Step S1321: Select a database connection from the database connection pool, wherein the number of database connections in the database connection pool is determined based on the number of threads in the thread pool.

[0131] Step S1322: Divide the parsing results into batches based on the message type of the Automatic Identification System (AIS) to obtain multiple batches of data of the same type;

[0132] Step S1323: Using a database connection, multiple batches of the same type of data are stored in batches through transaction commit.

[0133] In multi-threaded parallel processing, each thread needs to interact with the database to store the parsed data. To avoid the overhead of re-establishing and disconnecting connections for each database operation, a database connection pool can be used to manage these connections. Specifically, a series of database connections can be pre-created in the database connection pool and kept in an available state. When a thread needs to access the database, it can "borrow" a database connection from the pool and "return" it to the pool after use, instead of closing it immediately.

[0134] For example, the number of database connections in a database connection pool can be equal to the number of threads in a thread pool, thereby ensuring that each executing thread has an available database connection without waiting for a connection to be created.

[0135] For example, to improve the efficiency of batch data insertion, the parsed AIS message data can be classified according to its type, and messages of the same type can be grouped into the same batch to obtain multiple batches of data of the same type. Message data of the same type can be written to the database at once, thereby reducing the number of database operations and lowering I / O costs.

[0136] For example, a data structure or queue can be used to cache data of the same type. When a preset batch size is reached, a batch storage operation is triggered. Specifically, the batch size can be set to 200 records per batch, merging data of the same type into a batch SQL statement, and executing the batch insert in a single operation.

[0137] For example, a transaction can be created to execute batch SQL insert statements, adding multiple batches of data to a database table at once. The transaction is then committed upon completion. If any errors occur during transaction execution, the transaction is rolled back, ensuring that no data is partially inserted and maintaining database consistency.

[0138] Based on steps S1321 to S1323 above, a database connection is selected from the database connection pool. Then, the parsing results are batched based on the Automatic Identification System (AIS) message type, resulting in multiple batches of data of the same type. Finally, using the database connection and a transaction commit mechanism, multiple batches of the same type of data are stored in bulk. This reduces the number of database connection creation and disconnection cycles and lowers database operation overhead by utilizing the connection pool's connection reuse mechanism. Simultaneously, processing batches of data based on AIS message type makes batch writing more efficient, reducing I / O costs when inserting single data entries. The transaction commit mechanism ensures data consistency and integrity; even if errors occur during processing, rollback can maintain the correct data state, avoiding data redundancy or loss.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0140] This invention also provides a data processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0141] Figure 2 This is a structural block diagram of a data processing apparatus according to one embodiment of the present invention, such as... Figure 2 As shown, the device includes:

[0142] The acquisition module 201 is used to acquire the target data file, wherein the target data file is used to record the Automatic Identification System (AIS) message data to be processed during satellite communication.

[0143] Parsing module 202 is used to perform multi-threaded parallel parsing of the target data file based on the frame structure of the Automatic Identification System (AIS) message data to obtain the parsing results;

[0144] The processing module 203 is used to perform multi-dimensional error data classification statistics and / or batch storage of the parsing results.

[0145] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0146] According to another aspect of the present invention, a satellite device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the data processing method of the present invention during runtime.

[0147] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the storage medium is located to execute the data processing method of the present invention.

[0148] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0149] Step S11: Obtain the target data file, wherein the target data file is used to record the Automatic Identification System (AIS) message data to be processed during satellite communication;

[0150] Step S12: Based on the frame structure of the Automatic Identification System (AIS) message data, perform multi-threaded parallel parsing on the target data file to obtain the parsing results;

[0151] Step S13: Perform multi-dimensional error data classification statistics and / or batch storage of the parsing results.

[0152] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0153] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the data processing method of the present invention.

[0154] Optionally, in this embodiment, the above-mentioned computer program product can be configured as a computer program that performs the following steps:

[0155] Step S11: Obtain the target data file, wherein the target data file is used to record the Automatic Identification System (AIS) message data to be processed during satellite communication;

[0156] Step S12: Based on the frame structure of the Automatic Identification System (AIS) message data, perform multi-threaded parallel parsing on the target data file to obtain the parsing results;

[0157] Step S13: Perform multi-dimensional error data classification statistics and / or batch storage of the parsing results.

[0158] According to another aspect of the present invention, a chip system is also provided, comprising: a processor for calling and running a computer program from a memory, causing a communication device equipped with the chip system to perform the data processing method of the present invention.

[0159] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0160] Step S11: Obtain the target data file, wherein the target data file is used to record the Automatic Identification System (AIS) message data to be processed during satellite communication;

[0161] Step S12: Based on the frame structure of the Automatic Identification System (AIS) message data, perform multi-threaded parallel parsing on the target data file to obtain the parsing results;

[0162] Step S13: Perform multi-dimensional error data classification statistics and / or batch storage of the parsing results.

[0163] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0164] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0165] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0166] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0168] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0169] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: Obtain a target data file, wherein the target data file is used to record Automatic Identification System (AIS) message data to be processed during satellite communication; Based on the frame structure of the Automatic Identification System (AIS) message data, the target data file is parsed in a multi-threaded parallel manner to obtain the parsing results. The parsing results are then subjected to multidimensional error data classification statistics and / or batch data storage.

2. The data processing method according to claim 1, characterized in that, The frame structure of the Automatic Identification System (AIS) message data includes the following fields: frame control header field, start identifier field, message body field, start marker field for a single message, end marker field for a single message, and check field.

3. The data processing method according to claim 1, characterized in that, Obtaining the target data file includes: The original data file is loaded using a binary reading method; The binary data in the original data file is converted into a hexadecimal string to obtain the target data file.

4. The data processing method according to claim 2, characterized in that, Based on the frame structure of the Automatic Identification System (AIS) message data, the target data file is parsed in a multi-threaded parallel manner to obtain the parsing results, including: Based on the frame structure of the Automatic Identification System (AIS) message data, multiple valid data frames are extracted from the target data file. The multiple valid data frames are parsed in parallel using multiple threads to obtain the parsing results.

5. The data processing method according to claim 4, characterized in that, Based on the frame structure of the Automatic Identification System (AIS) message data, extracting the plurality of valid data frames from the target data file includes: Based on the start identifier field, the start marker field of the single message, and the end marker field of the single message, the plurality of valid data frames are extracted from the target data file, wherein the plurality of valid data frames simultaneously match the start identifier field, the start marker field of the single message, and the end marker field of the single message; The data processing method further includes: Based on the start identifier field, the start marker field of the single message, and the end marker field of the single message, at least one erroneous data frame is detected from the target data file, and the at least one erroneous data frame is classified and statistically analyzed. The at least one erroneous data frame matches the start identifier field, but the at least one erroneous data frame does not match the start marker field or the end marker field of the single message.

6. The data processing method according to claim 4, characterized in that, The parsing results obtained by performing multi-threaded parallel parsing on the multiple valid data frames include: Extract the message body fields from the plurality of valid data frames; The message body fields are validated based on the validation fields to obtain the validation results. The Automatic Identification System (AIS) message type is parsed from the message body field; Based on the verification results and the message type of the Automatic Identification System (AIS), the multiple valid data frames are classified into multiple task queues; The tasks allocated in the multiple task queues are processed in parallel to obtain the parsing results.

7. The data processing method according to claim 6, characterized in that, The data processing method further includes: The thread pool is configured based on the number of processor cores, wherein the thread pool includes: multiple producer threads and multiple consumer threads; Based on the verification results and the Automatic Identification System (AIS) message type, classifying the multiple valid data frames into the multiple task queues includes: Using the multiple producer threads, the multiple valid data frames are classified into multiple task queues based on the verification results and the Automatic Identification System (AIS) message type. The tasks allocated in the multiple task queues are processed in parallel to obtain the parsing results, including: The multiple consumer threads are used to process the tasks allocated in the multiple task queues in parallel to obtain the parsing results.

8. The data processing method according to claim 6, characterized in that, The data processing method further includes: In response to the failure of the message body field of a portion of the plurality of valid data frames to pass validation, the message body of the portion of data frames is retained and the validation status of the portion of data frames is marked.

9. The data processing method according to claim 2, characterized in that, The data processing method further includes at least one of the following: The frame control header fields are extracted from multiple valid data frames, and the frame control header fields are parsed to obtain a structured dictionary, wherein the structured dictionary is used to record multiple data frame traceability fields; For each of the plurality of valid data frames, an association mapping is established between the frame control header field and the message body field, wherein the association mapping is used to record the source information of each valid data frame; A dedicated control head is used to track the logs and perform full-link tracking of the data processing process for each of the multiple valid data frames.

10. The data processing method according to claim 1, characterized in that, The multidimensional error data classification and statistics of the parsing results include: Based on the parsing results, the error types and causes of multiple erroneous data frames are determined; Based on the error type and the error cause, multidimensional error data classification and statistics are performed on the multiple error data frames.

11. The data processing method according to claim 1, characterized in that, Batch storage of the parsing results includes: A database connection is selected from the database connection pool, wherein the number of database connections in the database connection pool is determined based on the number of threads in the thread pool. The parsing results are divided into batches based on the message type of the Automatic Identification System (AIS) to obtain multiple batches of data of the same type. Using the database connection, multiple batches of the same type of data are stored in batches through transaction commit.

12. A satellite device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the data processing method according to any one of claims 1 to 11.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1 to 11.

15. A chip system, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a communication device equipped with the chip system to perform the data processing method as described in any one of claims 1 to 11.

Images