A data processing method and device, electronic equipment and computer readable medium

Abstract

This application discloses a data processing method, apparatus, electronic device, and computer-readable medium, relating to the field of computer technology. The method includes: acquiring trajectory point data according to a data processing request to generate a data processing task; writing the data processing task to an in-memory database based on an asynchronous queue and thread pool; distributing the data processing task in a distributed manner to determine an allocation server; determining the role of the allocation server; and, based on the role, request reception data at the time of election timeout, and pre-candidates, determining a target server to perform denoising processing on the trajectory point data corresponding to the data processing task, obtaining denoised trajectory point data; comparing the number of trajectory points in the denoised trajectory point data with the number of trajectory points before denoising; if the number of trajectory points is inconsistent, updating the corresponding number of trajectory points before denoising in the in-memory database based on the denoised trajectory point data. This improves the efficiency, accuracy, and stability of data processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and computer-readable medium. Background Technology

[0002] Currently, in the logistics and transportation field, Redis is only used to store trajectory point data. However, trajectory point data is typically written at high frequency, which can put significant write pressure on Redis. Although Redis has high write performance, it may still face write bottlenecks under extremely high concurrency. Furthermore, high-frequency writes can also lead to frequent Redis persistence operations, thus affecting overall performance. While the common Raft algorithm can maintain data consistency when requests are encountered, Raft's election process is time-consuming, especially when a leader fails and an election is needed to select a new leader. During the election, the system may be unable to process client requests, leading to brief service interruptions. Trajectory denoising data processing suffers from high write pressure, low efficiency, poor stability, and slow response time. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data processing method, apparatus, electronic device, and computer-readable medium that can solve the problems of high writing pressure, low efficiency, poor stability, and slow response time in existing trajectory denoising data processing.

[0004] To achieve the above objectives, according to one aspect of the embodiments of this application, a data processing method is provided, comprising:

[0005] Obtain trajectory point data based on the received data processing request, and generate a data processing task based on the trajectory point data;

[0006] Data processing tasks are written to an in-memory database using asynchronous queues and thread pools, and the data processing tasks in the in-memory database are distributed in a distributed manner to determine the allocation server;

[0007] Determine the role of the allocation server, and based on the role, the request to receive data when the election timer expires, and the pre-candidates, determine the target server;

[0008] The target server is invoked to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on the trajectory denoising algorithm, so as to obtain denoised trajectory point data.

[0009] Compare the number of trajectory points in the denoised trajectory point data with the corresponding number of trajectory points before denoising. If the number of trajectory points is inconsistent, update the corresponding number of trajectory points before denoising in the in-memory database based on the denoised trajectory point data.

[0010] Optionally, the target server is determined, including:

[0011] The role of the allocation server is determined. If the role is follower, the corresponding data processing task is redirected to the server with the role of leader. If no request is received when the election timer of the allocation server expires, the role of the allocation server is changed to pre-candidate, and a pre-election request is sent to the server with the role of non-leader. In response to receiving more than a preset threshold of positive votes, the role of the allocation server is changed from pre-candidate to candidate. Based on the servers with the determined role of candidate, an election vote is performed to determine the target server with the role of leader.

[0012] Optionally, before distributing the data processing tasks in the in-memory database in a distributed manner, the method further includes:

[0013] Call the trajectory processing service to add data processing tasks to the asynchronous queue, and call the thread pool to write the data processing tasks in the asynchronous queue to the in-memory database.

[0014] Optionally, the data processing method further includes:

[0015] The target server, whose role is leader, periodically sends heartbeat messages to all servers whose role is follower.

[0016] During the election voting process, servers acting as candidates periodically send heartbeat messages to servers acting as non-leaders.

[0017] Optionally, the data processing method further includes:

[0018] In response to an interruption in the noise reduction processing service provided by the target server, obtain the service interruption time;

[0019] The election timer's timeout is dynamically adjusted based on the service interruption time.

[0020] Optionally, the trajectory point data is obtained according to the received data processing request, including:

[0021] Get the scheduled task corresponding to the data processing request, and in response to the scheduled task being triggered, extract the earliest trajectory point data of the preset duration from the data source.

[0022] In addition, this application also provides a data processing apparatus, including:

[0023] The acquisition unit is configured to acquire trajectory point data based on the received data processing request and generate a data processing task based on the trajectory point data.

[0024] The allocation server determination unit is configured to write data processing tasks to an in-memory database based on an asynchronous queue and a thread pool, and to distribute the data processing tasks in the in-memory database in a distributed manner to determine the allocation server;

[0025] The target server determination unit is configured to play the role of a target determination and allocation server. Based on the role, the request receiving data when the election timer expires, and the pre-candidates, it determines the target server.

[0026] The denoising unit is configured to call the target server to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on the trajectory denoising algorithm, so as to obtain denoised trajectory point data.

[0027] The update unit is configured to compare the number of trajectory points in the denoised trajectory point data with the corresponding undenoised trajectory point data. If the number of trajectory points is inconsistent, the corresponding undenoised trajectory point data in the in-memory database is updated based on the denoised trajectory point data.

[0028] Optionally, the target server determination unit is further configured to:

[0029] The role of the allocation server is determined. If the role is follower, the corresponding data processing task is redirected to the server with the role of leader. If no request is received when the election timer of the allocation server expires, the role of the allocation server is changed to pre-candidate, and a pre-election request is sent to the server with the role of non-leader. In response to receiving more than a preset threshold of positive votes, the role of the allocation server is changed from pre-candidate to candidate. Based on the servers with the determined role of candidate, an election vote is performed to determine the target server with the role of leader.

[0030] Optionally, the data processing apparatus further includes a writing unit configured to:

[0031] Call the trajectory processing service to add data processing tasks to the asynchronous queue, and call the thread pool to write the data processing tasks in the asynchronous queue to the in-memory database.

[0032] Optionally, the data processing device further includes a heartbeat information transmission unit, configured to:

[0033] The target server, whose role is leader, periodically sends heartbeat messages to all servers whose role is follower.

[0034] During the election voting process, servers acting as candidates periodically send heartbeat messages to servers acting as non-leaders.

[0035] Optionally, the data processing apparatus further includes a dynamic adjustment unit configured to:

[0036] In response to an interruption in the noise reduction processing service provided by the target server, obtain the service interruption time;

[0037] The election timer's timeout is dynamically adjusted based on the service interruption time.

[0038] Optionally, the acquisition unit is further configured to:

[0039] Get the scheduled task corresponding to the data processing request, and in response to the scheduled task being triggered, extract the earliest trajectory point data of the preset duration from the data source.

[0040] In addition, this application also provides a data processing electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the data processing method described above.

[0041] In addition, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described above.

[0042] To achieve the above objectives, according to another aspect of the embodiments of this application, a computer program product is provided.

[0043] A computer program product according to an embodiment of this application includes a computer program that, when executed by a processor, implements the data processing method provided in an embodiment of this application.

[0044] One embodiment of the above invention has the following advantages or beneficial effects: This application obtains trajectory point data according to the received data processing request, generates a data processing task based on the trajectory point data; writes the data processing task to an in-memory database based on an asynchronous queue and thread pool, distributes the data processing tasks in the in-memory database to determine the allocation server; determines the role of the allocation server, and determines the target server based on the role, the request receiving data when the election timer times out, and pre-candidates; calls the target server to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on the trajectory denoising algorithm to obtain denoised trajectory point data; compares the number of trajectory points in the denoised trajectory point data with the corresponding undenoised trajectory point data, and if the number of trajectory points is inconsistent, updates the corresponding undenoised trajectory point data in the in-memory database based on the denoised trajectory point data. This improves the efficiency and accuracy of data processing during trajectory denoising, increases data processing response time, avoids excessive write pressure on the in-memory database, reduces denoising processing service interruption time, and improves data processing stability.

[0045] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0046] The accompanying drawings are provided to better understand this application and do not constitute an undue limitation thereof. Wherein:

[0047] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application;

[0049] Figure 3 This is a schematic diagram illustrating the election process of a data processing method according to an embodiment of this application.

[0050] Figure 4 This is a schematic diagram of data writing according to a data processing method of one embodiment of this application;

[0051] Figure 5 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of the main units of a data processing apparatus according to an embodiment of this application;

[0053] Figure 7 This is an exemplary system architecture diagram to which embodiments of this application can be applied;

[0054] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0055] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solutions of this application comply with relevant national laws and regulations. It should also be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions. The collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions of this application all comply with relevant laws and regulations, are used for legal and reasonable purposes, do not violate public order and good morals, are not shared, disclosed, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to user personal information, safeguard user personal information security, cybersecurity, and national security, and ensure that those authorized to access personal information comply with relevant laws and regulations. Once this user personal information is no longer needed, risks should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0056] When used, including in certain relevant applications, data is deidentified to protect user privacy, for example by removing specific identifiers, controlling the amount or specificity of stored data, controlling how data is stored, and / or other methods.

[0057] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application, as shown below. Figure 1 As shown, the data processing method mainly includes the following steps S101-S105.

[0058] Step S101: Obtain trajectory point data according to the received data processing request, and generate a data processing task based on the trajectory point data.

[0059] In this embodiment, the entity executing the data processing method (e.g., a server) can receive data processing requests via wired or wireless connections. For example, the data processing request could be a request to perform real-time noise reduction on vehicle trajectory data. Based on the data processing request, the trajectory point data to be processed is obtained. To improve data processing efficiency, data processing tasks are generated based on the trajectory point data and effectively distributed across multiple servers.

[0060] Specifically, the process of obtaining trajectory point data based on the received data processing request includes: obtaining the timed task corresponding to the data processing request; and, in response to the timed task being triggered, extracting the earliest trajectory point data of a preset duration from the data source.

[0061] To control costs without reducing read speed, trajectory point data for the earliest preset duration (e.g., the earliest 20 minutes) can be extracted from the data source via scheduled tasks.

[0062] Step S102: Write data processing tasks to an in-memory database based on an asynchronous queue and thread pool, and distribute the data processing tasks in the in-memory database to determine the allocation server.

[0063] In-memory databases, such as Redis.

[0064] Specifically, before distributing the data processing tasks in the in-memory database, the data processing method also includes: calling the trajectory processing service to add the data processing tasks to the asynchronous queue, and calling the thread pool to write the data processing tasks in the asynchronous queue to the in-memory database.

[0065] By placing data processing tasks into an asynchronous queue, a thread pool reads the data processing tasks from the asynchronous queue and writes them to an in-memory database. Then, a distributed processing component distributes the data processing tasks in the in-memory database Redis to accurately determine the final allocation server for each data processing task.

[0066] Step S103: Determine the role of the allocation server, and determine the target server based on the role, the request to receive data when the election timer expires, and the pre-candidates.

[0067] After determining the final allocation server for each data processing task, the role of the allocation server can be determined by the message sending and receiving status of the allocation server. For example, the roles of the allocation server may include leader (responsible for handling all requests, such as read or write data processing requests), follower (if a data processing request is connected to the node corresponding to the follower, the node corresponding to the follower will redirect the received processing request to the node corresponding to the leader), pre-candidate, and candidate.

[0068] In this embodiment, the node mentioned refers to a server. The system retrieves request data received when the election timer of the allocation server times out, such as heartbeat messages, pre-election requests, and snapshot delivery requests. If the request data is empty, the allocation server is designated as a pre-candidate. If the request data is either a heartbeat message or a snapshot delivery request, there is no pre-candidate. If the request data is a pre-election request, the node that sent the pre-election request is designated as a pre-candidate. If the node corresponding to the pre-candidate receives a majority of votes, it officially enters the candidate state and begins the actual election process to ultimately determine the leader node, i.e., the target server. The target server performs real-time denoising processing on the trajectory point data. Specifically, the method further includes: the target server, acting as the leader, periodically sends heartbeat messages to all servers acting as followers to prevent all followers from initiating an election; during the election voting process, servers acting as candidates periodically send heartbeat messages to servers acting as non-leaders to ensure that non-leader servers are aware of the election process, reducing service interruptions.

[0069] Step S104: Call the target server to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on the trajectory denoising algorithm to obtain denoised trajectory point data.

[0070] Trajectory data denoising algorithms can be, for example, methods based on sliding window average filtering, median filtering, or Kalman filtering. The target server is then invoked to perform denoising processing on the trajectory point data corresponding to the data processing task using methods based on sliding window average filtering, median filtering, or Kalman filtering, resulting in denoised trajectory point data.

[0071] Step S105: Compare the number of trajectory points in the denoised trajectory point data with the corresponding number of trajectory points before denoising. If the number of trajectory points is inconsistent, update the corresponding number of trajectory points before denoising in the memory database based on the denoised trajectory point data.

[0072] If the number of trajectory points in the denoised trajectory point data changes compared to the number of trajectory points in the undenoised trajectory point data, it indicates an inconsistency in the amount of trajectory point data. In this case, the corresponding undenoised trajectory point data in the in-memory database Redis will be replaced with the denoised trajectory point data to ensure the accuracy of trajectory data processing.

[0073] This embodiment obtains trajectory point data based on received data processing requests, generates data processing tasks based on the trajectory point data, writes the data processing tasks to an in-memory database using an asynchronous queue and thread pool, and distributes the data processing tasks in the in-memory database to determine the allocation server. The role of the allocation server is determined based on the role, the request receiving data at the timeout of the election timer, and pre-candidates to identify the target server. The target server is then invoked to denoise the trajectory point data corresponding to the data processing task using a trajectory denoising algorithm to obtain denoised trajectory point data. The number of trajectory points in the denoised trajectory point data is compared with the number of trajectory points in the original trajectory point data. If the number of trajectory points is inconsistent, the corresponding original trajectory point data in the in-memory database is updated based on the denoised trajectory point data. This improves the efficiency and accuracy of data processing during trajectory denoising, increases data processing response time, avoids excessive write pressure on the in-memory database, reduces denoising service interruption time, and improves data processing stability.

[0074] Figure 2 This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application, as shown below. Figure 2 As shown, the data processing method mainly includes the following steps S201-S205.

[0075] Step S201: Obtain trajectory point data according to the received data processing request, and generate a data processing task based on the trajectory point data.

[0076] Trajectory point data can be vehicle trajectory point data generated by vehicles transporting goods in the logistics industry. Trajectory point data can be divided according to one or more dimensions such as time, vehicle travel area, or data volume. Based on the divided trajectory point data, various data processing tasks can be generated to improve data processing efficiency.

[0077] Step S202: Write data processing tasks to an in-memory database based on an asynchronous queue and thread pool, and distribute the data processing tasks in the in-memory database to determine the allocation server.

[0078] By placing data processing tasks into an asynchronous queue, the thread pool reads the data processing tasks from the asynchronous queue and writes them to an in-memory database. Then, the distributed processing component distributes the data processing tasks in the in-memory database Redis to accurately determine the final allocation server for each data processing task.

[0079] Step S203: Determine the role of the allocation server. If the role is follower, redirect the corresponding data processing task to the server with the role of leader. If no request is received when the election timer of the allocation server expires, change the role of the allocation server to a pre-candidate and send a pre-selection request to the server with the role of non-leader. In response to receiving more than a preset threshold of affirmative votes, change the role of the allocation server from pre-candidate to candidate. Based on the servers with the determined role of candidate, perform election voting to determine the target server with the role of leader.

[0080] If a data processing task is assigned to a server with the role of a follower, that server will redirect the data processing task to a server with the role of a leader to ensure orderly and accurate data processing and avoid duplicate data processing.

[0081] If no requests such as heartbeat messages, pre-selection requests, or snapshot delivery requests are received when the election timer of the allocation server expires, the role of the allocation server (e.g., C) is changed to that of a pre-candidate. The allocation server with the role of a pre-candidate sends pre-selection requests to servers with the role of non-leader (e.g., follower, other pre-candidates). If the allocation server (e.g., C) is the first to receive more than a preset threshold of affirmative votes, the role of the allocation server (e.g., C) is changed from pre-candidate to candidate. Election voting is performed based on the servers with the determined role of candidates to accurately determine the target server with the role of leader.

[0082] Step S204: Call the target server to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on the trajectory denoising algorithm to obtain denoised trajectory point data.

[0083] Specifically, the data processing method also includes: in response to a noise reduction service interruption provided by the target server, obtaining the service interruption time; and dynamically adjusting the timeout of the election timer based on the service interruption time. By dynamically adjusting the timeout of the election timer, election efficiency and system stability are significantly improved, and service interruption time is reduced.

[0084] Step S205: Compare the number of trajectory points in the denoised trajectory point data with the corresponding number of trajectory points before denoising. If the number of trajectory points is inconsistent, update the corresponding number of trajectory points before denoising in the memory database based on the denoised trajectory point data.

[0085] If the number of trajectory points in the denoised trajectory point data changes compared to the number of trajectory points in the undenoised trajectory point data, it indicates an inconsistency in the amount of trajectory point data. In this case, the corresponding undenoised trajectory point data in the in-memory database Redis will be replaced with the denoised trajectory point data to ensure the accuracy of trajectory data processing.

[0086] To achieve more accurate real-time storage and noise reduction of logistics vehicle trajectory points, Netty combined with Redis is used to store the trajectory points continuously uploaded by transportation vehicles nationwide. To more accurately display the real-time location of transportation vehicles, the distributed consensus algorithm C-Raft is used to ensure data consistency without affecting system stability.

[0087] Figure 3 This is a schematic diagram illustrating the election process of a data processing method according to an embodiment of this application. C-Raft Algorithm: Distributed systems typically consist of multiple nodes connected asynchronously via a network. Each node has independent computation and storage, and nodes collaborate through network communication. Distributed consistency refers to multiple nodes reaching a consensus on the value of a variable; once consensus is reached, the current value of the variable is determined. The C-Raft algorithm primarily optimizes the leader election process of the Raft algorithm. It introduces a pre-election phase and a fast election mechanism, improves the heartbeat mechanism, and dynamically adjusts the election timeout, significantly improving election efficiency and system stability, and reducing service interruption time. This improvement allows the distributed system to recover and operate normally more quickly when the leader fails. Regarding the C-Raft algorithm, in the pre-election phase: when a follower (e.g., ...) becomes the leader of a system, the leader is selected by the leader of the system. Figure 3 Node C (term number: 0, timeout: 300ms) did not receive a leader within the election timeout period (e.g., Figure 3 When node A (task number: 1) sends a heartbeat message (i.e., the leader: node A has not sent a heartbeat to the follower: node C), Figure 3 Followers that haven't received a heartbeat message: Node C will enter the pre-selection phase as a pre-candidate. This follower: Node C will transform itself into a pre-candidate and send messages to other nodes (e.g., ...). Figure 3 Other followers who receive heartbeat messages from the leader, such as Figure 3 Node B in the context: Term number: 0, Timeout: 200ms and Figure 3 Another node C (term number: 0, timeout: 300ms) sends a pre-vote request. Other nodes (e.g.) Figure 3 Other followers, such as Figure 3Node B in the context: Term number: 0, Timeout: 200ms and Figure 3 Another node C (term number: 0, timeout: 300ms) receives a pre-election request. If they believe the log of the node that sent the request (i.e., the pre-candidate) is up-to-date (i.e., the log index and term number are not lagging behind), they will vote in favor. The node that sent the request as a pre-candidate will then receive the votes of approval. If the pre-candidate receives the votes of approval from a majority of its followers (nodes), then the pre-candidate (node ​​C: term number: 0, timeout: 300ms) will officially enter the candidate state and begin the actual election process. The implementation principle is as follows... Figure 3 As shown, the leader periodically sends heartbeat messages (AppendEntries RPC) to all followers to prevent them from initiating an election. During the election, candidates can also send heartbeat messages to ensure that other nodes in the system are aware that an election is taking place, reducing service interruptions.

[0088] The Boss Group (also known as BossNioEventLoopGroup) is specifically responsible for receiving connection requests from clients. The Worker Group (also known as WorkerNioEventLoopGroup) is specifically responsible for handling read and write operations on the connections. Netty, in conjunction with Redis, improves performance in high-concurrency write scenarios by using Netty to process requests from both the Boss Group and Worker Groups and placing write requests into an asynchronous queue. A dedicated thread pool or process reads requests from the queue and writes them to Redis periodically. This balances the write load and prevents excessive write pressure on Redis. The implementation diagram is shown below. Figure 4 As shown.

[0089] Figure 5This is a schematic diagram of the main flow of a data processing method according to an embodiment of this application. Upon receiving a trajectory point message (i.e., a data processing message), logical processing is performed based on the corresponding task type. If it is a scheduled task, historical trajectories are transmitted to the processing node by sending an MQ message. If the message is successfully sent, the processing node processes the MQ message, stores the historical trajectory data in a Hive table, and then calls the Netty trajectory processing service to periodically delete the Redis data, ending the process. If it is a data processing task, the Netty trajectory processing service is called to store the data in Redis. The data processing task requiring denoising is effectively distributed to multiple servers using CRAF distributed processing. For each data processing task, trajectory data from the most recent 20 minutes is extracted from the assigned server and processed using an advanced trajectory denoising algorithm. The system then determines whether the trajectory has changed, specifically by comparing the number of trajectory points before and after denoising. If the number of trajectory points has changed, the Redis data is updated with the denoised data via the Netty trajectory processing service, ending the process. If the number of trajectory points has not changed, the update is skipped, and the process ends.

[0090] For example, after receiving trajectory point messages, the Netty framework (i.e., an asynchronous event-driven network application framework) is used to process requests, and trajectory data is processed concurrently through multi-threading. To reduce the write pressure on Redis, this embodiment of the application adopts a batch write queue mechanism to store the processed data in batches into the high-performance, fast-read-write, fast-access, and fast-response-time in-memory database Redis, thereby greatly meeting the requirements of data real-time performance. By adopting CRAF distributed processing technology, the data processing tasks that need to be denoised can be effectively distributed to multiple servers. This method significantly improves processing efficiency and greatly shortens the overall time of trajectory denoising. Distributed processing not only optimizes resource utilization but also ensures the stability and response speed of the system under high concurrency, thereby greatly improving the real-time performance and reliability of data processing. For each data processing task, trajectory data within the last 20 minutes (this is just an example, and this embodiment of the application does not specifically limit this time) is first extracted and processed using an advanced trajectory denoising algorithm. Then, the number of trajectory points before and after denoising is compared. If a change is found, the changed data is updated; otherwise, the update is skipped. This strategy significantly reduces unnecessary processing time, making the trajectory denoising process more efficient and timely, thereby improving the overall system's response speed and performance. To control costs without reducing read speed, MQ messages are sent and data is stored in Hive. The specific solution is as follows: A scheduled task is set up to execute every 20 minutes, retrieving the earliest 20 minutes of data from the data source. Then, distributed processing technology is used to distribute the task, transmitting data to processing nodes via MQ messages, and finally writing the processed data to Hive storage. Hive is a data warehouse built on Hadoop, providing a SQL-like query language, HiveQL, for querying and managing large-scale datasets. Hive can map structured data files to a database table and provide an SQL interface for data querying.

[0091] The C-Raft algorithm is a distributed consensus algorithm, an improvement and extension of the Raft algorithm. The Raft algorithm itself was designed to solve the consistency problem in distributed systems, aiming to achieve a fault-tolerant distributed system where multiple nodes can maintain a consistent state in the face of network partitions or node failures. Netty is a Java-based asynchronous event-driven network application framework for rapidly developing high-performance, highly reliable network servers and clients. It provides a rich set of APIs, simplifying the complexity of network programming, and is particularly suitable for handling high-concurrency and high-throughput network applications. This application introduces an improved C-Raft algorithm, which adds a new role, the candidate, enabling rapid leader election without affecting system stability, thus solving the problem of brief service interruptions caused by Raft during the election process. Replacing the distributed consensus algorithm Raft with the C-Raft algorithm optimizes the leader election process of the Raft algorithm. Firstly, a pre-election phase and a fast election mechanism are introduced into the leader election process. The heartbeat mechanism is improved, and the election timeout is dynamically adjusted, significantly improving election efficiency and system stability, and reducing service interruption time. Requests are handled using Netty, and write requests are placed in an asynchronous queue. A dedicated thread pool or process reads requests from the queue and writes them to Redis. This balances the write load and prevents excessive write pressure on Redis, ensuring the stability and quality of real-time storage of trajectory points during vehicle transportation.

[0092] Figure 6 This is a schematic diagram of the main units of a data processing apparatus according to an embodiment of this application. Figure 6 As shown, the data processing device 600 includes an acquisition unit 601, an allocation server determination unit 602, a target server determination unit 603, a noise reduction unit 604, and an update unit 605.

[0093] The acquisition unit 601 is configured to acquire trajectory point data according to the received data processing request and generate a data processing task based on the trajectory point data.

[0094] The allocation server determination unit 602 is configured to write data processing tasks to an in-memory database based on an asynchronous queue and a thread pool, and to distribute the data processing tasks in the in-memory database in a distributed manner to determine the allocation server.

[0095] The target server determination unit 603 is configured to play the role of a target determination and allocation server, and determines the target server based on the role, the request to receive data when the election timer expires, and the pre-candidates.

[0096] The denoising unit 604 is configured to call the target server to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on the trajectory denoising algorithm, so as to obtain denoised trajectory point data.

[0097] Update unit 605 is configured to compare the number of trajectory points in the denoised trajectory point data with the corresponding undenoised trajectory point data. If the number of trajectory points is inconsistent, the corresponding undenoised trajectory point data in the memory database is updated based on the denoised trajectory point data.

[0098] In some embodiments, the target server determination unit 603 is further configured to: determine the role of the allocation server; if the role is a follower, redirect the corresponding data processing task to the server with the role of a leader; if no request is received when the election timer of the allocation server expires, change the role of the allocation server to a pre-candidate; send a pre-selection request to the server with the role of a non-leader; in response to receiving more than a preset threshold of affirmative votes, change the role of the allocation server from a pre-candidate to a candidate; and perform an election vote based on the determined candidate server to determine the target server with the role of a leader.

[0099] In some embodiments, the data processing apparatus further includes Figure 6 The write unit, not shown, is configured to: call the trajectory processing service to add the data processing task to the asynchronous queue, and call the thread pool to write the data processing task in the asynchronous queue to the in-memory database.

[0100] In some embodiments, the data processing apparatus further includes Figure 6 The heartbeat information sending unit (not shown) is configured such that: the target server, which plays the role of leader, periodically sends heartbeat messages to all servers, which play the role of follower; during the election voting process, the server, which plays the role of candidate, periodically sends heartbeat messages to the server, which plays the role of non-leader.

[0101] In some embodiments, the data processing apparatus further includes Figure 6 The dynamic adjustment unit, not shown, is configured to: obtain the service interruption time in response to the interruption of the noise reduction processing service provided by the target server; and dynamically adjust the timeout time of the election timer based on the service interruption time.

[0102] In some embodiments, the acquisition unit 601 is further configured to: acquire a timed task corresponding to a data processing request, and in response to the timed task being triggered, extract the earliest preset duration trajectory point data from the data source.

[0103] It should be noted that the data processing method and data processing device in this application are related in specific implementation, so repeated content will not be described again.

[0104] Figure 7 An exemplary system architecture 700 is shown that can be applied to the data processing method or data processing apparatus of the embodiments of this application.

[0105] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0106] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0107] Terminal devices 701, 702, and 703 can be various electronic devices with data processing screens and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0108] Server 705 can be a server providing various services, such as a backend management server supporting data processing requests submitted by users using terminal devices 701, 702, and 703 (this is just an example). The backend management server can obtain trajectory point data based on the received data processing requests, generate data processing tasks based on the trajectory point data, write the data processing tasks to an in-memory database using an asynchronous queue and thread pool, distribute the data processing tasks in the in-memory database to determine the allocation server, determine the role of the allocation server, and identify the target server based on the role, request receiving data when the election timer times out, and pre-candidates. The target server is then invoked to denoise the trajectory point data corresponding to the data processing task using a trajectory denoising algorithm to obtain denoised trajectory point data. The number of trajectory points in the denoised trajectory point data is compared with the number of trajectory points in the original trajectory point data. If the number of trajectory points is inconsistent, the corresponding original trajectory point data in the in-memory database is updated based on the denoised trajectory point data. This improves the efficiency and accuracy of data processing during trajectory denoising, increases data processing response time, avoids excessive write pressure on the in-memory database, reduces denoising service interruption time, and improves data processing stability.

[0109] It should be noted that the data processing method provided in this application embodiment is generally executed by server 705, and correspondingly, the data processing device is generally located in server 705.

[0110] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0111] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device according to the embodiments of this application. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the computer system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0113] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0114] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this application.

[0115] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including an acquisition unit, an allocation server determination unit, a target server determination unit, a noise reduction unit, and an update unit. The names of these units do not necessarily limit the specific unit itself.

[0118] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to obtain trajectory point data according to a received data processing request, generate a data processing task based on the trajectory point data, write the data processing task to an in-memory database based on an asynchronous queue and thread pool, distribute the data processing tasks in the in-memory database to determine an allocation server, determine the role of the allocation server, and determine a target server based on the role, the request receiving data when the election timer times out, and pre-candidates; call the target server to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on a trajectory denoising algorithm to obtain denoised trajectory point data; compare the number of trajectory points in the denoised trajectory point data with the corresponding undenoised trajectory point data, and if the number of trajectory points is inconsistent, update the corresponding undenoised trajectory point data in the in-memory database based on the denoised trajectory point data.

[0119] The computer program product of this application includes a computer program that, when executed by a processor, implements the data processing method in the embodiments of this application.

[0120] According to the technical solution of the embodiments of this application, the efficiency and accuracy of data processing during trajectory denoising can be improved, the data processing response time can be increased, excessive write pressure on memory-type databases can be avoided, the interruption time of denoising processing services can be reduced, and the stability of data processing can be improved.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Obtain trajectory point data according to the received data processing request, and generate a data processing task based on the trajectory point data; The data processing tasks are written to an in-memory database based on asynchronous queues and thread pools, and the data processing tasks in the in-memory database are distributed to determine the allocation server. The role of the allocation server is determined, and the target server is determined based on the role, the request receiving data when the election timer expires, and the pre-candidates; The target server is invoked to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on the trajectory denoising algorithm, so as to obtain denoised trajectory point data. Compare the number of trajectory points in the denoised trajectory point data with the corresponding number of trajectory points before denoising. If the number of trajectory points is inconsistent, update the corresponding number of trajectory points before denoising in the memory database based on the denoised trajectory point data.

2. The method according to claim 1, characterized in that, The determination of the target server includes: The role of the allocation server is determined. If the role is follower, the corresponding data processing task is redirected to the server with the role of leader. If no request is received when the election timer of the allocation server expires, the role of the allocation server is changed to pre-candidate, and a pre-election request is sent to the server with the role of non-leader. In response to receiving more than a preset threshold of positive votes, the role of the allocation server is changed from pre-candidate to candidate. An election is performed based on the servers with the determined role of candidate to determine the target server with the role of leader.

3. The method according to claim 1, characterized in that, Before distributing the data processing tasks in the in-memory database in a distributed manner, the method further includes: The trajectory processing service is invoked to add the data processing task to the asynchronous queue, and the thread pool is invoked to write the data processing task in the asynchronous queue to the in-memory database.

4. The method according to claim 1, characterized in that, The method further includes: The target server, whose role is leader, periodically sends heartbeat messages to all servers whose role is follower. During the election voting process, servers acting as candidates periodically send heartbeat messages to servers acting as non-leaders.

5. The method according to claim 2, characterized in that, The method further includes: In response to the interruption of the noise reduction processing service provided by the target server, the service interruption time is obtained; The timeout of the election timer is dynamically adjusted based on the service interruption time.

6. The method according to claim 1, characterized in that, The step of obtaining trajectory point data according to the received data processing request includes: Get the scheduled task corresponding to the data processing request, and in response to the scheduled task being triggered, extract the earliest trajectory point data of the preset duration from the data source.

7. A data processing apparatus, characterized in that, include: The acquisition unit is configured to acquire trajectory point data according to the received data processing request, and generate a data processing task based on the trajectory point data. The allocation server determination unit is configured to write the data processing tasks to an in-memory database based on an asynchronous queue and a thread pool, and to distribute the data processing tasks in the in-memory database in a distributed manner to determine the allocation server; The target server determination unit is configured to determine the role of the allocation server and determine the target server based on the role, the request receiving data when the election timer expires, and the pre-candidates. The denoising unit is configured to call the target server to perform denoising processing on the trajectory point data corresponding to the corresponding data processing task based on the trajectory denoising algorithm, so as to obtain denoised trajectory point data. The update unit is configured to compare the number of trajectory points in the denoised trajectory point data with the corresponding undenoised trajectory point data. If the number of trajectory points is inconsistent, the corresponding undenoised trajectory point data in the memory database is updated based on the denoised trajectory point data.

8. A data processing electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images