Data processing methods, apparatus, storage media and electronic equipment
By using different memory mapping methods for full and incremental business data, flexible memory mapping is achieved, solving the problem of low flexibility in memory mapping in existing technologies and improving data access efficiency and concurrency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TRAVELSKY TECHNOLOGY LIMITED
- Filing Date
- 2026-03-12
- Publication Date
- 2026-06-02
Smart Images
Figure CN122131983A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of memory mapping, and more specifically, to a data processing method, apparatus, storage medium, and electronic device. Background Technology
[0002] In modern computer science, memory mapping has become an indispensable part of high-performance data processing systems. This technology allows files or data on disk to be directly mapped to a process's virtual memory space, bypassing traditional file read / write buffers, significantly improving data access speed and reducing the load on the CPU (Central Processing Unit) and I / O (Input / Output) systems. Especially in big data analytics, real-time trading systems, and high-performance computing applications, memory mapping technology, due to its zero-copy characteristic, has become a key means to improve data processing capabilities and response speed. Currently, related technologies typically employ a single method for memory mapping, resulting in low flexibility in memory mapping.
[0003] There is currently no effective solution to the aforementioned problems in the relevant technologies. Summary of the Invention
[0004] The main objective of this application is to provide a data processing method, apparatus, storage medium, and electronic device to solve the problem of low flexibility in memory mapping when performing memory mapping on data in related technologies.
[0005] To achieve the above objectives, according to one aspect of this application, a data processing method is provided. The method includes: upon receiving full-volume service data generated before a first time point, performing memory mapping processing on the full-volume service data using a first memory mapping method, wherein the full-volume service data mapped using the first memory mapping method is shared among multiple processes; and upon receiving incremental service data generated after the first time point, performing memory mapping processing on the incremental service data using a second memory mapping method, wherein the incremental service data refers to incremental service data relative to the full-volume service data, and the incremental service data mapped using the second memory mapping method supports independent modification by multiple processes based on a copy-on-write mechanism.
[0006] Optionally, the data processing method further includes: before performing memory mapping processing on the full amount of business data through the first memory mapping method, when the current time point reaches the first time point, sending a first data request to the data storage server, wherein the first data request is used to request to obtain the full amount of business data; and determining the data returned by the data storage server in response to the first data request as the full amount of business data.
[0007] Optionally, the data processing method further includes: after performing memory mapping processing on the full amount of business data through the first memory mapping method, sending a second data request to the data storage server at a preset first time interval, wherein the second data request is used to request incremental business data; and determining the data returned by the data storage server in response to the second data request as incremental business data.
[0008] Optionally, the data processing method further includes: receiving data from the data storage server in response to the second data request at a second time interval, and identifying the data as incremental business data, wherein the second time interval is greater than the first time interval, and the second time interval is the time interval at which the data storage server collects incremental business data.
[0009] Optionally, the data processing method also includes: performing data verification processing on incremental business data using preset data verification rules to obtain data verification results; and performing memory mapping processing on incremental business data through a second memory mapping method if the data verification passes.
[0010] Optionally, the data processing method further includes: determining a time point later than the first time point by a preset time interval to obtain a second time point; updating the first time point to the second time point, and performing memory mapping processing on all business data generated before the second time point through a first memory mapping method, and performing memory mapping processing on incremental business data generated after the second time point through a second memory mapping method.
[0011] Optionally, the data processing method further includes: before performing memory mapping processing on the incremental business data through the second memory mapping method, creating a log file corresponding to the incremental business data, and recording the data information of the incremental business data based on the log file; after performing memory mapping processing on the incremental business data through the second memory mapping method, recording the updated content of the incremental business data to the log file.
[0012] To achieve the above objectives, according to another aspect of this application, a data processing apparatus is provided. The apparatus includes: a first processing module, configured to perform memory mapping processing on the full volume of service data generated before a first time point using a first memory mapping method, wherein the full volume of service data mapped using the first memory mapping method is shared among multiple processes; and a second processing module, configured to perform memory mapping processing on the incremental service data generated after the first time point using a second memory mapping method, wherein the incremental service data refers to incremental service data relative to the full volume of service data, and the incremental service data mapped using the second memory mapping method supports independent modification by multiple processes based on a copy-on-write mechanism.
[0013] Optionally, the data processing device further includes: a first sending module, configured to send a first data request to the data storage server when the current time point reaches the first time point, wherein the first data request is used to request the acquisition of full business data; and a first determining module, configured to determine the data returned by the data storage server in response to the first data request as full business data.
[0014] Optionally, the data processing device further includes: a second sending module, configured to send a second data request to the data storage server at a preset first time interval, wherein the second data request is used to request incremental service data; and a second determining module, configured to determine the data returned by the data storage server in response to the second data request as incremental service data.
[0015] Optionally, the second determining module further includes: a determining submodule, used to receive data fed back by the data storage server in response to the second data request according to a second time interval, and to determine the data as incremental business data, wherein the second time interval is greater than the first time interval, and the second time interval is the time interval for the data storage server to count incremental business data.
[0016] Optionally, the second processing module further includes: a verification module, used to perform data verification processing on incremental business data using preset data verification rules to obtain data verification results; and a processing submodule, used to perform memory mapping processing on incremental business data through a second memory mapping method if the data verification passes.
[0017] Optionally, the data processing device further includes: a third determining module, used to determine a time point later than a preset time interval of the first time point to obtain a second time point; and a third processing module, used to update the first time point to the second time point, and to perform memory mapping processing on all business data generated before the second time point through a first memory mapping method, and to perform memory mapping processing on incremental business data generated after the second time point through a second memory mapping method.
[0018] Optionally, the data processing device further includes: a fourth processing module for creating a log file corresponding to the incremental business data and recording the data information of the incremental business data based on the log file; and a fifth processing module for recording the updated content of the incremental business data to the log file.
[0019] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided, which includes a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the above-described data processing method.
[0020] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described data processing method during runtime.
[0021] To achieve the above objectives, according to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data processing method described above.
[0022] In this embodiment, by employing a first mapping method to perform memory mapping on all business data, data is mapped using a shared mapping approach when the data access demand is relatively low, thereby reducing memory usage and copying costs and improving data access efficiency. By employing a second mapping method to perform memory mapping on incremental business data, data is mapped using a private mapping approach when the data access demand is relatively high, achieving private data storage. Each process can freely modify the data without affecting other processes, effectively preventing data contention and improving data processing efficiency and concurrency. Therefore, by using different mapping methods for different types of data, the flexibility of memory mapping is effectively improved.
[0023] Therefore, the method provided in this application achieves the goal of using different mapping methods for different types of data, realizes the technical effect of improving the flexibility of memory mapping, and solves the technical problem of low flexibility of memory mapping when related technologies perform memory mapping on data. Attached Figure Description
[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0025] Figure 1 This is a hardware structure block diagram of a computer terminal provided according to an embodiment of this application;
[0026] Figure 2 This is a flowchart of a data processing method provided according to an embodiment of this application;
[0027] Figure 3 This is a schematic diagram illustrating the acquisition of incremental service data according to an embodiment of this application;
[0028] Figure 4 This is a schematic diagram of the inspection data provided according to the embodiments of this application;
[0029] Figure 5This is a schematic diagram of a data processing method provided according to an embodiment of this application;
[0030] Figure 6 This is a schematic diagram of a data processing apparatus provided according to an embodiment of this application;
[0031] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0034] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, and necessary measures have been taken to ensure compliance with public order and good morals. Corresponding access points are provided for users to choose whether to authorize or refuse. For example, interfaces are established between this system and relevant users or organizations, providing users with corresponding access points to choose whether to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.
[0035] Example 1
[0036] According to an embodiment of this application, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0037] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor (MCU) or a field-programmable gate array (FPGA), etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output (I / O) interface, a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0038] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0039] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0040] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0041] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).
[0042] Under the aforementioned operating environment, this application provides the following: Figure 2 The data processing method shown. Figure 2 This is a flowchart of the data processing method according to Embodiment 1 of this application.
[0043] Step S201: Upon receiving the full amount of service data generated before the first time point, perform memory mapping processing on the full amount of service data using the first memory mapping method. The full amount of service data mapped using the first memory mapping method is shared among multiple processes.
[0044] Optionally, electronic devices, application systems, servers, and other devices can be used as the execution subject of this application. In this embodiment, the target processing system is used as the execution subject to execute the above-described data processing method.
[0045] Depending on the specific application scenario, the specific data content of the business data (i.e., full business data and incremental business data) in this application also varies. For example, in a civil aviation scenario, the business data may be ticket price data (i.e., fare data), and the target processing system may be a fare calculation service system. As another example, in a financial scenario, the business data may be transaction data, and the target processing system may be a transaction calculation system. In this embodiment, the data content of the business data is only described exemplarily.
[0046] In an optional embodiment, the first time point is a preset time point. After receiving the full amount of business data, the target processing system can use shared mapping (i.e., the first memory mapping method) to map the full amount of business data to a memory area shared by multiple processes, so that these processes can access and share the same dataset, that is, share the full amount of business data. The processes can perform corresponding business data processing based on the business data, such as fare calculation, data verification, etc.
[0047] For example, the system receives a text-formatted source data file (e.g., full business data), which is then preprocessed by a data preprocessing server to generate a memory-mapped database containing data records and various indexes. This database is then multicast to computing nodes (e.g., specific processes) to serve as the data source for the computing engine, enabling the engine to perform various data queries.
[0048] Optionally, shared mapping (i.e., the first memory mapping method) allows multiple processes to share the same memory region. When multiple processes share the same memory region, any modification made to this region by any one process is immediately reflected in the memory of all other processes. Shared mapping can reduce memory usage and copying costs, and improve data access efficiency.
[0049] Since full business data is accessed less frequently, less often, and less likely to be modified compared to incremental business data, using a shared mapping method to store full business data, rather than a private mapping method, can effectively reduce memory usage and save memory resources.
[0050] Step S202: Upon receiving incremental service data generated after the first time point, the incremental service data is processed by memory mapping using a second memory mapping method. Here, incremental service data refers to incremental service data relative to the full service data. The incremental service data mapped by the second memory mapping method supports multiple processes to modify independently based on the copy-on-write mechanism.
[0051] After the system completes loading of all business data at the first point in time, it can receive incremental business data for mapping the full business data for a period of time in the following period. This avoids the problem of excessive system resource consumption when frequently mapping the full business data.
[0052] When processing incremental business data, private mapping (i.e., the second memory mapping method) can be used. In this method, each process has an independent memory-mapped copy of the incremental business data, allowing independent data read and write operations based on copy-on-write, without affecting the data of other processes or the underlying data files. For example, when the system receives incremental business data, it maps the data to a shared memory region (e.g., a physical memory page, with the page table marked as read-only) through private mapping. The physical memory pages mapped to by multiple processes are identical, meaning they share a copy of the data in memory. When a process first attempts to modify the incremental business data in the memory region, the kernel does not immediately update the original file's content. Instead, the kernel creates a copy of the memory page containing the data to be modified for that process, allowing the process to perform the modification operation on the copy, while other processes still access the unmodified data. Processes are used for data processing based on business data, such as fare calculation and data verification.
[0053] Optionally, private mapping (i.e., a second memory mapping method) can build a robust data isolation barrier. Through a unique mapping mechanism, data is completely isolated from external processes, ensuring that data has its own dedicated storage space during the data construction process, unaffected by interference or influence from other processes. This isolation effectively avoids data conflicts and chaos caused by concurrent operations, providing a reliable guarantee for stable data construction.
[0054] Private mappings access file content directly from memory, reducing the system call overhead of traditional I / O and improving read / write efficiency. Simultaneously, private mappings achieve data isolation through copy-on-write technology; modifications made by a process to the mapped content are only visible to itself and do not affect other processes or underlying files. The kernel triggers the copy operation only when a process actually modifies the data, initially sharing physical memory pages to reduce memory usage. This on-demand copying mechanism is suitable for handling large files or high-frequency read / write scenarios, as well as scenarios requiring independent data copies, such as temporary data processing or sensitive operations, to avoid data conflicts caused by concurrent modifications by multiple processes. Since private mappings do not directly modify the original file, they inherently possess data tamper-proof characteristics.
[0055] Optionally, the core difference between private mappings and shared mappings lies in data visibility. Modifications to private mappings only take effect in the current process, while changes to shared mappings will be synchronized to other processes and underlying files.
[0056] Since incremental business data is accessed more frequently, is more likely to be modified, and is more likely to be accessed than full business data, using a private mapping method to store incremental business data, rather than a shared mapping method, can effectively improve the reliability of data access and avoid data conflicts caused by data modification when multiple processes access incremental business data.
[0057] In this embodiment, by employing a first mapping method to perform memory mapping on all business data, data is mapped using a shared mapping approach when the data access demand is relatively low, thereby reducing memory usage and copying costs and improving data access efficiency. By employing a second mapping method to perform memory mapping on incremental business data, data is mapped using a private mapping approach when the data access demand is relatively high, achieving private data storage. Each process can freely modify the data without affecting other processes, effectively preventing data contention and improving data processing efficiency and concurrency. Therefore, by using different mapping methods for different types of data, the flexibility of memory mapping is effectively improved.
[0058] Therefore, the method provided in this application achieves the goal of using different mapping methods for different types of data, realizes the technical effect of improving the flexibility of memory mapping, and solves the technical problem of low flexibility of memory mapping when related technologies perform memory mapping on data.
[0059] Optionally, in the data processing method provided in this application embodiment, before performing memory mapping processing on the full amount of business data through the first memory mapping method, the method further includes: when the current time point reaches the first time point, sending a first data request to the data storage server, wherein the first data request is used to request to obtain the full amount of business data; and determining the data returned by the data storage server in response to the first data request as the full amount of business data.
[0060] In an optional embodiment, the first time point is a pre-set time used to trigger a full update of business data. For example, it may be the time point when the target processing system starts or the starting point of a periodic update cycle.
[0061] Optionally, the target processing system can monitor the current time and, upon reaching the first time point, initiate a first data request to the data storage server. The first data request may include at least data requirement information, such as the data identifier (e.g., batch number, version number) of the full volume of business data to be acquired, and may also include the time range to which the full volume of business data belongs. Optionally, the first data request may also include authentication information, enabling the data storage server to identify the request source and provide the correct data.
[0062] After receiving the first data request, the data storage server can send the corresponding data back to the target processing system based on the data requirement information in the first data request, so that the target processing system can identify the data as the full amount of business data.
[0063] In an optional embodiment, before sending the first data request to the data storage server, the target processing system may first set the basic configuration for server data updates, which includes, but is not limited to, parameter configuration, runtime environment, and mount directory. For example, the startup parameter configuration may include the system version number, the address of the data storage server, the absolute path of the business data in the data storage server, the deadline for building the full set of business data (i.e., the start time of incremental updates, such as the first time point), and the log storage path.
[0064] It should be noted that by automatically sending a data request to obtain all business data when the current time point reaches the first time point, the timeliness and reliability of obtaining all business data are improved.
[0065] Optionally, in the data processing method provided in the embodiments of this application, after performing memory mapping processing on the full amount of business data through the first memory mapping method, the method further includes: sending a second data request to the data storage server according to a preset first time interval, wherein the second data request is used to request to obtain incremental business data; and determining the data returned by the data storage server in response to the second data request as incremental business data.
[0066] Optionally, after completing the memory mapping process of the full amount of business data at the first time point, the target processing system may periodically send a second data request to the data storage server at a preset first time interval to obtain any new or changed business data since the last full data update, i.e., incremental business data.
[0067] For example, after completing the memory mapping process for all business data, the target system starts a timer to automatically trigger the sending of a second data request based on a first time interval configured by the system (e.g., every 2 minutes). This request can contain explicit incremental data requirement information, such as the timestamp of the last data update and data identifiers (e.g., batch number, version number, etc.). The data storage server can then use this information to filter out incremental business data within the corresponding time range and provide feedback.
[0068] After receiving the second data request, the data storage server can send the corresponding data back to the target processing system based on the data requirement information in the second data request, so that the target processing system can identify the data as incremental business data.
[0069] In an optional embodiment, after receiving the data from the data storage server in response to the second data request, the target processing system may first compare the data with the timestamp information to confirm that the data does indeed contain all the new or changed records since the last request, and then treat it as incremental business data.
[0070] It should be noted that, through the above method, the latest incremental business data can be continuously and periodically obtained after processing all the business data, thereby improving the timeliness and reliability of obtaining incremental business data.
[0071] Optionally, in the data processing method provided in this application embodiment, determining the data fed back by the data storage server in response to the second data request as incremental service data includes: receiving the data fed back by the data storage server in response to the second data request according to a second time interval, and determining the data as incremental service data, wherein the second time interval is greater than the first time interval, and the second time interval is the time interval for the data storage server to count incremental service data.
[0072] In an optional embodiment, after the target processing system sends a second data request to the data storage server at a preset first time interval, the system waits for the data storage server to process and provide feedback on the incremental business data at a second time interval based on its own statistical incremental business data. For example, after the system sends the second data request to the data storage server, it does not immediately receive feedback. The data storage server continues to collect incremental business data from various business channels in the background until its internally set second time interval (e.g., every 30 minutes) is reached. During this period, the server performs data aggregation, compression, and integrity checks to ensure that the data fed back to the target processing system is complete and reliable incremental business data.
[0073] Optionally, the second time interval can be longer than the first time interval to improve the timeliness of acquiring incremental business data. For example, the first time interval is 2 minutes and the second time interval is 30 minutes.
[0074] For example, Figure 3 This is a schematic diagram illustrating the acquisition of incremental service data according to an embodiment of this application, such as... Figure 3 As shown, after the target processing system starts, the system sends data requests to the data storage server at a fixed frequency (i.e., the first time interval, e.g., every 2 minutes). Upon the first request, the system initiates a first data request, and the data storage server pushes all business data (including all historical records) to the system at once. When the data storage server detects that new business data is ready and receives a second data request from the system, it pushes the data added since the last synchronization (the latest version, i.e., incremental business data). In an optional embodiment, incremental business data is generated in batches every 30 minutes (i.e., the second time interval), and the system synchronization frequency is aligned with the data generation cycle, i.e., an incremental update is performed every 30 minutes. For example: If the current time is 10:00 AM on October 21, 2025, the system starts up and initially receives all business data from 00:00 on January 1, 2020 to 10:00 AM on October 21, 2025. Then, the system sends a second data request to the data storage server every two minutes. When the incremental business data in the data storage server is ready at 10:30 AM and the second data request is received, the data storage server pushes the latest batch of incremental business data updated at 10:30 AM to the target processing system.
[0075] In an optional embodiment, incremental service data can be stored on a date-by-date basis. That is, each date corresponds to a folder, and each folder stores all incremental service data for that date. For example, if the data storage server pushes incremental service data every half hour, the target processing system stores subfolders corresponding to the incremental service data pushed in half-hour increments within each date's folder. In other words, multiple subfolders are stored within each date's folder, and different subfolders are used to store incremental service data for different time periods.
[0076] Optionally, incremental business data may include multiple data tables. For each subfolder corresponding to the incremental business data, there may be multiple unit folders, each used to store data from different data tables. For example, a unit folder may store all data record files for its corresponding data table, and these data record files may contain the data table information required for freight rate calculation.
[0077] In an optional embodiment, the data storage server packages the incremental service data into a compressed file and then sends the corresponding compressed file to the target processing system. The target processing system can decompress the compressed file to obtain the corresponding incremental service data.
[0078] It should be noted that by setting the first time interval to be less than the second time interval, that is, by setting the frequency at which the target processing system sends the second data request to be greater than the frequency at which the data storage server compiles statistics, the timeliness of obtaining incremental business data can be further improved.
[0079] Optionally, in the data processing method provided in this application embodiment, the incremental service data is processed by memory mapping through a second memory mapping method, including: performing data verification processing on the incremental service data using a preset data verification rule to obtain a data verification result; and performing memory mapping processing on the incremental service data through the second memory mapping method if the data verification passes.
[0080] Optionally, after acquiring incremental business data, the system can perform data validation using preset data validation rules. Data validation can be used to verify whether the data path is correct, whether the updated data information is empty, whether the batch number is empty, whether the number of digits in the batch number string meets the standard, and whether the data format is correct, etc. The data validation result is used to indicate whether the incremental business data passes the validation.
[0081] For example, Figure 4 This is a schematic diagram of the inspection data provided according to the embodiments of this application, such as... Figure 4 As shown, after verifying the compliance of data batch numbers and other related information, if the data verification result indicates that the data verification passed, the incremental business data is processed through memory mapping using the second memory mapping method, i.e., data update is performed. If the data verification result indicates that the data verification failed, an error is reported.
[0082] For example, after acquiring incremental business data, the timestamp of the incremental business data to be mapped is read to determine the data construction batch. Then, a data file in a specific format is constructed based on the incremental business data, and the data file in the specific format is mapped into memory through a second memory mapping method. For example, if the update time point for the full business data is set to September 18, 2025, then the starting timestamp for the incremental business data update is 20250918, and it will be updated every half hour thereafter until the next time the full business data is to be statistically analyzed.
[0083] In an optional embodiment, during the mapping process, the target processing system can read the metadata of the data table in the incremental business data and then perform memory mapping processing on the metadata. The metadata includes, but is not limited to, the content of the data entries excluding large fields, the large fields, and the index values corresponding to the data entries. Large fields are data entries, referring to fields that occupy a significant amount of storage space in the data table. These fields may contain complex text, images, audio, or other large blocks of data. When processing fare data, large fields may contain detailed rules, complex combinations of conditions, or long text descriptions.
[0084] It should be noted that by performing data validation on incremental business data, the system can effectively prevent data errors or corruption from affecting subsequent services and maintain the high quality of data. After the data validation is successful, a second memory mapping method is used for memory mapping, allowing processes to access and update data independently. This maintains data isolation and security while effectively utilizing the performance advantages of memory mapping to accelerate data processing.
[0085] Optionally, in the data processing method provided in the embodiments of this application, the method further includes: determining a time point later than a preset time interval of the first time point to obtain a second time point; updating the first time point to the second time point, and performing memory mapping processing on all business data generated before the second time point through a first memory mapping method, and performing memory mapping processing on incremental business data generated after the second time point through a second memory mapping method.
[0086] Optionally, the system can calculate a second time point, which is later than the first full data load, based on a preset time interval, as the time benchmark for the next round of data processing. For example, after completing the first full load of business data, the system will automatically calculate the time point for the next full data update.
[0087] When the second time point arrives, the system can update the base time point (first time point) and reload all business data; after the new first time point, the system will continue to request and process new incremental business data according to the original mechanism.
[0088] For example, when the second time point arrives, the system will update the first time point to this new time point, and then use the first memory mapping method to reload and map all business data to a memory area shared by multiple processes for a new round of data processing. Following this, the system continues to send second data requests at preset first time intervals (e.g., every 30 minutes) to obtain incremental business data after the second time point, and uses the second memory mapping method for mapping processing.
[0089] It should be noted that, through the above method, periodic full data updates and timely incremental data loading are achieved. The system can periodically switch data with decreasing access demand to a shared mapping method, thereby further improving the flexibility of memory mapping.
[0090] Optionally, in the data processing method provided in this application embodiment, before performing memory mapping processing on the incremental service data through the second memory mapping method, the method further includes: creating a log file corresponding to the incremental service data, and recording the data information of the incremental service data based on the log file; after performing memory mapping processing on the incremental service data through the second memory mapping method, the method further includes: recording the updated content of the incremental service data to the log file.
[0091] Optionally, after each acquisition of incremental business data, a log file can be created in the subfolder corresponding to the incremental business data. This log file can be used to record relevant information about the incremental business data, such as data name, build time, version number, etc. The log file can also include information such as the start and end batches of the full update, the current update batch, the start and end times of the incremental update, and the data version.
[0092] Optionally, after the data mapping process is complete, the system will append the change information of the incremental business data to the previously created log file. This may involve recording the change type (such as update, insert, or delete) for each data record, as well as the timestamp of the data change. The system can use a transaction log to ensure that every data update is fully recorded, and the data state can be recovered through the log file even if an exception or failure occurs during the update process.
[0093] It should be noted that by using logs to record information during the memory mapping process, every data change can be accurately tracked, thus ensuring data integrity and recoverability.
[0094] In an alternative embodiment, Figure 5 This is a schematic diagram of a data processing method provided according to an embodiment of this application. Figure 5 An optional application process of this embodiment will be explained. For example... Figure 5 As shown, the target processing system can perform the following steps to perform data mapping:
[0095] 1. Before the freight calculation service is launched, a series of data update configurations are performed. For example, this includes determining the data storage server used to obtain business data, establishing a secure connection channel with the data storage server, determining whether the data construction adopts a full update mode, an incremental update mode, or a full + incremental hybrid update mode, and specifying the base timestamp of the first set of data shards, etc., to ensure the integrity and timeliness of the initial data loading.
[0096] 2. Regularly retrieve business data from the data storage server. For example, automatically execute a data collection task every 30 minutes to ensure that incremental business data on the server remains up-to-date.
[0097] 3. Data Construction and Preprocessing. In the data preprocessing stage, information such as the data batch number and the primary key set of the data table to be updated is obtained. Based on this data, the system confirms the task scheduling scheme and the data path of the incremental business data, which are then used to construct the data.
[0098] 4. Read and update data. The data table contains data entries, large fields and index values. When the data construction process starts, perform the following operations: (1) Batch read the data entries, large fields and index values and other metadata of each data table; (2) Use a private mapping mechanism to establish a virtual memory mapping to achieve zero-copy data loading.
[0099] For example, perform multi-threaded decompression operations on incremental business data and perform integrity verification on the data to ensure the standardization and integrity of the data; create log files to record information about the built files, including data name, build time, version number, etc.; read the timestamp of the data to be built to determine the data build batch, build the data according to the incremental business data, and map it through a private mapping mechanism; check the size of the memory mapping space, and if the space does not meet the requirements, perform expansion processing.
[0100] 5. Data Integration. In the private data storage model, memory mapping parameters are dynamically adjusted based on the memory distribution characteristics of each data table to achieve field-level memory alignment optimization. Simultaneously, storage log files are maintained or updated synchronously, employing a transactional recording mechanism to ensure that every data change operation is completely and accurately tracked and archived.
[0101] Therefore, the method provided in this application achieves the goal of using different mapping methods for different types of data, realizes the technical effect of improving the flexibility of memory mapping, and solves the technical problem of low flexibility of memory mapping when related technologies perform memory mapping on data.
[0102] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0103] Example 2
[0104] This application also provides a data processing apparatus. It should be noted that the data processing apparatus of this application can be used to execute the data processing method provided in this application. The data processing apparatus provided in this application will be described below.
[0105] According to an embodiment of this application, an apparatus for implementing the above-described data processing method is also provided, such as... Figure 6 As shown, the device includes:
[0106] The first processing module 601, upon receiving the full amount of service data generated before the first time point, performs memory mapping processing on the full amount of service data through a first memory mapping method, wherein the full amount of service data mapped through the first memory mapping method is shared among multiple processes;
[0107] The second processing module 602 is used to perform memory mapping processing on the incremental service data generated after the first time point by means of a second memory mapping method. The incremental service data refers to the incremental service data relative to the full service data. The incremental service data mapped by the second memory mapping method supports multiple processes to modify independently based on the copy-on-write mechanism.
[0108] In this embodiment, by employing a first mapping method to perform memory mapping on all business data, data is mapped using a shared mapping approach when the data access demand is relatively low, thereby reducing memory usage and copying costs and improving data access efficiency. By employing a second mapping method to perform memory mapping on incremental business data, data is mapped using a private mapping approach when the data access demand is relatively high, achieving private data storage. Each process can freely modify the data without affecting other processes, effectively preventing data contention and improving data processing efficiency and concurrency. Therefore, by using different mapping methods for different types of data, the flexibility of memory mapping is effectively improved.
[0109] Therefore, the method provided in this application achieves the goal of using different mapping methods for different types of data, realizes the technical effect of improving the flexibility of memory mapping, and solves the technical problem of low flexibility of memory mapping when related technologies perform memory mapping on data.
[0110] Optionally, in the data processing apparatus provided in this application embodiment, the first sending module is used to send a first data request to the data storage server when the current time point reaches the first time point, wherein the first data request is used to request to obtain the full amount of business data; and the first determining module is used to determine the data fed back by the data storage server in response to the first data request as the full amount of business data.
[0111] Optionally, in the data processing apparatus provided in this application embodiment, the second sending module is used to send a second data request to the data storage server at a preset first time interval, wherein the second data request is used to request incremental service data; and the second determining module is used to determine the data fed back by the data storage server in response to the second data request as incremental service data.
[0112] Optionally, in the data processing apparatus provided in this application embodiment, a determining submodule is used to receive data fed back by the data storage server in response to the second data request according to a second time interval, and to determine the data as incremental business data, wherein the second time interval is greater than the first time interval, and the second time interval is the time interval for the data storage server to count incremental business data.
[0113] Optionally, in the data processing apparatus provided in this application embodiment, the verification module is used to perform data verification processing on the incremental business data using preset data verification rules to obtain data verification results; the processing submodule is used to perform memory mapping processing on the incremental business data through a second memory mapping method when the data verification passes.
[0114] Optionally, in the data processing apparatus provided in this application embodiment, the third determining module is used to determine a time point later than the first time point by a preset time interval to obtain a second time point; the third processing module is used to update the first time point to the second time point, and to perform memory mapping processing on all business data generated before the second time point through a first memory mapping method, and to perform memory mapping processing on incremental business data generated after the second time point through a second memory mapping method.
[0115] Optionally, in the data processing apparatus provided in this application embodiment, the fourth processing module is used to create a log file corresponding to the incremental business data and record the data information of the incremental business data based on the log file; the fifth processing module is used to record the updated content of the incremental business data to the log file.
[0116] It should be noted that the first processing module 601 and the second processing module 602 mentioned above correspond to steps S201 to S202 in Embodiment 1. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.
[0117] Example 3
[0118] Embodiments of this application may provide an electronic device. Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 7 As shown, the electronic device may include: one or more ( Figure 7 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.
[0119] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0120] The processor can invoke information and applications stored in the memory through the transmission device to perform the following steps: upon receiving full service data generated before the first time point, performing memory mapping processing on the full service data using a first memory mapping method, wherein the full service data mapped using the first memory mapping method is shared among multiple processes; upon receiving incremental service data generated after the first time point, performing memory mapping processing on the incremental service data using a second memory mapping method, wherein the incremental service data refers to incremental service data relative to the full service data, and the incremental service data mapped using the second memory mapping method supports multiple processes to modify independently based on a copy-on-write mechanism.
[0121] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: before performing memory mapping processing on the full amount of business data through the first memory mapping method, when the current time point reaches the first time point, send a first data request to the data storage server, wherein the first data request is used to request to obtain the full amount of business data; determine the data returned by the data storage server in response to the first data request as the full amount of business data.
[0122] The processor can also invoke information and applications stored in the memory through the transmission device to perform the following steps: after performing memory mapping processing on the full amount of business data through the first memory mapping method, a second data request is sent to the data storage server at a preset first time interval, wherein the second data request is used to request incremental business data; the data returned by the data storage server in response to the second data request is determined as incremental business data.
[0123] The processor can invoke the information and application stored in the memory through the transmission device to perform the following steps: receiving data from the data storage server in response to the second data request at a second time interval, and identifying the data as incremental service data, wherein the second time interval is greater than the first time interval, and the second time interval is the time interval at which the data storage server counts incremental service data.
[0124] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: perform data verification processing on the incremental service data using preset data verification rules to obtain the data verification result; if the data verification passes, perform memory mapping processing on the incremental service data using a second memory mapping method.
[0125] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine a time point later than the first time point by a preset time interval to obtain a second time point; update the first time point to the second time point, and perform memory mapping processing on all business data generated before the second time point through a first memory mapping method, and perform memory mapping processing on incremental business data generated after the second time point through a second memory mapping method.
[0126] The processor can access the information and application stored in the memory via the transmission device to perform the following steps: before performing memory mapping processing on the incremental service data through the second memory mapping method, create a log file corresponding to the incremental service data, and record the data information of the incremental service data based on the log file; after performing memory mapping processing on the incremental service data through the second memory mapping method, record the updated content of the incremental service data to the log file.
[0127] In this embodiment, by employing a first mapping method to perform memory mapping on all business data, data is mapped using a shared mapping approach when the data access demand is relatively low, thereby reducing memory usage and copying costs and improving data access efficiency. By employing a second mapping method to perform memory mapping on incremental business data, data is mapped using a private mapping approach when the data access demand is relatively high, achieving private data storage. Each process can freely modify the data without affecting other processes, effectively preventing data contention and improving data processing efficiency and concurrency. Therefore, by using different mapping methods for different types of data, the flexibility of memory mapping is effectively improved.
[0128] Therefore, the method provided in this application achieves the goal of using different mapping methods for different types of data, realizes the technical effect of improving the flexibility of memory mapping, and solves the technical problem of low flexibility of memory mapping when related technologies perform memory mapping on data.
[0129] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 7 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.
[0130] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0131] Example 4
[0132] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in Embodiment 1.
[0133] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.
[0134] This application also provides a computer program product, which, when executed on a data processing device, is a program adapted to perform data processing method steps.
[0135] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0136] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0137] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0141] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data processing method, characterized in that, include: Upon receiving all service data generated before the first time point, the full service data is processed by memory mapping using a first memory mapping method, wherein the full service data mapped by the first memory mapping method is shared among multiple processes. Upon receiving incremental service data generated after the first time point, the incremental service data is processed by memory mapping using a second memory mapping method. The incremental service data refers to incremental service data relative to the full service data. The incremental service data mapped by the second memory mapping method supports multiple processes to modify independently based on the copy-on-write mechanism.
2. The method according to claim 1, characterized in that, Before performing memory mapping processing on the full volume of business data using the first memory mapping method, the method further includes: When the current time point reaches the first time point, a first data request is sent to the data storage server, wherein the first data request is used to request the acquisition of the full amount of business data; The data returned by the data storage server in response to the first data request is determined as the full volume of business data.
3. The method according to claim 1, characterized in that, After performing memory mapping processing on the full volume of service data using the first memory mapping method, the method further includes: A second data request is sent to the data storage server at a preset first time interval, wherein the second data request is used to request the acquisition of the incremental business data; The data returned by the data storage server in response to the second data request is identified as the incremental business data.
4. The method according to claim 3, characterized in that, Determining the data returned by the data storage server in response to the second data request as the incremental service data includes: The system receives data from the data storage server in response to the second data request at a second time interval, and identifies this data as the incremental service data. The second time interval is greater than the first time interval, and the second time interval is the time interval at which the data storage server collects incremental service data.
5. The method according to claim 1, characterized in that, The incremental service data is processed by memory mapping using a second memory mapping method, including: The incremental business data is processed using preset data verification rules to obtain the data verification results; If the data verification passes, the incremental service data is processed by memory mapping using the second memory mapping method.
6. The method according to claim 1, characterized in that, The method further includes: A second time point is obtained by determining a time point later than the first time point by a preset time interval; The first time point is updated to the second time point, and the full amount of business data generated before the second time point is processed by memory mapping using the first memory mapping method, and the incremental business data generated after the second time point is processed by memory mapping using the second memory mapping method.
7. The method according to claim 1, characterized in that, Before performing memory mapping processing on the incremental service data using the second memory mapping method, the method further includes: Create a log file corresponding to the incremental service data, and record the data information of the incremental service data based on the log file; After performing memory mapping processing on the incremental service data using the second memory mapping method, the method further includes: The updated content of the incremental business data is recorded to the log file.
8. A data processing apparatus, characterized in that, include: The first processing module, upon receiving all the service data generated before the first time point, performs memory mapping processing on the all the service data using a first memory mapping method, wherein the all the service data mapped using the first memory mapping method is shared among multiple processes. The second processing module is used to perform memory mapping processing on the incremental service data generated after the first time point by means of a second memory mapping method. The incremental service data refers to the incremental service data relative to the full service data. The incremental service data mapped by the second memory mapping method supports multiple processes to modify independently based on the copy-on-write mechanism.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the data processing method according to any one of claims 1 to 7.
10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data processing method according to any one of claims 1 to 7.