Data processing method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a nonvolatile storage medium. The method comprises the steps that a data processing task of a data file is received, and the data file is used for recording three-dimensional geographic information data; storing the data processing task into a task queue; according to the priority of the data processing task of the data file in the task queue and the load condition of the thread in the thread pool, a target thread in the thread pool is called to execute the data processing task, and data in the data file is directly mapped to the virtual address space of the process. According to the method and the device, the technical problems of low resource utilization rate and loading delay during traditional single-thread processing or common multi-thread data processing are solved.

Description

Technical Field

[0001] The present application relates to the field of electronic information technology, and in particular, to a data processing method, device, electronic device and non-volatile storage medium. Background Art

[0002] The OSGB file format is a format specifically used to store 3D geographic information data, and is widely used in virtual reality, geographic information systems (GIS), and other fields that require accurate 3D models. However, with the rapid growth of data size, traditional IO operations are becoming increasingly inefficient in processing large-scale OSGB files, making it difficult to meet the needs of efficient processing.

[0003] There are single-threaded and multi-threaded reading and writing methods in the related technologies. The traditional single-threaded method has obvious performance bottlenecks when facing huge amounts of data, including high processing delays and low system resource utilization. A single thread cannot handle multiple file read operations at the same time. Single-threaded processing requires all tasks to be completed in sequence, and different tasks cannot be processed in parallel, resulting in high processing delays. Each task must wait for the previous task to be completed before it can begin. This linear processing method exhibits significant delays when facing large-scale data; a single thread cannot fully utilize system resources such as multi-core CPUs and large-capacity memory, resulting in low resource utilization. In modern computer systems, multi-core processors have become mainstream, but a single thread can only use one CPU core, and the other cores are idle, resulting in a waste of resources.

[0004] In order to improve efficiency, some technical solutions have introduced multi-threaded processing, but these solutions usually only involve ordinary multi-threaded operations and fail to fully optimize thread scheduling, data processing strategies and system resource utilization, resulting in limited performance improvement. There is still a lot of room for improvement. Multiple threads are started to read multiple data files (for example, OSGB files) each time, and each thread performs file read and write operations independently. The loading of each file requires a large number of disk read operations. Due to the limitation of disk bandwidth and frequent disk seeks, the overall loading time will increase significantly. In the process of processing data, the system needs to continuously read file data for reading and writing. If each thread performs file read and write operations independently, frequent disk access will cause the loading speed of read and write data to not keep up with the speed of data processing, resulting in data processing jams.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a data processing method, device, electronic device and non-volatile storage medium to at least solve the technical problems of low resource utilization and loading delay in traditional single-threaded processing or ordinary multi-threaded data processing.

[0007] According to one aspect of an embodiment of the present application, a method for data processing is provided, including: receiving a data processing task of a data file, wherein the data file is used to record three-dimensional geographic information data; storing the data processing task in a task queue; and calling a target thread in a thread pool to execute the data processing task based on the priority of the data processing task of the data file in the task queue and the load of the threads in the thread pool, wherein the data in the data file is directly mapped to the virtual address space of the process.

[0008] In some embodiments of the present application, before receiving a data processing task for a data file, the method also includes: creating a task queue for executing the data processing task for the data file; determining a preset number of threads to be created in the thread pool based on the number of cores of the CPU of the processing device; and creating a preset number of threads in the thread pool, wherein the state of the threads is a waiting state.

[0009] In some embodiments of the present application, based on the priority of the data processing task of the data file in the task queue and the load condition of the threads in the thread pool, a target thread in the thread pool is called to execute the data processing task, including: when there are data processing tasks in the task queue, the data processing tasks are sorted in order from high to low according to the priority of the data processing tasks to obtain a task sequence; and the target thread in the thread pool is called to execute the data processing task in the order of the task sequence, wherein the target thread is determined based on the load condition of the threads in the thread pool and the state of the threads.

[0010] In some embodiments of the present application, a target thread in a thread pool is called in the order of a task sequence to perform a data processing task, including: determining a thread in the thread pool with the smallest load and in a waiting state as a target thread; calling the target thread to perform the data processing task, wherein when the target thread performs the data processing task, the target thread state is a wake-up state, and when the target thread completes the data processing task, the state is a waiting state.

[0011] In some embodiments of the present application, when a thread in a thread pool is in a waiting state for more than a preset time, the thread is closed and the resources occupied by the thread are released. When the number of threads in a waiting state in the thread pool is 0, a preset number of new threads are created in the thread pool again, and the state of the new threads is a waiting state.

[0012] In some embodiments of the present application, data in a data file is directly mapped to the virtual address space of a process in the following manner: obtaining a file descriptor in the data file; mapping the file descriptor to the virtual address space of the corresponding process through file mapping, and recording the mapping information in the data file in a file mapping record table.

[0013] In some embodiments of the present application, the method also includes: obtaining a cache data structure, wherein the cache data structure is used to store data files that need to be cached and reused, and the cache data structure is stored in a key-value pair storage manner; and caching data files that need to be shared and reused according to the cache data structure.

[0014] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: a receiving module for receiving a data processing task of a data file, wherein the data file is used to record three-dimensional geographic information data; a storage module for storing the data processing task in a task queue; and a processing module for calling a target thread in a thread pool to execute the data processing task according to the priority of the data processing task of the data file in the task queue and the load of the threads in the thread pool, wherein the data in the data file is directly mapped to the virtual address space of the process.

[0015] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute the above data processing method.

[0016] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the above data processing method is executed when the program is run.

[0017] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, and when the computer instructions are executed by a processor, the above data processing method is implemented.

[0018] In an embodiment of the present application, a data processing task of receiving a data file is adopted, wherein the data file is used to record three-dimensional geographic information data; the data processing task is stored in a task queue; according to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool, a target thread in the thread pool is called to execute the data processing task, wherein the data in the data file is directly mapped to the virtual address space of the process. By mapping the data in the data file directly to the virtual address space of the process, the problem of loading delay is solved. According to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool, the target thread in the thread pool is called to execute the data processing task, thereby achieving the purpose of improving resource utilization, thereby solving the technical problems of low resource utilization and loading delay in traditional single-threaded processing or ordinary multi-threaded data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a data processing method provided according to an embodiment of the application;

[0021] Figure 2 is a flowchart of a data processing method according to an embodiment of the present application;

[0022] Figure 3 is a flowchart of an optional data processing method according to an embodiment of the present application;

[0023] Figure 4 It is a structural diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0027] Open scene Figure 2Open Scene Graph Binary (OSGB): OSGB file format is a format specially used to store three-dimensional geographic information data. It is widely used in virtual reality, geographic information system (GIS) and other fields that require accurate three-dimensional models. It is suitable for various applications that require accurate three-dimensional models, such as urban planning, terrain analysis and game development. It can store multiple types of three-dimensional data, including geometric data, textures, lighting information, etc.

[0028] Input / Output (IO): This is the process of exchanging data between a computer system and the outside world (such as users, files, devices, etc.). In an operating system, IO operation is one of the core functions that allows the operating system to manage the flow of data, including reading data from input devices (such as keyboards and mice) and sending data to output devices (such as monitors and printers).

[0029] Least Recently Used (LRU): The LRU algorithm eliminates data that has not been accessed for the longest time. It is based on the assumption that if data has been accessed recently, it is more likely to be accessed in the future; it is usually implemented using hash tables and doubly linked lists. Hash tables store keys and pointers to corresponding nodes, and doubly linked lists store nodes in order of access. When accessing a piece of data, move its corresponding node to the head of the linked list. When data needs to be eliminated, simply remove the node from the end of the linked list. RU is suitable for scenarios where there is a "locality principle" in the access pattern, that is, recently accessed data is likely to be accessed in the future.

[0030] Least Frequently Used (LFU): The LFU algorithm eliminates the data with the least number of accesses. It believes that data with low access frequency is also less likely to be accessed in the future. Implementing LFU usually requires maintaining a hash table and a data structure (such as a heap or an ordered linked list) sorted by the number of accesses. The hash table stores keys and pointers to corresponding nodes, and the nodes contain the number of accesses and data. When a piece of data is accessed, its access count is increased, and its position in the data structure is adjusted according to the number of accesses. When data needs to be eliminated, select the node with the least number of accesses. LFU is suitable for scenarios where the data access frequency varies greatly in the access pattern. It can better utilize cache space and retain data that is infrequent but still important.

[0031] Mutex lock: used to protect access to shared resources (such as task queues) in a multi-threaded environment, ensuring that only one thread can access the task queue at the same time, avoiding data competition and inconsistency.

[0032] Condition variables: Used in conjunction with mutex locks, they allow a thread to remain blocked while waiting for a specific condition (such as a task queue is not empty) to be met until it is awakened by another thread.

[0033] In the related art, the traditional single-threaded method has obvious performance bottlenecks when facing a huge amount of data, including high processing delay and low system resource utilization. Ordinary multi-threaded operations fail to fully optimize thread scheduling, data processing strategies and system resource utilization, resulting in limited performance improvement. There is still a lot of room for improvement. Multiple threads are started to read multiple data files (for example, OSGB files) each time. Each thread performs file read and write operations independently. The loading of each file requires a large number of disk read operations. Due to the limitation of disk bandwidth and frequent disk seeks, the overall loading time will increase significantly. In the process of processing data, the system needs to continuously read file data for reading and writing. If each thread performs file read and write operations independently, frequent disk access will cause the loading speed of read and write data to not keep up with the speed of data processing, resulting in data processing jams. Therefore, there are problems of low resource utilization and loading delay when processing data with traditional single-threaded processing or ordinary multi-threaded processing. In order to solve this problem, a relevant solution is provided in the embodiment of the present application, which is described in detail below.

[0034] According to an embodiment of the present application, an embodiment of a method for data processing is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0036] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0040] In the above operating environment, the present application embodiment provides a flow chart of a data processing method, such as Figure 2 As shown, the method comprises the following steps:

[0041] Step S202, receiving a data processing task of a data file, wherein the data file is used to record three-dimensional geographic information data.

[0042] In the technical solution provided in step S202, the size of the thread pool directly affects the concurrency capability and resource utilization of the system. Therefore, before receiving the data processing task of the data file, it is necessary to create in advance a task queue for executing the data processing task of the data file (for example, OSGB file) and determine the preset number of threads to be created in the thread pool according to the number of cores of the CPU of the processing device, and create a preset number of threads in the thread pool, wherein the state of the thread is a waiting state. When the thread in the thread pool is in the waiting state for more than the preset time, the thread is closed and the resources occupied by the thread are released. When the number of threads in the waiting state in the thread pool is 0, a preset number of new threads are created in the thread pool again, and the state of the new thread is a waiting state.

[0043] The following is a specific embodiment:

[0044] The preset number of threads to be created in the thread pool can be set to a multiple of the number of CPU cores, such as 1.5 to 2 times, so that the computing power of multi-core processors can be fully utilized, excessive competition and context switching between threads can be avoided, and efficiency can be improved. Considering the number and size of OSGB files, if the number of files is very large, the size of the thread pool can be appropriately increased, not too much exceeding the number of CPU cores (for example, more than 4 times). Too many threads may cause more context switching, which will reduce efficiency; mutexes and conditional variables are used to manage threads in the thread pool to ensure the correctness and fairness of thread scheduling and the access security of task queues. Tasks are allocated through task queues to ensure that threads can quickly obtain tasks when idle, avoiding idle waiting of threads. When the threads in the thread pool are in a waiting state for more than a preset time (for example, 1 hour), the threads are closed and the resources occupied by the threads are released. When the number of threads in a waiting state in the thread pool is 0, a preset number of new threads are created in the thread pool again. The state of the new threads is a waiting state. When the threads execute data processing tasks, the state is a wake-up state. When the data processing tasks are completed, the state changes back to a waiting state.

[0045] The following is another optional embodiment:

[0046] The thread pool can be designed to be dynamically adjusted, that is, the number of threads can be increased or decreased according to the actual load (for example, the length of the task queue) and the execution efficiency of the threads (for example, the speed at which the tasks are completed). Continuously monitor resource usage, such as CPU usage and memory usage. When resource usage approaches the threshold, reduce the size of the thread pool. Conversely, increase the size of the thread pool.

[0047] Step S204, storing the data processing task into the task queue.

[0048] Step S206, according to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool, calling the target thread in the thread pool to execute the data processing task, wherein the data in the data file is directly mapped to the virtual address space of the process.

[0049] In the technical solution provided in step S206, there are multiple ways to implement calling the target thread in the thread pool to execute the data processing task according to the priority of the data processing task of the data file in the task queue and the load of the threads in the thread pool. For example, it can be implemented in the following way: when there are data processing tasks in the task queue, sort the data processing tasks in descending order according to the priority of the data processing tasks to obtain a task sequence; call the target thread in the thread pool to execute the data processing task in the order of the task sequence, wherein the target thread is determined based on the load of the threads in the thread pool and the state of the threads.

[0050] It should be noted that there are multiple ways to implement the above-mentioned calling the target thread in the thread pool to execute the data processing task in the order of the task sequence. For example, the target thread in the thread pool can be called to execute the data processing task in the order of the task sequence in the following way: the thread with the smallest load in the thread pool and in the waiting state is determined as the target thread; the target thread is called to execute the data processing task, wherein when the target thread executes the data processing task, the target thread is in the awakened state, and when the target thread completes the data processing task, the target thread is in the waiting state.

[0051] The following is a specific embodiment:

[0052] The task queue stores OSGB file data processing tasks. Each task can contain at least the following information:

[0053] Task ID: used to uniquely identify each task, OSGB file path: the location of the file to be processed, task priority: indicates the urgency of the task (for example, divided into high, medium, and low).

[0054] The task queue can be implemented using a priority queue (such as a heap structure), which ensures that high-priority tasks are processed first. When a new task is added, it is inserted into the appropriate position according to the priority, and high-priority tasks are preferentially assigned to waiting threads. The thread load can be evaluated by checking indicators such as the number of tasks being executed, processing time, and CPU usage.

[0055] The created thread will enter a waiting state, waiting for the wake-up notification of the condition variable. When there is no task, the thread is in a waiting state to save system resources. When a new task is added to the task queue, the condition variable is triggered, waking up a waiting thread to execute the task. When taking tasks, a mutex lock is used to protect the queue to ensure atomic operations, that is, an operation or a group of operations must be either completed or not completed, to prevent multiple threads from modifying the queue at the same time. After the thread completes the task, it releases the lock on the task queue, and then enters the waiting state again, ready to receive the next task.

[0056] The following is the specific implementation process:

[0057] When receiving notification of task queue changes, check the task queue, select the task with the highest priority from the task queue, determine the thread with the smallest load in the thread pool and in the waiting state as the target thread, and the thread starts to execute the task and sets the state to the awakened state. The thread updates its load state while executing the task, and changes the state back to the waiting state after completing the task. If the thread is not awakened within the specified time (timeout), it is considered that the thread is no longer needed, and it is closed and the resources are released.

[0058] Through the above steps, the IO operation speed and system resource utilization of OSGB files are significantly improved: multi-threaded parallel reading and writing of OSGB files speeds up the processing process, the use of mutex locks and conditional variables ensures synchronization between threads and data consistency, avoids deadlock, and improves the system's response speed and throughput through fast task allocation and wake-up mechanisms.

[0059] In order to avoid the problem of loading delay, the data in the data file used in the above steps is directly mapped to the virtual address space of the process, so that the reading and writing operations of the file can be as efficient as accessing the memory, thereby reducing disk IO operations and improving overall performance. For example, this can be achieved in the following way: obtain the file descriptor in the data file; map the file descriptor to the virtual address space of the corresponding process through file mapping, and record the mapping information in the data file in the file mapping record table.

[0060] The following is a specific embodiment:

[0061] Initialize the memory mapping file module according to the configuration parameters (for example, through the memory mapping file module), and prepare the necessary data structures and resources. When a data file needs to be read or written, the file content is mapped to the virtual address space of the corresponding process through the mapping controller. The file mapping record table records the mapping information of the file for subsequent management and access. Through memory mapping, the content of the file can be directly mapped to the memory space of the process, which greatly improves the data access speed, reduces the delay of traditional IO operations, and significantly improves the efficiency of data access. When the file is no longer needed, the memory mapping of the file is released through the mapping controller, and the file mapping record table is updated to release related resources. The following is the specific implementation process:

[0062] Before starting memory mapping, open the OSGB file first. This step is completed through the standard file operation API. After the file is successfully opened, a file descriptor (fd) is obtained. fd is the unique identifier assigned by the operating system to the opened file. It is a non-negative integer used to identify the opened file in the computer operating system and is used for file access communication between the kernel and the user space. In Unix and Unix-like systems (such as Linux), file descriptors are widely used, and each open file corresponds to a unique file descriptor. When a program opens a file, device, or socket, the operating system returns a file descriptor, and subsequent read and write operations are performed through this descriptor. File descriptors are usually used in programming as system call parameters for file reading, writing, closing, and other operations; in order to effectively manage memory mapping, it is necessary to track which files have been mapped into memory, which is usually done through a data structure, such as a file mapping record table, which records key data such as mapped file information, the starting address of the mapping, and the length of the file. When a file is mapped, this information is added to the table; when the file mapping is revoked, the information is removed from the table; since multiple threads may access the memory mapping of the same file at the same time, it is necessary to ensure that access between threads is safe and does not cause data competition or inconsistency. This is achieved by using mutex locks. When writing or updating the mapped memory, the thread needs to obtain the mutex lock first; after the operation is completed, the lock is released to allow other threads to access; when the thread needs to access data in the OSGB file, it can read and write directly through the memory pointer, which greatly reduces data access latency and improves processing speed, especially when processing large files.

[0063] In the process of OSGB file processing, we often encounter some data that is shared or requires repeated operations, such as data conversion and vertex data merging. When the amount of data is large, we can also set up a shared cache and put these data into the shared cache to improve data processing efficiency, avoid repeated processing, reduce disk IO operations and increase data access speed. For example, we can do this in the following ways: obtain the cache data structure, where the cache data structure is used to store data files that need to be cached and reused, and the cache data structure is stored in a key-value pair storage method; cache the data files that need to be shared and reused according to the cache data structure, and the thread safety mechanism ensures safe access to cache data in a multi-threaded environment. Use read-write locks or other synchronization mechanisms to ensure that multiple threads can read cache data concurrently, and avoid data competition and inconsistency when writing or updating cache data.

[0064] The following is a specific embodiment:

[0065] The cache data structure uses an efficient key-value pair storage method (such as a hash table) to ensure fast access to data. Each cache item includes the data's identification, content, and related metadata information (such as access time, frequency of use, etc.). The cache replacement strategy determines when the data in the cache is replaced. Replacement strategies include LRU (for example, creating an LRUCache cache), LFU, etc. The cache replacement strategy ensures that when cache space is limited, high-frequency access data is retained first to improve the cache hit rate, and then the hot data is cached. The cache memory can be implemented using a hash map hashMap, which can quickly find data. The specific implementation process is as follows:

[0066] Use efficient key-value pair storage methods, such as hash tables, to implement cache data structures. The main implementation process is:

[0067] Choose a suitable hash function to map the data identifier (such as file name, file ID or other unique identifier) ​​to the index position of the hash table. The hash function should be able to evenly distribute data and reduce conflicts; determine the size of the hash table, set according to the expected number of cache items and system performance requirements. The size of the hash table should be able to balance cache capacity and memory consumption;

[0068] When data is read for the first time, a cache entry is created, which contains the data identifier, data content and metadata (for example, access time, frequency of use, etc.); a hash function is used to calculate the hash value of the data identifier and find the corresponding position in the hash table; the cache entry is stored at that position, and if the position is occupied, a conflict resolution strategy (for example, open addressing, chain address method) is used to decide how to store the new item; when an application requests data, a hash value is calculated using the data identifier and the position in the hash table is located; a check is made to see if there is a corresponding cache entry at that position, and if so, the data content is returned directly; if the data is not in the cache, it needs to be re-read from the disk and the cache updated.

[0069] When the cache reaches a preset size or threshold, a cache replacement strategy (e.g., LRU, LFU) is applied to determine which data should be evicted from the cache to make room for new data. The LRU strategy requires maintaining an access list or array to record the most recently accessed data items. When a new data item needs to be inserted into the cache and the cache is full, the data item in the list or array that has not been accessed for the longest time is replaced. The LFU strategy requires recording the frequency of use of each cache item and replacing the least frequently used data item. In addition, since the cache may be accessed concurrently by multiple threads, it is necessary to implement read-write locks or other synchronization mechanisms to ensure data consistency and integrity, and set up a thread safety mechanism to ensure safe access to cached data in a multi-threaded environment: read operations can hold read locks to allow multiple threads to read data at the same time, but write operations (insert, update, delete) require holding write locks. Only one thread can perform write operations at a time. Condition variables or semaphores are used to control the waiting and wake-up of threads to ensure that threads follow the correct order and rules when accessing the cache. Each time data is accessed, its metadata (for example, access time) is updated to help implement cache replacement strategies. Atomic operations are used to update metadata to avoid data competition during multi-threaded access.

[0070] The above-mentioned shared cache can be implemented in the form of a dynamic link library (DLL library for short), which is a shared library implementation method in Microsoft Windows operating system. Multiple applications can share the code and data in the same DLL, reducing the use of disk space and memory occupation. The DLL can be dynamically loaded when the program is running, which means that the loading can be delayed until it is really needed, thereby improving the startup speed of the program.

[0071] Through the above steps, by dynamically adjusting the thread pool size, optimizing task scheduling, memory-mapped file access and cache management strategies, the efficiency and response speed of data processing are significantly improved, and the problems of low resource utilization and loading delay in traditional single-threaded processing or ordinary multi-threaded data processing are solved.

[0072] The present application embodiment provides a flow chart of an optional data processing method, such as Figure 3 As shown, it includes an application layer (user space), an operating system layer (kernel space), and a hardware layer (hard disk). The application layer is composed of an application and a thread pool module. The thread pool module includes a thread pool manager, a task queue, and a task scheduler. The operating system layer (kernel space) is composed of a shared cache module (including multiple shared data) and a memory mapping file module (including a memory mapping manager and a memory mapping record table (i.e., the above-mentioned file mapping record table)). The hardware layer (hard disk) is composed of multiple OSGB files and multiple files under the 3D Tiles specification (such as Batched 3D Model, referred to as b3dm). 3D Tiles is a dynamic, scalable transmission and rendering format for three-dimensional geographic spatial data. It consists of multiple small, streamable tiles, which can be in the formats of B3DM, Pnts (point cloud data), or I3DM (3D model with instance data). B3DM files are an optimized format obtained after processing OSGB data, which is more suitable for real-time rendering and network transmission.

[0073] The thread pool module is used to create a task queue for executing data processing tasks of data files (for example, OSGB files) in advance and determine the preset number of threads to be created in the thread pool according to the number of cores of the CPU of the processing device, and create a preset number of threads in the thread pool, wherein the state of the thread is a waiting state. When the thread in the thread pool is in the waiting state for more than a preset time, the thread pool manager closes the thread and releases the resources occupied by the thread. When the number of threads in the waiting state in the thread pool is 0, a preset number of new threads are created in the thread pool again, and the state of the new threads is a waiting state.

[0074] The task scheduler is used to call the target thread in the thread pool to execute the data processing task according to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool; determine the thread with the smallest load in the thread pool and the waiting state as the target thread; call the target thread to execute the data processing task, wherein when the target thread executes the data processing task, the target thread state is the awakened state, and when the target thread completes the data processing task, the state is the waiting state, wherein the data in the data file is directly mapped to the virtual address space of the process.

[0075] The memory mapping file module of the operating system layer (kernel space) is used to use the memory mapping manager to directly map the data in the used data files (for example, OSGB files and b3dm files in the hardware layer (hard disk)) to the virtual address space of the process, so that the reading and writing operations of the files can be as efficient as accessing the memory, thereby reducing disk IO operations and improving the overall performance. For example, it can be implemented in the following way: the memory mapping manager obtains the file descriptor in the data file; maps the file descriptor to the virtual address space of the corresponding process through file mapping, and records the mapping information in the data file in the file mapping record table.

[0076] The shared cache module of the operating system layer (kernel space) is used in the process of OSGB file processing. It often encounters some shared or repetitively operated data, such as data conversion and vertex data merging. When the amount of data is large, a shared cache can also be set up to put these data into the shared cache to improve data processing efficiency, avoid repeated processing, reduce disk IO operations and increase data access speed. For example, the following methods can be used: obtain the cache data structure, where the cache data structure is used to store data files that need to be cached and reused, and the cache data structure is stored in a key-value pair storage method; cache the data files that need to be shared and reused according to the cache data structure, and the thread safety mechanism ensures safe access to the cache data in a multi-threaded environment. Use read-write locks or other synchronization mechanisms to ensure that multiple threads can read cache data concurrently, and avoid data competition and inconsistency when writing or updating cache data.

[0077] The present application embodiment provides a structural diagram of a data processing device, such as Figure 4 As shown, including:

[0078] The receiving module 402 is used for receiving a data processing task of a data file, wherein the data file is used for recording three-dimensional geographic information data.

[0079] The receiving module 402 is also used to receive data processing tasks of data files.

[0080] The receiving module is also used to create a task queue for executing data processing tasks of data files; determine the preset number of threads to be created in the thread pool according to the core number of the CPU of the processing device; create a preset number of threads in the thread pool, wherein the state of the thread is a waiting state, and when the thread in the thread pool is in the waiting state for more than a preset time, the thread is closed and the resources occupied by the thread are released, and when the number of threads in the waiting state in the thread pool is 0, a preset number of new threads are created in the thread pool again, and the state of the new threads is a waiting state.

[0081] The storage module 404 is used to store the data processing task into the task queue.

[0082] Processing module 406 is used to call the target thread in the thread pool to execute the data processing task according to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool, wherein the data in the data file is directly mapped to the virtual address space of the process.

[0083] The processing module 406 is also used to call the target thread in the thread pool to execute the data processing task, including: when there is a data processing task in the task queue, sorting the data processing tasks in order from high to low priority to obtain a task sequence; calling the target thread in the thread pool to execute the data processing task in the order of the task sequence, wherein the target thread is determined based on the load situation and the state of the thread in the thread pool.

[0084] The processing module 406 is also used to determine the thread with the smallest load in the thread pool and in a waiting state as the target thread; call the target thread to execute the data processing task, wherein when the target thread executes the data processing task, the target thread state is a wake-up state, and when the target thread completes the data processing task, the state is a waiting state.

[0085] It should be noted that the various modules in the above-mentioned data processing device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0086] The embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to perform the above data processing method. For example, receiving a data processing task of a data file, wherein the data file is used to record three-dimensional geographic information data; storing the data processing task in a task queue; calling a target thread in a thread pool to perform the data processing task according to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool, wherein the data in the data file is directly mapped to the virtual address space of the process, by directly mapping the data in the data file to the virtual address space of the process.

[0087] The embodiment of the present application also provides an electronic device, the electronic device includes a processor, the processor is used to run a program, wherein the above data processing method is executed when the program is running. For example, a data processing task of a data file is received, wherein the data file is used to record three-dimensional geographic information data; the data processing task is stored in a task queue; according to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool, a target thread in the thread pool is called to execute the data processing task, wherein the data in the data file is directly mapped to the virtual address space of the process, by directly mapping the data in the data file to the virtual address space of the process.

[0088] According to another aspect of the embodiment of the present application, a computer program product is also provided, including a computer program, which implements the above data processing method when executed by a processor. For example, a data processing task of a data file is received, wherein the data file is used to record three-dimensional geographic information data; the data processing task is stored in a task queue; according to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool, a target thread in the thread pool is called to execute the data processing task, wherein the data in the data file is directly mapped to the virtual address space of the process, by directly mapping the data in the data file to the virtual address space of the process.

[0089] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0092] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0094] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for data processing, characterized in that: include: A data processing task for receiving a data file, wherein the data file is used to record three-dimensional geographic information data; Storing the data processing task in a task queue; According to the priority of the data processing task of the data file in the task queue and the load of the threads in the thread pool, the target thread in the thread pool is called to execute the data processing task, wherein the data in the data file is directly mapped to the virtual address space of the process.

2. The method according to claim 1, characterized in that Before receiving the data processing task of the data file, the method further includes: Creating the task queue for executing the data processing task of the data file; Determine the preset number of threads to be created in the thread pool according to the number of CPU cores of the processing device; The preset number of threads are created in the thread pool, wherein the state of the threads is a waiting state.

3. The method according to claim 1, characterized in that According to the priority of the data processing task of the data file in the task queue and the load of the threads in the thread pool, calling the target thread in the thread pool to execute the data processing task includes: In the case where the data processing tasks exist in the task queue, the data processing tasks are sorted in descending order according to their priorities to obtain a task sequence; The target thread in the thread pool is called in the order of the task sequence to execute the data processing task, wherein the target thread is determined based on the load condition of the threads in the thread pool and the state of the threads.

4. The method according to claim 3, characterized in that Calling the target thread in the thread pool to execute the data processing task in the order of the task sequence includes: Determine the thread in the thread pool that has the smallest load and is in a waiting state as the target thread; The target thread is called to execute the data processing task, wherein when the target thread executes the data processing task, the target thread is in an awake state, and when the target thread completes executing the data processing task, the target thread is in a waiting state.

5. The method according to claim 2, characterized in that: When the time that a thread in the thread pool is in the waiting state exceeds a preset time, the thread is closed and the resources occupied by the thread are released; when the number of threads in the waiting state in the thread pool is 0, the preset number of new threads are created in the thread pool again, and the state of the new threads is the waiting state.

6. The method according to claim 1, characterized in that The data in the data file is directly mapped to the virtual address space of the process in the following way: Get the file descriptor of the data file; The file descriptor is mapped to the virtual address space of the corresponding process through file mapping, and the mapping information in the data file is recorded in a file mapping record table.

7. The method according to claim 1, characterized in that The method further comprises: Acquire a cache data structure, wherein the cache data structure is used to store data files that need to be cached and reused, and the cache data structure is stored in a key-value pair storage manner; The data files that need to be shared and reused are cached according to the cache data structure.

8. A data processing device, characterized in that: include: A receiving module, used for receiving a data processing task of a data file, wherein the data file is used for recording three-dimensional geographic information data; A storage module, used for storing the data processing task into a task queue; A processing module is used to call a target thread in the thread pool to execute the data processing task according to the priority of the data processing task of the data file in the task queue and the load of the thread in the thread pool, wherein the data in the data file is directly mapped to the virtual address space of the process.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the data processing method according to any one of claims 1 to 7 when running.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the data processing method described in any one of claims 1 to 7 is implemented.

Images