Data processing method and device, computer equipment and computer readable storage medium

Abstract

This application discloses a data processing method, apparatus, computer equipment, and computer-readable storage medium, applicable to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and assisted driving. The method includes: acquiring data to be processed; determining a target storage space for the data to be processed based on a database engine file, wherein the database engine file includes a memory-mapped area and a non-memory-mapped area, the memory-mapped area providing memory data storage space and the non-memory-mapped area providing file storage space; and storing the data records corresponding to the data to be processed into the target storage space. This application provides a database capable of providing large-capacity data storage services while ensuring data security.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods, data processing apparatus, computer equipment, and computer-readable storage media. Background Technology

[0002] A key-value (KV) database is a database that stores data in key-value pairs, where each key corresponds to a unique value. Currently, most common KV databases use in-memory database engines to achieve high performance. A database engine is the core service used for storing, processing, and protecting data.

[0003] However, in the event of a power outage, memory-based database engines are at risk of data loss and have relatively limited capacity. Therefore, improving database capacity while ensuring data security are problems that current database engines need to address. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer equipment, and computer-readable storage medium, which can provide large-capacity data storage services while ensuring data security.

[0005] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0006] Obtain the data to be processed;

[0007] The target storage space for the data to be processed is determined based on the database engine file, which includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area is used to provide memory data storage space, and the non-memory-mapped area is used to provide file storage space.

[0008] Store the data records corresponding to the data to be processed in the target storage space.

[0009] On one hand, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0010] The acquisition unit is used to acquire data to be processed.

[0011] The determining unit is used to determine the target storage space for the data to be processed based on the database engine file, which includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area is used to provide memory data storage space, and the non-memory-mapped area is used to provide file storage space.

[0012] A storage unit is used to store the data records corresponding to the data to be processed into the target storage space.

[0013] On one hand, embodiments of this application provide a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the data processing method described above.

[0014] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned data processing method.

[0015] On one hand, embodiments of this application provide a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned data processing method.

[0016] In this embodiment, the data to be processed is first acquired; then, the target storage space for the data is determined based on the database engine file, which includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area provides memory data storage space, and the non-memory-mapped area provides file storage space. Finally, the data records corresponding to the data to be processed are stored in the target storage space. The database engine file used combines memory data storage space and file storage space, providing large-capacity data storage services while ensuring data security. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a memory data mapping provided in an embodiment of this application;

[0021] Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of a database engine file provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the structure of the header region of a database engine file provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of a free block linked list provided in an embodiment of this application;

[0025] Figure 8a This is a schematic diagram of a target storage space provided in an embodiment of this application;

[0026] Figure 8b This is a schematic diagram of a data record corresponding to data to be processed, provided in an embodiment of this application;

[0027] Figure 9 This is a schematic diagram of a hash table provided in an embodiment of this application;

[0028] Figure 10a This is a schematic diagram of a log file provided in an embodiment of this application;

[0029] Figure 10b This is a schematic diagram of a log file header provided in an embodiment of this application;

[0030] Figure 10c This is a schematic diagram of a data operation record provided in an embodiment of this application;

[0031] Figure 11 This is a schematic diagram of a data processing method provided in an embodiment of this application;

[0032] Figure 12 This is a schematic diagram of a game database acquisition interface provided in an embodiment of this application;

[0033] Figure 13 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0034] Figure 14 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.

[0037] This application involves cloud technology, which refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0038] Cloud storage is a new concept that extends and develops from cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of various types of storage devices (also called storage nodes) in a network through application software or application interfaces to work together and provide data storage and business access functions. Currently, the storage method of a storage system is as follows: Logical volumes are created. When creating a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of the disks of one or several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only data but also additional information such as a data identifier (ID entity). The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object. The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0039] A database, simply put, can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or Extensible Markup Language (XML); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as Structured Query Language (SQL); or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages. In particular, the embodiments of this application can be applied to key-value (KV) databases. A KV database is a database that stores data in key-value pairs, where each key corresponds to a unique value.

[0040] Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

[0041] This application also relates to blockchain technology. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0042] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operational monitoring. The user management module is responsible for managing the identity information of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between user real identities and blockchain addresses (access management). Under authorization, it also manages and audits transactions of certain real identities and provides risk control rule configuration (risk control audit). The basic services module is deployed on all blockchain node devices to verify the validity of business requests. After consensus is reached on valid requests, they are recorded in storage. For a new business request, the basic services first perform interface adaptation parsing and authentication (interface adaptation), and then encrypt the business information through a consensus algorithm (consensus management). After encryption, the data is transmitted completely and consistently to the shared ledger (network communication) and recorded and stored. The smart contract module is responsible for contract registration, issuance, triggering, and execution. Developers can define contract logic using a programming language and publish it to the blockchain (contract registration). According to the contract terms, the key or other events are invoked to trigger execution and complete the contract logic. It also provides functions for contract upgrades and cancellations. The operation and monitoring module is mainly responsible for deployment, configuration modification, contract settings, cloud adaptation, and real-time status visualization output during product release, such as alarms, network status monitoring, and node device health status monitoring.

[0043] The platform's product service layer provides the basic capabilities and implementation frameworks for typical applications. Developers can leverage these basic capabilities, along with the specific characteristics of their business needs, to implement blockchain-based business logic. The application service layer provides blockchain-based application services to business stakeholders.

[0044] Based on the aforementioned cloud and blockchain technologies, this application proposes a data processing method to provide high-performance, highly scalable data storage services while ensuring data security. Specifically, the general principle of this data processing method is as follows: First, the data to be processed is acquired; then, the target storage space for the data to be processed is determined based on a database engine file, where the database engine file includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area provides memory data storage space, and the non-memory-mapped area provides file storage space; finally, the data records corresponding to the data to be processed are stored in the target storage space.

[0045] In specific implementations, the data processing methods mentioned above can be executed by computer equipment, which can be a terminal device or a server. The terminal device can be, for example, a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, vehicle terminal, aircraft, etc., but is not limited to these. The server can be, for example, a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0046] Alternatively, the data processing methods mentioned above can be executed jointly by the terminal device and the server. For example, see [link to relevant documentation]. Figure 1 As shown: Terminal device 101 can first obtain the data to be processed and send it to server 102. Correspondingly, after receiving the data, server 102 can determine the target storage space for the data based on the database engine file, and then store the corresponding data record in the target storage space. Alternatively, terminal device 101 can also send operation instructions for the data to be processed to server 102, and server 102 can execute the operation corresponding to the operation instruction on the data record corresponding to the data to be processed.

[0047] In this embodiment, the database engine file combines in-memory data storage space and file storage space, providing high-performance and highly scalable data storage services while ensuring data is stored on disk, thus improving data security.

[0048] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0049] Based on the above explanation, the following will be combined with Figure 2 The flowchart shown further illustrates the data processing method proposed in the embodiments of this application. In this embodiment, the data processing method is mainly described using the aforementioned computer device as an example. Please refer to... Figure 2 The data processing method may specifically include steps S201 to S203:

[0050] S201. Obtain the data to be processed.

[0051] In the embodiments of this application, the data to be processed may refer to text data, audio data, image data, etc. This data to be processed may be extracted from a database by a computer device, or it may be obtained from other devices; no limitation is made here.

[0052] It should be noted that when the object of the data to be processed is a user, all data related to the data to be processed in this application embodiment are obtained after authorization by the user. Furthermore, when this application embodiment is applied to a specific product or technology, the data used must be authorized or agreed to by the user, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0053] S202. Determine the target storage space for the data to be processed based on the database engine file.

[0054] In this embodiment, the target storage space includes memory space or file space, and the database engine file includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area provides memory data storage space, and the non-memory-mapped area provides file storage space. The database engine file refers to the file that stores database records. The database engine file used in computer devices combines memory data storage space and file storage space, providing high-performance, highly scalable data storage services while improving data security.

[0055] Please see Figure 3 , Figure 3 This is a schematic diagram of memory data mapping provided in an embodiment of this application. The database engine file includes memory-mapped and non-memory-mapped areas. Computer devices can use mmap (memory-mapped file) mapping to map data in the memory-mapped areas of the database engine file into the memory address space. The memory-mapped area can be read and written using memory read / write methods. The operating system will also indirectly write the data in memory back to the database engine file, thereby reducing memory data copying and improving data processing efficiency. Specifically, mmap mapping is a method of memory-mapped files, mapping a file or other object to the address space of a process, establishing a one-to-one mapping between the file's disk address and a segment of virtual addresses in the process's virtual address space. After achieving this mapping, the process can use pointers to read and write to this memory segment, and the system will automatically write back to the corresponding file on the disk, thus completing the file operation without needing to call system call functions such as read / write.

[0056] S203. Store the data record corresponding to the data to be processed into the target storage space.

[0057] In this embodiment, the computer device generates a data record corresponding to the data to be processed. The data record may include various fields related to the data to be processed, such as a key field and a value field. After determining the target storage space for the data to be processed, the computer device can store the data record corresponding to the data to be processed in the target storage space, and the corresponding data can be obtained subsequently through the data record.

[0058] In summary, in this embodiment, the data to be processed is first acquired; then, the target storage space for the data to be processed is determined based on the database engine file, which includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area provides memory data storage space, and the non-memory-mapped area provides file storage space; finally, the data records corresponding to the data to be processed are stored in the target storage space. The database engine file used combines memory data storage space and file storage space, providing high-performance, highly scalable data storage services while ensuring data persistence to disk, thus improving data security.

[0059] The following is combined Figure 4 The flowchart shown further illustrates the data processing method proposed in the embodiments of this application. In this embodiment, the data processing method is mainly described using the aforementioned computer device as an example. Please refer to... Figure 4 The data processing method may specifically include steps S401 to S404.

[0060] in:

[0061] S401. Obtain the data to be processed.

[0062] S402. Determine the target storage space for the data to be processed from the target memory space corresponding to the memory-mapped region.

[0063] In this embodiment, the computer device first determines the target storage space for the data to be processed from the target memory space corresponding to the memory-mapped region. Only when the target memory space is insufficient is the target storage space determined from the file storage space corresponding to a non-memory-mapped region. This approach provides higher performance and greater scalability for data storage services.

[0064] In one possible implementation, the memory-mapped region includes a header region that includes a free block list. The method further includes: obtaining the free block list included in the header region, which includes a free memory block list and a free file block list; determining the target storage space for the data to be processed from the free block list; and if the target storage space for the data to be processed does not exist in the free block list, then performing the step of determining the target storage space for the data to be processed from the target memory space corresponding to the memory-mapped region. It should be noted that the computer device first determines the target storage space for the data to be processed from the free block list included in the header region. Only when the storage space in the free block list cannot satisfy the target storage space for the data to be processed does the computer device determine the target storage space from the target memory space corresponding to the memory-mapped region.

[0065] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a database engine file provided in an embodiment of this application. For example... Figure 5 As shown, the database engine file includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area includes a header area, a key memory block storage space, and a value memory block storage space. The non-memory-mapped area includes a key / value file storage space. The key and value memory block storage spaces refer to the aforementioned memory data storage space, while the key / value file storage space refers to the aforementioned file storage space. Specifically, the header area can be used to store metadata, statistics, hash tables, free block lists, Least Recently Used (LRU) lists, and other information. The memory-mapped area provides memory data storage space and can map hot data (such as LRU lists) to the memory address space using mmap mapping, providing high-performance read and write capabilities. The non-memory-mapped area provides file storage space; data read and write operations in this area can only be performed through ordinary file read / write interfaces.

[0066] In one possible implementation, the header region also includes engine parameter information, which includes the free block level. The computer device determines the target storage space for the data to be processed from the free block list. Specifically, the implementation involves: determining the level and number of free blocks to be requested based on the free block level and the size of the data to be processed; determining the target block from the free block list based on the level and number of free blocks to be requested; and using the target block as the target storage space for the data to be processed.

[0067] It should be noted that after the target block is designated as the target storage space for the data to be processed, the target block needs to be removed from the free block list; or an identifier needs to be added to the target block to indicate that the target block has been occupied, thereby avoiding the situation where multiple data to be processed occupy the same target block, which helps to reduce target block occupancy conflicts.

[0068] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of the header region of a database engine file provided in an embodiment of this application. For example... Figure 6 As shown, the header area of ​​the database engine file includes the magic number, header area size, engine parameter information, statistics, a hash table, and a free block list. The magic number is a meaningless parameter, typically an uninterpreted numeric constant or string; the header area size is the space occupied by the header area; the engine parameter information contains set engine parameters such as the number of hash buckets, the minimum free block size (minSize), and the free block level; the statistics are used to count the number of records in the database engine file, read / write operations, and other statistics; the hash table is used to store and retrieve data records; the free block list includes a free memory block list and a free file block list. The free memory block list includes a key free memory block list and a value free memory block list, and the free file block list includes a key free file block list and a value free file block list.

[0069] In the actual implementation, the size of free memory blocks or free file blocks is divided into levels, including small blocks and large blocks, all of which are integer multiples of minSize. For example... Figure 7 As shown, Figure 7 This is a schematic diagram of a free block linked list provided in an embodiment of this application. From left to right, the free block linked lists are from minSize*1 to minSize*n, where n represents the free block level (a parameter set in the engine parameter information). The database engine file first determines the target storage space at the free block linked list. Only when the storage space in the free block linked list is insufficient will the target storage space be determined from the memory-mapped region. Furthermore, the target storage space determined from the free block linked list must be an integer multiple of minSize. For example, if the target storage space for the data to be processed is a minimum blocks (minSize), the following algorithm is used:

[0070] a / n = b; a%n = c (1)

[0071] This means that b free memory blocks (or free file blocks) of size minSize*n and 1 free memory block (or free file block) of size minSize*c are needed.

[0072] For example, assuming the target storage space for the data to be processed is 11 minimum blocks and the free block level is 5, the target blocks determined by the above formula are: 2 free memory blocks (or free file blocks) at level 5 and 1 free memory block (or free file block) at level 1. Therefore, 2 free memory blocks (or free file blocks) are determined from the free block list Head_5, and 1 free memory block (or free file block) is determined from the free block list Head_1.

[0073] In one possible implementation, the computer device determines the target block from the free block list based on the level and quantity of the requested free blocks. Specifically, it determines the target free memory block from the free memory block list within the free block list based on the level and quantity of the requested free blocks, and uses this target free memory block as the target block. If the target free memory block does not exist in the free memory block list, it determines the target free file block from the free file block list within the free block list based on the level and quantity of the requested free blocks, and uses this target free file block as the target block. It should be noted that the computer device first checks the free memory block list within the free block list to determine if the target block exists; if not, it determines the target block from the free file block list within the free block list.

[0074] Optionally, the computer device can also determine the target free memory block from the free memory block list within the free block list; if the target free memory block does not exist in the free memory block list, it can determine the target free memory block from the target memory space corresponding to the memory-mapped region; if the target free memory block does not exist in the target memory space corresponding to the memory-mapped region, it can determine the target free file block from the free file block list within the free block list; if the target free file block does not exist in the free file block list, it can determine the target free memory block from the file storage space corresponding to the non-memory-mapped region. This method can effectively improve the performance of the database storage service.

[0075] In one possible implementation, the data record corresponding to the data to be processed includes a Key field and a Value field. The corresponding target blocks can be determined separately. Specifically, for the Key field, the computer device can first determine the target block (i.e., the Key free memory block) from the Key free memory block linked list, and use this Key free memory block to store the data record corresponding to the data to be processed, which is the Key memory block; alternatively, it can continue to determine the target block (i.e., the Key free file block) from the Key free file block linked list, and use this Key free file block to store the data record corresponding to the data to be processed, which is the Key file block. Similarly, for the Value field, the computer device can first determine the target block (i.e., the Value free memory block) from the Value free memory block linked list, and use this Value free memory block to store the data record corresponding to the data to be processed, which is the Value memory block; alternatively, it can continue to determine the target block (i.e., the Value free file block) from the Value free file block linked list, and use this Value free file block to store the data record corresponding to the data to be processed, which is the Value file block.

[0076] In one possible implementation, after the computer device stores the data record corresponding to the data to be processed into the target storage space, the method further includes: if it is determined that the data record corresponding to the data to be processed has been deleted, then adding the target block to the free block list.

[0077] For example, assuming the target storage space for the data to be processed consists of 11 minimum blocks and the free block level is 5, as mentioned above, the determined target blocks are two level 5 free memory blocks (or free file blocks) and one level 1 free memory block (or free file block). When it is detected that the data record corresponding to the data to be processed has been deleted, the target blocks need to be added to the free block list. That is, two level 5 free memory blocks (or free file blocks) are added to the free block list Head_5, and one level 1 free memory block (or free file block) is added to the free block list Head_1.

[0078] Optionally, after the computer device stores the data record corresponding to the data to be processed into the target storage space, the method further includes: if it is determined that the data record corresponding to the data to be processed has been deleted, determining the level and number of free blocks to be reclaimed based on the free block level and the size of the data to be processed; determining target blocks from the target storage space based on the level and number of free blocks to be reclaimed, and adding the target blocks to the free block list. It should be noted that when the computer device detects that the free block list is empty, that is, when processing data for the first time, the target storage space of the data to be processed is determined from the target memory space corresponding to the memory mapping region. When it is detected that the data record corresponding to the data to be processed has been deleted, the target blocks to be returned need to be determined first using formula (1), and then these target blocks are added to the free block list.

[0079] S403. If there is no target storage space for the data to be processed in the target memory space corresponding to the memory-mapped region, then the target storage space for the data to be processed shall be determined from the file storage space corresponding to the non-memory-mapped region.

[0080] In one possible implementation, please see Figure 8a If it is the first time writing data to the database, the computer device can choose not to query the free block list, but instead prioritize determining the target storage space for the data to be processed from the target memory space corresponding to the memory-mapped region. Only when the target memory space is insufficient will the computer device determine the target storage space for the data to be processed from the file storage space corresponding to the non-memory-mapped region, thereby achieving high-performance and highly scalable data storage services.

[0081] S404. Store the data record corresponding to the data to be processed into the target storage space.

[0082] The specific implementation methods of steps S401 and S404 can be referred to the specific implementation methods of steps S201 and S203 above, and will not be repeated here.

[0083] Please see Figure 8b , Figure 8b This is a schematic diagram of a data record corresponding to data to be processed, provided in an embodiment of this application. For example... Figure 8b As shown, a data record corresponding to a piece of data to be processed consists of a Key field and a Value field. The Key field consists of multiple Key memory blocks or multiple Key file blocks, and the Value field consists of multiple Value memory blocks or multiple Value file blocks.

[0084] In one possible implementation, the memory-mapped region includes a header region containing engine parameter information and a hash table. The engine parameter information includes the number of hash buckets. The method further includes: determining the target hash bucket corresponding to the data to be processed based on the key field of the data to be processed and the number of hash buckets; and adding the data record corresponding to the data to be processed to the binary tree structure corresponding to the target hash bucket. This approach, utilizing a hash table in conjunction with a binary tree structure, enables rapid location of data records, improving data processing efficiency.

[0085] Please see Figure 9 , Figure 9 This is a schematic diagram of a hash table provided in an embodiment of this application. For example... Figure 9 As shown, the hash table includes n hash buckets, where n represents the number of hash buckets. Hash calculations are performed based on the key field of the data to be processed and the number of hash buckets to find the target hash bucket corresponding to the data to be processed.

[0086] bucket_T=key% bucket_size (2)

[0087] Here, `key` represents the key field of the data to be processed, `bucket_size` represents the number of hash buckets, and `bucket_T` represents the identifier of the target hash bucket corresponding to the data to be processed. Further, the data record corresponding to the data to be processed is added to the binary tree structure corresponding to the target hash bucket.

[0088] In one possible implementation, after the computer device stores the data record corresponding to the data to be processed into the target storage space, the method further includes: obtaining an operation instruction for the data to be processed, the operation instruction including a read instruction or a delete instruction; determining the target hash bucket corresponding to the data to be processed based on the key field and the number of hash buckets of the data to be processed; querying the data record corresponding to the data to be processed from the binary tree structure corresponding to the target hash bucket; and executing the operation corresponding to the operation instruction on the data record corresponding to the data to be processed.

[0089] It should be noted that when the obtained operation instruction for the data to be processed is a read instruction, the target hash bucket corresponding to the data to be processed is determined according to the above formula (2). The data record corresponding to the data to be processed is quickly retrieved using a binary tree search method, and the data corresponding to that data record is read.

[0090] Similarly, when the obtained operation instruction for the data to be processed is a deletion instruction, the target hash bucket corresponding to the data to be processed is determined according to the above formula (2). The binary tree search method quickly finds the data record corresponding to the data to be processed, deletes the data record, and adds the target block corresponding to the data to be processed to the free block list.

[0091] In one possible implementation, the method further includes: retrieving a log file when the computer device restarts after a power outage, and recovering data based on the log file and the database engine file, wherein the log file is used to record the data operation log. It should be noted that storing data records simultaneously in the log file and the database engine file ensures that data is not lost even when the computer device loses power, improving data security. For example, the computer device exports and copies the database engine file to a remote machine every morning; when performing data recovery for a specific period, the database engine file from that morning and the log file (such as the Binlog file) for that period can be retrieved, and the performed operations (including writing and deleting data) can be redone to recover the data.

[0092] Please see Figure 10a , Figure 10a This is a schematic diagram of a log file provided in an embodiment of this application. For example... Figure 10a As shown, this log file (BinlogFile) consists of a header and data operation records (logRecord). Please refer to [link / reference]. Figure 10b , Figure 10b This is a schematic diagram of a log file header provided in an embodiment of this application. For example... Figure 10b As shown, this header includes the start and end times of the data operation, the smallest database write operation number (seq), and the largest database write operation number. Please refer to [link / reference]. Figure 10c , Figure 10c This is a schematic diagram of a data operation record provided in an embodiment of this application. For example... Figure 10c As shown, the data operation record (logRecord) includes the operation type (such as writing data, deleting data, etc.) performed on the data record corresponding to the data to be processed, the database write operation number, the execution time, and the key and value fields corresponding to the operation.

[0093] In summary, the data processing method described above primarily employs a database engine that combines in-memory data storage space with file storage space. For example... Figure 11 As shown, database record reading and writing are performed through the database engine. The database engine file stores the data records corresponding to various data. The database engine file used here combines memory (i.e., memory data storage space) and data files (i.e., file storage space). Furthermore, during data recovery, log files are also combined to ensure that data is persisted to disk while providing high-performance and highly scalable data storage services, thereby improving data security.

[0094] Please see Figure 12 , Figure 12This is a schematic diagram of a game database acquisition interface provided in an embodiment of this application. Figure 12 As shown, the game database retrieval interface includes a game database introduction box, an activity announcement box, a product introduction box, and a "Use Now" button. The game database introduction box contains a brief description of the game database; the activity announcement box lists current popular activities that users can participate in; and the product introduction box contains the game database product documentation, such as a user manual. Users can access the game database by triggering the "Use Now" button. The computer equipment utilizes the aforementioned data processing methods to implement a database engine, providing users with high-performance, low-cost, highly scalable, highly secure, and 24 / 7 data storage services.

[0095] In summary, in this embodiment, the data to be processed is first acquired; then, the target storage space for the data to be processed is determined based on the database engine file. This database engine file includes a memory-mapped area and a non-memory-mapped area, where the memory-mapped area provides memory data storage space, and the non-memory-mapped area provides file storage space; finally, the data records corresponding to the data to be processed are stored in the target storage space. The database engine file combines memory data storage space and file storage space. When memory data storage space is insufficient, data can continue to be written to the file storage space, providing large-capacity data storage services while ensuring data security.

[0096] Based on the above data processing method, this application provides a data processing apparatus. Please refer to... Figure 13 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. The data processing device 1300 can operate the following units:

[0097] Acquisition unit 1301 is used to acquire data to be processed;

[0098] The determining unit 1302 is used to determine the target storage space of the data to be processed based on the database engine file, the database engine file including a memory-mapped area and a non-memory-mapped area, the memory-mapped area being used to provide memory data storage space and the non-memory-mapped area being used to provide file storage space;

[0099] Storage unit 1303 is used to store the data record corresponding to the data to be processed into the target storage space.

[0100] In one embodiment, the determining unit 1302, when determining the target storage space of the data to be processed based on the database engine file, may specifically be used to: determine the target storage space of the data to be processed from the target memory space corresponding to the memory-mapped region; if there is no target storage space of the data to be processed in the target memory space corresponding to the memory-mapped region, then determine the target storage space of the data to be processed from the file storage space corresponding to the non-memory-mapped region.

[0101] In another embodiment, the memory-mapped region includes a header region, which includes a free block list. The determining unit 1302 is further configured to: obtain the free block list included in the header region, which includes a free memory block list and a free file block list; determine the target storage space of the data to be processed from the free block list; if there is no target storage space for the data to be processed in the free block list, then perform the step of determining the target storage space of the data to be processed from the target memory space corresponding to the memory-mapped region.

[0102] In another embodiment, the header region also includes engine parameter information, which includes free block level. The determining unit 1302, when determining the target storage space for the data to be processed from the free block list, can be specifically used to: determine the level and number of free blocks to be requested based on the free block level and the size of the data to be processed; determine the target block from the free block list based on the level and number of free blocks to be requested, and use the target block as the target storage space for the data to be processed.

[0103] In another embodiment, the determining unit 1302, when determining the target block from the free block list based on the level and quantity of the free block to be requested, may specifically be used to: determine the target free memory block from the free memory block list of the free block list based on the level and quantity of the free block to be requested, and use the target free memory block as the target block; if there is no target free memory block in the free memory block list, then determine the target free file block from the free file block list of the free block list based on the level and quantity of the free block to be requested, and use the target free file block as the target block.

[0104] In another embodiment, after storing the data record corresponding to the data to be processed into the target storage space, the storage unit 1303 further includes a processing unit, which is used to add the target block to the free block list if it is determined that the data record corresponding to the data to be processed has been deleted.

[0105] In another implementation, the memory-mapped region includes a header region, which includes engine parameter information and a hash table. The engine parameter information includes the number of hash buckets. The processing unit is further configured to: determine the target hash bucket corresponding to the data to be processed based on the key field of the data to be processed and the number of hash buckets; and add the data record corresponding to the data to be processed to the binary tree structure corresponding to the target hash bucket.

[0106] In another embodiment, after storing the data record corresponding to the data to be processed into the target storage space, the storage unit 1303 is further configured to: obtain an operation instruction for the data to be processed, the operation instruction including a read instruction or a delete instruction; determine the target hash bucket corresponding to the data to be processed based on the key field and the number of hash buckets of the data to be processed; query the data record corresponding to the data to be processed from the binary tree structure corresponding to the target hash bucket; and execute the operation corresponding to the operation instruction for the data record corresponding to the data to be processed.

[0107] According to another embodiment of this application, Figure 13 The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0108] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 2 or Figure 4 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 13 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0109] In this embodiment, the data to be processed is first acquired; then, the target storage space for the data to be processed is determined based on the database engine file. This target storage space includes memory space or file space. The database engine file here includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area is used to provide memory data storage space, and the non-memory-mapped area is used to provide file storage space. Finally, the data records corresponding to the data to be processed are stored in the target storage space. The database engine file used combines memory data storage space and file storage space, providing large-capacity data storage services while ensuring data security.

[0110] Based on the description of the above method and apparatus embodiments, this application also provides a computer device. Please refer to... Figure 14 The computer device 1400 includes at least a processor 1401, a communication interface 1402, and a computer storage medium 1403. The processor 1401, communication interface 1402, and computer storage medium 1403 can be connected via a bus or other means. The computer storage medium 1403 can be stored in the memory 1404 of the computer device 1400. The computer storage medium 1403 is used to store computer programs, which include program instructions. The processor 1401 is used to execute the program instructions stored in the computer storage medium 1403. The processor 1401 (or CPU (Central Processing Unit)) is the computing and control core of the computer device, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions.

[0111] In one embodiment, the processor 1401 described in this application embodiment can be used to perform a series of data processing operations, specifically including: acquiring data to be processed; determining the target storage space for the data to be processed based on a database engine file, wherein the database engine file includes a memory-mapped area and a non-memory-mapped area, wherein the memory-mapped area is used to provide memory data storage space and the non-memory-mapped area is used to provide file storage space; storing the data records corresponding to the data to be processed into the target storage space, etc.

[0112] This application embodiment also provides a computer storage medium (memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer storage medium provides storage space that stores the operating system of the computer device. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1401. These instructions can be one or more computer programs (including program code). It should be noted that the computer storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor.

[0113] In one embodiment, a processor may load and execute one or more instructions stored in a computer storage medium to achieve the aforementioned... Figure 2 or Figure 4 The corresponding steps of the method in the data processing method embodiment shown; in specific implementation, one or more instructions in the computer storage medium are loaded by the processor 1401 and executed as follows:

[0114] Obtain the data to be processed;

[0115] The target storage space for the data to be processed is determined based on the database engine file, which includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area is used to provide memory data storage space, and the non-memory-mapped area is used to provide file storage space.

[0116] Store the data records corresponding to the data to be processed in the target storage space.

[0117] In one implementation, when determining the target storage space for the data to be processed based on the database engine file, the one or more instructions can be loaded and specifically executed by the processor: determining the target storage space for the data to be processed from the target memory space corresponding to the memory-mapped region; if there is no target storage space for the data to be processed in the target memory space corresponding to the memory-mapped region, then determining the target storage space for the data to be processed from the file storage space corresponding to the non-memory-mapped region.

[0118] In another implementation, the memory-mapped region includes a header region, which includes a list of free blocks. The one or more instructions can be loaded by the processor and specifically executed as follows: obtaining the list of free blocks included in the header region, which includes a list of free memory blocks and a list of free file blocks; determining the target storage space for the data to be processed from the list of free blocks; if the target storage space for the data to be processed does not exist in the list of free blocks, then performing the step of determining the target storage space for the data to be processed from the target memory space corresponding to the memory-mapped region.

[0119] In another implementation, the header region also includes engine parameter information, which includes free block levels. When determining the target storage space for the data to be processed from the free block list, one or more instructions can be loaded and executed by the processor: determining the level and number of free blocks to be requested based on the free block level and the size of the data to be processed; determining the target block from the free block list based on the level and number of free blocks to be requested, and using the target block as the target storage space for the data to be processed.

[0120] In another implementation, when determining the target block from the free block list based on the level and quantity of the free block to be requested, the one or more instructions can be loaded and executed by the processor: based on the level and quantity of the free block to be requested, determine the target free memory block from the free memory block list of the free block list, and use the target free memory block as the target block; if there is no target free memory block in the free memory block list, then based on the level and quantity of the free block to be requested, determine the target free file block from the free file block list of the free block list, and use the target free file block as the target block.

[0121] In another implementation, after storing the data record corresponding to the data to be processed into the target storage space, the one or more instructions can be loaded and specifically executed by the processor: if it is determined that the data record corresponding to the data to be processed has been deleted, the target block is added to the free block list.

[0122] In another implementation, the memory-mapped region includes a header region, which includes engine parameter information and a hash table. The engine parameter information includes the number of hash buckets. One or more instructions can be loaded by the processor and specifically executed as follows: based on the key field of the data to be processed and the number of hash buckets, determine the target hash bucket corresponding to the data to be processed; add the data record corresponding to the data to be processed to the binary tree structure corresponding to the target hash bucket.

[0123] In another implementation, after storing the data record corresponding to the data to be processed into the target storage space, the one or more instructions can be loaded by the processor and specifically executed as follows: obtaining operation instructions for the data to be processed, including read instructions or delete instructions; determining the target hash bucket corresponding to the data to be processed based on the key field and the number of hash buckets of the data to be processed; querying the data record corresponding to the data to be processed from the binary tree structure corresponding to the target hash bucket; and executing the operation corresponding to the operation instruction for the data record corresponding to the data to be processed.

[0124] In this embodiment, the data to be processed is first acquired; then, the target storage space for the data to be processed is determined based on the database engine file. This target storage space includes memory space or file space. The database engine file here includes a memory-mapped area and a non-memory-mapped area. The memory-mapped area is used to provide memory data storage space, and the non-memory-mapped area is used to provide file storage space. Finally, the data records corresponding to the data to be processed are stored in the target storage space. The database engine file used combines memory data storage space and file storage space, providing large-capacity data storage services while ensuring data security.

[0125] It should be noted that, according to one aspect of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the aforementioned... Figure 2 or Figure 4 The data processing method embodiments shown are provided in various optional ways. Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, equivalent variations made in accordance with the claims of this application are still within the scope of this application.

Claims

1. A data processing method, characterized by, The method comprises: acquiring to-be-processed data; determining a target storage space of the to-be-processed data based on a database engine file, the database engine file comprising a memory mapping area and a non-memory mapping area, the memory mapping area being used to provide a memory data storage space, and the non-memory mapping area being used to provide a file storage space; storing a data record corresponding to the to-be-processed data into the target storage space; wherein the memory mapping area comprises a header area, the header area comprising engine parameter information and a hash table, the engine parameter information comprising a number of hash buckets; and the steps of determining the target storage space of the to-be-processed data based on the database engine file and storing the data record corresponding to the to-be-processed data into the target storage space comprise: determining a target hash bucket corresponding to the to-be-processed data based on a key field of the to-be-processed data and the number of hash buckets; adding the data record corresponding to the to-be-processed data into a binary tree structure corresponding to the target hash bucket.

2. The method of claim 1, wherein, The step of determining the target storage space of the to-be-processed data based on the database engine file comprises: determining the target storage space of the to-be-processed data from a target memory space corresponding to the memory mapping area; if the target storage space of the to-be-processed data does not exist in the target memory space corresponding to the memory mapping area, then determining the target storage space of the to-be-processed data from a file storage space corresponding to the non-memory mapping area.

3. The method of claim 2, wherein, The memory mapping area comprises a header area, and the header area comprises a free block linked list; the method further comprises: acquiring the free block linked list comprised by the header area, the free block linked list comprising a free memory block linked list and a free file block linked list; determining the target storage space of the to-be-processed data from the free block linked list; if the target storage space of the to-be-processed data does not exist in the free block linked list, then performing the step of determining the target storage space of the to-be-processed data from the target memory space corresponding to the memory mapping area.

4. The method of claim 3, wherein, The header area further comprises engine parameter information, and the engine parameter information comprises a free block level; the step of determining the target storage space of the to-be-processed data from the free block linked list comprises: determining a level and a number of to-be-applied free blocks based on the free block level and a size of the to-be-processed data; determining a target block from the free block linked list based on the level and the number of to-be-applied free blocks, and taking the target block as the target storage space of the to-be-processed data.

5. The method of claim 4, wherein, The step of determining the target block from the free block linked list based on the level and the number of to-be-applied free blocks comprises: determining a target free memory block from a free memory block linked list of the free block linked list based on the level and the number of to-be-applied free blocks, and taking the target free memory block as the target block; if the target free memory block does not exist in the free memory block linked list, then determining a target free file block from a free file block linked list of the free block linked list based on the level and the number of to-be-applied free blocks, and taking the target free file block as the target block.

6. The method according to claim 4 or 5, characterized in that, After the data record corresponding to the to-be-processed data is stored in the target storage space, the method further includes: If it is determined that the data record corresponding to the to-be-processed data is deleted, the target block is added to the free block linked list.

7. The method of claim 1, wherein, After the data record corresponding to the to-be-processed data is stored in the target storage space, the method further includes: An operation instruction for the to-be-processed data is obtained, the operation instruction including a read instruction or a delete instruction; Based on the key field of the to-be-processed data and the number of hash buckets, a target hash bucket corresponding to the to-be-processed data is determined; A data record corresponding to the to-be-processed data is queried from a binary tree structure corresponding to the target hash bucket; An operation corresponding to the operation instruction is performed on the data record corresponding to the to-be-processed data.

8. A data processing apparatus, characterized by, The apparatus includes: An obtaining unit is configured to obtain to-be-processed data. A determining unit is configured to determine a target storage space of the to-be-processed data based on a database engine file, the database engine file including an in-memory mapping region and a non-in-memory mapping region, the in-memory mapping region being configured to provide an in-memory data storage space, the non-in-memory mapping region being configured to provide a file storage space, the in-memory mapping region including a header region, the header region including engine parameter information and a hash table, the engine parameter information including a number of hash buckets, and the determining unit being specifically configured to determine a target hash bucket corresponding to the to-be-processed data based on a key field of the to-be-processed data and the number of hash buckets. A storing unit is configured to store a data record corresponding to the to-be-processed data in the target storage space, and the storing unit is specifically configured to add the data record corresponding to the to-be-processed data to a binary tree structure corresponding to the target hash bucket.

9. A computer device, comprising: The computer device includes a memory and a processor, and the memory stores a computer program, which, when executed by the processor, causes the processor to execute the data processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more computer programs, and the one or more computer programs are adapted to be loaded and executed by the processor to execute the data processing method according to any one of claims 1-7.

11. A computer program product or computer program, characterized in that, The computer program product includes a computer program, and the computer program is adapted to be loaded and executed by the processor to execute the data processing method according to any one of claims 1-7.

Images