Data access method and apparatus, data storage method and apparatus

Abstract

The application relates to data storage and data reading, in particular to a data access method and device. The method comprises the following steps: obtaining a data access instruction corresponding to an object to be accessed and carrying target index information; determining a first address of a target address space loaded with a shared memory file and initial position information of an initial object in the object to be accessed in the target shared space according to the target index information; the shared memory file is obtained by mapping each data object in the target shared space; adjusting the initial position information based on the first address to obtain object position information matched with the target address space; obtaining data content corresponding to the initial object according to the object position information and by calling the target address space; and sequentially polling each target object in the object to be accessed except the initial object to obtain data content corresponding to each target object. The method can save computer resources.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data access method and apparatus, and a data storage method and apparatus. Background Technology

[0002] With the development of computer technology, an off-heap memory space can be allocated to store frequently used data. However, due to the limitations of the JVM (Java Virtual Machine, Java Execution Engine), data objects cannot be directly stored in the off-heap memory space; only object byte streams can be stored. When storing data objects in off-heap memory, the data objects need to be serialized to convert them into object byte streams. When accessing off-heap memory data is needed through the JVM, deserialization is required to convert the object byte stream back into a data object.

[0003] However, using serialization and deserialization algorithms to access the JVM's internal data consumes CPU and other computer resources, resulting in a waste of resources. Summary of the Invention

[0004] Therefore, it is necessary to provide a data access method, apparatus, computer equipment, storage medium, and computer program that saves computer resources, as well as a data storage method, apparatus, computer equipment, storage medium, and computer program, to address the aforementioned technical problems.

[0005] A data access method, the method comprising:

[0006] Obtain a data access instruction corresponding to the object to be accessed and carrying target index information, wherein the object to be accessed is at least a part of the data object written to the target shared space, and the object to be accessed includes at least one target object;

[0007] Based on the target index information, the first starting address of the target address space containing the shared memory file and the initial position information of the initial object in the target shared space are determined; the shared memory file is obtained by mapping each data object in the target shared space;

[0008] The initial location information is adjusted based on the first address to obtain object location information that matches the target address space;

[0009] Based on the object location information, and by calling the target address space, the data content corresponding to the initial object is obtained;

[0010] The method sequentially polls each target object in the object to be accessed, excluding the initial object, to obtain the data content corresponding to each target object. Specifically, for each polled target object, the object position information of the current target object is determined based on the object position information of the preceding target object polled in the previous poll. Then, based on the object position information of the current target object, the data content corresponding to the current target object is obtained by calling the target address space. In one embodiment, the method further includes:

[0011] When an object to be accessed in the target physical memory corresponding to the target address space meets the deletion conditions, the object to be accessed is deleted from the target physical memory.

[0012] In one embodiment, obtaining the data content corresponding to the current target object by calling the target address space based on the object location information of the current target object includes:

[0013] Based on the object location information of the current target object, the data content in the current target object is obtained by calling the target address space;

[0014] When the data content in the current target object indicates that the current target object references at least one layer of reference objects, the data content included in each layer of reference objects is determined based on the reference position information corresponding to each layer of reference objects;

[0015] The data content in the current target object and the data content included in each layer of referenced objects are combined as the data content corresponding to the current target object.

[0016] In one embodiment, when the data content in the current target object indicates that the current target object references at least one level of reference objects, determining the data content included in each level of reference objects based on the reference position information corresponding to each level of reference objects includes:

[0017] When the data content in the current target object indicates that the current target object references at least one layer of reference objects, for each layer of reference objects, the reference position information of the reference objects at the corresponding level is adjusted based on at least one of the first starting address and the second starting address of the target shared space, so as to obtain the object position information corresponding to the reference objects at the corresponding level and matching the target address space.

[0018] Based on the object location information corresponding to each level of the referenced object, and by calling the target address space, the data content corresponding to each level of the referenced object is obtained.

[0019] In one embodiment, before obtaining a data access instruction corresponding to the object to be accessed and carrying target index information, the method further includes:

[0020] Obtain the file storage path of the shared memory file, load the corresponding shared memory file into the target address space through the file storage path, and obtain the first starting address of the target address space;

[0021] Based on the first address of the target address space, obtain the verification information in the shared memory file, and verify the integrity of the shared memory file based on the verification information.

[0022] A data access device, comprising:

[0023] An initial position determination module is used to obtain a data access instruction corresponding to the object to be accessed and carrying target index information. The object to be accessed is at least a part of the data objects written to the target shared space, and the object to be accessed includes at least one target object. Based on the target index information, the module determines the first starting address of the target address space loaded with the shared memory file and the initial position information of the initial object in the target shared space. The shared memory file is obtained by mapping each data object in the target shared space.

[0024] The initial object acquisition module is used to adjust the initial position information based on the first starting address to obtain object position information that matches the target address space; and to obtain the data content corresponding to the initial object based on the object position information and by calling the target address space.

[0025] The target object acquisition module is used to sequentially poll each target object in the object to be accessed, excluding the initial object, to obtain the data content corresponding to each target object. Specifically, for each polled target object, the object position information of the current target object is determined based on the object position information of the preceding target object polled in the previous poll, and the data content corresponding to the current target object is obtained by calling the target address space based on the object position information of the current target object.

[0026] In one embodiment, the target index information includes the file storage path of the shared memory file and the target fragment identifier; the initial position determination module further includes a target fragment determination module, configured to determine the first starting address of the target address space containing the shared memory file based on the file storage path, and determine the file metadata corresponding to the shared memory file based on the first starting address; determine the target fragment corresponding to the object to be accessed based on the file metadata and the target fragment identifier; and determine the initial position information of the initial object in the target shared space based on the fragment content of the target fragment.

[0027] In one embodiment, the file metadata includes the memory space occupied by the header of the shared memory file; the target segment determination module is further configured to determine, based on the target segment identifier and the space size, the first offset position information of the target segment corresponding to the object to be accessed in the target shared space; based on the first starting address and the first offset position information, determine the segment position information corresponding to the target segment and matching the target address space; and based on the segment position information of the target segment, determine the segment content of the target segment.

[0028] In one embodiment, the data access instruction is triggered by a data access process, and the initial object acquisition module is further configured to respond to the data access instruction by, according to the object location information corresponding to the initial object, and by calling the target address space through the data access process, reading the data content of the initial object from the target physical memory corresponding to the target address space and feeding it back; wherein, the data content in the target physical memory is loaded and stored from the shared memory file by the computer kernel.

[0029] In one embodiment, the data access device further includes a deletion module, configured to delete the object to be accessed from the target physical memory when the object in the target physical memory corresponding to the target address space meets the deletion conditions.

[0030] In one embodiment, the target object acquisition module is further configured to determine the object location information of the preceding target object polled in the previous poll and the memory space occupied by the preceding target object; and based on the object location information of the preceding target object and the memory space occupied by the preceding target object, determine the object location information of the current target object polled in the current poll.

[0031] In one embodiment, the target object acquisition module includes a reference object acquisition module, configured to acquire the data content of the current target object by calling the target address space based on the object location information of the current target object; when the data content of the current target object indicates that the current target object references at least one level of reference objects, the data content included in each level of reference objects is determined based on the reference location information corresponding to each level of reference objects; the data content of the current target object and the data content included in each level of reference objects are combined as the data content corresponding to the current target object.

[0032] In one embodiment, the reference object acquisition module is further configured to, when the data content in the current target object indicates that the current target object references at least one level of reference objects, adjust the reference position information of the reference objects at the corresponding level based on at least one of the first starting address and the second starting address of the target shared space for each level of reference objects, to obtain object position information corresponding to the reference objects at the corresponding level and matching the target address space; and obtain the data content corresponding to each level of reference objects by calling the target address space based on the object position information corresponding to each level of reference objects.

[0033] In one embodiment, the reference object acquisition module is further configured to: when the data content of the current target object includes reference location information of the reference object referenced by the current target object at the current level, adjust the reference location information of the reference object at the current level based on at least one of the first starting address and the second starting address of the target shared space, to obtain object location information corresponding to the reference object at the current level that matches the target address space; obtain the data content corresponding to the reference object at the current level by calling the target address space based on the object location information of the reference object at the current level; when the data content of the reference object at the current level includes reference location information of the reference object at the next level in the target shared space, use the reference object at the next level as the reference object at the current level in the next round, and return to the step of adjusting the reference location information of the reference object at the current level based on at least one of the first starting address and the second starting address of the target shared space to obtain object location information corresponding to the reference object at the current level that matches the target address space, until the data content of the reference object at the final level is obtained.

[0034] In one embodiment, the data access device is further configured to obtain the file storage path of the shared memory file, load the corresponding shared memory file into the target address space through the file storage path to obtain the first starting address of the target address space; obtain the verification information in the shared memory file according to the first starting address of the target address space, and verify the integrity of the shared memory file according to the verification information.

[0035] In one embodiment, the object to be accessed is commodity transaction data collected by the OLAP engine component through a big data platform; the data access device is also used to perform multi-dimensional analysis on the target commodity transaction data when the target commodity transaction data corresponding to the object to be accessed is obtained, and to obtain a commodity transaction analysis report.

[0036] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0037] Obtain a data access instruction corresponding to the object to be accessed and carrying target index information, wherein the object to be accessed is at least a part of the data object written to the target shared space, and the object to be accessed includes at least one target object;

[0038] Based on the target index information, the first starting address of the target address space containing the shared memory file and the initial position information of the initial object in the target shared space are determined; the shared memory file is obtained by mapping each data object in the target shared space;

[0039] The initial location information is adjusted based on the first address to obtain object location information that matches the target address space;

[0040] Based on the object location information, and by calling the target address space, the data content corresponding to the initial object is obtained;

[0041] The target objects, excluding the initial object, are polled sequentially to obtain the data content corresponding to each target object. For each target object polled, the object position information of the current target object is determined based on the object position information of the preceding target object polled in the previous poll. Based on the object position information of the current target object, the data content corresponding to the current target object is obtained by calling the target address space.

[0042] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0043] Obtain a data access instruction corresponding to the object to be accessed and carrying target index information, wherein the object to be accessed is at least a part of the data object written to the target shared space, and the object to be accessed includes at least one target object;

[0044] Based on the target index information, the first starting address of the target address space containing the shared memory file and the initial position information of the initial object in the target shared space are determined; the shared memory file is obtained by mapping each data object in the target shared space;

[0045] The initial location information is adjusted based on the first address to obtain object location information that matches the target address space;

[0046] Based on the object location information, and by calling the target address space, the data content corresponding to the initial object is obtained;

[0047] The target objects, excluding the initial object, are polled sequentially to obtain the data content corresponding to each target object. For each target object polled, the object position information of the current target object is determined based on the object position information of the preceding target object polled in the previous poll. Based on the object position information of the current target object, the data content corresponding to the current target object is obtained by calling the target address space.

[0048] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the following steps:

[0049] Obtain a data access instruction corresponding to the object to be accessed and carrying target index information, wherein the object to be accessed is at least a part of the data object written to the target shared space, and the object to be accessed includes at least one target object;

[0050] Based on the target index information, the first starting address of the target address space containing the shared memory file and the initial position information of the initial object in the target shared space are determined; the shared memory file is obtained by mapping each data object in the target shared space;

[0051] The initial location information is adjusted based on the first address to obtain object location information that matches the target address space;

[0052] Based on the object location information, and by calling the target address space, the data content corresponding to the initial object is obtained;

[0053] The target objects, excluding the initial object, are polled sequentially to obtain the data content corresponding to each target object. For each target object polled, the object position information of the current target object is determined based on the object position information of the preceding target object polled in the previous poll. Based on the object position information of the current target object, the data content corresponding to the current target object is obtained by calling the target address space.

[0054] The aforementioned data access method, apparatus, computer device, storage medium, and computer program, by acquiring data access instructions, can accurately determine the initial location information of the initial object based on the target index information in the data access instructions. By determining the initial location information, it can be adjusted based on the first address of the target address space to obtain object location information matching the target address space. By determining the object location information, the data content of the initial object can be obtained based on the object location information, and each target object other than the initial object in the objects to be accessed can be polled sequentially to obtain the data content corresponding to each target object. Since the shared memory file is directly loaded into the target address space during the data access process, the corresponding data content can be obtained by calling the target address space without serializing or deserializing any objects to be accessed. Compared to the traditional method that requires serialization and deserialization of the objects to be accessed before obtaining the corresponding data content, this method saves computer resources.

[0055] Furthermore, since a large number of temporary objects are generated during the serialization and deserialization of data objects, which consume a lot of memory resources, this method can effectively reduce the memory space occupied by a large number of temporary objects, thereby improving memory utilization.

[0056] A data access method, the method comprising:

[0057] Obtain the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes a content storage area and a fragment storage area;

[0058] Write the data object into the content storage area of ​​the target shared space;

[0059] Generate an original fragment corresponding to the data object, and write the original fragment into the fragment storage area of ​​the shared memory space;

[0060] Based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space, determine the second offset position information of the initial storage object in the target shared space;

[0061] Based on the second offset position information, the fragment metadata in the original fragment is adjusted to obtain a target fragment corresponding to the data object; wherein, the fragment metadata in the target fragment is used to indicate the storage location of the initial storage object in the data object in the target shared space.

[0062] In one embodiment, the method further includes:

[0063] When the target shared space meets the mapping conditions, the data objects stored in the target shared space are mapped to the disk according to the preset file storage path, resulting in a shared memory file stored on the disk.

[0064] A data storage device, comprising:

[0065] The data writing module is used to acquire the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes a content storage area and a fragment storage area; and write the data object into the content storage area of ​​the target shared space.

[0066] The fragment writing module is used to generate an original fragment corresponding to the data object and write the original fragment into the fragment storage area of ​​the shared memory space;

[0067] The fragment adjustment module is used to determine the second offset position information of the initial storage object in the target shared space based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space; and to adjust the fragment meta-information in the original fragment based on the second offset position information to obtain a target fragment corresponding to the data object; wherein the fragment meta-information in the target fragment is used to indicate the storage position of the initial storage object in the data object in the target shared space.

[0068] In one embodiment, the data writing module is further configured to determine the reference object referenced by each storage object in the data object; and write the storage object and the reference object into the content storage area.

[0069] In one embodiment, the fragment writing module is further configured to generate an original fragment corresponding to the data object; determine the occupied position information of the last-order fragment stored in the fragment storage area in the target shared space; determine the target unoccupied position in the fragment storage area based on the occupied position information, and write the original fragment to the target unoccupied position.

[0070] In one embodiment, the target shared space includes a file metadata storage area; the data storage device is further configured to, when the data object is written to the content storage area in the target shared space, read the total number of fragments and the total amount of data stored in the file metadata storage area; and increment the total number of fragments and the total amount of data respectively.

[0071] In one embodiment, the data storage device is further configured to, when the target shared space meets the mapping conditions, map the data objects stored in the target shared space to the disk according to a preset file storage path, thereby obtaining a shared memory file stored on the disk.

[0072] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0073] Obtain the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes a content storage area and a fragment storage area;

[0074] Write the data object into the content storage area of ​​the target shared space;

[0075] Generate an original fragment corresponding to the data object, and write the original fragment into the fragment storage area of ​​the shared memory space;

[0076] Based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space, determine the second offset position information of the initial storage object in the target shared space;

[0077] Based on the second offset position information, the fragment metadata in the original fragment is adjusted to obtain a target fragment corresponding to the data object; wherein, the fragment metadata in the target fragment is used to indicate the storage location of the initial storage object in the data object in the target shared space.

[0078] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0079] Obtain the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes a content storage area and a fragment storage area;

[0080] Write the data object into the content storage area of ​​the target shared space;

[0081] Generate an original fragment corresponding to the data object, and write the original fragment into the fragment storage area of ​​the shared memory space;

[0082] Based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space, determine the second offset position information of the initial storage object in the target shared space;

[0083] Based on the second offset position information, the fragment metadata in the original fragment is adjusted to obtain a target fragment corresponding to the data object; wherein, the fragment metadata in the target fragment is used to indicate the storage location of the initial storage object in the data object in the target shared space.

[0084] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the following steps:

[0085] Obtain the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes a content storage area and a fragment storage area;

[0086] Write the data object into the content storage area of ​​the target shared space;

[0087] Generate an original fragment corresponding to the data object, and write the original fragment into the fragment storage area of ​​the shared memory space;

[0088] Based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space, determine the second offset position information of the initial storage object in the target shared space;

[0089] Based on the second offset position information, the fragment metadata in the original fragment is adjusted to obtain a target fragment corresponding to the data object; wherein, the fragment metadata in the target fragment is used to indicate the storage location of the initial storage object in the data object in the target shared space.

[0090] The aforementioned data storage method, apparatus, computer equipment, storage medium, and computer program, by writing data objects into the content storage area of ​​the target shared space and storing the generated original fragments in the fragment storage area, can adjust the original fragments based on the second starting address and the initial storage address of the target shared space to obtain the target fragment corresponding to the data object. By obtaining the target fragment, when access to the data object is needed, the storage location of the initial storage object in the target shared space can be determined based on the target fragment data object. Since the data object can be directly stored in the target memory space and read directly through the target fragment, compared to the traditional method which requires serialization and deserialization of the object to be accessed before obtaining the corresponding data content, this method eliminates the need for any serialization and deserialization, thus saving computer resources. Attached Figure Description

[0091] Figure 1 This is an application environment diagram of a data access method in one embodiment;

[0092] Figure 2 This is a flowchart illustrating a data access method in one embodiment;

[0093] Figure 3 This is a schematic diagram of the architecture of a computer device in one embodiment;

[0094] Figure 4 This is a schematic diagram of the internal layout of a shared memory file in one embodiment;

[0095] Figure 5 This is a schematic diagram of the data access process in one embodiment;

[0096] Figure 6 This is a flowchart illustrating a data storage method in one embodiment;

[0097] Figure 7 This is a flowchart illustrating a data access method in a specific embodiment;

[0098] Figure 8 This is a flowchart illustrating a data storage method in a specific embodiment;

[0099] Figure 9 This is a structural block diagram of a data access device in one embodiment;

[0100] Figure 10 This is a structural block diagram of a data storage device in one embodiment;

[0101] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0102] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0103] Figure 1 This is a diagram illustrating the application environment of a data access method in one embodiment. (Refer to...) Figure 1 This data access method is applied to a data access system. The data access system includes a computer device 102, a first terminal 104, and a second terminal 106. The first terminal 104 and the computer device 102 are connected via a network, and the second terminal 106 and the computer device 102 are also connected via a network. The first terminal 104 is used to write data objects to the computer device 102, thereby allowing the computer device to map the written data objects to a shared memory file and load the shared memory file into a target address space. The second terminal 106 is used to send target index information to the computer device 102, thereby allowing the computer device 102 to access the object to be accessed by calling the target address space based on the target index information. It is easy to understand that the first terminal 104 can also be used to send target index information to the computer device 102, and the second terminal 106 can also be used to write data objects to the computer device 102.

[0104] Computer device 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The first terminal 104 and the second terminal 106 can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these. The terminals and servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this connection.

[0105] This application relates to data storage, specifically cloud storage. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of storage devices (also called storage nodes) of various types in a network to work together and jointly provide data storage and business access functions.

[0106] Currently, the storage method of the storage system is as follows: Logical volumes are created. When a logical volume is created, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. The front-end stores data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume, and it records the storage location information of each object. Therefore, when the front-end requests access to data, the file system can use the storage location information of each object to allow the front-end to access the data.

[0107] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0108] The data access method provided in this application can be applied to a data access system, which can be a distributed system formed by access devices and multiple connected nodes. Specifically, a distributed architecture system can be a blockchain system. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0109] Traditional solutions rely on mechanisms provided by the JDK, using I / O (Input / Output) functions, data serialization, and deserialization functions to transfer data objects. This process generates numerous temporary data objects during both storage and retrieval, placing significant pressure on the process's garbage collection (GC) and severely impacting operational efficiency. Furthermore, reading data objects typically involves large datasets, often in the petabyte (PB) range, with individual data access processes maintaining terabyte (TB) levels of data. Due to limited computer memory, each data access requires loading data from disk into the LRU cache (Least Recently Used cache) and updating the cache. These frequent I / O operations place heavy read / write pressure on the disk. Consequently, data is copied multiple times between user-space and kernel-space memory during data retrieval, affecting system efficiency and disk lifespan. Furthermore, even if the off-heap function is implemented by requesting off-heap memory through the JVM's unsafe API and placing the data off-heap, it is still impossible to completely avoid copying data between the heap and off-heap during data reading.

[0110] This application provides new data reading and access interfaces at the JVM level, enabling the front-end to directly map corresponding data objects to a shared memory file (mmap memory file) on disk. This shared memory file can then be accessed via third-party tools such as SCP, or stored directly on a network storage device accessible from both the computer's front-end and back-end, thus avoiding file read / write operations in the data access process. When using this shared memory file, the back-end can directly call the provided data reading interface and the mmap system call to map the data from the shared memory file to the data access process in one go. During this process, no serialization or deserialization of any data objects is required, and the use and release of the target physical memory corresponding to the data access process are controlled through the corresponding interface, avoiding the traditional method of controlling data transfer between memory and disk using an LRU cache.

[0111] To better implement the data access and data storage methods in this application, data reading and data storage interfaces can be pre-built before implementing data access and data reading. The key interface functions involved are described below:

[0112] (1)void*JVM_Cache_Init(size_t size):

[0113] The computer front-end requests a shared memory space of size `size` bytes (page-size aligned) from the computer kernel. This shared memory space is completely off-heap and not managed by the JVM's garbage collection (GC) thread. During the call to the `void*JVM_Cache_Init(size_t size)` function, the function completes the allocation of the shared memory space and the initialization of the header (meta).

[0114] (2)int JVM_Cache_write(Object obj):

[0115] This function takes the input parameter `obj` as the root object. Starting from the root object `obj`, it performs a breadth-first traversal, copying `obj` and all data objects referenced by `obj` to off-heap memory (shared memory space) allocated through `JVM_Cache_Init` using the glibc function `void* memcpy(void*dest, const void*src, size_t n)`. To ensure data consistency, this operation is a `VM_operation`, meaning the function will first cause the JVM to enter a safepoint before performing the relevant object operations. The result is that `obj` is completely written to the shared memory file, and all references to non-primitive variables within `obj` point to objects within the shared memory (data objects referenced by the root object `obj`), thus excluding any heap objects except for `klass`. Furthermore, the initialized meta header is updated when this function is called. It's important to note that this function is only called from the front end of the computer device.

[0116] (3)int JVM_Cache_Load(const char*mmap_file_path, boolean data_read_only=true):

[0117] This function is called only by the backend of the computer device to load a shared memory file into the target address space of the data access process in read-only mode. It returns 0 if loading is successful, and -1 otherwise. Before loading is complete, this function verifies the metadata (magic) in the header and each segment of the shared memory file to ensure that the file is not corrupted.

[0118] (4)object JVM_Cache_Read(const char*mmap_file_path, int index):

[0119] This function is only called by the backend of the computer device. It extracts the target segment (index) from the shared memory file, which is the metadata information of the data object to be accessed. Then, it determines the offset between the storage address of the data object in the target shared space and its second starting address in the target shared space, using the target segment as the basis. Combined with the first starting address of the shared memory file in the target address space, it calculates the object address information (actual address) of the data object to be accessed. Afterward, this memory can be accessed directly using the object address information. Simultaneously, when the JVM obtains the data content of the initial object in the data object to be accessed, it calls the `oop_iterate` function for each target object in each data object to iterate through the `oop` type member variables in each target object. Based on the first and second starting addresses, it corrects the values ​​of the `oop` type member variables to obtain the object location information of the referenced object, and then sets the memory region pointed to by the referenced object location information to read-only.

[0120] (5)JVM_Cache_Madvise(const char*mmap_file_path, int offset):

[0121] This function is used to determine when a data object in a shared memory file is no longer in use and needs to be discussed from the LRU cache, and then directly releases the physical memory occupied by the corresponding data object.

[0122] (6)int JVM_Cache_Unload(const void*address,const char*mmap_file_path)

[0123] When a shared memory file is no longer needed, this function can be called directly to completely unload the shared memory file from the target address space.

[0124] (7)int JVM_Cache_Cleanup(const void*address,const char*mmap_file_path):

[0125] This function performs data cleanup, releasing all target physical memory occupied by shared memory files and closing the descriptors of the shared memory files.

[0126] The above is only a brief introduction to the key functions. The application of each function in specific embodiments will be explained in more detail later.

[0127] In one embodiment, a data access method is provided. This embodiment mainly illustrates the application of this method to computer device 102. (Refer to...) Figure 2 The data access method specifically includes the following steps:

[0128] Step S202: Obtain a data access instruction corresponding to the object to be accessed and carrying target index information. The object to be accessed is at least a part of the data object written to the target shared space, and the object to be accessed includes at least one target object.

[0129] The object to be accessed refers to the data object that needs to be accessed. It includes at least one target object, and each target object stores corresponding data content. For example, the object to be accessed could be a hashmap storing multiple key-value pairs. Each key-value pair can be called a target object, and the value corresponding to the key can be a Java object. Index information indicates the position of the initial object within the shared memory file; the initial object is the target object located at the beginning of the list of objects to be accessed.

[0130] The target shared space refers to the computer memory space used to receive written data objects. Simply put, when an object needs to be accessed, it can be called an object to be accessed; similarly, when an object is written to the target shared space, it can be called a data object. The object to be accessed is at least a portion of the data objects written to the target shared space. For example, the target shared space may store 10 data objects, and the object to be accessed could be the third data object among those 10.

[0131] Specifically, when it is necessary to obtain the data content of the object to be accessed, the data access party can input the target index information of the object to be accessed. Then, the backend in the computer device can determine the initial position information of the initial object in the target shared space through the target index information, and then the object to be accessed can be accessed based on the initial position information.

[0132] In one embodiment, reference Figure 3The computer device includes a front-end and a back-end. The front-end writes data objects to the target shared space, while the back-end accesses the objects to be accessed through a data access process. When the target shared space meets the mapping conditions, the computer device can map each data object in the target shared space to obtain a shared memory file stored on the computer's disk. Before accessing the object to be accessed, the back-end loads the shared memory file into the target address space of the data access process, obtaining the correspondence between the shared memory file and the target address space. Thus, when accessing the object to be accessed is needed, the data access process in the back-end can determine the corresponding shared memory file based on the target index information and obtain the data content of the object to be accessed through the correspondence between the shared memory file and the target address space. Figure 3 A schematic diagram of the architecture of a computer device in one embodiment is shown.

[0133] In one embodiment, a computer device may store multiple shared memory files. When a data writer writes a data object to a target shared space, the writer can record the file storage path of the shared memory file mapped through this target shared space, thereby establishing an association between the data object and the file storage path. When a data reader needs to access an object, the reader can determine the target file storage path corresponding to the object based on the association between the data object and the file storage path, and input this target file storage path to obtain target index information containing the file storage path.

[0134] In one embodiment, the data access instruction can be an operation instruction automatically initiated by the data access process based on operational needs, or it can be an operation instruction initiated by the data access party based on data reading needs. This embodiment does not impose any limitations on this.

[0135] In one embodiment, before obtaining a data access instruction corresponding to the object to be accessed and carrying target index information, the data access method further includes: obtaining the file storage path of the shared memory file, loading the corresponding shared memory file into the target address space through the file storage path to obtain the first address of the target address space; obtaining the verification information in the shared memory file based on the first address of the target address space, and verifying the integrity of the shared memory file based on the verification information.

[0136] Specifically, before accessing the object to be accessed, the backend data access process can determine whether the shared memory file is loaded into the corresponding target address space based on the file storage path of the shared memory file in the target index information. If it is determined that it is not loaded into the target address space, the shared memory file is loaded into the target address space, thereby obtaining the first address of the shared memory file in the target address space, and the correspondence between each data object in the shared memory file and its virtual address. Here, the address space refers to the process's virtual address space, which corresponds to multiple virtual addresses. These virtual addresses are mapped to the physical memory space through a page table, allowing the process to read the corresponding data stored in the physical memory space based on the virtual addresses. (Reference) Figure 3 When a shared memory file is mapped to a target address space, the data access process can obtain the first base address of the shared memory file in the target address space, which is the virtual address corresponding to the data object in the first order of the shared memory file. The data access process can also determine the one-to-one correspondence between each data object in the shared memory file and the virtual address. For example, it can determine that the second-order virtual address, which is adjacent to and follows the first base address, corresponds to the data object in the second order of the shared memory file; the third-order virtual address, which is adjacent to and follows the second-order virtual address, corresponds to the data object in the third order of the shared memory file, and so on.

[0137] It is worth noting that when a shared memory file is mapped to the target address space, the computer device only obtains the correspondence between the virtual address and the data object. At this time, the data content of the shared memory file is not actually copied to the physical memory space mapped to the virtual address through the page table.

[0138] Furthermore, once the correspondence between each data object and its virtual address in the shared memory file is known, the data access process can determine the target virtual address corresponding to the verification information based on this correspondence, and then read the verification information from the shared memory file based on this target virtual address. The integrity of the shared memory file is then verified according to the verification information. For example, when the verification information is obtained, the data access process can compare the verification information with preset standard information. If the verification information matches the standard information, the shared memory file is determined to be complete and error-free; if the verification information does not match the standard information, the shared memory file is determined to be corrupted.

[0139] In one embodiment, when the target virtual address corresponding to the verification information is determined based on the correspondence between virtual addresses and data objects—that is, when the verification location information corresponding to the verification information in the shared memory file is determined—the data access process can directly read the corresponding verification information based on the verification location information by accessing memory. At this time, the computer kernel can query the physical memory space mapped to the verification location information through the page table and read the verification information in the physical memory space. When it is determined that no verification information is stored in the physical memory space, the computer kernel generates a page fault and, based on the page fault, loads the verification information from the shared memory file into the physical memory space mapped to the target virtual address, and returns the verification information stored in the physical memory to the data access process.

[0140] In one embodiment, when a shared memory file is mapped to a target address space, a data access process can read all data objects in the shared memory file and store all the read data objects in the corresponding physical memory space, so that subsequent data access processes can directly read the data objects in the physical memory.

[0141] In one embodiment, when the target shared memory file is not loaded into the target address space, the target file can be loaded into the target address space by calling the following code: int JVM_Cache_Load(const char* mmap_file_path, boolean data_read_only=true). Here, mmap_file_path is the file storage path of the shared memory file on disk.

[0142] In the above embodiments, by mapping the shared memory file to the target address space, the correspondence between each data object in the shared memory file and the virtual address in the target address space can be known, so that the corresponding data object can be correctly found based on this correspondence.

[0143] Step S204: Based on the target index information, determine the first starting address of the target address space containing the shared memory file, and the initial position information of the initial object in the target shared space; the shared memory file is obtained by mapping each data object in the target shared space.

[0144] The target index information includes the file storage path of the shared memory file on disk and the target segment identifier. The target segment identifier is information used to indicate the target segment corresponding to the object to be accessed. The target segment refers to a data segment that stores the metadata of the object to be accessed.

[0145] In one embodiment, reference Figure 4 , Figure 4 A schematic diagram of the internal layout of a shared memory file in one embodiment is shown. The shared memory file includes a file metadata storage area (meta), a segment storage area, and a content storage area. The file metadata storage area is the header (meta) of the shared memory file, used to store metadata such as verification information (magic), the memory size occupied by the header (meta_size), the number of segments contained in the shared memory file, the memory size occupied by the data objects contained in the shared memory file (data_size), and the second base address of the target shared space (object_base_address). The segment storage area stores at least one segment, with each segment corresponding to a data object. For example, segment 1 corresponds to data object 1, and segment 1 stores the metadata of data object 1: the memory size occupied by the segment (segment_size), the number of target objects contained in the corresponding data object (object_count), and the offset of the storage address of the initial object in the target shared space relative to the second base address (first_object_offset_from_meta), etc. It is easy to understand that since the shared memory file is obtained through mapping the target shared memory, it is a mirror image of the target shared memory. Therefore, the target shared memory also includes the file metadata storage area (meta), fragment storage area, and content storage area as described above, and the data arrangement in each storage area is consistent with the shared memory file. For ease of description, the metadata stored in the file metadata storage area will be referred to as file metadata, and the metadata stored in the fragment storage area will be referred to as fragment metadata.

[0146] Specifically, when the target index information is obtained, the data access process determines the target fragment corresponding to the object to be accessed based on the target index information, and determines the initial position information of the initial object in the object to be accessed based on the fragment metadata in the target fragment. Here, the initial object refers to the target object in the first order of the object to be accessed, for example... Figure 4 In the context of obj1, the initial location information indicates the offset of the initial object's storage address in the target shared space relative to the second starting address.

[0147] In one embodiment, the data access process can determine the first address of the target address space containing the shared memory file based on the file storage path in the target index information and the correspondence between the file storage path and the first address. It then reads the file metadata based on the first address and determines the second address of the target shared space based on the file metadata. Further, the data access process determines the storage address of the target segment in the target shared space based on the target segment identifier and the second address in the target index information. It adjusts the storage address of the target segment in the target shared space based on the first and second addresses to obtain the segment location information of the target segment. For example, it subtracts the first address from the second address and adds the storage location of the target segment in the target shared space to obtain the segment location information of the target segment that matches the target address space. Thus, the data access process reads the segment content of the target segment by calling the target address space, i.e., reads the segment metadata, and determines the initial location information of the initial object in the target shared space based on the segment metadata.

[0148] In one embodiment, the data accessor can access the object to be accessed through the following code: object JVM_Cache_Read(const char* mmap_file_path, int index). Here, mmap_file_path is the file storage path of the shared memory file; index is the identifier of the target fragment.

[0149] In one embodiment, when a data object is stored in a target shared space, the data storage provider can record the target fragment identifier corresponding to the data object, so that when the data object needs to be accessed, target index information can be generated based on the target fragment identifier of the target fragment.

[0150] Step S206: Adjust the initial location information based on the first starting address to obtain object location information that matches the target address space.

[0151] Step S208: Based on the object location information, obtain the data content corresponding to the initial object by calling the target address space.

[0152] Specifically, the data access process adjusts the initial location information based on the first address. For example, the data access process superimposes the first address with the initial location information to obtain the object location information of the initial object that matches the target address space. Then, by accessing memory through the target address space, the process reads the data content of the initial object from the target physical memory corresponding to the target address space based on the object location information of the initial object. Here, the object location information refers to the storage address that matches the target address space and points to the corresponding data object. It is easy to understand that when the data content of the initial object does not exist in the target physical memory, the computer kernel will automatically load the data content of the initial object from the shared memory file into the corresponding location in the target physical memory.

[0153] Because the data objects stored in the shared memory file can be directly mapped to the data access process, the data objects can be read directly from the disk into the data access process. Accessing data objects this way avoids the normal Linux I / O stack. Normally, accessing the data object via the Linux I / O stack requires at least two copies of the data object in the shared memory file before the data content can be read. Therefore, this solution significantly reduces the number of data copies, thereby improving data reading efficiency.

[0154] Step S210: Poll each target object in the object to be accessed in turn, except for the initial object, to obtain the data content corresponding to each target object; wherein, for the current target object polled each time, the object position information of the current target object polled in the previous poll is determined according to the object position information of the preceding target object polled in the previous poll, and the data content corresponding to the current target object is obtained by calling the target address space according to the object position information of the current target object.

[0155] Specifically, after reading the data content of the initial object, the data access process sequentially determines the object location information of the corresponding unaccessed target object based on the object location information of the accessed target object, and obtains the data content of the unaccessed target object based on the object location information of the unaccessed target object.

[0156] To facilitate understanding, the following explanation uses the currently polled target object as an example. When it is necessary to read the currently polled target object, the data access process determines the object location information of the previously polled predecessor target object and the default memory space size occupied by the target object. Based on the object location information of the predecessor target object and the default space size, it determines the object location information of the current target object and retrieves the data content corresponding to the current target object by calling the target address space. For example, the default memory space size occupied by each target object can be preset to 2KB. Thus, the data access process adds 2KB to the object location information of the previously polled predecessor target object to obtain the object location information of the currently polled target object.

[0157] In one embodiment, for each polled target object, the object location information of the current target object is determined based on the object location information of the previous polled target object. This includes: determining the object location information of the previous polled target object and the memory space occupied by the previous target object; and determining the object location information of the current polled target object based on the object location information of the previous target object and the memory space occupied by the previous target object.

[0158] Specifically, for the preceding target object polled in the previous polling, when the object location information of the preceding target object is obtained, the data access process can store this object location information through a pointer, so that the data content of the preceding target object can be read based on the pointer, and the space size occupied by the preceding target object can be determined based on the read data content.

[0159] For each polled target object, the data access process determines the size of the memory space occupied by the previous target object and reads the object location information from the pointer corresponding to the previous target object. Based on the size of the memory space occupied by the previous target object and the object location information of the previous target object, the process determines the object location information of the current target object polled in this round. For example, the object location information of the previous target object is superimposed with the size of the memory space occupied by the previous target object to obtain the object location information of the current target object polled in this round.

[0160] In this embodiment, by determining the object location information of the previous polled target object and the memory space occupied by the previous target object, the object location information of the current polled target object can be accurately determined based on the object location information of the previous target object and the memory space occupied by the previous target object. Thus, the object data of the current object can be read based on the accurately determined object location information of the current target object.

[0161] In the aforementioned data access method, by obtaining the data access instruction, the initial location information of the initial object can be accurately determined based on the target index information in the data access instruction. By determining the initial location information, it can be adjusted based on the first address of the target address space to obtain the object location information matching the target address space. By determining the object location information, the data content of the initial object can be obtained based on the object location information, and each target object (excluding the initial object) in the objects to be accessed is polled sequentially to obtain the data content corresponding to each target object. Since the shared memory file is directly loaded into the target address space during the data access process, the corresponding data content can be obtained by calling the target address space without needing to serialize or deserialize any objects to be accessed. Compared to the traditional method that requires serialization and deserialization of the objects to be accessed before obtaining the corresponding data content, this method saves computer resources.

[0162] Furthermore, since a large number of temporary objects are generated during the serialization and deserialization of data objects, which consume a lot of memory resources, this method can effectively reduce the memory space occupied by a large number of temporary objects, thereby improving memory utilization.

[0163] In one embodiment, the target index information includes the file storage path of the shared memory file and the target fragment identifier. Based on the target index information, determining the first starting address of the target address space containing the shared memory file and the initial position information of the initial object in the target shared space includes: determining the first starting address of the target address space containing the shared memory file based on the file storage path, and determining the file metadata corresponding to the shared memory file based on the first starting address; determining the target fragment corresponding to the object to be accessed based on the file metadata and the target fragment identifier; and determining the initial position information of the initial object in the target shared space based on the fragment content of the target fragment.

[0164] Specifically, the computer device stores a mapping between file storage paths and first starting addresses. When the target index information is obtained, the data access process can determine the first starting address of the target address space containing the shared memory file based on this mapping and the file storage path in the target index information. Further, the data access process reads the file metadata corresponding to the shared memory file based on the first starting address, and determines the target segment corresponding to the object to be accessed based on the file metadata and the target segment identifier in the target index information. It then reads the segment content and determines the initial position information of the initial object in the target shared space based on the segment content. For example, when the segment content is obtained, the data access process can determine the initial position information of the initial object in the target shared space based on `first_object_offset_from_meta` in the segment content.

[0165] In this embodiment, since the target fragment contains the initial position information of the initial object in the object to be accessed, the corresponding target fragment can be determined based on the target index information, thereby determining the initial position information based on the target fragment.

[0166] In one embodiment, the file metadata includes the size of the memory space occupied by the header in the shared memory file; determining the target segment corresponding to the object to be accessed based on the file metadata and the target segment identifier includes: determining the first offset position information of the target segment corresponding to the object to be accessed in the target shared space based on the target segment identifier and the space size; determining the segment position information corresponding to the target segment and matching the target address space based on the first starting address and the first offset position information; and determining the segment content of the target segment based on the segment position information of the target segment.

[0167] Specifically, the data access process determines the first offset position information of the target segment corresponding to the object to be accessed in the target shared space based on the memory space occupied by the header (meta_size) and the target segment identifier in the file metadata. The first offset position information is the offset of the target segment's storage address in the target shared space relative to the second starting address. For example, when the memory space occupied by the header (meta_size), the target segment identifier (index), and the memory space occupied by each segment are all determined to be 4KB, the first offset position information is: meta_size + (index - 1) * 4KB.

[0168] Furthermore, the data access process determines the segment location information corresponding to the target segment and matching the target address space based on the first starting address and the first offset position information, and reads the segment content of the target segment by calling the target address space according to the segment location information. For example, the data access process superimposes the first starting address and the first offset position information to obtain the segment location information, and reads the segment content of the target segment according to the segment location information.

[0169] In this embodiment, by determining the first starting address and the first offset position information, the segment position information can be determined based on the first starting address and the first offset position information, thereby obtaining the segment content of the target segment based on the segment position information.

[0170] In one embodiment, a data access instruction is triggered by a data access process, and based on object location information, the data content corresponding to the initial object is obtained by calling the target address space. This includes: responding to the data access instruction, based on the object location information corresponding to the initial object, calling the target address space through the data access process, reading the data content of the initial object from the target physical memory corresponding to the target address space, and feeding it back; wherein the data content in the target physical memory is loaded from a shared memory file and stored by the computer kernel.

[0171] Specifically, the data access process in the background of the computer device responds to data access commands and, based on the object location information of the initial object, calls the target address space, reads the object content of the initial object from the target physical memory corresponding to the target address space, and then returns it. It's easy to understand that the data content in the target physical memory space is loaded and stored by the computer kernel from a shared memory file.

[0172] In one embodiment, such as Figure 5 As shown, when a data access command is received, the background process determines whether the data object to be accessed is stored in the target physical memory. If the data object is stored, the background process reads it directly from the target physical memory. If the data object is not stored, the background process checks whether the memory space of the target physical memory has reached a preset value, i.e., whether the upper limit has been reached. If the preset value has been reached, the background process deletes some data objects from the target physical memory, allowing the computer kernel to read the data to be accessed from the shared memory file and store the data in the target physical memory. Figure 5 A schematic diagram of the data access process in one embodiment is shown.

[0173] In the above embodiments, since the data object is stored in a shared memory file and the shared memory file is loaded into the target address space, and the data content is read through the target address space, compared with the traditional method of storing data in the heap and reading the corresponding data from the heap through an IO stream (InputStream-OutputStream), this method can also reduce the memory copy overhead during IO operations.

[0174] In one embodiment, after reading the data content of the initial object from the target physical memory corresponding to the target address space and feeding it back, the above data access method further includes: deleting the object to be accessed from the target physical memory when the object to be accessed in the target physical memory corresponding to the target address space meets the deletion conditions.

[0175] Specifically, when the computer device's backend determines that an object to be accessed in the target physical memory has not been accessed within a preset time period—for example, if it is determined that the object has not been accessed by the data access process for a week—the backend can determine that the object meets the deletion criteria and delete it from the target physical memory. It is easily understood that the object can be deleted through the data access process or through other processes, such as a deletion process. This embodiment does not limit this. It is also easily understood that the computer device only deletes the object to be accessed and does not delete the corresponding target segment from the target physical memory.

[0176] In one embodiment, when the background determines that the shared memory file has not been accessed within a preset time period, the background can delete all the data content corresponding to the shared memory file in the target physical memory.

[0177] In one embodiment, the data accessor can also delete the file object to be sent using the following code: JVM_Cache_Madvise(const char* mmap_file_path, int offset). Here, offset is the offset of the storage address of the accessed object in the target shared space relative to the second starting address.

[0178] In one embodiment, the data access party can also delete all data content corresponding to the shared memory file from the target physical memory using the following code: `int JVM_Cache_Cleanup(const void* address, const char* mmap_file_path)`. Here, `address` is the first address; `mmap_file_path` is the file storage path.

[0179] In one embodiment, the data access party can also unload the shared memory file from the target address space using the following code: `int JVM_Cache_Cleanup(const void* address, const char* mmap_file_path)`. `address` is the first address; `mmap_file_path` is the file storage path.

[0180] In one embodiment, the background can also monitor the size of the target physical memory in real time. When the size of the target physical memory exceeds a preset value, in order to ensure the normal operation of the computer device, the background can delete some data objects in the target physical memory.

[0181] Existing technologies can store data in the heap and directly rely on the JVM's GC (Garbage Collection) mechanism to clean it up. However, this does not reduce the heap size, so theoretically it will not reduce GC time or reduce GC pressure. This solution modifies the JVM, allowing us to move some data that was originally stored in the heap to off-heap memory. Furthermore, this data bypasses the heap and is directly loaded into the data access process via a shared memory file, thus avoiding GC management. Therefore, this solution can significantly reduce the heap size, thereby reducing GC time.

[0182] In the above embodiments, the utilization rate of the target physical memory can be improved by deleting the corresponding data objects from the target physical memory. Furthermore, since specific data objects can be deleted, compared to the traditional method of storing data in a fixed-size heap and deleting all data in the heap at once, this embodiment can significantly reduce the time spent deleting data, thereby improving the efficiency of data deletion. Moreover, since this application can directly store data objects in a shared memory file and directly read data objects from the shared memory file, compared to the traditional method of serializing and deserializing data objects before they can be read, this embodiment can significantly reduce the large number of temporary objects generated during serialization and deserialization, thus eliminating the time spent deleting temporary objects and further improving the efficiency of data deletion.

[0183] In one embodiment, obtaining the data content corresponding to the current target object by calling the target address space based on the object location information of the current target object includes: obtaining the data content in the current target object by calling the target address space based on the object location information of the current target object; when the data content in the current target object indicates that the current target object references at least one level of reference objects, determining the data content included in each level of reference objects based on the reference location information corresponding to each level of reference objects; and using the data content in the current target object and the data content included in each level of reference objects together as the data content corresponding to the current target object.

[0184] In this context, a referenced object refers to a data object that is directly or indirectly referenced by the target object.

[0185] Specifically, a shared memory file can store the target object and the reference objects at various levels referenced by the target object. The target object and its references can be viewed as an object tree composed of "edges" and "nodes," where the target object is the root node. The reference objects referenced by the target object constitute nodes directly connected to the root node and located at the next level (first level). The reference objects referenced by the reference objects corresponding to the first-level nodes constitute the second-level nodes, and so on, until the final level node. For ease of description, the reference objects constituting the first-level nodes will be referred to as first-level reference objects, the reference objects constituting the second-level nodes as second-level reference objects, and so on, until the final level reference object.

[0186] When the backend retrieves the data content of the current target object by calling the target address space based on the object location information of the current target object, the backend can analyze the data content to determine whether the current target object has referenced objects. If it is determined from the data content that the current target object references a referenced object, the backend determines the reference location information corresponding to each level of referenced objects, and based on the reference location information, determines the data content included in each level of referenced objects. The data content in the current target object and the data content included in each level of referenced objects are combined as the data content corresponding to the current target object. Here, the reference location information refers to the storage address of the referenced object in the target shared space.

[0187] In one embodiment, when writing a data object to be written to the target shared memory, the front end can obtain each reference object referenced by the data object to be written using a breadth-first search method and store each reference object in the target shared memory. Further, the front end determines the storage location of each reference object in the target shared memory and writes the storage location of each reference object in the target shared memory as reference location information into the data object to be written, so that the data object to be written can contain the reference location information of the reference objects. Similarly, the reference object at the current level can also contain the reference location information of the referenced objects at the next lower level.

[0188] In one embodiment, a data object may contain at least one storage object. For example, a data object may be a hashmap storing multiple key-value pairs, each key-value pair being referred to as a storage object. The front end may determine each storage object in the data object to be written, as well as the reference objects referenced by each storage object, and store the storage objects and reference objects in the target shared memory, and write the reference position information of the reference objects into the corresponding storage objects. The front end may also uniformly store the reference position information of all reference objects in the initial object of the data object to be written. This embodiment is not limited thereto.

[0189] In one embodiment, the reference position information of the referenced object can be stored through a member variable of a specific type. For example, the reference position information of the referenced object can be stored through a member variable of type oop (non-primitive). Thus, when it is necessary to determine whether the current target object has a referenced object, the data content of the target object can be traversed through the function oop_iterate to determine whether it contains a member variable of non-primitive data type, and the value in the member variable of non-primitive data type is used as the corresponding reference position information.

[0190] In this embodiment, by acquiring the data content of the current target object, it can be determined whether the current target object has a reference object. If a reference object is determined to exist, the data content of the reference object is read. This allows for a more comprehensive reading of the data content.

[0191] In one embodiment, when the data content in the current target object indicates that the current target object references at least one level of reference objects, the data content included in each level of reference objects is determined based on the reference location information corresponding to each level of reference objects. This includes: when the data content in the current target object indicates that the current target object references at least one level of reference objects, for each level of reference objects, the reference location information of the corresponding level of reference objects is adjusted based on at least one of the first starting address and the second starting address of the target shared space, to obtain the object location information corresponding to the corresponding level of reference objects and matching the target address space; and according to the object location information corresponding to each level of reference objects, the data content corresponding to each level of reference objects is obtained by calling the target address space.

[0192] Specifically, when the data content of the current target object indicates that the current target object references at least one level of referenced objects, that is, when the data content of the current target object stores the storage address of the referenced object in the target shared space, the backend determines the first address and the second address, and adjusts the reference position information of each referenced object according to the first address and the second address to obtain the object position information corresponding to the referenced object at the corresponding level and matching the target address space, and obtains the data content corresponding to each level of referenced object according to the object position information.

[0193] In one embodiment, the backend determines the address difference between the first address and the second address, and determines the object location information of the corresponding referenced object based on the address difference and the reference location information of the referenced object. For example, the backend subtracts the first address from the second address to obtain the address difference, and then adds the address difference to the reference location information of the referenced object to obtain the object location information of the corresponding referenced object.

[0194] In one embodiment, the backend can subtract the reference location information of the referenced object from the second address to obtain the offset of the storage address of the referenced object in the target shared space relative to the second address, and then add the offset of the storage address of the referenced object in the target shared space relative to the second address to the first address to obtain the corresponding object location information.

[0195] In one embodiment, when the class corresponding to the data object is loaded, the JVM creates a klass instance. The klass instance stores all information defined in the class, including variables, methods, parent classes, interfaces, constructors, etc. The member variables of type oop contain a pointer to the klass instance. Through this pointer, the JVM can obtain the class meta-information of the data object at runtime. Since JDK 8 already has CDS (Class Data Sharing) functionality for the Java core class library, CDS uses a memory-mapped file containing the internal representation of the core classes. During the actual runtime of the data access process, this memory-mapped file is directly mapped by the JVM to address 0x800000000. Thus, in different data access processes, the pointer to the klass instance in the data file does not need to be updated; it can be used directly. Subsequent updates only require correcting the reference location information of the referenced object.

[0196] On older versions of the JDK, a class storage area can be added to each fragment storage area to store the instance identifier of the klass instance of the corresponding data object. On the backend, after the shared memory file is mapped into the target address space, the corresponding target fragment can be found based on the target index information, and the instance identifier of the klass instance can be extracted from the class storage area in the target fragment. Based on the instance identifier of the klass instance, the corresponding pointer can be found, and the pointer pointing to itself can be updated to the target address based on address 0x800000000.

[0197] In the above embodiments, the reference location information is adjusted by at least one of the first starting address and the second starting address of the target shared space, so that the adjusted object location information can match the target address space, and then each referenced object can be directly read based on the object location information.

[0198] In one embodiment, when the data content in the current target object indicates that the current target object references at least one level of referenced objects, the data content included in each level of referenced objects is determined based on the reference location information corresponding to each level of referenced objects. This includes: when the data content in the current target object includes the reference location information of the referenced objects referenced by the current target object at the current level, adjusting the reference location information of the referenced objects at the current level based on at least one of a first starting address and a second starting address of the target shared space to obtain object location information corresponding to the referenced objects at the current level that matches the target address space; and adjusting the reference location information of the referenced objects at the current level based on the referenced objects at the current level. The process involves obtaining the object's location information and retrieving the data content corresponding to the referenced object at the current level by calling the target address space. When the data content of the referenced object at the current level includes the reference location information of the referenced object at the next level in the target shared space, the referenced object at the next level is used as the referenced object at the current level in the next round. The process returns at least one of the first address and the second address of the target shared space to adjust the reference location information of the referenced object at the current level. The process continues until the data content of the referenced object at the final level is obtained.

[0199] Specifically, when the data content of the current target object is obtained, the backend can determine the reference location information of the current-level referenced object based on the data content. Then, based on at least one of the first and second address keys, the reference location information of the current-level referenced object is adjusted to obtain the object location information corresponding to the current-level referenced object and matching the target address space. Based on the object location information of the current-level referenced object, the data content of the current-level referenced object is obtained. When it is determined that the current-level referenced object contains the reference location information of the next-level referenced object in the target shared space, the backend adjusts the reference location information of the next-level referenced object based on at least one of the first and second address keys to obtain the object location information of the next-level referenced object. Based on the object location information of the next-level referenced object, the data content of the next-level referenced object is read. This process iterates until the data content of the final-level application object is obtained.

[0200] In this embodiment, by reading the data content of the referenced objects at each level, the data content of the object to be accessed is obtained more comprehensively.

[0201] In one embodiment, the object to be accessed is commodity transaction data collected by the OLAP engine component through a big data platform; the data access method further includes: when the target commodity transaction data corresponding to the object to be accessed is obtained, performing multi-dimensional analysis on the target commodity transaction data to obtain a commodity transaction analysis report.

[0202] Among them, OLAP (Online Analytical Processing) is specifically designed for handling complex data analysis. It can quickly and flexibly perform complex queries on large amounts of data according to the requirements of analysts, and provide the query results to the data users so that they can accurately grasp the business situation of the enterprise, understand the needs of the target audience, and formulate the right solutions.

[0203] Specifically, commodity transaction data can be obtained through OLAP engine components and mapped to a shared memory file using the aforementioned method. When analysis of the commodity transaction data is required, the shared memory file can be loaded into the target address space, and the target commodity transaction data to be accessed can be read using a memory query method. Furthermore, the computer device can obtain a preset data analysis model, analyze the target commodity transaction data through the data analysis model, obtain a commodity transaction analysis report, and provide feedback.

[0204] In this embodiment, by acquiring the transaction data of the target product to be accessed and generating a corresponding transaction analysis report, the data access party can understand the transaction situation from multiple dimensions based on the transaction analysis report. Furthermore, since the target product transaction data is read from a shared memory file, the efficiency of acquiring the target product transaction data can be improved.

[0205] In one embodiment, a data storage method is provided. This embodiment mainly illustrates the application of this method to a computer device, which can be the aforementioned... Figure 1 Terminal 102 or server 104. (See reference...) Figure 6 The data storage method specifically includes the following steps:

[0206] Step S602: Obtain the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes the content storage area and the fragment storage area.

[0207] Step S604: Write the data object to the content storage area in the target shared space.

[0208] The content storage area can store at least one data object, and the fragment storage area can store at least one fragment. There is a one-to-one correspondence between fragments and data objects. A data object can contain at least one storage object. For example, a data object can be a hashmap that stores multiple key-value pairs, and each key-value pair can be called a storage object.

[0209] Specifically, the front-end can pre-allocate a target shared space of a preset size, so that the computer kernel can return the second starting address of the target shared space to the front-end when the target shared space is successfully allocated. Furthermore, the data writer can divide the successfully allocated target shared space into a memory storage area for storing data objects and a fragment storage area for storing fragments. Thus, when the data writer inputs the data object to be written, the front-end can write the data object into the content storage area of ​​the target shared space according to the second starting address.

[0210] In one embodiment, the frontend can request the target shared space using the following code: `void *JVM_Cache_Init(size_t size)`. Here, `size_t size` represents the size of the target shared space, for example, 40GB. It is easy to understand that the target shared space resides off-heap and is not managed by the JVM's (Java Virtual Machine, Java execution engine) garbage collection mechanism.

[0211] Step S606: Generate the original fragment corresponding to the data object and write the original fragment to the fragment storage area of ​​the shared memory space.

[0212] Specifically, when a data object is written to the content storage area, the front end can generate a corresponding original fragment and write the original fragment to the fragment storage area. For example, an original fragment can be generated containing the memory space occupied by the fragment (segment_size), the number of target objects contained in the corresponding data object (object_count), and the offset of the storage address of the initial object in at least one target object in the target shared space relative to the second first address (first_object_offset_from_meta). In the original fragment, segment_size, object_count, and first_object_offset_from_meta are all random values.

[0213] Step S608: Determine the second offset position information of the initial storage object in the target shared space based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space.

[0214] Step S610: Adjust the fragment metadata in the original fragment according to the second offset position information to obtain the target fragment corresponding to the data object; wherein, the fragment metadata in the target fragment is used to indicate the storage location of the initial storage object in the data object in the target shared space.

[0215] Specifically, the front-end determines the second offset position information of the initial storage object in the target shared space based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object within the target shared space. For example, the front-end subtracts the initial storage address of the initial storage object in the target shared space from the second starting address to obtain the second offset position information. Here, the initial object refers to the storage object in the first order of the data objects stored in the target shared space; the initial storage address refers to the storage address of the initial object in the target shared space; and the second offset position information refers to the offset of the initial object's storage address in the target space relative to the second starting address.

[0216] Furthermore, the front-end can adjust the fragment metadata in the original fragment based on the second offset position information to obtain the target fragment corresponding to the data object. For example, the value of first_object_offset_from_meta in the original fragment can be modified to the second offset position information, so that when it is necessary to read the data to be accessed, the back-end can determine the storage location of the initial object in the target shared space based on the fragment metadata in the target fragment.

[0217] In one embodiment, the front end can also adjust the memory space occupied by the segment in the original segment (segment_size) and the number of target objects contained in the corresponding data object (object_count) to adjust the random value to the actual value, so as to obtain the target segment corresponding to the data object.

[0218] The aforementioned data storage method, by writing the data object into the content storage area of ​​the target shared space and storing the generated original fragment in the fragment storage area, can adjust the original fragment based on the second starting address and the initial storage address of the target shared space to obtain the target fragment corresponding to the data object. By obtaining the target fragment, when accessing the data object is needed, the storage location of the initial storage object in the target fragment data object within the target shared space can be determined. Since the data object can be directly stored in the target memory space and read directly through the target fragment, compared to the traditional method which requires serialization and deserialization of the object to be accessed before obtaining the corresponding data content, this method eliminates the need for any serialization and deserialization, thus saving computer resources.

[0219] In one embodiment, writing a data object to a content storage area in a target shared space includes: determining the reference object referenced by each storage object in the data object; and writing the storage object and the reference object to the content storage area.

[0220] Specifically, when the data object to be written is obtained, the frontend can use this data object as the root object and write the root object and all referenced objects referenced by the root object into the content storage area. That is, each storage object in the data object, and each referenced object referenced by each storage object, is written into the content storage area. For example, in the example above, when storing a hashmap containing multiple key-value pairs in the target shared space, the frontend can store not only the key-value pairs in the content storage area, but also the referenced objects referenced by each key-value pair in the content storage area.

[0221] In one embodiment, the front end can write the data object to be written to the target shared space using the following code: int JVM_Cache_write(Object obj), where obj is the data object, such as the hashmap in the example above.

[0222] In the above embodiments, by writing both the stored object and the referenced objects referenced by the stored object into the content storage area, the data content of the object to be accessed read subsequently is more comprehensive.

[0223] In one embodiment, generating an original fragment corresponding to a data object and writing the original fragment to a fragment storage area in a shared memory space includes: generating an original fragment corresponding to a data object; determining the occupied position information of the last-order fragment stored in the fragment storage area in the target shared space; determining the target unoccupied position in the fragment storage area based on the occupied position information, and writing the original fragment to the target unoccupied position.

[0224] Specifically, when generating the original fragment, the front end determines the occupied location information of the last-order fragment stored in the fragment storage area within the target shared space. The occupied location information refers to the storage address of the last-order fragment in the target shared space. Further, based on the occupied location information, the front end determines the target in the fragment storage area as the occupied location and writes the original fragment to the unoccupied location of the target. For example, the front end writes the original fragment after the occupied location, that is, after the last-order fragment.

[0225] In this embodiment, by determining the occupied location information, the original fragments can be written to the target unoccupied location based on the occupied location information, so that each original fragment can be stored in the correct location without overwriting the stored fragments.

[0226] In one embodiment, the target shared space includes a file metadata storage area; the method further includes: when a data object is written to the content storage area in the target shared space, reading the total number of fragments and the total amount of data stored in the file metadata storage area; and increasing the total number of fragments and the total amount of data by values ​​respectively.

[0227] Specifically, the target shared space may also include a file metadata storage area for storing metadata of shared memory files. When requesting the target shared space, the data storage provider can correspondingly allocate the file metadata storage area, allowing the front-end to initialize file metadata and store the initialized file metadata in the file metadata storage area. Further, when writing data objects to the content storage area, the front-end obtains the number of storage objects contained in the data object and adjusts the total amount of data in the file metadata storage area based on the number of storage objects. For example, it adds the total amount of data in the file metadata storage area to the number of storage objects to obtain the adjusted total amount of data. The front-end also adjusts the total number of fragments in the file metadata storage area, for example, by adding 1 to the total number of fragments to obtain the adjusted total number of fragments.

[0228] In this embodiment, by adjusting the total number of segments and the total amount of data in the file metadata, the adjusted total number of segments and the total amount of data can point to the correct true value, so that subsequent processing can be performed based on the true value.

[0229] In one embodiment, the data access method further includes: when the target shared space meets the mapping conditions, mapping the data objects stored in the target shared space to the disk according to the preset file storage path, so as to obtain the shared memory file stored on the disk.

[0230] Specifically, the front end determines whether the target shared space meets the mapping conditions, and when it is determined that the mapping conditions are met, it maps the data objects stored in the target shared space to the computer disk according to the preset file storage path, so as to obtain a shared memory file with the preset file storage path.

[0231] In one embodiment, when the size of the target shared space reaches a preset value, the front end maps the data objects stored in the target shared space to obtain a shared memory file.

[0232] In one embodiment, when no new data objects are written to the target shared space within a preset time period, it can be assumed that the data writer no longer needs to operate on the target shared space. At this time, the front end maps the data objects stored in the target shared space to obtain a shared memory file.

[0233] In the above embodiments, by determining whether the target shared space meets the mapping conditions, the data objects stored in the target shared space can be mapped in a timely manner when the mapping conditions are met, thereby obtaining a shared memory file.

[0234] In one embodiment, such as Figure 7 As shown, a flowchart of a data access method in a specific embodiment is provided:

[0235] S702, obtain the file storage path of the shared memory file, load the corresponding shared memory file into the target address space through the file storage path, and obtain the first address of the target address space; based on the first address of the target address space, obtain the verification information in the shared memory file, and verify the integrity of the shared memory file based on the verification information.

[0236] S704, obtain a data access instruction corresponding to the object to be accessed and carrying target index information, wherein the object to be accessed is at least a part of the data object written to the target shared space, and the object to be accessed includes at least one target object.

[0237] S706, the target index information includes the file storage path of the shared memory file and the target fragment identifier; the shared memory file is obtained by mapping each data object in the target shared space; based on the file storage path, the first address of the target address space loaded with the shared memory file is determined, and the file metadata corresponding to the shared memory file is determined based on the first address.

[0238] S708, the file metadata includes the size of the memory space occupied by the header in the shared memory file; based on the target fragment identifier and the size of the space, the first offset position information of the target fragment corresponding to the object to be accessed in the target shared space is determined.

[0239] S710, based on the first starting address and the first offset position information, determine the segment position information corresponding to the target segment and matching the target address space; based on the segment position information of the target segment, determine the segment content of the target segment.

[0240] S712, based on the content of the target fragment, determine the initial location information of the initial object in the target shared space within the object to be accessed. Adjust the initial location information based on the first starting address to obtain object location information that matches the target address space.

[0241] S714, the data access instruction is triggered by the data access process; in response to the data access instruction, based on the object location information corresponding to the initial object, the target address space is called through the data access process, the data content of the initial object is read from the target physical memory corresponding to the target address space and fed back; wherein, the data content in the target physical memory is loaded from the shared memory file and stored by the computer kernel.

[0242] S716, sequentially poll each target object in the object to be accessed except for the initial object to obtain the data content corresponding to each target object; wherein, determine the object position information of the previous target object polled in the previous poll and the memory space occupied by the previous target object; based on the object position information of the previous target object and the memory space occupied by the previous target object, determine the object position information of the current target object polled in the current poll.

[0243] S718, based on the object location information of the current target object, the data content in the current target object is obtained by calling the target address space; when the data content in the current target object indicates that the current target object references at least one level of reference objects, for each level of reference objects, the reference location information of the reference objects at the corresponding level is adjusted based on at least one of the first starting address and the second starting address of the target shared space, so as to obtain the object location information corresponding to the reference objects at the corresponding level and matching the target address space.

[0244] S720: Based on the object location information corresponding to each level of referenced objects, and by calling the target address space, obtain the data content corresponding to each level of referenced objects; combine the data content in the current target object and the data content included in each level of referenced objects as the data content corresponding to the current target object.

[0245] S722: When the object to be accessed in the target physical memory corresponding to the target address space meets the deletion conditions, the object to be accessed is deleted from the target physical memory.

[0246] In the aforementioned data access method, by obtaining the data access instruction, the initial location information of the initial object can be accurately determined based on the target index information in the data access instruction. By determining the initial location information, it can be adjusted based on the first address of the target address space to obtain the object location information matching the target address space. By determining the object location information, the data content of the initial object can be obtained based on the object location information, and each target object (excluding the initial object) in the objects to be accessed is polled sequentially to obtain the data content corresponding to each target object. Since the shared memory file is directly loaded into the target address space during the data access process, the corresponding data content can be obtained by calling the target address space without needing to serialize or deserialize any objects to be accessed. Compared to the traditional method that requires serialization and deserialization of the objects to be accessed before obtaining the corresponding data content, this method can greatly improve the efficiency of data access.

[0247] Furthermore, since a large number of temporary objects are generated during the serialization and deserialization of data objects, which consume a lot of memory resources, this method can effectively reduce the memory space occupied by a large number of temporary objects, thereby improving memory utilization.

[0248] In one embodiment, such as Figure 8 As shown, a flowchart of a data storage method in a specific embodiment is provided:

[0249] S802, obtain the data object to be written, and determine the target shared space corresponding to the data object; the target shared space includes the content storage area and the fragment storage area.

[0250] S804, determine the reference object referenced by each storage object in the data object; write the storage object and the reference object to the content storage area.

[0251] S806, generate the original fragment corresponding to the data object; determine the occupied position information of the last stored fragment in the fragment storage area in the target shared space; determine the unoccupied position of the target in the fragment storage area based on the occupied position information, and write the original fragment to the unoccupied position of the target.

[0252] S808, based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space, determine the second offset position information of the initial storage object in the target shared space.

[0253] S810, based on the second offset position information, the fragment metadata in the original fragment is adjusted to obtain the target fragment corresponding to the data object; wherein, the fragment metadata in the target fragment is used to indicate the storage location of the initial storage object in the data object in the target shared space.

[0254] S812, when a data object is written to the content storage area in the target shared space, read the total number of fragments and the total amount of data stored in the file metadata storage area; and increment the total number of fragments and the total amount of data respectively.

[0255] S814, when the target shared space meets the mapping conditions, the data objects stored in the target shared space are mapped to the disk according to the preset file storage path, and a shared memory file is obtained stored on the disk.

[0256] For the purpose of facilitating understanding by those skilled in the art, the present application is described in its entirety below:

[0257] The computer device front-end (client side) can store data objects in pre-allocated off-heap memory (target shared memory) through the interface (JVM_Cache_Init). When the data object in the target shared memory meets the loading conditions, it is loaded onto the computer disk, resulting in a shared memory file. When the data object needs to be read, the computer device back-end (server side) can read it through the interface (JVM_Cache_Read). At this point, the data access process checks if the data object exists in the target physical memory. If it exists, it is read directly through memory access. If it does not exist, the computer kernel loads the data object from the target shared file into the target physical memory, allowing data access to retrieve the data object from the target physical memory.

[0258] On the backend, when reading data from the disk, the shared memory file can be called directly, which is more efficient than the traditional method of reading data through the JVM's I / O functions. In addition, the user space and kernel space share the same memory, so the data object does not need to go through the user space cache multiple times, thereby improving the efficiency of data reading.

[0259] During the process of mapping data objects from disk to the target address space in the backend, no deserialization operation is required on the data objects. This makes the data objects passed from the frontend available in the data access process. In this way, the disadvantages of reading data objects through serialization and deserialization are avoided, thereby effectively reducing GC pressure.

[0260] Since the memory occupied by shared memory files is completely outside the JVM Heap and outside the management of JVM GC threads, this technique can effectively reduce heap size and reduce global GC stop-the-world time.

[0261] This application also provides an application scenario in which the above-described data access method is applied. Specifically, the data access method is applied in this scenario as follows:

[0262] The data writer can store a large amount of commodity transaction data generated by the commodity trading platform into a target shared space, and then map the commodity transaction data in the target shared space to obtain a shared memory file. When it is necessary to read the target commodity transaction data, the data reader can input the target index information, so that the computer backend can determine the object location information corresponding to the object to be accessed based on the target index information, and then read the target commodity transaction data based on the object location information. Once the target commodity transaction data is obtained, the computer equipment can perform multi-dimensional analysis and processing on the target commodity transaction data to obtain a commodity transaction analysis report.

[0263] This application also provides another application scenario where the above-described data access method is applied. Specifically, the data access method is applied in this scenario as follows:

[0264] The data writer can input page access data into the target shared space in the manner described above. The computer backend can then read the target page access data from the target address space that has loaded the shared memory file, and perform subsequent processing based on the read target page access data, such as generating a page access report.

[0265] It is understood that the above application scenarios are only used to illustrate the solution of this application. The data access method and / or data storage method in this application can also be applied to other scenarios without limitation. For example, storing and / or accessing user data collected by an application to build user profiles. Another example is accessing and / or storing media data (such as audio and video data) to perform subsequent processing based on the accessed media data, such as media data classification or recommendation.

[0266] It should be understood that, although Figure 2 , 6 The steps in flowchart -8 are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps, and they can be executed in other orders. Furthermore, Figure 2 , 6At least some of the steps in -8 may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0267] In one embodiment, such as Figure 9 As shown, a data access device 900 is provided. This device can be a software module, a hardware module, or a combination of both, integrated into a computer device. Specifically, the device includes: an initial position determination module 902, an initial object acquisition module 904, and a target object acquisition module 906, wherein:

[0268] The initial position determination module 902 is used to obtain a data access instruction corresponding to the object to be accessed and carrying target index information. The object to be accessed is at least a part of the data objects written to the target shared space, and the object to be accessed includes at least one target object. Based on the target index information, the module determines the first starting address of the target address space loaded with the shared memory file and the initial position information of the initial object in the target shared space. The shared memory file is obtained by mapping each data object in the target shared space.

[0269] The initial object acquisition module 904 is used to adjust the initial position information based on the first starting address to obtain the object position information that matches the target address space; and to obtain the data content corresponding to the initial object by calling the target address space based on the object position information.

[0270] The target object acquisition module 906 is used to sequentially poll each target object in the object to be accessed, excluding the initial object, to obtain the data content corresponding to each target object. Specifically, for each polled target object, the object position information of the current target object is determined based on the object position information of the previous polled target object, and the data content corresponding to the current target object is obtained by calling the target address space based on the object position information of the current target object.

[0271] In one embodiment, the target index information includes the file storage path of the shared memory file and the target fragment identifier; the initial position determination module 902 further includes a target fragment determination module 9021, which is used to determine the first starting address of the target address space containing the shared memory file according to the file storage path, and determine the file metadata corresponding to the shared memory file based on the first starting address; determine the target fragment corresponding to the object to be accessed according to the file metadata and the target fragment identifier; and determine the initial position information of the initial object in the target shared space according to the fragment content of the target fragment.

[0272] In one embodiment, the file metadata includes the size of the memory space occupied by the header in the shared memory file; the target fragment determination module 9021 is further configured to determine the first offset position information of the target fragment corresponding to the object to be accessed in the target shared space according to the target fragment identifier and the space size; determine the fragment position information corresponding to the target fragment and matching the target address space according to the first starting address and the first offset position information; and determine the fragment content of the target fragment according to the fragment position information of the target fragment.

[0273] In one embodiment, the data access instruction is triggered by the data access process. The initial object acquisition module 904 is also used to respond to the data access instruction, according to the object location information corresponding to the initial object, and by calling the target address space through the data access process, to read the data content of the initial object from the target physical memory corresponding to the target address space and then feed it back. The data content in the target physical memory is loaded from the shared memory file and stored by the computer kernel.

[0274] In one embodiment, the data access device 900 further includes a deletion module 908, which deletes the object to be accessed from the target physical memory when the object in the target physical memory corresponding to the target address space meets the deletion conditions.

[0275] In one embodiment, the target object acquisition module 906 is further configured to determine the object location information of the previous target object polled in the previous poll and the memory space occupied by the previous target object; and based on the object location information of the previous target object and the memory space occupied by the previous target object, determine the object location information of the current target object polled in the current poll.

[0276] In one embodiment, the target object acquisition module 906 includes a reference object acquisition module 9061, which is used to acquire the data content in the current target object by calling the target address space based on the object location information of the current target object; when the data content in the current target object indicates that the current target object references at least one level of reference objects, the data content included in each level of reference objects is determined based on the reference location information corresponding to each level of reference objects; and the data content in the current target object and the data content included in each level of reference objects are used together as the data content corresponding to the current target object.

[0277] In one embodiment, the reference object acquisition module 9061 is further configured to, when the data content in the current target object indicates that the current target object references at least one level of reference objects, adjust the reference position information of the reference objects at the corresponding level based on at least one of the first starting address and the second starting address of the target shared space for each level of reference objects, so as to obtain the object position information corresponding to the reference objects at the corresponding level and matching the target address space; and obtain the data content corresponding to each level of reference objects by calling the target address space according to the object position information corresponding to each level of reference objects.

[0278] In one embodiment, the reference object acquisition module 9061 is further configured to: when the data content of the current target object includes the reference location information of the reference object referenced by the current target object at the current level, adjust the reference location information of the reference object at the current level based on at least one of the first address and the second address of the target shared space to obtain the object location information corresponding to the reference object at the current level and matching the target address space; obtain the data content corresponding to the reference object at the current level by calling the target address space based on the object location information of the reference object at the current level; when the data content of the current level reference object includes the reference location information of the reference object at the next level in the target shared space, take the reference object at the next level as the reference object at the current level in the next round, and continue to execute the step of adjusting the reference location information of the reference object at the current level based on at least one of the first address and the second address of the target shared space to obtain the object location information corresponding to the reference object at the current level and matching the target address space, until the data content of the reference object at the final level is obtained.

[0279] In one embodiment, the data access device 900 is further configured to obtain the file storage path of the shared memory file, load the corresponding shared memory file into the target address space through the file storage path to obtain the first address of the target address space, obtain the verification information in the shared memory file based on the first address of the target address space, and verify the integrity of the shared memory file based on the verification information.

[0280] In one embodiment, the object to be accessed is commodity transaction data collected by the OLAP engine component through a big data platform; the data access device 900 is also used to perform multi-dimensional analysis on the target commodity transaction data when the target commodity transaction data corresponding to the object to be accessed is obtained, and to obtain a commodity transaction analysis report.

[0281] In one embodiment, such as Figure 10 As shown, a data storage device 1000 is provided. This device can be a software module, a hardware module, or a combination of both, integrated into a computer device. Specifically, the device includes: a data writing module 1002, a fragment writing module 1004, and a fragment adjustment module 1006, wherein:

[0282] The data writing module 1002 is used to obtain the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes a content storage area and a fragment storage area; and writes the data object into the content storage area of ​​the target shared space.

[0283] The fragment writing module 1004 is used to generate the original fragment corresponding to the data object and write the original fragment to the fragment storage area of ​​the shared memory space.

[0284] The fragment adjustment module 1006 is used to determine the second offset position information of the initial storage object in the target shared space based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space; and to adjust the fragment meta-information in the original fragment based on the second offset position information to obtain the target fragment corresponding to the data object; wherein, the fragment meta-information in the target fragment is used to indicate the storage position of the initial storage object in the data object in the target shared space.

[0285] In one embodiment, the data writing module 1002 is further configured to determine the reference object referenced by each storage object in the data object; and write the storage object and the reference object into the content storage area.

[0286] In one embodiment, the fragment writing module 1004 is further configured to generate an original fragment corresponding to the data object; determine the occupied position information of the last-order fragment stored in the fragment storage area in the target shared space; determine the target unoccupied position in the fragment storage area based on the occupied position information, and write the original fragment to the target unoccupied position.

[0287] In one embodiment, the target shared space includes a file metadata storage area; the data storage device 1000 is further configured to, when a data object is written to the content storage area in the target shared space, read the total number of fragments and the total amount of data stored in the file metadata storage area; and increment the total number of fragments and the total amount of data respectively.

[0288] In one embodiment, the data storage device 1000 is further configured to, when the target shared space meets the mapping conditions, map the data objects stored in the target shared space to the disk according to a preset file storage path, thereby obtaining a shared memory file stored on the disk.

[0289] For specific limitations regarding data access devices and data storage devices, please refer to the limitations regarding data access methods and data storage methods above, which will not be repeated here. Each module in the aforementioned data access device and data storage device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0290] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores and accesses data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data access method.

[0291] Those skilled in the art will understand that Figure 11The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0292] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0293] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0294] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0295] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0296] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0297] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data access method, characterized in that, The method includes: Obtain a data access instruction corresponding to the object to be accessed and carrying target index information, wherein the object to be accessed is at least a part of the data object written to the target shared space, and the object to be accessed includes at least one target object; Based on the target index information, the first starting address of the target address space containing the shared memory file and the initial position information of the initial object in the target shared space are determined; the shared memory file is obtained by mapping each data object in the target shared space; The initial location information is adjusted based on the first address to obtain object location information that matches the target address space; Based on the object location information, and by calling the target address space, the data content corresponding to the initial object is obtained; The process involves sequentially polling each target object in the object to be accessed, excluding the initial object, to obtain the data content corresponding to each target object. Specifically, for each target object polled to the current target object, the object position information of the current target object is determined based on the object position information of the preceding target object polled in the previous poll. Then, based on the object position information of the current target object, the data content corresponding to the current target object is obtained by calling the target address space. The step of obtaining the data content corresponding to the current target object by calling the target address space based on the object location information of the current target object includes: Based on the object location information of the current target object, the data content in the current target object is obtained by calling the target address space; When the data content in the current target object includes the reference location information of the referenced object referenced by the current target object at the current level, the reference location information of the referenced object at the current level is adjusted based on at least one of the first starting address and the second starting address of the target shared space, so as to obtain the object location information corresponding to the referenced object at the current level and matching the target address space. Based on the object location information of the referenced object at the current level, and by calling the target address space, the data content corresponding to the referenced object at the current level is obtained; When the data content of the current level reference object includes the reference position information of the next level reference object in the target shared space, the next level reference object is used as the current level reference object in the next round. The step of adjusting the reference position information of the current level reference object based on at least one of the first starting address and the second starting address of the target shared space to obtain the object position information corresponding to the current level reference object and matching the target address space continues to be executed until the data content of the final level reference object is obtained. The data content in the current target object and the data content included in each layer of referenced objects are combined as the data content corresponding to the current target object.

2. The method according to claim 1, characterized in that, The target index information includes the file storage path of the shared memory file and the target fragment identifier; The step of determining the first starting address of the target address space containing the shared memory file and the initial position information of the initial object in the target shared space based on the target index information includes: Based on the file storage path, determine the first starting address of the target address space containing the shared memory file, and determine the file metadata corresponding to the shared memory file based on the first starting address; Based on the file metadata and the target fragment identifier, determine the target fragment corresponding to the object to be accessed; Based on the content of the target fragment, determine the initial location information of the initial object in the target shared space.

3. The method according to claim 2, characterized in that, The file metadata includes the memory space occupied by the header of the shared memory file; determining the target segment corresponding to the object to be accessed based on the file metadata and the target segment identifier includes: Based on the target fragment identifier and the space size, determine the first offset position information of the target fragment corresponding to the object to be accessed in the target shared space; Based on the first starting address and the first offset position information, determine the segment position information that corresponds to the target segment and matches the target address space; Based on the segment position information of the target segment, the segment content of the target segment is determined.

4. The method according to claim 1, characterized in that, The data access instruction is triggered by the data access process; the step of obtaining the data content corresponding to the initial object based on the object location information and by calling the target address space includes: In response to the data access instruction, based on the object location information corresponding to the initial object, the data access process calls the target address space to read the data content of the initial object from the target physical memory corresponding to the target address space and then feeds it back; wherein, the data content in the target physical memory is loaded and stored from the shared memory file by the computer kernel.

5. The method according to claim 1, characterized in that, For each polled target object, the object position information of the current target object is determined based on the object position information of the preceding target object polled in the previous poll, including: Determine the object location information of the previous polled target object and the memory space occupied by the previous target object; Based on the object location information of the preceding target object and the memory space occupied by the preceding target object, the object location information of the current target object in the current polling is determined.

6. The method according to any one of claims 1 to 5, characterized in that, The object to be accessed is commodity transaction data collected by the OLAP engine component through a big data platform; the method further includes: When the target product transaction data corresponding to the object to be accessed is obtained, the target product transaction data is analyzed in multiple dimensions to obtain a product transaction analysis report.

7. A data storage method, characterized in that, The method includes: Obtain the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes a content storage area and a fragment storage area; Write the data object into the content storage area of ​​the target shared space; Generate an original fragment corresponding to the data object, and write the original fragment into the fragment storage area of ​​the target shared space; Based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space, determine the second offset position information of the initial storage object in the target shared space; Based on the second offset position information, the fragment metadata in the original fragment is adjusted to obtain a target fragment corresponding to the data object; wherein, the fragment metadata in the target fragment is used to indicate the storage location of the initial storage object in the data object in the target shared space; The method further includes: When a data access instruction corresponding to the object to be accessed and carrying target index information is obtained, the data content in the object to be accessed is obtained by the method described in any one of claims 1 to 6; the object to be accessed is at least a part of the data object written to the target shared space.

8. The method according to claim 7, characterized in that, The step of writing the data object into the content storage area of ​​the target shared space includes: Determine the reference object referenced by each storage object in the data object; Write the stored object and the referenced object into the content storage area.

9. The method according to claim 7, characterized in that, The step of generating the original fragment corresponding to the data object and writing the original fragment into the fragment storage area of ​​the target shared space includes: Generate the original fragment corresponding to the data object; Determine the occupied position information of the last-order fragment stored in the fragment storage area within the target shared space; Based on the occupied location information, determine the target unoccupied location in the fragment storage area, and write the original fragment to the target unoccupied location.

10. The method according to any one of claims 7 to 9, characterized in that, The target shared space includes a file metadata storage area; the method further includes: When the data object is written to the content storage area in the target shared space, the total number of fragments and the total amount of data stored in the file metadata storage area are read. The total number of segments and the total amount of data are increased numerically, respectively.

11. A data access device, characterized in that, The device includes: An initial position determination module is used to obtain a data access instruction corresponding to the object to be accessed and carrying target index information. The object to be accessed is at least a part of the data objects written to the target shared space, and the object to be accessed includes at least one target object. Based on the target index information, the module determines the first starting address of the target address space loaded with the shared memory file and the initial position information of the initial object in the target shared space. The shared memory file is obtained by mapping each data object in the target shared space. The initial object acquisition module is used to adjust the initial position information based on the first starting address to obtain object position information that matches the target address space; and to obtain the data content corresponding to the initial object based on the object position information and by calling the target address space. The target object acquisition module is used to sequentially poll each target object in the object to be accessed, excluding the initial object, to obtain the data content corresponding to each target object. Specifically, for each polled target object, the object position information of the current target object is determined based on the object position information of the preceding target object polled in the previous poll, and the data content corresponding to the current target object is obtained by calling the target address space based on the object position information of the current target object. The target object acquisition module is further configured to: acquire data content in the current target object by calling the target address space based on the object location information of the current target object; when the data content in the current target object includes reference location information of a referenced object referenced by the current target object at the current level, adjust the reference location information of the referenced object at the current level based on at least one of the first address and the second address of the target shared space, to obtain object location information corresponding to the referenced object at the current level that matches the target address space; and acquire data content corresponding to the referenced object at the current level by calling the target address space based on the object location information of the referenced object at the current level. When the data content of the current level reference object includes the reference location information of the next level reference object in the target shared space, the next level reference object is used as the current level reference object in the next round. The step of adjusting the reference location information of the current level reference object based on at least one of the first starting address and the second starting address of the target shared space to obtain the object location information corresponding to the current level reference object and matching the target address space continues to be executed until the data content of the final level reference object is obtained. The data content of the current target object and the data content included in each level reference object are used together as the data content corresponding to the current target object.

12. The apparatus according to claim 11, characterized in that, The target index information includes the file storage path of the shared memory file and the target fragment identifier; the initial position determination module is further configured to determine the first starting address of the target address space containing the shared memory file based on the file storage path, and determine the file metadata corresponding to the shared memory file based on the first starting address; determine the target fragment corresponding to the object to be accessed based on the file metadata and the target fragment identifier; and determine the initial position information of the initial object in the target shared space based on the fragment content of the target fragment.

13. The apparatus according to claim 12, characterized in that, The file metadata includes the memory space occupied by the header of the shared memory file; the initial position determination module is further configured to determine the first offset position information of the target segment corresponding to the object to be accessed in the target shared space according to the target segment identifier and the space size; determine the segment position information corresponding to the target segment and matching the target address space according to the first starting address and the first offset position information; and determine the segment content of the target segment according to the segment position information of the target segment.

14. The apparatus according to claim 11, characterized in that, The data access instruction is triggered by the data access process; the initial object acquisition module is also used to respond to the data access instruction, according to the object location information corresponding to the initial object, and through the data access process to call the target address space, read the data content of the initial object from the target physical memory corresponding to the target address space and feed it back; wherein, the data content in the target physical memory is loaded and stored from the shared memory file by the computer kernel.

15. The apparatus according to claim 11, characterized in that, The target object acquisition module is further configured to determine the object location information of the previous target object polled in the previous poll and the memory space occupied by the previous target object; and based on the object location information of the previous target object and the memory space occupied by the previous target object, determine the object location information of the current target object polled in the current poll.

16. The apparatus according to claim 11, characterized in that, The object to be accessed is commodity transaction data collected by the OLAP engine component through the big data platform; the data access device is also used to perform multi-dimensional analysis on the target commodity transaction data when the target commodity transaction data corresponding to the object to be accessed is obtained, and to obtain a commodity transaction analysis report.

17. A data reading device, characterized in that, The device includes: The data writing module is used to acquire the data object to be written and determine the target shared space corresponding to the data object; the target shared space includes a content storage area and a fragment storage area; and write the data object into the content storage area of ​​the target shared space. The fragment writing module is used to generate an original fragment corresponding to the data object and write the original fragment into the fragment storage area of ​​the target shared space; The fragment adjustment module is used to determine the second offset position information of the initial storage object in the target shared space based on the second starting address of the target shared space and the initial storage address of the initial storage object in the data object in the target shared space; and to adjust the fragment meta-information in the original fragment based on the second offset position information to obtain a target fragment corresponding to the data object; wherein, the fragment meta-information in the target fragment is used to indicate the storage position of the initial storage object in the data object in the target shared space; The object to be accessed is at least a portion of the data object written to the target shared space; when a data access instruction corresponding to the object to be accessed and carrying target index information is obtained, the data content in the object to be accessed is obtained by the apparatus as described in any one of claims 11 to 16.

18. The apparatus according to claim 17, characterized in that, The data writing module is further configured to determine the reference object referenced by each storage object in the data object; and write the storage object and the reference object into the content storage area.

19. The apparatus according to claim 17, characterized in that, The fragment writing module is also used to generate an original fragment corresponding to the data object; determine the occupied position information of the fragments stored in the fragment storage area that are in the last order in the target shared space; determine the target unoccupied position in the fragment storage area based on the occupied position information, and write the original fragment to the target unoccupied position.

20. The apparatus according to claim 17, characterized in that, The target shared space includes a file metadata storage area; the data reading device is also used to read the total number of fragments and the total amount of data stored in the file metadata storage area when the data object is written to the content storage area in the target shared space. The total number of segments and the total amount of data are increased numerically, respectively.

21. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

22. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images