Data processing method and device, computer equipment and computer readable storage medium

Abstract

This application discloses a data processing method, apparatus, computer device, and computer-readable storage medium. By performing memory mapping processing on a first file based on a target process, a mapped region of the first file in the virtual address space of the target process is obtained. The first file contains first target data and storage space-related data associated with the first target data. In response to a first data access instruction, the storage space-related data in the first file is obtained based on the mapped region. A first data object is generated based on the storage space-related data. Based on the first data object and the storage space-related data encapsulated in the first data object, a portion of the first target data in the first file is obtained. Responding to the first data access instruction based on the portion of the first target data allows the first data object generated in the target process to contain a small amount of storage space-related data, rather than all the data in the first file, thus reducing data usage in memory.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data processing method, apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] The existing process loading of data files involves different processes loading data files separately, each process loading its own private data file resource, which is stored in the system's physical memory. As the number of processes increases, the physical memory occupied by the data file resources grows linearly. This is particularly problematic for devices, especially servers using a multi-process distributed architecture, where numerous processes result in data file resources consuming a significant amount of memory. Summary of the Invention

[0003] This application provides a data processing method, apparatus, computer device, and computer-readable storage medium, which can enable the first data object generated by the target process to contain storage space-related data with a small amount of data, rather than containing all the data of the first file, thereby reducing the data's memory usage.

[0004] This application provides a data processing method, including:

[0005] Based on the target process, the first file is memory mapped to obtain the mapped region of the first file in the virtual address space of the target process, wherein the first file contains first target data and storage space associated data of the first target data;

[0006] In response to a first data access instruction, the storage space associated data in the first file is obtained based on the mapped region;

[0007] A first data object is generated based on the storage space associated data;

[0008] Based on the first data object and the storage space associated data encapsulated by the first data object, a portion of the first target data in the first file is obtained;

[0009] The system responds to the first data access instruction based on the aforementioned portion of the first target data.

[0010] Accordingly, this application also provides a data processing apparatus, including:

[0011] The mapping unit is used to perform memory mapping processing on the first file based on the target process to obtain the mapping region of the first file in the virtual address space of the target process, wherein the first file contains first target data and storage space associated data of the first target data;

[0012] The first acquisition unit is configured to, in response to a first data access instruction, acquire the storage space associated data in the first file based on the mapping region;

[0013] The generation unit is used to generate a first data object based on the storage space associated data;

[0014] The second acquisition unit is used to acquire a portion of the first target data in the first file based on the first data object and the storage space associated data encapsulated in the first data object;

[0015] A response unit is used to respond to the first data access instruction based on the portion of the first target data.

[0016] Accordingly, this application also provides a computer device including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute any of the data processing methods provided in this application.

[0017] Accordingly, embodiments of this application also provide a computer-readable storage medium for storing a computer program, which is loaded by a processor to execute any of the data processing methods provided in embodiments of this application.

[0018] This application embodiment obtains a mapped region of the first file in the virtual address space of the target process by performing memory mapping processing on the first file based on the target process. The first file contains first target data and storage space associated data of the first target data. In response to a first data access instruction, the storage space associated data in the first file is obtained based on the mapped region. A first data object is generated based on the storage space associated data. Based on the first data object and the storage space associated data encapsulated in the first data object, a portion of the first target data in the first file is obtained. The first data access instruction is responded to based on the portion of the first target data. This can achieve the goal of the first data object generated in the target process containing a small amount of storage space associated data, rather than containing all the data of the first file, thereby reducing the data's memory occupation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the data processing method provided in the embodiments of this application;

[0021] Figure 2 This is a schematic diagram showing the corresponding data objects and structures provided in the embodiments of this application;

[0022] Figure 3 This is a schematic diagram of serialization processing provided in an embodiment of this application;

[0023] Figure 4(1) is a UML diagram provided in the embodiment of this application;

[0024] Figure 4(2) is a UML diagram provided in the embodiments of this application;

[0025] Figure 4(3) is a UML diagram provided in the embodiments of this application;

[0026] Figure 5 This is a schematic diagram of the data processing apparatus provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] This application provides a data processing method, apparatus, computer device, and computer-readable storage medium. The data processing apparatus can be integrated into a computer device, which may be a server or a terminal, etc.

[0030] The terminal may include mobile phones, wearable smart devices, tablets, laptops, personal computers (PCs), and in-vehicle computers, etc.

[0031] The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0032] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0033] This embodiment will be described from the perspective of a data processing device, which can be integrated into a computer device, such as a server or a terminal.

[0034] This application provides a data processing method, such as... Figure 1 As shown, the specific flow of this data processing method can be as follows:

[0035] 101. Based on the target process, perform memory mapping processing on the first file to obtain the mapping region of the first file in the virtual address space of the target process, wherein the first file contains the first target data and the storage space associated data of the first target data.

[0036] Among them, memory mapping (MMAP) can map the contents of the first file into the virtual address space of the process. The contents of the file can be directly accessed as part of the process's memory. The process can directly read and write the contents of the first file. Optionally, the first file can be mapped into the virtual address space of the target process by using the mmap system call.

[0037] The first file may include first target data and data related to the storage location of the first target data, i.e., storage space associated data. The first target data may be, for example, a game attribute configuration table, which may include data such as the skills and attributes of game characters and the effects of game items. The content of the first target data may vary in different application scenarios and can be flexibly configured. The storage space associated data may include the storage address of the key-value pairs of the first target data, hash tables, etc. The specific settings can be made according to the information required for querying data, and are not limited here.

[0038] There can be one or more target processes that map the first file into the virtual address space of the target process. This allows the target process to obtain the storage space associated data based on the mapping region of the first file in the virtual address space. Then, it can access the target data in the first file based on the storage space associated data, without having to fully load the target data into the target process, which can reduce the data's occupation of device memory.

[0039] 102. In response to the first data access instruction, obtain the storage space associated data in the first file based on the mapped region.

[0040] The first data access instruction can be an access instruction for the first target data in the first file. The target process can access the data in the first file according to the mapping area. The target process can access the storage space associated data in the first file according to the first access instruction. Compared with the first target data, the amount of data in the storage space is very small.

[0041] In one embodiment, the storage space data may include address mapping data, which may include hash tables, search trees, linked lists, etc. The address mapping data is used to quickly determine the location information of the required data in the storage space associated data based on the data identifier. The step "based on the first data object and the storage space associated data encapsulated by the first data object, obtain a portion of the first target data in the first file" includes:

[0042] The data identifier for the data required to respond to the first data access instruction is determined according to the first data access instruction;

[0043] Based on the first data object and address mapping data, determine the storage address corresponding to the data identifier;

[0044] Retrieve a portion of the first target data from the first file based on the storage address.

[0045] The data identifier can be an index, a key value, etc. For example, if the first target data is stored in a key-value pair data structure, then the data identifier can be the key value of the first target data. The first data access instruction contains the data identifier of the required data.

[0046] In one embodiment, the first data object may encapsulate address mapping data and the storage address of the first target data. Based on the address mapping data in the first data object, the storage address corresponding to the data identifier can be queried, and the position of the data encapsulated in the first data object can be found, thereby allowing the corresponding storage address to be queried.

[0047] After finding the storage address, the required data, i.e., part of the first target data, can be obtained from the first file.

[0048] In one embodiment, the storage space associated data further includes the starting storage address of the first file and the relative address of the first target data, that is, the address offset of the first target data relative to the starting storage address of the first file. The step "determining the storage address corresponding to the data identifier based on the first data object and the address mapping data" may include:

[0049] Based on the first data object and address mapping data, determine the target address offset corresponding to the data identifier;

[0050] Based on the starting storage address and the target address offset, the storage address of a portion of the first target data is determined.

[0051] The first data object may encapsulate address mapping data and the address offset of the first target data. Based on the address mapping data encapsulated in the first data object, the target address offset corresponding to the data identifier in the first data object is determined.

[0052] Based on the starting storage address and the address offset, the storage address of a portion of the first target data is determined. Specifically, the storage address of the first target data can be obtained by adding the address offset to the starting storage address.

[0053] 103. Generate the first data object based on the data associated with the storage space.

[0054] The first data object can be an object from object-oriented programming, such as a Python object. The first data object is generated based on the associated data within the storage space, and it encapsulates that associated data.

[0055] 104. Based on the first data object and the storage space associated data encapsulated by the first data object, obtain a portion of the first target data in the first file.

[0056] Based on the first data object and the storage space associated data encapsulated by the first data object, a portion of the first target data in the first file can be accessed in the first file. The portion of the first target data may be the data required to respond to the first data access instruction.

[0057] 105. Respond to the first data access instruction based on a portion of the first target data.

[0058] Responding to the first data access instruction based on a portion of the first target data can either directly return a portion of the first target data, or it can perform data operations on the portion of the first target data, obtain the operation result, and then return the operation result. Data operations on the first target data can include arithmetic operations, intersection calculations, etc.

[0059] In one embodiment, the first file further includes second target data. The first target data and the second target data can be data of different data types. For example, the first target data can be a tuple, array, set, dictionary, or other data type, while the second target data can be an integer, floating-point number, string, or other data type. The first target data usually contains multiple data items; for example, an array may contain multiple data items, and only a portion of these data may need to be accessed. The second target data is usually accessed as a whole. A second data object can be generated based on the second target data. That is, the data processing method provided in this application may further include:

[0060] In response to the second data access instruction, the second target data is obtained from the first file based on the mapped region;

[0061] If a second data object is generated based on the second target data, then the second data access command is responded to based on the second data object.

[0062] The specific process of obtaining the second target data can be referred to the process of obtaining the storage space associated data of the first target data, and will not be elaborated here.

[0063] After generating a second data object based on the obtained second target data, the second data access command can be responded to based on the second data object.

[0064] After performing memory mapping on the first file, the mapped region can be managed. When the mapped region is no longer needed, it is released. That is, in one embodiment, after the step of "performing memory mapping on the first file based on the target process to obtain the mapped region of the first file in the virtual address space of the target process", the data processing method provided in this application may further include:

[0065] A management object is generated for the mapped region and the first file. The management object is used to detect the number of data objects generated based on the first file that still exist in the target process.

[0066] If the detected quantity is zero, the recycling mechanism is triggered to destroy the mapping area corresponding to the first file.

[0067] The management object can encapsulate methods for managing the mapped region. The management object can detect the number of data objects generated by the target process based on the first file that still exist. If the number is not zero, it means that there are still data objects referencing the data in the first file. If the number is zero, it means that there are no data objects referencing the data in the first file. When the number is zero, the recycling mechanism is triggered, and the first file is destroyed in the mapped region corresponding to the target process through the recycling mechanism.

[0068] In one embodiment, the first file can be memory-mapped using a preset module object, and a preset module object reference management object can be set to ensure that after the data table is loaded, this mapped area will not be reclaimed unless the module object actively releases the reference. That is, the step "Memory mapping of the first file based on the target process" includes:

[0069] The first file is memory-mapped based on the target process using a pre-defined module object;

[0070] After the step of "generating a management object for the mapped region and the first file", the data processing method provided in this application embodiment may further include:

[0071] Set a preset module object reference management object so that the preset module object triggers the recycling mechanism based on the management object.

[0072] Specifically, this could involve creating a custom object `MmapTable` that inherits from `types.ModuleType`. This object would be responsible for loading, referencing, and reloading the first file, mapping the data object of the first file into memory via the `mmap` system call. Furthermore, an import hook mechanism could be used, with custom Finder and Loader, to ensure that module imports in the root directory of the first file return `MmapTable` objects.

[0073] When a shallow copy function is called to copy a data object, the corresponding copying method can be executed according to the data type of the first object. That is, in one embodiment, after the step of "generating the first data object according to the storage space associated data", the data processing method provided in this application embodiment may further include:

[0074] If the data object is a first-type data object, in response to a call to the shallow copy function on the data object, the data object is returned to the shallow copy function;

[0075] If the data object is a second data type object, in response to the call to the shallow copy function on the data object, a new data object is generated based on the data object, and the pointer of the new data object points to the same memory address as the pointer of the data object.

[0076] The data object generated in step 103 can be a custom object. For example, for Tuple, Set, and Dict types, the generated custom object can be a custom ShmTuple, ShmSet, or ShmDict type.

[0077] The first data type of object can be ShmSet or ShmTuple. Treating ShmSet and ShmTuple as immutable objects, i.e., singleton classes, when a shallow copy function (such as the copy function) is called, it returns the ShmSet or ShmTuple itself.

[0078] The second data type object can be a ShmDict. When a shallow copy function (such as the copy function) is called, a new ShmDict is returned. The elements in the new ShmDict, which is the original ShmDict, all point to the mapped region of the first file.

[0079] When a shallow copy function is called to copy a data object, the corresponding copying method can be executed according to the data type of the first object. That is, in one embodiment, after the step of "generating the first data object according to the storage space associated data", the data processing method provided in this application may further include:

[0080] If the data object is a third type of data, in response to calling the deep copy function on the data object, the data object is copied level by level based on the deep copy function to obtain a new data object;

[0081] If the data object is a fourth data type object, in response to the call to the deep copy function on the data object, the data in the data object is accumulated and copied through the accumulator function to obtain a new data object, so that operations can be performed based on the new data object.

[0082] The third data type can include ShmTuple and ShmDict. When a deep copy function (such as the deepcopy function) is called, ShmTuple and ShmDict adopt the relevant implementations of tuple and dict, and use a recursive method to call the copy method layer by layer to achieve deep copy.

[0083] The fourth data type can refer to ShmSet. When a deep copy function (such as the deepcopy function) is called, and a deep copy is achieved through a reduce function (i.e., an accumulator function), the original deepcopy function can perform a deep copy of the ShmSet based on the return value of this method.

[0084] Optionally, the first file in step 101 can be obtained through the following steps:

[0085] Obtain a second file, which contains at least one first target data object;

[0086] Perform data format conversion on the data encapsulated in the first target data object to obtain the first target data;

[0087] The storage space associated data of the first target data is used to perform structure transformation processing to obtain the first structure data;

[0088] The first file is generated based on the first target data and the first structure data.

[0089] The data format of the second file may differ from that of the first file. For example, the second file may be a Python file, and the first file may be a binary file. The first and second files may also be in other data formats, which are not limited here. In one embodiment, the second file may be a file obtained by converting a table in Excel or CSV format. For example, the second file may be a Python file obtained by converting an Excel or CSV table, and the first file may be a binary file obtained by serializing the second file.

[0090] The second file may include at least one first target data object. The data encapsulated in the first target object is subjected to data format conversion processing to obtain the first target data. The storage space associated data obtained in step 102 may specifically be the storage space data corresponding to one of the first target data, so as to generate a data object based on the storage space data, and obtain the data encapsulated in the first target data object, which is then converted into the first target data.

[0091] The storage space associated data of the first target data is obtained and processed into a structure to convert the storage space associated data of the first target data into a structure, thus obtaining the first structure data.

[0092] Furthermore, the first structure data can be converted into a new format, and a first file can be generated based on the processed first structure data and the first target data.

[0093] Optionally, the second file also includes a second target data object. The first target data object and the second target data object can be objects of different data types. Before generating the first file, the first target data object can be processed. That is, before the step "generating the first file based on the first target data and the first structure data", the data processing method provided in this application can also include:

[0094] The data encapsulated in the second target data object is processed by a structure conversion to obtain the second structure data.

[0095] The step "Generate a first file based on the first target data and the first structure data" may include:

[0096] The first file is generated based on the first target data, the first structure data, and the second structure data.

[0097] For example, the data encapsulated in the second target data object can be processed by structure conversion to obtain the second structure data; the first file can be generated based on the first target data, the first structure data, the second structure data, and the structure data of the second type of data object.

[0098] For example, the second file is a Python file, and the corresponding structure for various Python objects in the second file is as follows: Figure 2 As shown, for basic types (None, Bool, Int, Float, String, etc.), the struct stores the necessary data for deserialization into a Python object. For container types (Tuple, Set, Dict), the internal structure of the container corresponding to the Python type can be preserved, and the pointers are replaced with relative addresses relative to the starting address of the first file mapping.

[0099] For container types Tuple, Set, and Dict, custom Python objects (RefShmTuple, RefShmSet, and RefShmDict) are used to reference shared memory data (i.e., the first target data in the first file). Access is achieved through the macro definition PyObject_NEW, which only adds a small amount of memory (Python object header data, shared memory object pointer, and data start position pointer). The actual memory occupied by the data is shared.

[0100] As can be seen from the above, the embodiments of this application obtain a mapped region of the first file in the virtual address space of the target process by performing memory mapping processing on the first file based on the target process. The first file contains first target data and storage space associated data of the first target data. In response to the first data access instruction, the storage space associated data in the first file is obtained based on the mapped region. A first data object is generated based on the storage space associated data. Based on the first data object and the storage space associated data encapsulated by the first data object, a portion of the first target data in the first file is obtained. The first data access instruction is responded to based on the portion of the first target data. This can achieve the goal of the first data object generated in the target process containing a small amount of storage space associated data, rather than containing all the data of the first file, thereby reducing the data's memory occupation.

[0101] To further illustrate the data processing method provided in this application, the following example uses a binary file as the first file and a Python file as the second file.

[0102] The data processing method provided in this application may include two parts: one part is to serialize the second file to obtain the first file, and the other part is to perform memory mapping based on the first file and perform deserialization to obtain a Python object.

[0103] First, you can obtain an Excel or CSV spreadsheet. This spreadsheet could be a configuration table from a game or a data table from other application scenarios. Then, convert the Excel or CSV spreadsheet into a Python file. Next, convert the Python objects within the Python file into a serializable struct. Finally, write the struct sequence into a binary file to obtain the first file. For example, the second file and the first file can be... Figure 3 As shown in the figure, the first file is only a schematic diagram and does not represent the actual stored data content. In the first row of the table, dict is the data type of the Python file, and k1, v1, k2, and v2 are the storage addresses of the key values ​​in the table (the storage addresses can be relative addresses).

[0104] The method provided in this application for serializing a second file to obtain a first file can be output as a Python module. Projects only need to import the module to use it without modifying the project's logic code.

[0105] To ensure that the project scripts do not require additional adjustments after importing modules, and that the data manipulation methods remain consistent with those used for native Python types, this invention primarily addresses the following compatibility issues:

[0106] A custom class, MmapTable, inheriting from types.ModuleType, is responsible for loading, referencing, and reloading (reload) data table objects mapped into memory via the mmap system call. Furthermore, it employs an import hook mechanism, using custom Finder and Loader to ensure that importing modules from the root directory of the specified configuration table returns objects of the custom type MmapTable.

[0107] This ensures that the isinstance function returns the correct type. Specifically, when the custom container types ShmTuple, ShmSet, and ShmDict call the built-in isinstance function on the original container types tuple, set, and dict respectively, the return result will still be True. This way, the logic code that originally called this function to determine the instance type can still run correctly.

[0108] To ensure compatibility with data serialization modules (such as BSON, JSON, and msgpack), allowing custom container data to be serialized by these modules, this can be achieved through object type detection by the data serialization module. For example, since ShmDicts inherits from `collections.abc.Mapping`, the JSON module can serialize ShmDicts using a serialization scheme that uses the `Mapping` type. Alternatively, serialization can be performed using a custom encoding method. The data serialization module is used to serialize a primary file into data suitable for network transmission, not to serialize Python files into binary files.

[0109] By serializing the second file, you can process the Python basic data types (None, Bool, Int, Float, String, etc.) and container types (Tuple, Set, Dict, etc.) in the second file separately.

[0110] For Python's basic data types, such as storing type information and numerical information, the precision of the data can also be controlled. For example, using C's long long type to store Python int type data objects, although the range of representation is reduced, is sufficient for configuration tables and can reduce the memory usage of data storage.

[0111] For container types, three data structures—ShmTuple, ShmSet, and ShmDict (these are custom names used in this application)—represent binary data of type tuple, set, and dict, respectively. During data serialization, pointers to elements within the container are replaced with relative addresses. When retrieving an element, the corresponding data on the shared file is found by adding an offset to the data's starting address. In the serialization structure design of container objects, the implementation of structures such as hash tables is kept consistent with the Python engine, maintaining the same high performance as the engine code.

[0112] For the various Python types in the second file, the corresponding structures are as follows: Figure 2 As shown, for basic types (None, Bool, Int, Float, String, etc.), the struct stores the necessary data for deserialization into a Python object. For container types (Tuple, Set, Dict), the internal structure of the corresponding Python type container can be preserved, some data related to deletion and modification operations can be deleted, pointers can be replaced with relative addresses relative to the starting address of the first file mapping, and List can be converted into Tuple for processing.

[0113] Based on the first file, memory mapping and deserialization are performed to obtain a Python object. Specifically, for basic Python data types, the deserialization operation generates a native Python object. For example, the Python int type data object can be stored directly using the C long long type, and the deserialization uses the PyLong_FromLongLong function in the Python C API.

[0114] Generating native Python objects is compatible with binary operations on Python's basic data types, such as addition, subtraction, multiplication, and division. Furthermore, Python's basic data types (such as None, True, False, and small integers) are reusable and do not occupy extra space. Python's garbage collection mechanism will immediately reclaim basic data types when their reference count reaches 0, and they will not remain resident in memory unless logically blocked. When generating basic data type objects, memory allocation and data copying overhead are small, and efficiency is controllable.

[0115] For container types, the deserialization operation generates a custom Python object that references the binary data corresponding to the shared file. The deserialization operation does not use tools like msgpack, pickle, or json because it generates native Python objects. Although the first file is shared, the generated Python objects are private to each process, and each generated Python object contains all the data from the first file. This is equivalent to directly importing a Python data table and generating a module object, thus failing to achieve the goal of saving memory.

[0116] UML diagrams of basic data types and container types are as follows: Figures 4(1)-4(3) As shown, `ShmObject` is the basic type that all classes possess, where the `type` field stores an enumeration value indicating the type of data the class represents. The data within the class structure varies depending on the data type. For basic types, it stores the necessary data for deserialization into Python objects. For container types, the internal structure of the corresponding Python type is retained, some data related to deletion and modification operations is removed, and pointers are replaced with relative addresses relative to the starting address of the file mapping. For container types tuples, sets, and dictionaries, custom Python types (`RefShmTuple`, `RefShmSet`, and `RefShmDict`) are used to reference shared memory data. Access is achieved by generating a custom object using the macro `PyObject_NEW`, adding only a small amount of memory (Python object header data, shared memory object pointer, and data start position pointer); the actual memory occupied by the data is shared.

[0117] If the first file needs to be updated, a new binary file can be generated for sharing, the process can remap the new file, or the shared data table can be degraded back to the original Python file.

[0118] For the mapping space and mapping file, a dedicated Python object (named MmapObj) is generated for management. This object holds a pointer to the address of the mapping space and a handle to the mapping file, respectively. The mapping area is released and the mapping file is closed only when this object is destroyed. MMapObj is referenced by the MmapTable module object, ensuring that this part of the mapping area will not be reclaimed as long as the module object does not actively dereference it after the data table is loaded. When accessing container data ShmTuple, ShmSet, and ShmDict in the shared file and generating their corresponding reference objects, the reference count of MmapObj is incremented by 1. Conversely, when the container reference object is destroyed, the reference count of MmapObj is decremented by 1. In this way, as long as the shared data is being used, the reference count of MmapObj is always greater than 0, thus avoiding destruction due to Python's garbage collection mechanism. The corresponding shared memory area will not be released, and the MmapTable module object can then load and reference new data, allowing old and new data to coexist.

[0119] In summary, it is possible to generate a first data object in the target process that contains a small amount of associated data in the storage space, rather than containing all the data in the first file, thereby reducing the data's memory usage.

[0120] To facilitate better implementation of the data processing method provided in the embodiments of this application, a data processing apparatus is also provided in one embodiment. The meanings of the terms used are the same as in the data processing method described above, and specific implementation details can be found in the description of the method embodiments.

[0121] This data processing device can be integrated into a computer device, such as... Figure 5 As shown, the data processing device may include: a mapping unit 301, a first acquisition unit 302, a generation unit 303, a second acquisition unit 304, and a response unit 305, as detailed below:

[0122] (1) Mapping unit 301 is used to perform memory mapping processing on the first file based on the target process to obtain the mapping area of ​​the first file in the virtual address space of the target process, wherein the first file contains first target data and storage space associated data of the first target data.

[0123] (2) The first acquisition unit 302 is used to acquire the storage space associated data in the first file based on the first data access instruction in response to the first data access instruction.

[0124] (3) Generation unit 303 is used to generate a first data object based on the above-mentioned storage space associated data.

[0125] (4) The second acquisition unit 304 is used to acquire part of the first target data in the first file based on the first data object and the storage space associated data encapsulated by the first data object.

[0126] In one embodiment, the aforementioned storage space associated data includes address mapping data, and the second acquisition unit 304 can also be used for:

[0127] The data identifier of the data required to respond to the first data access instruction is determined according to the first data access instruction.

[0128] Based on the first data object and address mapping data mentioned above, determine the storage address corresponding to the data identifier mentioned above;

[0129] Based on the aforementioned storage address, the aforementioned portion of the first target data is obtained from the aforementioned first file.

[0130] In one embodiment, the aforementioned storage space associated data further includes the starting storage address of the first file and the address offset of the first target data. The second acquisition unit 304 can also be used for:

[0131] Based on the first data object and the address mapping data, determine the target address offset corresponding to the data identifier.

[0132] Based on the aforementioned starting storage address and the aforementioned target address offset, the storage address of the aforementioned portion of the first target data is determined.

[0133] (5) Response unit 305 is used to respond to the first data access instruction based on the first target data.

[0134] In one embodiment, the first document further includes second target data, and the data processing apparatus provided in this application may further include:

[0135] The third acquisition unit is used to acquire second target data from the first file based on the above-mentioned mapping area in response to the second data access instruction;

[0136] The instruction response unit is used to generate a second data object based on the second target data, and then respond to the second data access instruction based on the second data object.

[0137] In one embodiment, after performing memory mapping processing on the first file based on the target process to obtain the mapped region of the first file in the virtual address space of the target process, the data processing method provided in this application may further include:

[0138] The management object creation unit is used to generate a management object for the above-mentioned mapping area and the above-mentioned first file. The management object is used to detect the number of data objects generated based on the first file that still exist in the above-mentioned target process.

[0139] The recycling unit is used to trigger the recycling mechanism if the above quantity is detected to be zero, so as to destroy the mapping area corresponding to the first file based on the above recycling mechanism.

[0140] In one embodiment, the mapping unit 301 described above can also be used for:

[0141] The first file is memory-mapped based on the target process using a pre-defined module object;

[0142] The data processing apparatus provided in this application may further include:

[0143] The preset module object is set to reference the management object, so that the preset module object triggers the recycling mechanism based on the management object.

[0144] In one embodiment, the data processing apparatus provided in this application may further include:

[0145] The first copy unit is configured to, if the data object is an object of a first data type, return the data object to the shallow copy function in response to calling the shallow copy function for the data object.

[0146] The second copy unit is used to generate a new data object based on the data object if the data object is an object of the second data type, in response to calling the shallow copy function on the data object, and the pointer of the new data object and the pointer of the data object point to the same memory address.

[0147] In one embodiment, the data processing apparatus provided in this application may further include:

[0148] The third copy unit is used to, if the data object is a third type of data object, respond to the call of the deep copy function for the data object, and perform a layer-by-layer copying process on the data object based on the deep copy function to obtain a new data object;

[0149] The fourth copy unit is used to, if the data object is an object of a fourth data type, respond to the call to the deep copy function for the data object, and perform an accumulator function to accumulate and copy the data in the data object to obtain a new data object, so as to perform operations based on the new data object.

[0150] In one embodiment, the data processing apparatus provided in this application may further include:

[0151] The file acquisition unit is used to acquire a second file, wherein the second file contains at least one first target data object;

[0152] The format conversion unit is used to perform data format conversion processing on the data encapsulated in the first target data object to obtain the first target data.

[0153] The first conversion unit is used to perform structure conversion processing on the storage space associated data of the first target data to obtain the first structure data.

[0154] The generation unit is used to generate the first file based on the first target data and the first structure data.

[0155] In one embodiment, the second document further includes a second target data object, and the data processing apparatus provided in this application may further include:

[0156] The second conversion unit is used to perform structure conversion processing on the data encapsulated by the second target data object to obtain the second structure data.

[0157] The above-mentioned generation unit can be used for:

[0158] The first file is generated based on the first target data, the first structure data, and the second structure data.

[0159] As can be seen from the above, the data processing device in this embodiment performs memory mapping processing on the first file based on the target process by the mapping unit 301, thereby obtaining the mapping region of the first file in the virtual address space of the target process. The first file contains first target data and storage space associated data of the first target data. The first acquisition unit 302, in response to the first data access instruction, acquires the storage space associated data in the first file based on the mapping region. The generation unit 303 generates a first data object based on the storage space associated data. The second acquisition unit 304 acquires a portion of the first target data in the first file based on the first data object and the storage space associated data encapsulated in the first data object. The response unit 305 responds to the first data access instruction based on the portion of the first target data. This can achieve the goal of the first data object generated in the target process containing a small amount of storage space associated data, rather than containing all the data of the first file, thereby reducing the data's memory occupation.

[0160] Accordingly, embodiments of this application also provide a computer device, which can be a terminal. For example... Figure 6 As shown, Figure 6This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 500 includes a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, and a computer program stored on the memory 502 and executable on the processor. The processor 501 and the memory 502 are electrically connected. Those skilled in the art will understand that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0161] The processor 501 is the control center of the computer device 500. It connects various parts of the computer device 500 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, it performs various functions of the computer device 500 and processes data, thereby monitoring the computer device 500 as a whole.

[0162] In this embodiment, the processor 501 in the computer device 500 loads the instructions corresponding to the processes of one or more applications into the memory 502 according to the following steps, and the processor 501 runs the applications stored in the memory 502 to achieve various functions:

[0163] Based on the target process, the first file is memory mapped to obtain the mapped region of the first file in the virtual address space of the target process. The first file contains the first target data and the storage space associated data of the first target data.

[0164] In response to the first data access instruction, the storage space associated data in the first file is obtained based on the mapped region;

[0165] A first data object is generated based on the storage space association data; based on the first data object and the storage space association data encapsulated in the first data object, a portion of the first target data in the first file is obtained;

[0166] The system responds to the first data access command based on a portion of the first target data.

[0167] As can be seen from the above, the embodiments of this application obtain a mapped region of the first file in the virtual address space of the target process by performing memory mapping processing on the first file based on the target process. The first file contains first target data and storage space associated data of the first target data. In response to the first data access instruction, the storage space associated data in the first file is obtained based on the mapped region. A first data object is generated based on the storage space associated data. Based on the first data object and the storage space associated data encapsulated by the first data object, a portion of the first target data in the first file is obtained. The first data access instruction is responded to based on the portion of the first target data. This can achieve the goal of the first data object generated in the target process containing a small amount of storage space associated data, rather than containing all the data of the first file, thereby reducing the data's memory occupation.

[0168] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0169] Optional, such as Figure 6 As shown, the computer device 500 also includes: a touch screen display 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch screen display 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507. Those skilled in the art will understand that... Figure 6 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0170] The touch display screen 503 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 503 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the computer device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 501. It can also receive and execute commands from the processor 501. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 501 to determine the type of touch event. Subsequently, the processor 501 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 503 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 503 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 503 can also be used as part of the input unit 506 to achieve input functions.

[0171] The radio frequency circuit 504 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0172] Audio circuitry 505 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 505 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 505, converted back into audio data, and output to processor 501 for processing. The audio data is then transmitted via radio frequency circuitry 504 to, for example, another computer device, or output to memory 502 for further processing. Audio circuitry 505 may also include an earphone jack to facilitate communication between peripheral headphones and the computer device.

[0173] The input unit 506 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0174] Power supply 507 is used to supply power to various components of computer device 500. Optionally, power supply 507 can be logically connected to processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 507 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0175] although Figure 6 As not shown in the diagram, the computer device 500 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0176] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0177] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0178] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute steps in any of the data processing methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0179] Based on the target process, the first file is memory mapped to obtain the mapped region of the first file in the virtual address space of the target process. The first file contains the first target data and the storage space associated data of the first target data.

[0180] In response to the first data access instruction, the storage space associated data in the first file is obtained based on the mapped region;

[0181] A first data object is generated based on the storage space association data; based on the first data object and the storage space association data encapsulated in the first data object, a portion of the first target data in the first file is obtained;

[0182] The system responds to the first data access command based on a portion of the first target data.

[0183] As can be seen from the above, the embodiments of this application obtain a mapped region of the first file in the virtual address space of the target process by performing memory mapping processing on the first file based on the target process. The first file contains first target data and storage space associated data of the first target data. In response to the first data access instruction, the storage space associated data in the first file is obtained based on the mapped region. A first data object is generated based on the storage space associated data. Based on the first data object and the storage space associated data encapsulated by the first data object, a portion of the first target data in the first file is obtained. The first data access instruction is responded to based on the portion of the first target data. This can achieve the goal of the first data object generated in the target process containing a small amount of storage space associated data, rather than containing all the data of the first file, thereby reducing the data's memory occupation.

[0184] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0185] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0186] The data processing method, apparatus, computer equipment, and computer storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, include: Based on the target process, the first file is memory mapped to obtain the mapped region of the first file in the virtual address space of the target process, wherein the first file contains first target data and storage space associated data of the first target data; In response to a first data access instruction, the storage space associated data in the first file is obtained based on the mapped region; A first data object is generated based on the storage space associated data; The data identifier of the data required to respond to the first data access instruction is determined according to the first data access instruction, and the data identifier is determined based on the data structure of the first target data; Based on the address mapping data indicated by the first data object and the storage space associated data encapsulated in the first data object, determine the storage address corresponding to the data identifier; Based on the storage address, obtain a portion of the first target data required in the first file to respond to the first data access instruction; The system responds to the first data access instruction based on the aforementioned portion of the first target data.

2. The method according to claim 1, characterized in that, The storage space association data also includes the starting storage address of the first file and the address offset of the first target data. The step of determining the storage address corresponding to the data identifier based on the first data object and the address mapping data includes: Based on the first data object and the address mapping data, determine the target address offset corresponding to the data identifier; Based on the starting storage address and the target address offset, the storage address of the portion of the first target data is determined.

3. The method according to claim 1, characterized in that, The first file also includes second target data, and the method further includes: In response to a second data access instruction, the second target data is obtained from the first file based on the mapped region; If a second data object is generated based on the second target data, then the second data access instruction is responded to based on the second data object.

4. The method according to claim 1, characterized in that, After performing memory mapping processing on the first file based on the target process to obtain the mapped region of the first file in the virtual address space of the target process, the method further includes: A management object is generated for the mapping region and the first file. The management object is used to detect the number of data objects generated based on the first file that still exist in the target process. If the detected quantity is zero, a recycling mechanism is triggered to destroy the mapping area corresponding to the first file based on the recycling mechanism.

5. The method according to claim 4, characterized in that, The memory mapping process of the first file based on the target process includes: The first file is memory-mapped based on the target process using a pre-defined module object; After generating the management object for the mapped region and the first file, the method further includes: The preset module object is set to reference the management object, so that the preset module object triggers the recycling mechanism based on the management object.

6. The method according to claim 1, characterized in that, After generating the first data object based on the storage space associated data, the method further includes: If the data object is an object of a first data type, in response to calling the shallow copy function for the data object, the data object is returned to the shallow copy function; If the data object is an object of the second data type, in response to calling the shallow copy function on the data object, a new data object is generated based on the data object, and the pointer of the new data object points to the same memory address as the pointer of the data object.

7. The method according to claim 1, characterized in that, After generating the first data object based on the storage space associated data, the method further includes: If the data object is a third type of data object, in response to calling the deep copy function for the data object, the data object is copied level by level based on the deep copy function to obtain a new data object; If the data object is an object of the fourth data type, in response to calling the deep copy function on the data object, the data in the data object is accumulated and copied through the accumulator function to obtain a new data object, so that operations can be performed based on the new data object.

8. The method according to any one of claims 1-7, characterized in that, The first file was obtained through the following steps: Obtain a second file, the second file containing at least one first target data object; The data encapsulated in the first target data object is converted to a new data format to obtain the first target data. The storage space associated data of the first target data is processed by structure transformation to obtain the first structure data; The first file is generated based on the first target data and the first structure data.

9. The method according to claim 8, characterized in that, The second file also includes a second target data object. Before generating the first file based on the first target data and the first structure data, the method further includes: The data encapsulated in the second target data object is subjected to structure transformation processing to obtain the second structure data; The process of generating the first file based on the first target data and the first structure data includes: The first file is generated based on the first target data, the first structure data, and the second structure data.

10. A data processing apparatus, characterized in that, include: The mapping unit is used to perform memory mapping processing on the first file based on the target process to obtain the mapping region of the first file in the virtual address space of the target process, wherein the first file contains first target data and storage space associated data of the first target data; The first acquisition unit is configured to, in response to a first data access instruction, acquire the storage space associated data in the first file based on the mapping region; The generation unit is used to generate a first data object based on the storage space associated data; The second acquisition unit is configured to determine, according to the first data access instruction, a data identifier of the data required to respond to the first data access instruction, wherein the data identifier is determined based on the data structure of the first target data; determine the storage address corresponding to the data identifier based on the address mapping data indicated by the first data object and the storage space associated data encapsulated by the first data object; and acquire, based on the storage address, a portion of the first target data required to respond to the first data access instruction in the first file. A response unit is configured to respond to the first data access instruction based on the portion of the first target data.

11. A computer device, characterized in that, It includes a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to perform the data processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which is loaded by a processor to perform the data processing method according to any one of claims 1 to 9.

Images