Image data separation caching method and device, equipment and storage medium

Abstract

The invention discloses an image data separation caching method and device, equipment and a storage medium, and the method comprises the steps: setting a serialized context according to a unique cache instance of a current operation and a function type of the current operation; a user object is obtained, separation serialization is carried out on the user object based on the serialized context, a metadata structure and an associated memory list are generated, an image data object in the user object is stored in the memory list, and index information is generated at a corresponding position in the metadata structure; the metadata structure and the memory list are stored in an associated mode to construct a memory cache; and in a disk, writing the image data object in the memory cache into an independent binary file, and replacing the index information in the metadata structure with file reference information to generate a metadata file. By means of the method, memory occupation and I / O loads are remarkably reduced, and the cache read-write speed is increased.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image data separation and caching method, apparatus, device, and storage medium. Background Technology

[0002] In the fields of computer vision and image processing, intermediate computation results and final outputs of algorithm models typically need to be cached for reuse in subsequent processing or for backtracking analysis of historical results. This type of data generally exhibits hybrid characteristics: on the one hand, it contains structured metadata, such as sets of facial landmark coordinates, algorithm configuration parameters, and function type identifiers. This data is relatively small in size, has a clear structural hierarchy, and is suitable for serialization and storage using text formats such as JSON or XML; on the other hand, it contains unstructured image data, typically represented by the cv::Mat object in the OpenCV library, which stores a large amount of pixel information, and its data volume is usually much larger than the metadata portion.

[0003] Currently, the industry primarily relies on two common serialization strategies for handling this type of mixed data. The first strategy is the Base64 encoding scheme, which converts the binary data of the image into a Base64 string and embeds it directly into a specific field of the JSON document. The second strategy is a binary serialization scheme, which uses libraries such as Boost.Serialization to package all data (including metadata and image data) into a single custom binary file format.

[0004] However, both existing technologies have significant drawbacks. Base64 encoding performs extremely poorly in terms of memory efficiency. Due to the inherent characteristics of the Base64 encoding mechanism, image data size increases by approximately 33%, leading to a sharp increase in memory consumption for high-resolution images. Simultaneously, JSON parsers experience a severe performance degradation when processing documents containing excessively long Base64 strings, and fragmentation caused by discontinuous memory allocation becomes increasingly prominent. Binary serialization sacrifices data accessibility and flexibility, encapsulating all data into a single binary format, preventing developers from directly viewing or editing critical metadata and greatly increasing debugging difficulty. Furthermore, the binary format is extremely sensitive to changes in data structure; even minor adjustments to the metadata structure can render historical cached files completely unparseable, compromising system compatibility.

[0005] Faced with these technical obstacles, developers are often forced to adopt a manual data separation workaround: writing code to save image data separately as disk files, obtaining the storage path, and then writing the path information into a JSON structure. This stopgap measure not only introduces a large amount of repetitive code and increases the probability of errors, but also scatters cache management logic throughout the business code, significantly reducing system maintainability. Summary of the Invention

[0006] The purpose of this invention is to provide an image data separation and caching method, apparatus, device, and storage medium, which aims to solve the problems of high memory consumption and low serialization efficiency in existing data caching.

[0007] In a first aspect, embodiments of the present invention provide an image data separation and caching method, comprising: Set the serialization context based on the unique cache instance of the current operation and the function type of the current operation; Obtain a user object, and perform deserialization on the user object based on the serialization context to generate a metadata structure and an associated memory list. The image data objects within the user object are stored in the memory list, and index information is generated at the corresponding position in the metadata structure. The metadata structure is associated with the memory list and stored to build a memory cache; Within the disk, the image data objects in the memory cache are written to a separate binary file, and the index information in the metadata structure is replaced with file reference information to generate a metadata file.

[0008] In a second aspect, embodiments of the present invention provide an image data separation and caching device, comprising: The setting unit is used to set the serialization context based on the unique cache instance of the current operation and the function type of the current operation; The separation unit is used to obtain the user object and perform separation serialization on the user object based on the serialization context to generate a metadata structure and an associated memory list. The image data object in the user object is stored in the memory list, and index information is generated at the corresponding position in the metadata structure. A construction unit is used to associate and store the metadata structure with the memory list to build a memory cache; The writing unit is used to write the image data object in the memory cache to a separate binary file on the disk, and replace the index information in the metadata structure with file reference information to generate a metadata file.

[0009] Thirdly, embodiments of the present invention provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image data separation and caching method described in the first aspect above.

[0010] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the image data separation and caching method described in the first aspect.

[0011] This invention discloses an image data separation and caching method, apparatus, device, and storage medium. The method includes: setting a serialization context based on the unique cache instance of the current operation and the function type of the current operation; obtaining a user object and performing separation serialization on the user object based on the serialization context to generate a metadata structure and an associated memory list, wherein image data objects within the user object are stored in the memory list, and index information is generated at the corresponding position in the metadata structure; associating the metadata structure with the memory list to construct a memory cache; and writing the image data objects in the memory cache to an independent binary file on disk, replacing the index information in the metadata structure with file reference information to generate a metadata file. This invention intelligently intercepts image data objects during serialization using a thread-local context, storing them in an independent memory list and recording only a lightweight index in the metadata; during persistence, it automatically saves the image as an independent file in its original binary format and replaces the index with a file reference containing metadata such as the path. This directly avoids the data bloat and parsing burden caused by Base64 encoding, eliminates the readability and compatibility barriers of the full binary scheme, and achieves full automation of the storage and retrieval process. This significantly reduces memory usage and I / O load, improving cache read / write speed; while maintaining metadata readability for easy debugging. This invention also provides an image data separation and caching device, a computer-readable storage medium, and a computer device, all possessing the aforementioned beneficial effects, which will not be elaborated further here. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the method for separating and caching image data; Figure 2 Another flowchart illustrating the image data separation and caching method; Figure 3 A schematic block diagram of an image data separation and buffering device. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be understood that, when used in this specification and the appended claims, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more of its features, integrals, steps, operations, elements, components and / or collections thereof.

[0016] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0017] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the relevant listed items and all possible combinations, and includes such combinations.

[0018] Please see Figure 1 and Figure 2 This embodiment provides an image data separation and caching method, including: S101: Set the serialization context based on the unique cache instance of the current operation and the function type of the current operation; In this embodiment, setting the serialization context based on the unique cache instance of the current operation and the function type of the current operation includes: When a user calls a generic interface to save any custom data object, a serialization context instance is created and set in the thread-local storage area based on the unique cache instance of the current operation and the function type of the current operation.

[0019] This embodiment achieves multi-dimensional technical advantages by dynamically binding the serialization context to a unique cache instance and function type, and managing it using a thread-local storage area. This mechanism ensures strict isolation between different cache operations, avoids data confusion, and provides the ability to customize serialization strategies for different function types. The thread-local storage design naturally solves the concurrency safety problem in a multi-threaded environment, ensuring data consistency without the need for additional locking mechanisms. The architecture of automatically associating generic interfaces with context eliminates the need for developers to manually pass context parameters, greatly simplifying the call complexity. This design maintains system scalability, allowing seamless support for various custom data objects, while optimizing resource utilization through precise context management, significantly improving code maintainability and system stability while ensuring high performance.

[0020] Specifically, when a user program needs to save a custom structure instance containing image data, it will call a function like CacheSave. <t>The system uses a generic interface. Upon receiving this call, it first parses and determines the unique cache instance identifier associated with this operation, such as a specific LocalCache object, and also clarifies the specific function type (FunctionType) of this operation, such as face detection or image enhancement. Based on these two key pieces of information, the system immediately creates a SerializationContext instance dynamically in the thread-local storage area (thread_local). The internal structure of this instance is initialized, containing an nlohmann::jsondata object for storing JSON-formatted data, and a std::unordered_map object. <TString,std::vector <cv::mat>The `matLists` structure is used to map `cv::Mat` arrays corresponding to different functional types. This setting operation makes the context instance globally accessible throughout the subsequent execution lifecycle of the entire thread, but strictly isolated from similar operations in other threads, thus providing a precise and safe execution environment for the automatic redirection and separation management of data streams in the subsequent serialization process.

[0021] S102: Obtain the user object, and perform deserialization on the user object based on the serialization context to generate a metadata structure and an associated memory list, wherein the image data object in the user object is stored in the memory list, and index information is generated at the corresponding position in the metadata structure; In this embodiment, obtaining the user object and performing deserialization on the user object based on the serialization context to generate the metadata structure and associated memory list includes: Iterate through the user objects and serialize them; When an image data object is detected, a deep copy operation is performed on the image data object; The deep-copyed image data object is stored in a memory list associated with the serialization context, and the index pointing to the image data position in the memory list is written in the corresponding position in the JSON structure to obtain the metadata structure and the associated memory list.

[0022] This embodiment completes serialization by automatically traversing user objects, eliminating the need for manual data splitting, significantly reducing code redundancy and lowering development and maintenance costs. Performing a deep copy on image data objects prevents subsequent operations from tampering with the original data, ensuring data integrity. Storing image data in an associated memory list, with the JSON structure only written to the index, completely solves the problems of data bloat and inefficient parsing associated with Base64 encoding, while maintaining metadata readability and avoiding compatibility defects of the full binary scheme. The index accurately points to the image data location, providing reliable support for subsequent disk storage and deserialization data reassembly, ensuring the accuracy of data association.

[0023] Specifically, after the system initiates the serialization process for the user object, the standard serialization library begins to traverse all member variables of the object. For common metadata such as integers, floating-point numbers, strings, or custom structures, the library functions convert them into corresponding JSON key-value pairs according to default rules, gradually building a tree-structured JSON object.

[0024] When the iteration process encounters a member variable of a specific type, such as `cv::Mat`, the system triggers a custom serialization function (e.g., `to_json`) pre-registered for that type. This function immediately intercepts the `cv::Mat` object, preventing it from executing the default serialization process, specifically preventing its conversion to inefficient and space-consuming string formats (e.g., Base64 encoding), thus ensuring the simplicity of the JSON file structure. Then, a deep copy operation is performed on the passed-in `cv::Mat` object (i.e., the image data object), creating a new `Mat` object with completely independent data content but the same pixel matrix, thereby ensuring the data integrity and security of the original user object. Next, the system retrieves a pre-set serialization context instance from the current thread's local storage and adds the deep-copy `Mat` object to a dedicated memory list managed by the serialization context, namely a `std::vectorcv::Mat`. This addition operation returns the specific index of the `Mat` object in the list. Subsequently, the custom serialization function does not populate the JSON tree with the original or copied Mat data itself. Instead, it constructs and writes a lightweight index object, such as a MatIndex structure containing an integer index value and a null flag, at the current traversed JSON node position.

[0025] At this point, the traversal and serialization process is complete. The output is a clean, complete JSON data structure containing only the original metadata and a lightweight index, and an independent memory list that stores all the separated image data objects in sequence under the management of the serialization context. Together, they form the logical basis for subsequent operations.

[0026] The cv::Mat object itself is a container that contains binary pixel data, and the pixel data in its internal structure is pure binary.

[0027] S103: Associate the metadata structure with the memory list to build a memory cache; Once the metadata structure is built and all separate image data is stored in the memory list, the system establishes a strong association between the two through a shared pointer mechanism. A memory cache object is created as the top-level container, containing two key members: a smart pointer to the serialized JSON metadata structure and a shared pointer to the std::vectorcv::Mat memory list storing the image data. These two pointers are encapsulated within the same object, and object lifecycle management ensures they are always synchronized in existence or destruction. The system uses the cache instance identifier and function type tag of the current operation as a composite key to register the newly constructed memory cache object in the global cache management table. This table is implemented using a thread-safe hash mapping structure, supporting fast retrieval with minimal time complexity. The memory cache object internally maintains a reference counter, which increments whenever a component accesses the cache and decrements when it is released. When the counter reaches zero, memory reclamation is automatically triggered. To optimize memory layout, the image data list uses a contiguous memory allocation strategy, maintaining the original alignment between the header information and pixel data of each cv::Mat object to avoid additional copying overhead. The MatIndex indexes retained in the metadata structure correspond precisely to their positions in the memory list, forming an accurate mapping. The entire associative storage process completes the core structure construction in a single memory allocation, avoiding performance losses caused by fragmented allocation. When an application needs to access cached content, it can simultaneously obtain metadata and image data through a unified interface. The system automatically parses the index relationships internally and transparently reassembles the complete data structure, freeing developers from worrying about the underlying separate storage details and greatly simplifying the cache usage process.

[0028] In some embodiments, the image data list employs a contiguous memory allocation strategy, and the header information of each cv::Mat object maintains its original alignment with the pixel data, including: Calculate the total memory requirements for all cv::Mat objects, including header information and pixel data. Iterate through the set of images to be stored, accumulating the pixel data size for each matrix as rows × cols × elemSize (elemSize is a member variable of the cv::Mat object in OpenCV, representing the size of a single element in bytes) × channels (number of channels). Considering memory alignment requirements, the step size must be a multiple of 16 bytes. The allocator requests a sufficiently large contiguous block of memory at once. The header region stores all cv::Mat header information sequentially, with each header strictly adhering to the memory layout defined by the OpenCV library, including the precise location and size of member variables such as flags, dims, rows, cols, data pointers, and reference counts. Following the header region is a contiguous storage area for all pixel data. The pixel data for each matrix is ​​arranged in row-major order, with padding bytes inserted between adjacent matrices to ensure the starting address of the next matrix satisfies 16-byte boundary alignment. During the data pointer correction phase, all header information is traversed, and the data pointer of each cv::Mat is redirected to the starting position of the corresponding pixel data in a contiguous memory block. At the same time, the correct step[0] value is set to reflect the actual step size.

[0029] In some embodiments, it also includes: The hash mapping structure of the global cache management table is divided into M independent segments based on the high N bits of the hash value of the composite key. A dedicated read-write lock is configured for each segment to form a segment lock group. The system receives cache registration requests for composite keys containing cache instance identifiers and function type labels, calculates the hash value of the composite key, extracts the high N bits to determine the target segment index, obtains the corresponding read-write lock in the segment lock group based on the target segment index, and applies the write lock. The system uses the composite key and the associated memory cache object to perform a registration operation in the target segment to generate an updated segment data structure, releases the corresponding read-write lock to complete the registration process, monitors the access frequency of each segment to generate a hot segment identifier list, and when it is detected that the segment to which the composite key accessed by the current thread belongs exists in the hot segment identifier list, the query result is copied to the thread local storage area to form a thread cache copy. Subsequent query requests for the same composite key will preferentially obtain the memory cache object from the thread cache copy.

[0030] The segmented locking mechanism divides the global hash map into multiple logical segments based on the hash value of the key, and equips each segment with an independent lock, distributing the originally global lock contention to each segment. This allows write operations for different composite keys to be fully parallel as long as they fall into different segments, significantly improving the overall write throughput and scalability of the system, and effectively avoiding the serialization queuing delay caused by a single write lock.

[0031] Secondly, hotspot monitoring based on access frequency and thread-local caching constitute a dynamic performance optimization layer. The system automatically identifies frequently accessed hotspot segments through monitoring and copies the query results of these segments to the local storage of the thread that initiated the request. Subsequent requests for the same data can be read directly from local memory, achieving zero lock contention and extremely low latency access. This not only significantly reduces the latency of accessing hotspot data but also indirectly reduces contention for global segment locks, freeing up more resources for other concurrent operations.

[0032] Specifically, in the implementation of high-concurrency global cache management, the system first performs structural initialization. The core hash mapping structure of the global cache management table is divided into M (e.g., 256) independent logical segments based on the high N bits (e.g., the high 8 bits) of the composite key hash value. Each segment is an independent hash mapping sub-table used to store the mapping relationship between composite keys and memory cache objects within that segment. Simultaneously, the system configures a dedicated read-write lock for each of these M segments, and these locks together form a segment lock group.

[0033] When the system receives a cache registration request, the request contains a composite key consisting of a cache instance identifier and a function type label, along with a complete in-memory cache object to be associated. The system first applies a hash function to the composite key to calculate an integer hash value. Then, the system extracts the high N bits of this hash value and maps it to an integer between 0 and M-1 using a modulo operation; this integer is the target segment index. Based on this index, the system locates the corresponding dedicated read-write lock from the segment lock group. To perform a write operation, the system applies a write lock to this lock, gaining exclusive access to the target segment hash mapping sub-table.

[0034] While holding a write lock, the system performs a registration operation in the hash map sub-table of the target segment: inserting or updating the sub-table with the input composite key as the key and the pointer or reference of the associated memory cache object as the value. This operation generates and updates the data structure of the target segment. After the registration operation is completed, the system immediately releases the write lock applied to the segment, thus completing the core process of this cache registration.

[0035] To continuously optimize performance, a monitoring thread runs in the system background. This thread periodically counts the frequency of access requests (including reads and writes) received by each segment within a unit of time. Based on this statistical data, the system generates a list of hotspot segment identifiers, which lists the indexes of segments whose access frequency exceeds a preset threshold.

[0036] When any worker thread subsequently initiates a query request for a composite key, after calculating the target segment index, the system first checks if the index exists in the current hot segment identifier list. If it exists, after performing standard segment read lock acquisition, querying, and retrieving the memory cache object, the system copies the object content (or a lightweight reference) of the query result to the current thread's thread-local storage area, forming a thread-private cache copy. Subsequently, within the lifetime of the same thread, if a query request for the exact same composite key is received, the system will preferentially retrieve the memory cache object directly from that thread's local cache copy, thus completely bypassing contention for the global segment lock and access to the shared hash map, achieving extremely low-latency data retrieval.

[0037] S104: On the disk, the image data object in the memory cache is written to a separate binary file, and the index information in the metadata structure is replaced with file reference information to generate a metadata file.

[0038] In this embodiment, writing the image data object in the memory cache to a separate binary file includes: Traverse the metadata structure to find all index information; Based on each index information found, the corresponding image data object is retrieved from the memory list; Each acquired image data object is written to a binary file in its raw binary format, and a unique filename is generated for the binary file based on the hierarchical path and index value in the metadata structure.

[0039] This embodiment achieves batch generation and precise mapping of image data files by automatically traversing the metadata structure and extracting index information, avoiding the tediousness and errors of manually managing file paths. Writing image data directly in its raw binary format completely avoids the data volume expansion problem inherent in Base64 encoding, saving approximately one-third of storage space. Simultaneously, it utilizes the operating system's underlying file buffering mechanism to maximize disk write performance optimization. Dynamically generating unique filenames based on hierarchical paths and index values ​​ensures the systematic and unique nature of file naming, establishing a clear and stable correspondence between image data files and index nodes in the metadata structure, greatly enhancing the overall maintainability and traceability of the cached files. This separate storage strategy allows massive amounts of image data to be processed independently in batches, while lightweight metadata files remain readable, facilitating direct viewing and debugging, thus achieving a good balance between system performance and development / maintenance efficiency.

[0040] Specifically, after the memory cache is built, the system initiates the disk persistence process. First, the system performs a depth-first recursive traversal of the generated clean JSON metadata structure in memory, systematically searching for all nodes with specific markers, such as the MatIndex type. For each located MatIndex node, the system reads its internally stored integer index value. Based on this index value, the system accurately extracts the corresponding cv::Mat image object from the dedicated memory list associated with the current serialization context. Subsequently, the system writes the pixel data matrix contained in the cv::Mat object, strictly following its row-major order in memory, into a newly created, independent .bin file in its raw binary format without any encoding conversion. The file name is dynamically generated by combining the current cache operation's function type, the complete hierarchical key path of the MatIndex node in the JSON tree, and its index number, for example, forming a filename like face_detection_layer1_landmarks_0.bin, thus ensuring the file's uniqueness and the traceability of the data source. Next, the system replaces the original MatIndex nodes in the in-memory JSON structure with a new, more informative file reference node. This new node not only contains the disk storage path of the .bin file corresponding to the image data, but also includes key metadata necessary to reconstruct the cv::Mat object, such as the number of rows, columns, and data type of the image. After all MatIndex nodes have been replaced by the corresponding file reference nodes, the system saves this processed final JSON structure data to disk, naming it a file such as cache_data.json. This JSON file serves as the metadata description and address directory for the entire cached dataset, while all the large original image data is stored separately in a series of independent .bin files, collectively forming a persistent cache on disk.

[0041] In some embodiments, it also includes: The system traverses the JSON metadata structure in memory to identify array containers containing multiple MatIndex nodes. For each identified array container, a fast index area is built to record the position offset and index value of each MatIndex node in the array, generating an index mapping table. The constructed index mapping table is embedded into the root node of the JSON metadata structure to form an enhanced metadata structure. The enhanced metadata structure is losslessly compressed using the GZIP compression algorithm (GZIP compression algorithm is a lossless data compression technology based on the DEFLATE algorithm, combined with the LZ77 algorithm and Huffman coding) to generate a compressed byte stream. The compressed byte stream is associated with the corresponding binary image file and stored on disk. During loading, the enhanced metadata structure is restored by decompressing the compressed byte stream, and the target MatIndex node in the array container is quickly located according to the index mapping table in the enhanced metadata structure.

[0042] At the data retrieval level, this embodiment optimizes linear queries that previously required traversing the array into direct location based on offsets by building a dedicated index mapping table for dense arrays, significantly reducing the time complexity of accessing large arrays. At the storage efficiency level, GZIP lossless compression technology is used to process the enhanced metadata, effectively reducing disk storage space usage in scenarios with massive numbers of small objects while maintaining data integrity and recoverability. At the system performance level, the bundled storage design of the index table and metadata allows for direct retrieval of index information after decompression, achieving seamless integration of data loading and query preparation. This mechanism of dynamically identifying hot arrays and building adaptive indexes ensures efficient processing of regular data while specifically optimizing performance bottlenecks in particular scenarios, resulting in a dual optimization effect of storage space and access speed.

[0043] Specifically, the system iterates through the JSON metadata structure in memory, examining node types level by level to accurately identify array containers containing multiple MatIndex nodes. For each identified array container, the system constructs a dedicated fast index area, traversing all MatIndex nodes within the array, recording the position offset of each node in the array container and its corresponding index value, and organizing this information into a key-value pair index mapping table. Subsequently, the index mapping tables corresponding to all array containers are integrated and embedded into the root node of the JSON metadata structure, forming an enhanced metadata structure containing fast location information. Next, the GZIP compression algorithm is used to perform lossless compression processing on the enhanced metadata structure, generating a smaller compressed byte stream. This compressed byte stream is then associated with the corresponding binary image file and stored together on the disk according to a preset storage path. When data needs to be loaded, the system reads the compressed byte stream from the associated storage on disk, decompresses it using the GZIP decompression algorithm, and fully restores the enhanced metadata structure. Then, based on business requirements, it determines the array container to which the target MatIndex node belongs. Using the index mapping table in the root node of the enhanced metadata structure, it directly queries the position offset and index value of the target MatIndex node, quickly locating the target node within the array container without traversing the entire array container. After the data is loaded and the corresponding memory cache object is built, the system optimizes the hash mapping structure of the global cache management table. Based on the high N bits of the hash value of the composite key, it divides it into multiple independent segments, configuring a dedicated read-write lock for each segment. All segment read-write locks are integrated to form a segment lock group. When the system receives a cache registration request containing a composite key with a cache instance identifier and a function type label, along with the associated memory cache object, it calculates the hash value of the composite key using a preset hash algorithm, extracts the high N bits of the hash value, converts them into an integer as the target segment index, matches and obtains the corresponding dedicated read-write lock from the segment lock group based on this index, and applies a write lock to this read-write lock to ensure thread safety. Subsequently, the composite key and the memory cache object are written as key-value pairs into the hash map of the target segment, generating the updated segment data structure. The write lock is released immediately after writing, ending the cache registration process. During this process, the system continuously monitors the access frequency of each segment, counts the number of queries and access duration for each segment within a preset time window, and organizes the segments with access frequencies exceeding a threshold into a hotspot segment identifier list, which is updated in real time. When a cache query request initiated by the current thread detects that the segment to which the composite key belongs exists in the hotspot segment identifier list, the system, after completing the query and obtaining the corresponding memory cache object, copies the query result to the current thread's local storage area to form a thread cache copy. Subsequent query requests for the same composite key from this thread will prioritize retrieving the thread cache copy from its own thread local storage area. Only when the copy does not exist or is invalid will the target segment in the global cache management table be accessed again to perform the query operation.

[0044] In this embodiment, within the disk, the image data objects in the memory cache are written to a separate binary file, and the index information in the metadata structure is replaced with file reference information. After generating the metadata file, the process includes: Load the metadata file and load the corresponding binary file according to the file reference information to reconstruct the memory list, while restoring the file reference information in the metadata structure to index information; Based on the restored metadata structure and the reconstructed memory list, a reorganization and deserialization process is performed to obtain the user object.

[0045] This embodiment first loads a lightweight metadata file and identifies file references. The system can accurately locate and load only independent binary image files as needed. This reference-based loading mode avoids reading useless data and improves the overall deserialization startup speed. The accurate metadata stored in the file references is used to reconstruct the memory list, ensuring the accuracy and integrity of the image data conversion from binary stream to memory object. The most crucial step is restoring the file reference nodes to lightweight inodes. This allows the subsequent deserialization process to reuse the same efficient index access mechanism as the serialization stage. By directly extracting image data from the reconstructed memory list, the need for expensive disk I / O operations or complex parsing during deserialization is completely avoided. The entire process is logically closed-loop, automatically completing the accurate recovery from persistent storage to the complete business object in memory, ensuring correct data reorganization and maximizing cache read performance.

[0046] Specifically, when the system needs to recover user objects from the disk cache, it first reads and parses the metadata file `cache_data.json` on the disk. The system recursively traverses the JSON structure loaded into memory, locating all file reference nodes containing information such as the disk file path, the number of image rows, the number of columns, and the data type. For each such node, the system finds the corresponding `.bin` file based on the recorded path, reads its raw binary data, and uses the row, column, and type information stored in the node to accurately reconstruct the `cv::Mat` image object in memory; all reconstructed image objects are stored sequentially into a newly created memory list. Next, the system replaces each file reference node in the JSON structure with a lightweight `MatIndex` node, which only contains the position index of the corresponding image object in the reconstructed memory list. At this point, the system obtains a restored JSON structure containing only the original metadata and a lightweight index, as well as a complete reconstructed image data memory list. Subsequently, the system initiates the deserialization process based on this restored JSON structure and the reconstructed memory list. When the deserialization library traverses the JSON structure, a custom deserialization function is triggered whenever a MatIndex node is encountered. This function reads the index value from the node and directly retrieves the corresponding cv::Mat object from the reconstructed memory list using this index value. It then assigns the object to the corresponding member variable in the user object, thereby efficiently and completely restoring the original user object to memory.

[0047] In some embodiments, loading a metadata file and, based on file reference information, loading the corresponding binary file to reconstruct the memory list includes: Parse the metadata file to obtain the metadata structure; Traverse the metadata structure to identify all file reference information; The corresponding binary file is read based on the file path in each file reference information. The image data object is reconstructed using the number of rows, columns, and data types contained in the same file reference information. The image data objects are then stored in the reconstructed memory list in sequence.

[0048] This embodiment parses lightweight metadata files, enabling the system to quickly obtain structured information and avoiding the initial overhead of processing massive binary data. Automatic traversal of the metadata structure to identify file reference information ensures that all related data can be systematically located. A key advantage lies in the process of loading binary files based on file reference information: utilizing embedded metadata such as row and column numbers and data types, the original binary stream can be accurately reconstructed into complete image data objects, guaranteeing data integrity and format correctness. Finally, the reconstructed objects are stored sequentially in an in-memory list, not only reproducing the memory layout during serialization but also providing a constant-time access basis for subsequent deserialization, greatly improving data loading efficiency and overall performance.

[0049] Specifically, the cache loading process begins with file system operations. The system opens and reads the contents of the `cache_data.json` file, using a streaming JSON parser to progressively build the in-memory metadata structure. This parser employs an event-driven model, triggering callbacks when it encounters the start of an object, key-value pairs, and array elements, avoiding memory spikes caused by loading large files all at once. After the metadata structure is built, a recursive traverser initiates a depth-first search, checking each field layer by layer from the root node. When an object containing the fields `file_path`, `rows` (number of rows), `cols` (number of columns), and `type_code` is detected, it is identified as a file reference node, and its complete path coordinates and all metadata information are extracted. The system concatenates the cache root directory with the relative path `file_path` to obtain the absolute path, performing security checks to ensure the path does not contain sequences or system directory traversal characters, preventing security vulnerabilities. The file loading component uses memory-mapped I / O to open the binary file, first verifying the 4-byte header signature CMAT, then reading the actual file size and comparing it with the expected size in the metadata to verify data integrity. The cv::Mat reconstruction module accurately calculates the required memory size based on rows, cols, and type_code information, allocates a contiguous memory buffer, sets the step size and data pointers, and directly copies the data from the file-mapped region to the matrix's internal buffer, maintaining the original row-major order. The reconstruction process rigorously verifies the matrix dimensions; rows and cols values ​​must be greater than 0 and less than 10000, and type_code must be within a predefined range to prevent crashes caused by malicious data. Each successfully reconstructed cv::Mat object is appended to the end of a std::vectorcv::Mat type memory list in the order of discovery, while simultaneously recording its index position in the list. To optimize performance, the system pre-allocates memory list capacity, reserving space based on the total number of file reference nodes in the metadata, avoiding the copying overhead caused by dynamic expansion. Large image data employs a block-based reading strategy, processing 64KB data segments at a time to reduce the impact of single I / O latency. During traversal, the system maintains a stack of file reference node positions to ensure that nodes in nested structures are processed in the correct order, and that the reconstructed memory list index is completely consistent with the original serialization index. The error handling mechanism sets up independent exception capture for each file loading operation. The corruption of a single file will not interrupt the overall process. The system records the error location and uses an empty cv::Mat object as a placeholder, maintaining the index mapping relationship unchanged. After loading is complete, the memory list contains all reconstructed image data in the exact same order as during serialization, providing a precise data source for the subsequent deserialization stage.

[0050] In some embodiments, when reconstructing image data using the row number, column number, and data type information in the file reference information, the core is to rely on this metadata to restore the structure and data of the OpenCV cv::Mat object: First, the system reads the raw pixel data from the binary file and temporarily stores it in a memory buffer to ensure the integrity of the pixel data and its consistency with the original storage format; then, using the row number and column number in the file reference information as dimension parameters, an empty image data object is initialized, with the row number corresponding to the height dimension and the column number corresponding to the width dimension, to determine the two-dimensional matrix framework of the image data; next, the data type information in the file reference information is parsed, which corresponds to the OpenCV preset image data type identifier, and the system uses this information... The pixel data type of the image data object is set, including key attributes such as pixel bit depth and number of channels. For example, if the data type is CV_8UC3, the image data object is set to an 8-bit unsigned character type and a 3-channel pixel storage format. After the object structure is initialized, the original pixel data in the memory buffer is copied to the memory space of the image data object in row-major order to ensure that the arrangement order of the pixel data is consistent with the original image. Finally, it is verified whether the number of rows, columns and data type of the reconstructed image data object completely match the parameters in the file reference information. If there is a mismatch, the data verification exception process is triggered. If they match, the reconstruction of the image data object is completed, ensuring that the reconstructed image data is completely consistent with the original data before serialization.

[0051] Furthermore, when verifying the matching of the reconstructed image data object parameters with the file reference information parameters, the system first extracts the baseline parameters from the file reference information, including the baseline row count, baseline column count, and baseline data type identifier. These parameters are then temporarily stored in a separate verification buffer to ensure that the baseline parameters are complete and have not been tampered with. Next, the system reads the actual attribute parameters of the reconstructed image data object through the built-in attribute access interface of the image data object, specifically including the actual row count, actual column count, and the actual data type identifier representing the pixel storage format. Subsequently, numerical comparison verification is performed on the row count and column count. The actual row count is verified against the baseline row count, and the actual column count is verified against the baseline column count one by one. If any value is not equal, the corresponding row count or column count is immediately marked as an anomaly. For the data type identifier, the system first converts the baseline data type identifier and the actual data type identifier into a standardized numerical encoding format. For example, the type identifier representing pixel bit depth and channel number is converted into the corresponding integer encoding, and then a full comparison of the encoded values ​​is performed to avoid misjudgment due to differences in the format of the type identifier. After all parameters are compared, the system summarizes the verification results. If the number of rows, columns, and data types all match the baseline parameters, the verification is considered successful. If any parameter does not match, the system triggers an exception handling process, records the type of the mismatched parameter, the baseline value, and the actual value, terminates the subsequent use of the image data object, and prompts for troubleshooting issues such as binary file reading errors or metadata recording deviations, ensuring that only image data objects with matching parameters can be stored in the reconstructed memory list.

[0052] In some embodiments, based on the restored metadata structure and the reconstructed memory list, a reconstructed deserialization is performed to obtain user objects, including: A deserialization context is constructed using the restored metadata structure and the reconstructed memory list; Calling library functions converts the deserialization context into a preliminary user object; The index information in the metadata structure is read by overloaded functions, and the corresponding image data object is extracted from the reconstructed memory list based on the index information. The image data object is assigned to the corresponding variable of the initial user object to obtain the final user object.

[0053] This embodiment provides a complete runtime environment for data reconstruction by constructing a deserialization context containing a reconstructed memory list. Calling standard library functions to generate an initial user object effectively separates the processes of regular metadata loading and large image data injection, improving execution efficiency. The most critical advantage lies in the use of a custom overloaded function, which automatically reads index information from the metadata and accurately extracts the corresponding image data object directly from the reconstructed memory list with constant time complexity using the index value. This mechanism avoids time-consuming disk I / O operations or complex data format parsing during the deserialization stage. Finally, the extracted image data object is assigned to the corresponding member of the initial user object, ensuring that the final user object is completely consistent with the original object before serialization in terms of data content and structure. The entire process efficiently, automatically, and reliably completes the reconstruction from persistent storage to in-memory objects.

[0054] Specifically, at the start of the deserialization process, the system retrieves the previously reconstructed metadata structure and memory list from the thread-local storage area, constructs a deserialization context object, and binds the two together. This context contains a smart pointer to the memory list and a current processing state flag. The context object manages its lifecycle through the RAII mechanism, ensuring that temporary resources are automatically cleaned up when the operation ends. The standard JSON deserialization library function is called, passing in the metadata structure and context reference. The recursive parsing algorithm traverses the JSON nodes, directly mapping basic data types (integers, strings, floating-point numbers) to the corresponding fields of the user object, generating a preliminary user object skeleton without image data. When the parser encounters a MatIndex node containing the mat_index key, it triggers the overloaded from_json function for the cv::Mat type. This function extracts the index value from the node and performs boundary checks to verify that the index is within the valid range of the memory list. After successful verification, the function retrieves the corresponding cv::Mat object from the specified position in the memory list, performs a shallow copy operation to create a new matrix header, shares the original data buffer but maintains an independent reference count. The shallow copy process preserves all attributes of the original matrix, including the number of rows, columns, channels, step size, and data type identifier. The obtained cv::Mat object is assigned to the corresponding member variable of the initial user object. This process automatically matches the field type through template metaprogramming, without requiring the user to specify the target location. After the assignment operation is completed, the system updates the loading status flag of the field to avoid duplicate processing. For nested structures, the recursive algorithm ensures that child objects are reconstructed before parent objects, maintaining the complete object graph reference relationship. Then, the reconstruction process of each field is monitored. When an index goes out of bounds or the type does not match, an exception with detailed path information is thrown, recording the error location without interrupting the overall process. After all MatIndex nodes have been processed, the initial user object is fully populated and converted into the final user object. At this point, the memory management component releases the temporary reference of the deserialization context to the memory list, but maintains a strong hold on the cv::Mat data by the user object, ensuring the correct continuation of the data lifecycle.

[0055] In some embodiments, when the parser encounters a MatIndex node containing the key mat_index, it triggers the overloaded from_json function for the cv::Mat type. This function extracts the index value from the node and performs a boundary check to verify that the index is within the valid range of the memory list, including: The process involves: obtaining a reconstructed memory list containing multiple cv::Mat objects stored sequentially; obtaining a deserialization context bound to the reconstructed memory list and the current processing state flag; traversing the JSON nodes in the metadata structure to identify the MatIndex node containing the key mat_index; calling the overloaded from_json function for the cv::Mat type, passing the MatIndex node and the deserialization context as parameters; extracting the index value from the MatIndex node (an integer); obtaining the size information of the reconstructed memory list; comparing the index value with the size information of the reconstructed memory list to perform a boundary check and generating a boundary check result; determining whether the index value is within the valid range of the reconstructed memory list based on the boundary check result; and retrieving the cv::Mat object corresponding to the index value from the reconstructed memory list if the index value is within the valid range.

[0056] Specifically, when the deserialization process starts, the reconstructed memory list is obtained from the cache loading phase. This list is a `std::vectorcv::Mat` type container, which internally stores 128 pre-loaded `cv::Mat` objects in the original order during serialization. Each object contains complete image data and matrix header information. The deserialization context is created through a context factory, internally holding a smart pointer reference to the reconstructed memory list and a 32-bit status flag field. The initial value of the status flag is 0x01, indicating that parsing is in progress. The JSON parser uses a SAX-style event-driven model to traverse the metadata structure, checking the key set of each object node. When the `mat_index` key is detected and the node type is a JSON object, it is determined to be the target `MatIndex` node. The parser's internal type registry matches the special processing rules for the `cv::Mat` type and automatically calls the pre-registered overloaded `from_json` function, passing in a `const` reference to the current `MatIndex` node and a non-constant reference to the deserialization context. The overloaded function internally parses nodes of type `rapidjson::Value`, extracting the integer value corresponding to the `mat_index` key. This value represents the position index of the target image in the reconstructed memory list. The function obtains the return value of the `size()` method of the reconstructed memory list through the deserialization context, obtaining the current list capacity of 128. The boundary check logic performs double verification: first, it confirms that the index value is greater than or equal to 0; second, it confirms that the index value is strictly less than the list capacity of 128. If both conditions are met, the index is considered valid. During the verification process, atomic operations are used to update the context's status flag, setting the second bit to 1 to indicate that the boundary check is complete. When the index value is 42 and within the valid range [0, 127], the function accesses the `cv::Mat` object at the 42nd position through the `operator[]` method of the reconstructed memory list, performs a shallow copy to create a new matrix header, shares the original pixel data buffer but maintains an independent reference count. The shallow copy process preserves all attributes of the original matrix, including the number of rows, columns, channels, data type, and step size information. If the index value is out of range, the function throws an exception with detailed path information, including the invalid index value, list capacity, and JSON node position, and resets the error bits of the context state flag.

[0057] After extracting and assembling the cv::Mat object into a memory cache object, the system introduces a customized lock-free hash table based on CAS operations to replace the original segmented lock mechanism. The core of this hash table includes a bucket array, an atomic version number variable associated with each bucket, and a CAS atomic operation interface adapted to the underlying instruction set. When performing write operations such as cache registration, update, or deletion, the system first calculates the hash value of the cache composite key (cache instance identifier + function type label) to locate the target bucket in the lock-free hash table. Then, it reads the currently stored memory cache object (old value) and version number (old version) from that bucket, constructs a new memory cache object to be written (registration / update scenario) or a deletion flag (deletion scenario) according to business requirements, and generates a new version number (old version number + 1). Subsequently, it performs a CAS atomic operation to compare the current actual value and version number of the target bucket with the read old value and version. If they match, the bucket's value is updated to the new value and version number, completing the write operation. If they do not match (indicating a concurrent write conflict), the current value and version number of the bucket are read again, and the above CAS operation is repeated until successful. When performing cache read operations, an optimistic locking + version number mechanism is adopted: First, the current version number of the target bucket is read and recorded. Then, the memory cache object data in the bucket is read. After the read is completed, the version number of the bucket is queried again to compare whether the version numbers read twice are consistent. If they are consistent, it means that the data has not been concurrently modified during the read process, and the read cache object is returned directly. If they are inconsistent, it means that the data has been modified, and the version number read and data read process is re-executed. Compared with the segmented locking mechanism, this lock-free design does not require explicit application or release of read and write locks, completely avoiding write operation blocking queuing and context switching overhead caused by lock contention in high-concurrency scenarios.

[0058] This embodiment saves approximately 33% of storage space and memory usage by directly storing binary data. Simultaneously, since JSON does not contain giant strings, it significantly reduces the memory pressure and fragmentation risk of the JSON parser.

[0059] Secondly, image data is read and written directly to the disk in the form of raw binary blocks, utilizing the operating system's file buffering mechanism, which is much faster than text encoding and decoding. Real-world testing showed a significant improvement in read and write speed when processing caches containing multiple high-resolution face images.

[0060] Then through template technology (CacheLoad) <t> / CacheSave <t>The upper-level business logic does not need to be aware of the existence of separate storage. Developers only need to define a structure containing cv::Mat to cache it like a regular variable, without having to write any file read / write or path management code, which greatly improves development efficiency.

[0061] In addition, the generated cache_data.json maintains a clear structure, containing only metadata and file references. Developers can directly open the JSON file to view parameter configurations or inspect image file properties (size, channels, etc.), facilitating debugging and troubleshooting.

[0062] Please see Figure 3 This embodiment provides an image data separation and caching device 200, including: Setting unit 201 is used to set the serialization context based on the unique cache instance of the current operation and the function type of the current operation; The separation unit 202 is used to obtain a user object and perform separation serialization on the user object based on the serialization context to generate a metadata structure and an associated memory list, wherein the image data object in the user object is stored in the memory list and index information is generated at the corresponding position in the metadata structure; Construction unit 203 is used to associate and store the metadata structure with the memory list to build a memory cache; The writing unit 204 is used to write the image data object in the memory cache to a separate binary file on the disk, and replace the index information in the metadata structure with file reference information to generate a metadata file.

[0063] Furthermore, the setting unit 201 includes: Create a sub-unit to create and set a serialization context instance in the thread-local storage area based on the unique cache instance of the current operation and the function type of the current operation when the user calls the generic interface to save any custom data object.

[0064] Furthermore, the separation unit 202 includes: The traversal subunit is used to traverse the user object and serialize the user object; A deep copy subunit is used to perform a deep copy operation on the image data object when an image data object is detected. The storage sub-unit is used to store the deep-copyed image data object into a memory list associated with the serialization context, and write the index pointing to the image data position in the memory list at the corresponding position in the JSON structure to obtain the metadata structure and the associated memory list.

[0065] Furthermore, the writing unit 204 includes: The search sub-unit is used to traverse the metadata structure and find all the index information; The object retrieval subunit is used to retrieve the corresponding image data object from the memory list based on each index information found. The filename generation subunit is used to write each acquired image data object into a binary file in its original binary format, and generate a unique filename for the binary file based on the hierarchical path and index value in the metadata structure.

[0066] Furthermore, the writing unit 204 includes: The file loading subunit is used to load the metadata file, load the corresponding binary file according to the file reference information to reconstruct the memory list, and restore the file reference information in the metadata structure to the index information. The deserialization subunit is used to perform recombinant deserialization based on the restored metadata structure and the reconstructed memory list to obtain the user object.

[0067] Furthermore, the file loading subunit includes: The file parsing subunit is used to parse the metadata file and obtain the metadata structure. An information identification subunit is used to traverse the metadata structure and identify all file reference information. The object storage subunit is used to read the corresponding binary file according to the file path in each file reference information, and reconstruct the image data object using the number of rows, columns and data types contained in the same file reference information, and store the image data object into the reconstructed memory list in sequence.

[0068] Furthermore, the deserialization subunit includes: The context building subunit is used to build the deserialization context from the restored metadata structure and the reconstructed list of memory. The conversion subunit is used to call library functions to convert the deserialization context into a preliminary user object; The information reading subunit is used to read the index information in the metadata structure through overloaded functions, and extract the corresponding image data object from the reconstructed memory list based on the index information; The assignment subunit is used to assign the image data object to the corresponding variable of the initial user object to obtain the final user object.

[0069] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0070] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can implement the methods provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0071] The present invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the methods provided in the above embodiments. Of course, the computer device may also include various network interfaces, power supplies, and other components.

[0072] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

[0073] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusivity.

[0074] The term "comprises" implies that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.< / t> < / t> < / cv::mat> < / t>

Claims

1. An image data separation and caching method, characterized in that, include: Set the serialization context based on the unique cache instance of the current operation and the function type of the current operation; Obtain a user object, and perform deserialization on the user object based on the serialization context to generate a metadata structure and an associated memory list. The image data objects within the user object are stored in the memory list, and index information is generated at the corresponding position in the metadata structure. The metadata structure is associated with the memory list and stored to build a memory cache; Within the disk, the image data objects in the memory cache are written to a separate binary file, and the index information in the metadata structure is replaced with file reference information to generate a metadata file.

2. The image data separation and caching method according to claim 1, characterized in that, Setting the serialization context based on the unique cache instance of the current operation and the function type of the current operation includes: When a user calls a generic interface to save any custom data object, a serialization context instance is created and set in the thread-local storage area based on the unique cache instance of the current operation and the function type of the current operation.

3. The image data separation and caching method according to claim 1, characterized in that, The step of obtaining the user object and performing deserialization on the user object based on the serialization context to generate the metadata structure and the associated memory list includes: Iterate through the user objects and serialize them; When an image data object is detected, a deep copy operation is performed on the image data object; The deep-copyed image data object is stored in a memory list associated with the serialization context, and an index pointing to the image data position in the memory list is written in the corresponding position in the JSON structure to obtain the metadata structure and the associated memory list.

4. The image data separation and caching method according to claim 1, characterized in that, The step of writing the image data object in the memory cache to a separate binary file includes: Traverse the metadata structure to find all the index information; Based on each index information found, the corresponding image data object is retrieved from the memory list; Each acquired image data object is written to a binary file in its original binary format, and a unique filename is generated for the binary file based on the hierarchical path and index value in the metadata structure.

5. The image data separation and caching method according to claim 1, characterized in that, The process of writing the image data objects from the memory cache to a separate binary file on the disk, and replacing the index information in the metadata structure with file reference information to generate the metadata file, includes: Load the metadata file, and load the corresponding binary file according to the file reference information to reconstruct the memory list, while restoring the file reference information in the metadata structure to the index information; Based on the restored metadata structure and the reconstructed memory list, a reorganization and deserialization process is performed to obtain the user object.

6. The image data separation and caching method according to claim 5, characterized in that, The step of loading the metadata file and loading the corresponding binary file according to the file reference information to reconstruct the memory list includes: Parse the metadata file to obtain the metadata structure; Traverse the metadata structure to identify all file reference information; The corresponding binary file is read according to the file path in each file reference information, and the image data object is reconstructed using the number of rows, columns and data types contained in the same file reference information. The image data object is then stored in the reconstructed memory list in sequence.

7. The image data separation and caching method according to claim 5, characterized in that, Based on the restored metadata structure and the reconstructed memory list, the reorganization and deserialization are performed to obtain the user object, which includes: A deserialization context is constructed using the restored metadata structure and the reconstructed memory list; The library function is called to convert the deserialization context into a preliminary user object; The index information in the metadata structure is read by overloaded functions, and the corresponding image data object is extracted from the reconstructed memory list based on the index information. The image data object is assigned to the corresponding variable of the initial user object to obtain the final user object.

8. An image data separation and buffering device, characterized in that, include: The setting unit is used to set the serialization context based on the unique cache instance of the current operation and the function type of the current operation; The separation unit is used to obtain the user object and perform separation serialization on the user object based on the serialization context to generate a metadata structure and an associated memory list. The image data object in the user object is stored in the memory list, and index information is generated at the corresponding position in the metadata structure. A construction unit is used to associate and store the metadata structure with the memory list to build a memory cache; The writing unit is used to write the image data object in the memory cache to a separate binary file on the disk, and replace the index information in the metadata structure with file reference information to generate a metadata file.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image data separation and caching method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the image data separation and caching method as described in any one of claims 1 to 7.

Images