Image big data file storage method, reading method and electronic equipment

Abstract

The invention discloses an image big data file storage method, an image big data file reading method and electronic equipment, and belongs to the technical field of computer operation. According to the method, the data of each image block of the image big data file is merged and written into the binary file to be stored, the data element information is recorded, and meanwhile, dynamic capacity expansion is performed when the capacity of the binary file is insufficient, so that a large number of random and fine-grained read-write operations can be converted into sequential and large-batch read-write operations, and the read-write efficiency is improved. Therefore, the external memory addressing overhead and the I / O scheduling delay are effectively reduced, and the read-write efficiency of the image big data file is improved. Meanwhile, writing and coding sub-processes in the storage process are distributed to corresponding threads, reading and decoding sub-processes in the reading process are distributed to corresponding threads, and the sub-processes are scheduled in parallel and asynchronous assembly lines to ensure that the sub-processes can continuously process respective tasks, so that calculation and I / O resources are utilized to the maximum extent, and the efficiency of the system is improved. And the data processing throughput rate is obviously improved.

Description

Technical Field

[0001] This invention belongs to the field of computer operation technology, and more specifically, relates to a method for storing and reading large image data files, as well as an electronic device. Background Technology

[0002] With the rapid development of high-definition imaging technology and visual perception applications, image data has become one of the largest and fastest-growing types of big data resources. Its massive file size and complex structure pose serious challenges to the capacity, efficiency, and manageability of storage systems. Therefore, researching efficient, stable, and scalable image big data file storage methods is of great significance for improving the access performance of massive image data, reducing storage costs, and supporting upper-level image analysis and intelligent processing applications.

[0003] Currently, the Zarr format is a typical storage method for processing large amounts of image data. The core idea of ​​this method is to logically divide the complete image dataset into regular N-dimensional data blocks, storing each data block as an independent file (such as a binary file in the .zarr directory), supplemented by metadata files describing the overall dimensions and block information. While this file system-based block storage model facilitates parallel reading and writing and on-demand access to local data, its drawback lies in the "one data block corresponds to one independent file" implementation mechanism. When processing extremely large-scale images, this method generates a massive number of fragmented small files in the storage system, leading to an increase in metadata management overhead and requiring frequent addressing operations by the disk read / write head when accessing different data blocks. This restricts the input / output throughput efficiency (i.e., read / write efficiency) and scalability of the storage system, becoming a bottleneck affecting the high-speed access and processing performance of massive image data. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the prior art, the present invention provides a method for storing and reading image big data files, and an electronic device, so as to solve the technical problem of low reading and writing efficiency of image big data files in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for storing large image data files, comprising: The large image data file to be stored is converted into a multi-resolution pyramid structure, and the images at each resolution in the pyramid structure are divided into blocks of the same size. create One write thread and One coding thread; ; ; use Each encoding thread encodes each image block in parallel and stores them in a shared queue Q1 until all image blocks of the large image data file to be stored have been encoded. Simultaneously, the storage task for each encoded image block is divided into... Groups were created and assigned one-to-one with each other. In a single write thread; using Each write thread executes its corresponding storage task in parallel; The process of executing the corresponding storage task in each write thread includes: Obtain the encoded binary data of each image block under the corresponding storage task from the shared queue Q1, and use it as the target data for the corresponding image block; Create a binary file and its corresponding metadata text file in external storage; Open the binary file and write the target data of each image block under the corresponding storage task to the binary file sequentially; before writing the target data of each image block, obtain the used capacity C of the current binary file and calculate the remaining capacity of the binary file. When the target data size of the current image patch is greater than At the same time, expand the capacity of the binary file. After writing the target data for each image block, update the used capacity C of the current binary file, and write the name of the image block, its storage location in the binary file, and the size of the target data to the metadata text file.

[0006] More preferably, the above storage method further includes: after each write thread has completed its corresponding storage task, determining the remaining capacity of the current binary file. If the capacity redundancy exceeds the preset threshold, then release the unused areas in the current binary file.

[0007] More preferably, the shared queue Q1 is a non-blocking queue or a blocking queue.

[0008] More preferably, when the external storage speed When the speed exceeds the first preset speed, queue Q1 is selected as a non-blocking queue.

[0009] More preferably, This refers to the number of physical cores in the CPU processor.

[0010] More preferably, This refers to the number of physical cores in the CPU processor.

[0011] More preferably, the above encoding is JPEG encoding, LZW encoding, PackBits encoding, Zstd encoding, LZ4 encoding, or DEFLATE encoding.

[0012] More preferably, the process of executing the corresponding storage task in each write thread further includes: After obtaining the encoded binary data of each image block under the corresponding storage task from the shared queue Q1, calculate the check code of the encoded binary data of each image block under the corresponding storage task, and fill it after the encoded binary data of the corresponding image block to update the target data of the corresponding image block.

[0013] Secondly, the present invention provides a method for reading large image data files, including: create One read thread and One decoding thread; ; ; use Each read thread performs image block reading operations in parallel until the encoded binary data of all image blocks in the large image data file to be read is read. The large image data file to be read is a file stored in external storage using the storage method provided in the first aspect of this invention. The image block reading operation performed in each read thread includes: reading the binary file of an image data file and the corresponding metadata text file from the external storage, and storing the binary file in a shared queue Q2 in memory; obtaining the name of the image block recorded in the binary file, its storage location in the binary file, and the size of the target data from the corresponding metadata text file, and then dividing the binary file in queue Q2 into the target data of each image block. At the same time, utilizing Each decoding thread reads the encoded binary data of the image patch from queue Q2 in parallel and decodes it.

[0014] More preferably, when the external storage read speed When the speed is less than the second preset speed, When the external storage read speed When the speed is greater than the third preset speed, The second preset speed is less than the third preset speed.

[0015] More preferably, The number of binary files in the large image data file to be read is the smaller of the number of physical cores of the CPU processor.

[0016] More preferably, This refers to the number of physical cores in the CPU processor.

[0017] More preferably, the queue Q2 is a non-blocking queue or a blocking queue.

[0018] More preferably, when the read speed of the external storage... When the speed is less than the fourth preset speed and the ratio of the size of the image big data file to the memory capacity is less than the preset ratio, the above queue Q2 shall be a non-blocking queue.

[0019] More preferably, when the image big data file to be read is a file stored in external storage that carries a checksum, the image block reading operation performed in each reading thread further includes: For the target data of each image block: use the checksum at the end as the first checksum, and use the remaining binary data excluding the checksum at the end as the encoded binary data of the image block; calculate the checksum of the encoded binary data of the image block as the second checksum; compare the first checksum with the second checksum to perform integrity verification.

[0020] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first or second aspect of the present invention.

[0021] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: 1. The first aspect of the present invention provides a method for storing large image data files. By merging the data of each image block of the large image data file and writing it into a binary file for storage and recording data metadata, and dynamically expanding the capacity of the binary file when the capacity is insufficient, a large number of random, fine-grained read and write operations can be converted into sequential, large-volume read and write operations, thereby effectively reducing the overhead of external memory addressing and I / O scheduling latency, and improving the read and write efficiency of large image data files.

[0022] 2. The first aspect of the present invention provides a method for storing large image data files, which allocates the writing and encoding sub-processes in the storage process to corresponding threads and schedules them in a parallel asynchronous pipeline to ensure that they can continuously process their respective tasks, thereby maximizing the utilization of computing and I / O resources and significantly improving the data processing throughput.

[0023] 3. The first aspect of the present invention provides a method for storing large image data files, which supports multiple compression methods and has strong versatility.

[0024] 4. The first aspect of this invention provides a method for storing large image data files. Each binary file has a corresponding metadata text file, which records the name of each image block in the binary file, its storage location in the binary file, and the size of the target data. This allows for rapid location of the image to be read within the binary file and the corresponding binary encoded data to be read into the memory buffer by consulting the metadata text file during the reading process, maximizing I / O throughput and further improving data processing throughput. Simultaneously, when writing the target data of each image block under the corresponding storage task to the binary file, the order of the image blocks does not need to be considered, simplifying the asynchronous mechanism and further improving data processing efficiency.

[0025] 5. Furthermore, the first aspect of the present invention provides a method for storing large image data files. When storing the binary data encoded by each image block, a check code for verifying data integrity is further added, so that the integrity of the binary data encoded by each image block can be verified during the file reading process, making it more reliable.

[0026] 6. The second aspect of the present invention provides a method for reading large image data files, which allocates the reading and decoding sub-processes in the reading process to corresponding threads and schedules them in a parallel asynchronous pipeline to ensure that they can continuously process their respective tasks, thereby maximizing the utilization of computing and I / O resources and significantly improving data processing throughput. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating the storage process of image big data files provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the storage process of image big data files based on binary files, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the process of reading large image data files according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the process of reading large image data files based on binary files, as provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0029] To achieve the above objectives, in a first aspect, the present invention provides a method for storing large image data files, comprising: The large image data file to be stored is converted into a multi-resolution pyramid structure, and the images at each resolution in the pyramid structure are divided into blocks of the same size. create One write thread and One coding thread; ; ; use Each encoding thread encodes each image block in parallel and stores them in a shared queue Q1 until all image blocks of the large image data file to be stored have been encoded. Simultaneously, the storage task for each encoded image block is divided into... Groups were created and assigned one-to-one with each other. In a single write thread; using Each write thread executes its corresponding storage task in parallel; The process of executing the corresponding storage task in each write thread includes: Obtain the encoded binary data of each image block under the corresponding storage task from the shared queue Q1, and use it as the target data for the corresponding image block; Create a binary file (denoted as Bin file) and its corresponding metadata text file in external storage; Open the binary file and write the target data of each image block under the corresponding storage task to the binary file sequentially; before writing the target data of each image block, obtain the used capacity C of the current binary file and calculate the remaining capacity of the binary file. When the target data size of the current image patch is greater than At the same time, expand the capacity of the binary file. After writing the target data for each image block, update the used capacity C of the current binary file, and write the name of the image block, its storage location in the binary file, and the size of the target data to the metadata text file.

[0030] It should be noted that a Bin file is essentially a large binary file. It sequentially packages and stores a large number of image byte streams in lossy compression JPEG or lossless compression TIFF formats in a predefined order, making it appear as a single file to the outside world. The metadata text file can be in text formats such as TXT, JSON, and XML; there are no restrictions here.

[0031] In one optional implementation, the above storage method further includes: after each write thread has completed its corresponding storage task, determining the remaining capacity of the current binary file. If the value exceeds a preset capacity redundancy threshold (in one optional implementation, the preset capacity redundancy threshold is 100MB), then the unused area in the current binary file is released.

[0032] In one optional implementation, the shared queue Q1 is either a non-blocking queue or a blocking queue. Preferably, in one optional implementation, when the external storage speed... At higher speeds, i.e., when the external storage speed is... When the speed exceeds the first preset speed, the above queue Q1 will preferentially use a non-blocking queue.

[0033] Preferably, in one optional implementation, This refers to the number of physical cores in the CPU processor.

[0034] Preferably, in one optional implementation, This refers to the number of physical cores in the CPU processor.

[0035] It should be noted that the above encoding can be JPEG encoding, LZW encoding, PackBits encoding, Zstd encoding, LZ4 encoding, DEFLATE encoding, etc., and can be further extended through the format adaptation module. No limitation is made here.

[0036] In one optional implementation, the process of executing the corresponding storage task in each write thread further includes: After obtaining the encoded binary data of each image block under the corresponding storage task from the shared queue Q1, calculate the check code of the encoded binary data of each image block under the corresponding storage task, and fill it after the encoded binary data of the corresponding image block to update the target data of the corresponding image block.

[0037] It should be noted that the above checksum can be a cyclic redundancy check (CRC) checksum, MD5 checksum, etc., and there is no limitation here; among them, MD5 checksum is suitable for large amounts of data.

[0038] To further illustrate the image big data file storage method provided by the present invention, a specific embodiment is described in detail below: like Figure 1 As shown, the image big data file storage method provided in this embodiment includes the following parts: The large image data file to be stored is converted into a multi-resolution pyramid structure, and the images at each resolution in the pyramid structure are divided into blocks of the same size. Initialization phase: Detect the external storage write speed (i.e., external storage speed), and specify the number of encoding threads required for the encoded data. The number of threads writing encoded data If the external storage uses a mechanical hard drive with a speed of 20-200 MBps, set... If the external storage uses a solid-state drive with a speed of 200MBps or higher, set... In both cases, This refers to the number of physical cores in a CPU processor.

[0039] Establish One data encoding thread and A data writing thread is created. A shared queue Q1 is established. If a blocking queue is used, a synchronization lock is set in front of the queue; if a non-blocking queue is used, atomic operations are introduced to maintain the read and write indexes of the queue.

[0040] use Each encoding thread encodes each image block in parallel and stores them in a shared queue Q1 until all image blocks of the large image data file to be stored have been encoded. Simultaneously, the storage task for each encoded image block is divided into... Groups were created and assigned one-to-one with each other. In a single write thread; using Each write thread executes its corresponding storage task in parallel.

[0041] Specifically, Each data encoding thread runs concurrently after startup, and each thread executes the following steps in a loop: (1) Acquire the image block and take ownership of it.

[0042] (2) Perform image encoding operations (such as LZW, LZ4 and other compression encoding methods) on the obtained image blocks to generate encoded image blocks (denoted as encoded images) and store them in the memory array EL.

[0043] (3) Transfer ownership of EL's data to shared queue Q1, complete the current encoding task, release ownership of EL, and continue to the next step.

[0044] (4) The thread repeats the above steps until all tasks are completed.

[0045] In this embodiment, the shared queue Q1 is either a non-blocking queue or a blocking queue.

[0046] Blocking mode: When a thread transfers the encoded image block to Q1, it remains in a waiting state until the queue allows enqueueing; when enqueueing is allowed, the thread acquires the synchronization lock, transfers the encoded data to Q1; after enqueueing is complete, it releases ownership of the data and continues to the next task.

[0047] Non-blocking mode: When a thread transfers encoded data to Q1, it does not wait for the queue status and directly puts the encoded result into the queue through atomic operations.

[0048] The "transfer of data ownership" mentioned in this embodiment can be implemented using rvalue passing and pointer passing in C++, and there is no limitation here.

[0049] Specifically, when using a blocking shared queue, the data operation process is as follows: The thread storing data first acquires the synchronous read-write lock corresponding to the blocking shared queue.

[0050] If the task is to store data, the thread transfers ownership of the data to the blocking shared queue through methods such as pointer passing or rvalue passing; the blocking shared queue then enqueues the data.

[0051] For data transfer tasks, blocking shared queues transfer data ownership to threads through methods such as pointer passing or rvalue passing; blocking shared queues then dequeue data.

[0052] The thread unlocks the synchronized read-write lock.

[0053] When using a non-blocking shared queue, the data operation process is as follows: The thread that is storing data requests the data to be stored in the non-blocking shared queue.

[0054] A non-blocking shared queue atomically increments its internal atomic index to obtain a candidate storage location.

[0055] The non-blocking shared queue performs a Compare-And-Swap (CAS) atomic operation, checking whether the position has been preempted by other threads. If not preempted, the thread is allowed to store or remove data at that position, and the index position information is updated; if preempted, the index is retrieved again and the process is retried.

[0056] Each data writing thread runs concurrently after startup, and each loops through the following steps: (1) Obtain an available encoded image block from the shared queue Q1 and take ownership of the data for that entry.

[0057] (2) Write the binary data of the obtained encoded image block to the specified external storage path and write the file data into the external storage.

[0058] (3) After completing the write operation to external storage, release the current data resources and continue to the next step.

[0059] Blocking mode: When a thread retrieves data from Q1, it remains in a waiting state until there is available data in the queue; when there is available data, it acquires the synchronization lock, retrieves the data, and performs a write operation to external storage; after completing the write operation to external storage, it releases the holding rights and continues to the next task.

[0060] Non-blocking mode: When a thread retrieves data from Q1, it does not wait for the queue state and attempts to directly obtain an available data entry; after completing the write operation to external storage, it releases ownership through an atomic operation and continues to the next task.

[0061] The thread repeats the above steps until all encoded image blocks in Q1 have been processed.

[0062] When processing datasets consisting of numerous fragmented image files, existing technologies often face additional system overhead due to repeated and frequent external storage read / write operations. When the number of images is large and the size of individual files is small, the system must frequently perform disk seeks and data read / write operations, each I / O request accompanied by significant latency and context switching costs. Especially when the data distribution is relatively random or the storage medium itself has limited I / O concurrency capabilities, a large number of small file read requests can easily cause repeated disk arm swings or uneven load distribution on SSD read / write units, further exacerbating access latency. This not only weakens the access locality advantage originally expected from fragmented storage but also increases the overall I / O wait time required for the system to process a unit of data, severely restricting the efficiency of the data loading phase.

[0063] To mitigate the negative impact of fragmented files on I / O efficiency, this invention constructs a novel file structure for aggregating fragmented files. After aggregating the encoded images into a binary file (denoted as a Bin file), the data is written to external storage. Within the binary file, the binary data of multiple encoded image blocks are stored contiguously.

[0064] Specifically, such as Figure 2 As shown, the corresponding storage task is executed in each write thread in the following manner: Obtain the binary data of each encoded image block under the corresponding storage task from the shared queue Q1; calculate the CRC check code of the binary data of each encoded image block under the corresponding storage task, and fill it after the binary data of the corresponding encoded image block as the target data of the corresponding image block; Create a binary file and its corresponding metadata text file in external storage; Open the binary file and write the target data of each image block under the corresponding storage task to the binary file sequentially; before writing the target data of each image block, obtain the used capacity C of the current binary file and calculate the remaining capacity of the binary file. When the target data size of the current image patch is greater than At the same time, expand the capacity of the binary file. (In this embodiment, the capacity of the binary file is expanded by a preset multiple) After writing the target data for each image block, update the used capacity C of the current binary file, and write the name of the image block, its storage location in the binary file, and the size of the target data to the metadata text file.

[0065] Specifically, Each data writing thread runs concurrently after startup, and each loops through the following steps: 1. Initialization Phase: A compressed image large data aggregation file (BIN file) and its corresponding metadata information file (txt or json) are pre-created for each data writing thread. The initial size of the binary file is estimated based on the amount of data handled by each thread. The file is created in external storage. The internal organization of a binary file consists of the binary contents of multiple compressed images stored contiguously. 2. Establish Each thread writes data and specifies a shared queue Q1 as the data source for these threads. If a blocking queue is used, a synchronization lock is set in front of the queue; if a non-blocking queue is used, atomic operations are introduced to maintain the read and write indexes of the queue.

[0066] 3. Each data writing thread runs concurrently after startup, and each thread executes the following steps in a loop: (1) Obtain a copy of the encoded image block's binary data from the shared queue Q1 and claim ownership of the data. Calculate the CRC checksum of the encoded image block's binary data and append it to the end of the corresponding encoded image block's binary data as the target data for that image block; (2) Open the binary file and its metadata information file corresponding to the thread, and read the end position P of the current binary file to obtain the used capacity C of the current binary file.

[0067] (3) Based on the current capacity of the binary file Calculate the remaining capacity of a binary file .like If the data is large enough to hold the data to be written, then P is used as the starting position for writing, and the target data of the corresponding image block is written to the binary file; if If insufficient, the binary file size will be expanded by a preset multiple. And continue writing operations with the expanded capacity.

[0068] (4) Record the name of the current image block, its storage location in the binary file, and the size of the target data in the metadata information file.

[0069] (5) Release possession of the current data and proceed to the next step.

[0070] (6) The thread repeats the above steps until all tasks are completed.

[0071] Blocking mode: When a thread retrieves encoded image block data from Q1, it remains in a waiting state until there is available data in the queue; after retrieving the data, it executes the write process and uses a synchronization lock to ensure that the file write order is consistent when appending metadata; after completing the current round of writing, it continues to the next round of tasks.

[0072] Non-blocking mode: When a thread retrieves data from Q1, it does not wait for the queue state but attempts to obtain an available data entry using atomic indexes. After obtaining the data, it performs a write operation and manages the file offset and capacity expansion process through atomic operations. After completing the write, it continues to the next task.

[0073] The transfer of data ownership and release of possession mentioned above can be implemented using rvalue passing or pointer passing in C++.

[0074] 4. The thread continuously executes step 3 until all encoded image block binary data in shared queue Q1 has been processed. After all data has been written, the data writing thread saves and closes the corresponding binary file and its metadata information file, and then terminates the process.

[0075] In this embodiment, the expansion of the remaining capacity of the binary file is implemented as follows: The system first checks whether there is enough contiguous available space at the end of the existing binary file. If conditions permit, a contiguous allocated area is directly added to the end of the file. If the original location cannot meet the demand, a sufficiently large free area is found on the external storage medium, and the original data is migrated to the new location. After the migration is completed, new storage space is expanded. In contrast to expansion, when the actual data volume of the binary file is less than the initially reserved capacity, the system also supports shrinking it. Shrinking is implemented as follows: The system first confirms the actual boundary position of the valid data in the file, and then releases the unused redundant area at the end, thereby reducing the physical size of the file. During this process, the logical position of the original valid data remains unchanged; the system only removes excess padding space, avoiding waste of storage resources. By pre-allocating space and reallocating a larger space when space is insufficient, efficiency is traded for space, avoiding the time-consuming operation of frequently expanding the file capacity. The expansion size can be allocated according to a certain proportion of the original file size (e.g., 1.5 times, 2 times) or by estimating the progress of the currently encoded blocks, to reserve sufficient write capacity.

[0076] Secondly, the present invention provides a method for reading large image data files, including: create One read thread and One decoding thread; ; ; use Each read thread executes image block reading operations in parallel until the encoded binary data of all image blocks in the large image data file to be read has been read. The large image data file to be read is a file stored in external storage using the storage method provided in the first aspect of this invention. The image block reading operation performed in each read thread includes: reading the binary file and corresponding metadata text file of the large image data file from external storage, and storing the binary file in a shared queue Q2 in memory; obtaining the names of the image blocks recorded in the binary file, their storage locations in the binary file, and the size of the target data from the corresponding metadata text file, and then dividing the binary file in queue Q2 into target data for each image block; for the target data of each image block: At the same time, utilizing Each decoding thread reads the encoded binary data of the image patch from queue Q2 in parallel and decodes it.

[0077] In one alternative implementation, when the external memory read speed At lower speeds (i.e., external storage read speeds) (when less than the second preset speed) When the external storage read speed At higher speeds (i.e., external storage read speeds) (when the speed is greater than the third preset speed) The second preset speed is less than the third preset speed.

[0078] Preferably, in one optional implementation, The number of binary files in the large image data file to be read is the smaller of the number of physical cores of the CPU processor.

[0079] Preferably, in one optional implementation, This refers to the number of physical cores in the CPU processor.

[0080] In one alternative implementation, the queue Q2 is either a non-blocking queue or a blocking queue.

[0081] Preferably, in one optional implementation, when the external memory read speed When the ratio of large image data file size to memory capacity is relatively small, i.e., when the external storage read speed is high... When the speed is less than the fourth preset speed and the ratio of the size of the image big data file to the memory capacity is less than the preset ratio, the above queue Q2 will preferentially use a non-blocking queue.

[0082] In one optional implementation, when the image big data file to be read is a file stored in external storage that carries a checksum, the image block reading operation performed in each reading thread further includes: For the target data of each image block: use the checksum at the end as the first checksum, and use the remaining binary data excluding the checksum at the end as the encoded binary data of the image block; calculate the checksum of the encoded binary data of the image block as the second checksum; compare the first checksum with the second checksum to perform integrity verification.

[0083] It should be noted that the above-mentioned image big data file to be read can be a file stored in external storage using the storage method provided in the first aspect of the present invention (referred to as the first case), or a file that has been stored in external storage using the storage method provided in the first aspect of the present invention and then migrated to another external storage through file copying (such as copying to a medium such as a USB flash drive or a disk) or transmission operation (such as downloading from the network) (referred to as the second case). There is no limitation here.

[0084] For the above integrity verification, the failure of the verification often occurs in the second case. Considering that it may be due to the corruption of the binary file during the transmission process, the owner of the image big data file is usually notified to recopy or retransmit the binary file containing the image block and its corresponding metadata information file.

[0085] To further illustrate the method for reading large image data files provided by the present invention, a specific embodiment is described in detail below: like Figure 3 As shown, the method for reading large image data files provided in this embodiment includes the following parts: 1. Initialization Phase: Detect external storage read speed and specify the required number of data read threads. and the number of threads for decoding compressed data If the external storage uses a mechanical hard drive with a speed of 20-200 MBps, then set... If the external storage uses a solid-state drive with a speed of 200MBps or higher, then set... In both cases, The number of physical cores in the CPU processor. A list L of paths to all images to be read, used for allocating tasks among subsequent data loading threads.

[0086] 2. Establish One data read thread and One data decoding thread. Create a shared queue Q2. If a blocking queue is used, set a synchronization lock in front of the queue; if a non-blocking queue is used, introduce atomic operations to maintain the queue's read and write indexes.

[0087] 3. Each data reading thread runs concurrently after startup, and each thread executes the following steps in a loop: (1) Obtain the target compressed image file path through the pre-allocation in step 2 or the atomic lock of list L in step 1, load the binary data after image block encoding from external storage, and store it in the memory array RL.

[0088] (2) Transfer ownership of the data in RL to the shared queue Q2, complete the data loading, release RL, and continue to the next step.

[0089] (3) The thread repeats the above steps until all tasks are completed.

[0090] Blocking mode: When a thread transfers data to Q2, it remains in a waiting state until the queue allows enqueueing; when enqueueing is allowed, the thread acquires the synchronization lock, transfers the data to Q2; after enqueueing is complete, it releases ownership of the data and continues to the next task.

[0091] Non-blocking mode: When a thread transfers data to Q2, it does not wait for the queue status and directly puts the data into the queue through atomic operations.

[0092] The "transfer of data ownership" mentioned above, and which will also appear below, provides one way to achieve this: rvalue passing and pointer passing in C++.

[0093] 4. Each data decoding thread runs concurrently after startup, and each thread executes the following steps in a loop: (1) Obtain an available data entry from the shared queue Q2 and acquire data ownership of that entry.

[0094] (2) Perform image decoding operation on the acquired data to generate decoding results.

[0095] (3) Write the decoding result to the write index position of queue Q3 to complete the data placement; release the ownership and memory and continue to the next step.

[0096] Blocking mode: When a thread retrieves data from the shared queue Q2, it remains in a waiting state until there is available data in the queue; when there is available data, the thread acquires the synchronization lock and transfers the data to Q2; after decoding is completed, the result is placed into the queue Q3 in the same blocking mode.

[0097] Non-blocking mode: When a thread retrieves data from Q2, it does not wait for the queue state and attempts to retrieve an available data entry; after decoding, it uses an atomic operation to put the result into queue Q3.

[0098] The thread repeats the above steps until all data in Q2 has been processed.

[0099] To mitigate the negative impact of fragmented files on I / O efficiency, the first aspect of this invention constructs a novel file structure for aggregating fragmented files, where encoded images are aggregated into binary files and then written to external storage. For example... Figure 4 As shown, in this embodiment, the process of reading a file stored in external storage using the storage method provided in the first aspect of the present invention is as follows: 1. Initialization Phase. Specify the list of binary files L_b to be read, read the corresponding metadata file (txt or json), and parse the recorded image patch names, their storage locations within the binary files, and the size of the target data. The number of data reading threads is determined based on the number of binary files and the number of CPU physical cores. ,in And set the number of data decoding threads. This refers to the number of CPU cores. If... Then each data reading thread is allocated a binary file; if Then, multiple binary files are allocated to each thread according to the principle of average distribution. The number of binary files in the large image data file to be read.

[0100] 2. Establish One data read thread and One data decoding thread. A shared queue Q2 is created as the storage area for the read results. If a blocking queue is used, a synchronization lock is set in front of the queue; if a non-blocking queue is used, atomic operations are introduced to maintain the queue's read and write indexes.

[0101] 3. Each data reading thread runs concurrently after startup, and each thread executes the following steps in a loop: (1) Based on the list of binary files it is responsible for, retrieve the corresponding metadata information file content to obtain the name of each image block, its storage location in the binary file, and the size of the target data.

[0102] (2) Determine the loading method based on the compression ratio of the binary file: (a) When the compression ratio is high, the data reading thread reads the entire binary content of the binary file from the external storage at once and stores it in the RL array. Then, based on the metadata information, it maps and records the segment range of each image data in the RL in the internal structure of this thread. The thread then transfers the data ownership of the RL to the shared queue Q2 and continues to load the next task.

[0103] (b) When the compression ratio is low, the data reading thread reads the extended data of the binary file one by one in the image block, and puts the remaining binary data in the extended data except for the check code at the end into the RL array; then the data ownership of RL is transferred to queue Q2, and the data loading task of the next image block continues.

[0104] (3) The thread repeats the above steps until all binary files it is responsible for have been processed.

[0105] Blocking mode: When a thread transfers data to Q2, it remains in a waiting state until the queue allows enqueueing; when enqueueing is allowed, it acquires the synchronization lock, writes the data to queue Q2, releases ownership of the data after enqueueing is complete, and continues to execute the next task.

[0106] Non-blocking mode: When a thread transfers data to queue Q2, it does not wait for the queue state, but writes the data directly to queue Q2 through atomic operations, and continues subsequent tasks after the data placement is completed.

[0107] The transfer of data ownership mentioned above can be achieved using rvalue passing or pointer passing in C++.

[0108] The embodiments provided by this invention aggregate massive amounts of small files into contiguous binary files and combine them with metadata text files to achieve efficient read and write operations, while also supporting dynamic expansion when the binary file capacity is insufficient. Furthermore, with a sufficiently high compression ratio, it allows for batch reading of complete binary file data into a buffer followed by decomposition and decoding; or batch decoding followed by storage in a buffer, and then writing to the external storage binary file in smaller, more frequent increments. This strategy significantly reduces the inefficient operation time of external storage I / O.

[0109] 4. Each data decoding thread runs concurrently after startup, and each thread executes the following steps in a loop: (1) Obtain the available data entry from the shared queue Q2 and take ownership of the data.

[0110] (2) Based on the metadata information corresponding to the data, perform image decoding operation and generate decoding results.

[0111] (3) Place the decoding result at the write index position of the shared queue Q3 to complete the data placement and release the possession.

[0112] (4) The thread repeats the above steps until all data in queue Q2 has been processed.

[0113] Blocking mode: When a thread retrieves data from queue Q2, it waits until the queue contains available data; after retrieving the data, it performs decoding and writes the result to queue Q3 in a blocking mode.

[0114] Non-blocking mode: When a thread retrieves data from queue Q2, it does not wait for the queue state and attempts to obtain an available entry through atomic operations; after decoding, it writes the data to queue Q3 in a non-blocking manner.

[0115] In one specific implementation of this embodiment, an example of a read process without a lock mechanism is provided, including: After system startup, multiple data reading threads, data decoding threads, and a non-blocking shared queue Q2 are established. The queue maintains read and write indexes through atomic operations.

[0116] The data reading thread reads encoded image blocks from external storage according to the task allocation. After each thread completes loading one piece of data, it uses the atomic enqueue operation provided by the queue to put the data into the non-blocking shared queue Q2, and continues to process the next piece of allocated data until all tasks are completed.

[0117] The data decoding thread repeatedly performs atomic dequeue operations to retrieve data from Q2. After decoding the retrieved data, the thread atomically enqueues the decoding result into a non-blocking shared queue Q3, and then continues to retrieve the next piece of data until all data in Q2 has been processed.

[0118] In another specific implementation of this embodiment, the system uses a pre-caching structure to accelerate data flow when reading and writing binary files. During the reading process, the system first pre-allocates a large contiguous cache area in memory and reads the binary file from external storage into this cache area as a whole. Subsequently, each decoding thread retrieves data from the cache sequentially and executes the decoding process.

[0119] In summary, addressing the performance bottleneck caused by repetitive and frequent external storage read / write operations resulting from massive amounts of fragmented image data files, this invention aims to fundamentally reduce the number of I / O requests through innovation in data format and access patterns. By designing an integrated big data format, a large number of small files are merged and stored in a few pre-allocated large-capacity data files (BINs), thereby transforming a large number of random, fine-grained read / write operations into sequential, large-volume read / write operations. This effectively reduces disk addressing overhead and I / O scheduling latency, enabling efficient and stable streaming transmission of data from storage media to computing units.

[0120] Meanwhile, addressing the issue of low system resource utilization caused by the inherent serial execution mode between input / output (I / O) operations and data encoding / decoding operations in the data loading and saving process, this invention completely decouples the traditional "read first, then decode" or "encode first, then write" serial process. It allocates sub-processes such as reading, writing, decoding, and encoding to fixed threads and schedules them in a parallel asynchronous pipeline. This ensures that each work unit can continuously process its own tasks, thereby maximizing the utilization of computing and I / O resources and significantly improving data processing throughput.

[0121] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first or second aspect of the present invention.

[0122] The related technical solutions are the same as those provided in the first and second aspects of this invention, and are not limited here.

[0123] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for storing large image data files, characterized in that, include: The large image data file to be stored is converted into a multi-resolution pyramid structure, and the images at each resolution in the pyramid structure are divided into blocks of the same size. create One write thread and One coding thread; ; ; use Each encoding thread encodes each image block in parallel and stores them in a shared queue Q1 until all image blocks of the large image data file to be stored have been encoded. Simultaneously, the storage task for each encoded image block is divided into... Groups were created and assigned one-to-one with each other. In a single write thread; using Each write thread executes its corresponding storage task in parallel; The process of executing the corresponding storage task in each write thread includes: Obtain the encoded binary data of each image block under the corresponding storage task from the shared queue Q1, and use it as the target data of the corresponding image block; Create a binary file and its corresponding metadata text file in external storage; Open the binary file and write the target data of each image block under the corresponding storage task to the binary file sequentially; before writing the target data of each image block, obtain the used capacity C of the current binary file and calculate the remaining capacity of the binary file. When the target data size of the current image patch is greater than At the same time, expand the capacity of the binary file. After writing the target data for each image block, update the used capacity C of the current binary file, and write the name of the image block, its storage location in the binary file, and the size of the target data to the metadata text file.

2. The method for storing large image data files according to claim 1, characterized in that, Also includes: After each write thread completes its corresponding storage task, the remaining capacity of the current binary file is determined. If the capacity redundancy exceeds the preset threshold, then release the unused areas in the current binary file.

3. The method for storing large image data files according to claim 1, characterized in that, The shared queue Q1 can be a non-blocking queue or a blocking queue; when the external storage speed... When the speed exceeds the first preset speed, the shared queue Q1 is selected as a non-blocking queue.

4. The method for storing image big data files according to any one of claims 1, characterized in that, The encoding is JPEG encoding, LZW encoding, PackBits encoding, Zstd encoding, LZ4 encoding, or DEFLATE encoding.

5. The method for storing image big data files according to any one of claims 1-4, characterized in that, The process of executing the corresponding storage task in each write thread also includes: After obtaining the encoded binary data of each image block under the corresponding storage task from the shared queue Q1, the check code of the encoded binary data of each image block under the corresponding storage task is calculated and filled after the encoded binary data of the corresponding image block to update the target data of the corresponding image block.

6. A method for reading large image data files, characterized in that, include: create One read thread and One decoding thread; ; ; use Each read thread performs image block reading operations in parallel until the encoded binary data of all image blocks in the large image data file to be read has been read. The large image data file to be read is a file stored in external storage using the image data file storage method described in any one of claims 1-5. The image block reading operation performed in each read thread includes: reading the binary file of an image data file and its corresponding metadata text file from external storage, and storing the binary file in a shared queue Q2 in memory; obtaining the name of the image block recorded in the binary file, its storage location in the binary file, and the size of the target data from the corresponding metadata text file, and then dividing the binary file in the shared queue Q2 into target data for each image block. At the same time, utilizing Each decoding thread reads the encoded binary data of the image patch from the shared queue Q2 in parallel and decodes it.

7. The method for reading large image data files according to claim 6, characterized in that, When external storage read speed When the speed is less than the second preset speed, ; When external storage read speed When the speed is greater than the third preset speed, The second preset speed is less than the third preset speed.

8. The method for reading large image data files according to claim 6, characterized in that, The shared queue Q2 can be a non-blocking queue or a blocking queue; when the external storage read speed When the speed is less than the fourth preset speed and the ratio of the size of the image big data file to the memory capacity is less than the preset ratio, the shared queue Q2 is selected as a non-blocking queue.

9. The method for reading large image data files according to any one of claims 6-8, characterized in that, When the image big data file to be read is a file stored in external storage using the image big data file storage method described in claim 5, the image block reading operation executed in each reading thread further includes: For the target data of each image block: use the checksum at the end as the first checksum, and use the remaining binary data excluding the checksum at the end as the encoded binary data of the image block; calculate the checksum of the encoded binary data of the image block as the second checksum; compare the first checksum with the second checksum to perform integrity verification.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method according to any one of claims 1-9.

Images