Neuron pair point cloud extraction acceleration system and method based on sparse perception

Abstract

This invention discloses an accelerated system and method for neuron pairwise point cloud extraction based on sparse perception. These are corresponding solutions that can penetrate compression coding barriers, intelligently avoid background data in the pure compression domain, decompress on demand, and fully utilize the extreme sparsity of point cloud extraction for faster extraction. Specifically, the solution introduces extremely lightweight empty block identifiers and non-zero identifiers to support fast sparse filtering of target neuron fragments in the non-decompressed state. Furthermore, it can directly compare and filter target neuron identifiers in the palette dictionary of the compression domain without instantiating a large number of dense background voxels, thereby eliminating the request amplification effect during the data loading stage, significantly reducing memory bandwidth consumption, maximizing the performance of modern hardware, and achieving efficient execution of the neuron pairwise point cloud extraction task.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of large-scale brain science data processing, connectomics, computational neuroscience and high-performance parallel distributed computing, and in particular to an accelerated system and method for extracting paired point clouds of neurons based on sparse perception. Background Technology

[0002] In the cutting-edge field of modern brain science and neurobiology research, connectomics aims to comprehensively reconstruct the billions of neurons and their synaptic connections in the biological brain at nanometer-scale resolution. With the development of high-throughput three-dimensional electron microscopy imaging technology, the size of raw image datasets of single biological brain tissues has exceeded the petabyte (PB) level. To efficiently store and analyze these massive amounts of data, they are typically pre-cut into three-dimensional blocks and encoded using complex compression algorithms (such as the Compressed-Segmentation format, a segmented compression format). Researchers need to frequently initiate random queries to the storage backend to extract paired point clouds and surface meshes of specific neurons in three-dimensional space.

[0003] However, existing extraction systems face extremely severe dual bottlenecks when handling the above tasks: Bottleneck 1: Request Amplification Effect. Existing systems typically employ a static, coarse-grained data loading strategy. When a user specifies a tiny bounding box to query a segment of extremely sparse neuron branches, the system is forced to completely load the vast block reached by the bounding box into main memory and trigger full voxel decompression. Since the actual target neuron often accounts for less than 5% of the total voxels, this full decompression mechanism instantly inflates the original tiny query into an enormous, dense matrix, causing the memory bus to be occupied by a large number of irrelevant background voxels, resulting in significant bandwidth waste.

[0004] Bottleneck 2: Lack of sparsity awareness. After completing the expensive full decompression, downstream operators of the system (such as mask generation and breadth-first search) still treat the massive dense tensor as a uniform data field, launching a brute-force traversal computation across the entire space. Because the sparsity distribution information inherent in the compressed state itself is discarded, a massive amount of CPU (central processing unit) computation is needlessly wasted in the background zero-value region, which not only worsens the end-to-end latency of a single extraction, but also causes subsequent queued query tasks to fall into long-tail blocking.

[0005] Furthermore, traditional distributed storage clients generally rely on multi-threaded pool concurrency mechanisms. However, when dealing with fragmented and highly localized tasks such as point cloud extraction, if the underlying system is equipped with a high-speed solid-state drive (SSD), the large thread pool environment will introduce huge system overhead due to thread wake-ups, context switching, and mutex lock contention, leading to severe performance degradation.

[0006] In summary, academia and industry urgently need a point cloud extraction acceleration solution that can penetrate compression coding barriers, intelligently avoid background data in the pure compression domain, decompress on demand, and make full use of extreme sparsity, fundamentally breaking down memory walls and improving the end-to-end execution efficiency of the extraction pipeline.

[0007] In view of this, the present invention is hereby proposed. Summary of the Invention

[0008] The purpose of this invention is to provide a system and method for accelerating point cloud extraction of paired neurons based on sparse perception. This system can penetrate the compression coding barrier, intelligently avoid background data in the pure compression domain, decompress on demand, and make full use of the extreme sparsity of the data to accelerate point cloud extraction.

[0009] The objective of this invention is achieved through the following technical solution: A system for accelerating the extraction of paired point clouds of neurons based on sparse sensing, comprising: The data layer is used to store the raw compressed block data involved in the request bounding box read from the storage cluster, wherein the raw compressed block data is a compressed representation of a three-dimensional connected omics voxel segmentation image block; The C++ execution layer adopts a block-level memory architecture. It uses a preset structure object to retain the original compressed block data format and introduces empty block identifiers and non-zero identifiers. At the same time, it configures a sparse perception operator suite to complete the point cloud extraction task under the call of the Python interface layer. Among them, the empty block identifier is used to mark whether all three-dimensional connectivity voxels inside the block are background mask values, and the non-zero identifier is used to mark whether the block contains neuron identifiers that meet the extraction conditions. The Python interface layer processes point cloud extraction tasks from upper-layer applications by calling the C++ execution layer. This includes calling the sparse perception operator suite of the C++ execution layer, which uses empty block identifiers and non-zero identifiers to skip raw compressed block data that is irrelevant to the target neuron identifier. For non-empty raw compressed block data containing the target neuron identifier, the matching and comparison of the target neuron identifier is performed directly in the palette dictionary array contained therein, thereby completing the spatial localization and feature extraction of the point cloud. Both C++ and Python are computer programming languages.

[0010] An accelerated system for extracting paired point clouds of neurons based on sparse sensing, based on the aforementioned system implementation, includes the following methods: The data layer stores the raw compressed block data involved in the request bounding box read from the storage cluster. The raw compressed block data is a compressed representation of a three-dimensional connected omics voxel segmentation image block. By employing a block-level memory architecture in the C++ execution layer, and utilizing a pre-defined structure object to preserve the original compressed block data format, empty block identifiers and non-zero identifiers are introduced. Simultaneously, a sparse perception operator suite is configured to complete the point cloud extraction task under the call of the Python interface layer. Among them, the empty block identifier is used to mark whether all three-dimensional connectomics voxels inside the block are background mask values, and the non-zero identifier is used to mark whether the block contains neuron identifiers that meet the extraction conditions. The point cloud extraction task from the upper application is processed by calling the C++ execution layer through the Python interface layer. This includes calling the sparse perception operator suite of the C++ execution layer, which uses empty block identifiers and non-zero identifiers to skip the original compressed block data that is irrelevant to the target neuron identifier, and directly performing the matching and comparison of the target neuron identifier in the palette dictionary array contained in the non-empty original compressed block data containing the target neuron identifier, thereby completing the spatial localization and feature extraction of the point cloud. Both C++ and Python are computer programming languages.

[0011] As can be seen from the technical solution provided by the present invention, by introducing extremely lightweight empty block identifiers and non-zero identifiers, it supports fast sparse filtering of target neuron fragments in a non-decompressed state. Furthermore, it can directly perform target comparison and filtering location in the palette dictionary of the compressed domain without instantiating a large number of dense background voxels, thereby eliminating the request amplification effect in the data loading stage from the root, significantly reducing memory bandwidth consumption, and maximizing the performance of modern hardware, thus achieving efficient execution of the neuron pair point cloud extraction task. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a neuron-based paired point cloud extraction acceleration system provided in an embodiment of the present invention.

[0014] Figure 2This is an overview diagram of the overall system architecture provided in the embodiments of the present invention.

[0015] Figure 3 The core data structure and mapping diagram of the block-level memory architecture provided in the embodiments of the present invention.

[0016] Figure 4 The flowchart for on-demand sparse loading execution provided in the embodiments of the present invention is shown.

[0017] Figure 5 This is a flowchart illustrating the execution of sparse-aware masking in an embodiment of the present invention.

[0018] Figure 6 This is a flowchart of seed-driven in-situ isolated execution provided in an embodiment of the present invention.

[0019] Figure 7 This is a flowchart of the compressed domain contour extraction process provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0021] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0022] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0023] The following is a detailed description of the accelerated system and method for extracting paired point clouds of neurons based on sparse sensing provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all conventional products that can be purchased commercially.

[0024] Example 1 like Figure 1 As shown, an embodiment of the present invention provides a sparse-aware neuron pairwise point cloud extraction acceleration system, which mainly includes: (1) a data layer, used to store the original compressed block data involved in the request bounding box read from the storage cluster. The original compressed block data is a compressed representation of the three-dimensional connectomics voxel segmentation image block. Each original compressed block data corresponds to a structure object; (2) a C++ execution layer, which adopts a block-level memory architecture. It uses a preset structure object to introduce empty block identifiers and non-zero identifiers while retaining the format of the original compressed block data. At the same time, it configures a sparse-aware operator suite to complete the point cloud extraction task under the call of the Python interface layer. Among them, the empty block identifier is used to mark whether all three-dimensional connectomics voxels inside the block are background mask numbers. The value, the non-zero identifier is used to mark whether the block contains a neuron identifier that meets the extraction conditions; (3) Python interface layer, which provides docking shielding across language boundaries, transparently exposes the structure object of the C++ execution layer to the upper layer application, and calls the C++ execution layer to process the point cloud extraction task from the upper layer application, including: calling the sparse perception operator suite exposed by the C++ execution layer, the sparse perception operator suite calls the empty block identifier and the non-zero identifier to skip the original compressed block data that is unrelated to the target neuron identifier, and for the non-empty original compressed block data containing the target neuron identifier, the matching and comparison of the target neuron identifier is directly performed in the palette dictionary array contained therein, thereby completing the spatial localization and feature extraction of the point cloud; where C++ and Python are both computer programming languages.

[0025] In this embodiment of the invention, the original compressed block data is encoded using a compressed segmented format, which includes bit width information, a palette dictionary array, and a compressed bit stream.

[0026] In this embodiment of the invention, the system executes a three-layer hierarchical avoidance framework when processing point cloud extraction tasks: The first layer is a block-granularity zero-decompression layer, which is implemented through a sparse-aware operator suite in the C++ execution layer. It retains the original compressed block data read by the third layer as the basic storage unit, and directly skips the original compressed block data that is not intersecting outside the requested bounding box specified in the point cloud extraction task through three-dimensional geometric bounding box mapping calculation, thus maintaining the compressed representation of the original compressed block data and not triggering decompression; The second layer is a sparse-aware skip layer, which is implemented through a sparse-aware operator suite in the C++ execution layer, utilizing empty block identifiers and non-zero... The first layer identifies whether the original compressed block data contains the target neuron. For non-empty original compressed block data containing the target neuron identifier, the matching and comparison of the target neuron identifier is directly performed in the palette dictionary array contained within it. A zero-copy proxy data structure is dynamically constructed for the matched original compressed block data. Finally, the spatial localization and feature extraction of the point cloud are completed. The third layer is a single-threaded I / O read optimization layer, which is configured with a deterministic single-threaded cache reader. The original compressed block data involved in the request bounding box is read sequentially in a single thread according to the spatial physical storage order in the storage cluster. Here, I / O refers to input / output.

[0027] The aforementioned three-layer hierarchical avoidance framework is completed collaboratively by the data layer and the C++ execution layer. The first two layers reside in memory, while the third layer is responsible for reading data into memory. The first layer, using a sparsity-aware operator suite from the C++ execution layer, skips disjoint raw compressed block data based on the geometric relationship between the request bounding box and block coordinates, thus achieving zero-decompression filtering at the block level. The second layer, also using the sparsity-aware operator suite from the C++ execution layer, performs sparsity-aware judgment based on empty block identifiers, non-zero identifiers, and a palette dictionary array, and constructs a zero-copy proxy data structure for the matched raw compressed block data, enabling subsequent processes to directly access the compressed domain data. The third layer, through a deterministic single-threaded reader from the data layer, reads the raw compressed blocks sequentially according to the physical storage order, avoiding the scheduling and synchronization overhead caused by multi-threaded I / O. Therefore, irrelevant data is avoided step-by-step in the three stages of data reading, candidate block filtering, and subsequent C++ execution, reducing full decompression, data copying, and invalid computation.

[0028] In this embodiment of the invention, the zero-copy proxy data structure includes: a pointer to the original compressed state block data block palette dictionary array, a pointer to the compressed bit stream, and a non-zero flag; the zero-copy proxy data structure is uniformly managed in a one-dimensional vector array, and the sparse awareness operator suite performs native C++ computation by iterating over the zero-copy proxy data structure.

[0029] In this embodiment of the invention, the preset structure object includes a global master data structure (StoredBlock). The global master data structure, while retaining the original compressed block data format, additionally introduces an empty block identifier and a non-zero identifier. The empty block identifier and the non-zero identifier are both Boolean variables. When the original compressed block data is first loaded into memory and when it encounters an update operation, the two are automatically configured and initialized by the sparsity awareness mechanism at the system's underlying level.

[0030] In this embodiment of the invention, the C++ execution layer further includes a block central manager structure (BlockArena), which centrally stores the global master data structure and the corresponding zero-copy proxy data structure of all instantiated blocks in the form of a contiguous memory vector.

[0031] The instantiated block here represents a dense matrix of data obtained after decompressing the original compressed block data, representing the voxel data of that region; specifically, by accessing the palette dictionary array and bit width information, the decompressed value of each compressed bit stream (compressed binary data stream) is obtained, and dense matrix data is generated.

[0032] In this embodiment of the invention, the sparse-aware operator suite includes: an on-demand sparse loading module, a sparse-aware masking module, a seed-driven in-situ isolation module, and a compressed domain contour extraction module. The on-demand sparse loading module corresponds to the first and second layers of the aforementioned three-layer hierarchical avoidance framework and can invoke the third layer to obtain the original compressed state block data involved in the requested bounding box. The sparse-aware masking module, the seed-driven in-situ isolation module, and the compressed domain contour extraction module correspond to the second layer of the aforementioned three-layer hierarchical avoidance framework.

[0033] (1) On-demand sparse loading module. Its inputs include: request bounding box, two target neuron identifiers and original compressed block data; the original compressed block data related to the two target neuron identifiers is filtered by request bounding box, a global master data structure is created for the original compressed block data that intersects with the request bounding box, forming a master data structure array, and different non-zero identifiers are configured according to whether the original compressed block data contains target neuron identifiers. At the same time, a corresponding zero-copy proxy data structure is generated for the original compressed block data containing target neuron identifiers, forming a zero-copy proxy array.

[0034] The on-demand sparse loading module's operation includes: acquiring the original compressed block data involved in the request bounding box read by the deterministic single-threaded cache reader; calculating the physical coordinates corresponding to the original compressed block data and performing intersection calculations with the request bounding box; for original compressed block data with overlapping boundaries, applying a clipping mask operation to erase the 3D connectomics voxels extending outside the request bounding box boundary on the palette dictionary array and forcibly overwrite them with background mask values; then, directly reading the palette dictionary array of the original compressed block data and matching it with the target neuron identifier, determining the original compressed block data containing the target neuron identifier based on the matching result; if an original compressed block data does not contain two... If any one of the target neuron identifiers is present, it belongs to the original compressed block data without the target neuron. If it contains any target neuron identifier, it belongs to the non-empty original compressed block data containing the target neuron. For the original compressed block data that overlaps with the request bounding box, a global master data structure is constructed in the block collection manager of the C++ execution layer. The non-zero identifier of the corresponding global master data structure is configured so that the original compressed block data containing the target neuron identifier and the original compressed block data without the target neuron identifier have different non-zero identifiers. The corresponding zero-copy proxy data structure is added to the synchronous atomication of the original compressed block data containing the target neuron identifier to form a zero-copy proxy array.

[0035] (2) Sparse perception masking module, whose input is the main data structure array and zero-copy proxy array output by the on-demand sparse loading module. By accessing the global main data structure and zero-copy proxy data structure, it performs in-situ masking on the palette dictionary array, overwrites the non-target neuron identifiers with the background mask values, and outputs the compressed domain target mask block set (or the target neuron compressed domain fragment set).

[0036] The sparse-aware masking module works as follows: It reads the global master data structure from the master data structure array as the data for the current operation; it calculates the memory index of the corresponding original compressed state block data based on the coordinate mapping of the data for the current operation; it queries the non-zero identifiers of the original compressed state block data corresponding to the memory index, and if no target neuron identifier is found, it exits the processing of the corresponding original compressed state block data; for non-empty original compressed state block data containing target neuron identifiers, it obtains a pointer to the palette dictionary array through a zero-copy proxy data structure, initiates a sequential scan of the palette dictionary array, and overwrites the non-target neuron identifiers in the currently operated data with background mask values, completing the in-situ logical removal in the compressed domain state. After processing all global master data structures in the master data structure array, the target mask block set for the compressed domain is obtained.

[0037] (3) Seed-driven in-situ isolation module, whose input is the compressed domain target mask block set output by the sparse perception mask module. By locating the effective starting anchor point and performing sparse-guided breadth-first search based on the anchor point, the target neuron connected branch block set (or connected domain isolation result set) is obtained.

[0038] The seed-driven in-situ isolation module operates in two phases: a sparse anchor point localization phase and a sparse-guided breadth-first search phase. The sparse anchor point localization phase includes: initiating a block-by-block traversal, traversing each global master data structure in the compressed domain target mask block set, and calling empty block identifiers and non-zero identifiers to filter the global master data structures of the original compressed block data that do not contain target neuron identifiers. The point cloud extraction task input includes: two target neuron identifiers, the center coordinates of the target neuron pair (the geometric center of the target neuron pair), and the given requested space size. For the global master data structure corresponding to the original compressed block data containing target neuron identifiers, the distance between the internal voxel coordinates and the geometric center of the target neuron pair is calculated and continuously updated. The effective neuron closest to the center is output as the starting anchor point; that is, the three-dimensional connectomics voxel closest to the geometric center of the target neuron pair is used as the effective neuron starting anchor point. The sparse-guided breadth-first search phase includes: expanding the search from the starting anchor point, calculating the non-zero identifier of the global master data structure to which the original compressed block data corresponding to the voxel coordinates of the expanded neighboring points belongs; if it is determined by the non-zero identifier that the target neuron identifier is not present, the response search branch is truncated; if it is determined that the target neuron identifier is present, the corresponding global master data structure is further verified and included in the target neuron connected branch block set, and the data in the target neuron connected branch block set are all global master data structures corresponding to the original compressed block data containing the target neuron.

[0039] In this embodiment of the invention, during the search process, blocks connected to the starting anchor point and containing the target neuron identifier are identified as target neuron connected branch blocks, and all target neuron connected branch blocks are combined into a target neuron connected branch block set. Each target neuron connected branch block is represented by a global master data structure corresponding to the original compressed block data. Specifically, a target neuron connected branch block refers to a global master data structure that, during the sparse-guided breadth-first search process, is determined to contain the target neuron identifier by a non-zero identifier and is further included in the target neuron connected branch block set after connectivity verification. This global master data structure corresponds to an original compressed block data containing the target neuron identifier, and this original compressed block data is in the same connected branch as the starting anchor point.

[0040] Preferably, the C++ execution layer is configured with a local dynamic decompression cache pool. During the sparse-guided breadth-first search phase, when the feature value of the nearest point is obtained (i.e., the coordinates of the nearest voxels to be expanded generated during the sparse-guided breadth-first search), the local dynamic decompression cache pool configured in the C++ execution layer is queried first. If a match is found, the target feature value (the voxel identifier value stored at the coordinates of the voxel to be expanded, i.e., the neuron segmentation label value) is extracted in constant time complexity. That is, the corresponding original compressed block data already exists in the local dynamic decompression cache pool of the C++ execution layer, so the corresponding voxel identifier value is directly read from the corresponding original array. If no match is found, the corresponding original compressed block data is decompressed in reverse to generate the original array and written to the local dynamic decompression cache pool for subsequent access. When the local dynamic decompression cache pool triggers the threshold (i.e., the threshold for triggering insufficient remaining size), the least recently used strategy is used to eliminate redundant original arrays.

[0041] (4) Compression domain contour extraction module, whose input is the target neuron connected branch block set output by the seed-driven in-situ isolation module. By decompressing and extracting the effective blocks corresponding to each target neuron connected branch on demand according to slices, the surface contour information of the target neuron is obtained, and finally the point cloud extraction results of paired neurons are output.

[0042] The compressed domain contour extraction module adopts a compromise execution scheme based on fine-grained layer slicing. During initialization, based on the size of a single original compressed block data on a specified axis, it requests an aligned (i.e., the same size as the original compressed block data on the specified axis) blank physical reuse cache from the operating system. Each target neuron connected branch block in the target neuron connected branch block set corresponds to a global master data structure of an original compressed block data. Contour extraction is performed layer by layer according to the slice level (processed layer by layer on the specified axis). That is, the three-dimensional space covered by the target neuron connected branch block set is divided into multiple slice levels, and the zero-copy proxy data structure corresponding to the target neuron connected branch block is used for further processing. The process involves determining whether a slice intersects with the current slice level. Specifically, when processing the current slice level, the zero-copy proxy data structure corresponding to the target neuron's connected branch block is used to determine whether it intersects with the current slice level by checking the coordinate range (e.g., by checking if the start and end ranges of the corresponding original compressed block data on a specified axis overlap with the coordinate range of the current slice level). If an intersection is determined, the original compressed block data of the corresponding target neuron's connected branch block is decompressed into a blank physical multiplexing cache. Then, the surface mesh (i.e., surface contour data) of the current slice level is generated and extracted. Afterward, a memory reset instruction is called to flush the blank physical multiplexing cache to zero, preparing it for decompression and reuse of the next slice level.

[0043] In this embodiment of the invention, the system also includes a microkernel scheduling engine located in a unified task processing container. Here, the unified task processing container refers to a collection of processes. The processes mentioned here are concurrent execution units of the Python interface layer when performing the task of extracting point clouds from paired neurons. The microkernel scheduling engine encapsulates the underlying C++ execution layer interface, shields the details of sparse index calculation and memory pool reclamation, binds the operation lifecycle, and automatically releases memory when the task ends.

[0044] In this embodiment of the invention, the above system can serve as a pluggable middleware platform, compatible with and replacing the data request interface of existing open-source point cloud processing baselines (such as Cloud Volume).

[0045] The system provided in this embodiment of the invention mainly achieves the following beneficial effects: (1) Break through the memory bandwidth bottleneck: refine the data extraction granularity to the Block level; and by introducing lightweight empty block identifiers and non-zero identifiers, high-precision filtering is achieved in the non-decompression state, which fundamentally blocks the uncontrollable request amplification effect in the storage loading stage and greatly reduces the bandwidth occupation of injecting invalid data into the bus.

[0046] (2) Unleashing computational potential: The sparse perception operator suite of the present invention has unique computational capabilities within the compressed domain, directly penetrating the compression barrier and performing target neuron identifier comparison and in-situ memory overwrite filtering in the compact palette dictionary at the bottom layer. This breakthrough completely avoids the resource waste of being forced to instantiate a large and dense matrix to perform masking, resulting in a significant leap in computational efficiency.

[0047] (3) Significantly improve concurrent throughput: Eliminate concurrent scheduling deadlock through single-threaded hardware optimization layer, and eliminate language barriers by combining zero-copy structure. It demonstrates excellent scalability in hundreds of thousands of extraction tasks in the real world, and increases end-to-end throughput to dozens of times that of the traditional baseline.

[0048] To more clearly demonstrate the technical solution and its effects provided by the present invention, the system provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0049] I. Overall Overview of the Plan

[0050] Addressing the common challenges faced by existing connectionomics data processing platforms in extracting small, discrete, and extremely sparse pairwise point clouds—such as request amplification, performance collapse during full decompression, wasted computing power, and high scheduling overhead—this invention proposes a sparse-aware neuron pairwise point cloud extraction acceleration system. The core concept lies in reshaping the memory-resident representation structure of data to construct a high-performance execution ecosystem independent of full dense decompression. The system comprises, from bottom to top, the following layers: a data layer, acting as a basic buffer for storing raw compressed block data retrieved from the storage cluster at runtime; a C++ execution layer, serving as the core computing engine and control center, built on a block-level memory architecture; while retaining the original specific compression format of the raw compressed block data, it introduces extremely lightweight empty block identifiers (is_null) and non-zero identifiers (has_nonzero) in the data header, thereby supporting ultra-fast sparse filtering in an absolutely undecompressed state; and a Python interface layer, enabling cross-language... The interface boundary provides a shield, transparently exposing the underlying C++ structure and memory reclamation mechanism to the upper-layer application. It is responsible for receiving extraction requests and coordinating the global automated scheduling. The C++ execution layer is equipped with a sparse awareness operator suite, which is tightly embedded on the block-level memory architecture. The algorithm is customized for the subtle dendrite and axon features of neurons. By calling empty blocks and non-zero identifiers, it skips irrelevant original compressed state block data and directly enters the compact palette dictionary of non-empty original compressed state block data to perform comparison and in-situ modification operations, completing point cloud extraction under near-zero decompression extreme conditions.

[0051] II. Detailed introduction of the plan.

[0052] like Figure 2 and Figure 3 As shown, the system developed in this embodiment of the invention is named SABER (Sparsity-Aware Block-level Extraction Runtime) system. Its physical design is closely built around the block-level memory architecture. The system is strictly divided into three core layers in memory: the data layer, the C++ execution layer, and the Python interface layer. Figure 2 This is an overview diagram of the overall system architecture. Figure 3 This is the core data structure and mapping diagram of the block-level memory architecture.

[0053] The data layer acts as a high-speed buffer between the storage medium and main memory, storing raw, tiny data chunks of data fetched from the underlying system.

[0054] The C++ execution layer is the engine of the accelerated system and the central hub for metadata control. Centered around a core C++ class data structure called "StoredBlock" (also known as the global master data structure mentioned earlier), this layer securely maintains all block-level metadata information. StoredBlock not only preserves the data bit width, compact palette dictionary set, and compressed bitstream array required by compression format standards (such as Compressed-Segmentation), but its most groundbreaking feature is the introduction of two extremely lightweight Boolean status flags fixed at the memory header. One is an empty block flag (is_null), globally marking whether the space corresponding to the original compressed block data is completely devoid of any organizational information; the other is a non-zero flag (has_nonzero), dynamically marking whether the original compressed block data contains the target neuron identifier currently of interest to the user. During data initialization, initial loading into memory, and operations that trigger data changes, the system forcibly updates these two flags through a low-overhead background sniffing mechanism (sparse sensing mechanism). These two flags directly enable light-speed sparse filtering of randomly requested target regions without touching massive amounts of compressed bitstream data.

[0055] The Python interface layer exposes high-dimensional encapsulated objects and methods through Cython's cross-language technology, presenting a seamless interface environment to upper-layer applications (such as PyBlockStore), while internally it transparently directs hardware computing tasks through C++ core data structures.

[0056] Based on this block-level memory architecture, the SABER system has rigorously deployed a three-tiered, multi-level filtering framework to address the persistent problem of request amplification: The first layer is a block-level zero-decompression layer. When processing request bounding boxes, SABER uses StoredBlock for physical block control. First, it uses a geometric space coordinate mapping algorithm to calculate the intersection between the bounding box and the physical mesh. All mathematically non-intersecting edge raw compressed block data are identified as "dead zones" and discarded. The remaining StoredBlock objects perfectly maintain the compressed representation of the data, and the entire process never triggers data decompression.

[0057] The second layer is the sparsity-aware skip layer. When faced with raw compressed block data, the sparsity-aware masking module first examines the header's identifier state to quickly determine whether the raw compressed block data is worth processing. For non-empty raw compressed block data, the sparsity-aware masking module rapidly searches a tiny palette dictionary array to accurately determine whether the target neuron identifier exists. Furthermore, to prevent call latency caused by numerous dense loop checks running in the Python engine, the system constructs a zero-copy proxy data structure named "cBlock" in real-time within the C++ execution layer for valid raw compressed block data. cBlock strips away unnecessary management state, retaining only the execution pointer and non-zero identifier variables, enabling the sparsity-aware masking module to run efficiently in the native C++ language environment.

[0058] The third layer optimizes single-threaded I / O scheduling. Point cloud extraction tasks are compact and have a small workload. Blindly starting a multi-threaded pool for scheduling could lead to lock contention and context switching, which would ultimately degrade the performance of high-speed SSDs (Solid State Drives). Therefore, this system specifically designs a purely deterministic single-threaded reader that smoothly retrieves data according to the optimal sequential access characteristics of physical storage, maximizing the utilization of the extremely low latency of the physical hardware. The read data is then stored in a compressed voxel container.

[0059] like Figure 4As shown, the on-demand sparse loading execution process mainly includes: ① A single-threaded reader reads the data blocks involved in the request bounding box from the cache or solid-state drive. ② Block-level coordinate mapping is performed on the read data to convert the coarse-grained data block range into a finer-grained block range, and candidate raw compressed block data that spatially intersects with the request bounding box is selected. ③ Boundary checks are performed on the candidate raw compressed block data to determine whether they exceed the physical boundary range of the user request. ④ A clipping mask is applied to the blocks at the boundary overlap position, and irrelevant voxels or irrelevant identifiers outside the request bounding box are processed into background mask values ​​at the palette level. ⑤ Target identifier checks are performed, that is, the palette dictionary array of the candidate blocks is scanned to determine whether there is a target neuron identifier in the raw compressed block data. ⑥ For the candidate raw compressed block data determined in step ②, initialize the corresponding global master data structure in the C++ execution layer, recording bit width information, palette dictionary array, compressed bit stream, empty block identifier, and non-zero identifier; wherein, according to the target identifier check result in step ⑤, configure the non-zero identifier of the corresponding global master data structure so that the raw compressed block data containing the target neuron identifier and the raw compressed block data not containing the target neuron identifier have different non-zero identifier states, for example, if the target neuron identifier is included, it is set to 1, otherwise it is set to 0. ⑦ Synchronously generate the corresponding zero-copy proxy structure and mount it to the zero-copy proxy array in BlockArena (block central manager) so that the subsequent sparse perception operator can directly access the palette dictionary and compressed bit stream through pointers.

[0060] In the subsequent deep processing pipeline, the sparse sensing mask module first initiates a reverse mapping based on the processing space coordinate system to calculate the memory index of the corresponding original compressed block data. Then, it uses non-zero identifiers to achieve instruction-level conditional transitions, instantly and seamlessly skipping blank original compressed block data without the target neuron identifier. When faced with non-empty original compressed block data containing the target neuron identifier, it does not perform full decompression by allocating memory, but instead uses a zero-copy proxy data structure to directly obtain the pointer to the palette dictionary array in the underlying physical heap memory. The algorithm logic scans this array of only a few dozen bytes, and if any miscellaneous identifier that does not match the target neuron is found, it directly overwrites the data at that address with a predefined background mask value, achieving perfect logical filtering without touching the massive underlying bit stream.

[0061] like Figure 5As shown, the sparse-aware masking execution process mainly includes: ① The sparse-aware masking module maps the corresponding block index according to the spatial coordinates of the current processing area and accesses the cBlock proxy structure in BlockArena. ② The fast check function is called to check the non-zero identifiers in cBlock; if the current original compressed block data does not contain the target neuron identifier, it is skipped directly without further processing. ③ For the original compressed block data containing the target neuron identifier, the palette dictionary pointer is directly obtained without decompressing the underlying compressed bitstream. ④ In-situ update is performed within the palette dictionary array, overwriting the non-target neuron identifiers with the background mask values, thereby completing the logical masking operation within the compressed domain and synchronously affecting the effective data representation in the corresponding global master data structure.

[0062] To address the extremely messy and discontinuous neuronal fragments commonly found in biological slices, this invention introduces an intelligent seed-driven in-situ isolation module, employing a two-stage connected domain exploration isolation architecture: The first stage is a sparse anchor point localization stage, where the geometric center of interest (the geometric center of the target neuron pair) is marked as a virtual origin. Then, leveraging sparse sensing characteristics, a block-by-block rapid scan is initiated, quickly skipping redundant raw compressed data blocks. For raw compressed data blocks containing target neuron identifiers, the absolute physical coordinates of the effective voxels within them are extracted and their distance from the virtual origin is calculated. The coordinates of the nearest point are iteratively updated, ultimately outputting an accurate "starting anchor point" at lightning speed. The second stage is a sparse-guided breadth-first search stage, initiating breadth-first expansion from the starting anchor point. When expanding to unknown neighboring coordinate points, the non-zero identifiers of the raw compressed data blocks to which the neighboring coordinate points belong are checked. If a non-zero identifier is empty, the expansion path is instantly halted. To completely eliminate the performance degradation caused by repeated queries of the same original compressed block data during the search, the system is specially configured with a dynamically controlled local dynamic decompression cache pool in the C++ execution layer. When it is necessary to obtain the voxel value, the voxel value is first checked in the local dynamic decompression cache pool. If the voxel value is found, it is retrieved in O(1) time complexity (constant time complexity). Only when the voxel value is not found is a very small amount of actual decompression triggered and the finished product is placed in the pool. With the LRU strategy (Least Recently Used strategy), ultra-high isolation execution efficiency is achieved with minimal memory consumption.

[0063] like Figure 6As shown, the seed-driven in-situ isolation execution process mainly includes: ① Traversing each global master data structure in BlockArena to prepare for sparse anchor point localization. ② Performing sparse-aware judgment on the original compressed block data of each global master data structure; if it is empty or does not contain a target identifier, it is skipped directly; if it contains a target neuron identifier, necessary local data access is triggered. ③ Preparing to obtain target voxel information. ④ When necessary, performing on-demand decompression to obtain local original compressed block data. ⑤ Updating the effective starting anchor point closest to the geometric center of the target based on the decompressed local effective voxels. ⑥ Starting with the starting anchor point as the starting point, initiating sparse-guided breadth-first search (BFS). ⑦ When BFS expands to neighboring voxels, performing sparse-aware judgment on the block where the neighboring point is located; if the neighboring block is empty or does not contain a target identifier, the search branch is truncated. ⑧ If the neighboring block contains a target identifier, first querying the local dynamic decompression cache pool to determine whether the decompressed data of the required block already exists. ⑨ If a cache miss occurs, the operation is decompressed on demand to generate the original array. ⑩ The newly decompressed original array is written to a local dynamic decompression cache pool for reuse in subsequent accesses from neighboring nodes. If a cache hit occurs, the required voxel values ​​are read directly from the local dynamic decompression cache pool to avoid repeated decompression. The next expansion point is updated based on the obtained neighboring voxel values, and the BFS search front is continued to expand, ultimately preserving the target neuron branch connected to the starting anchor point.

[0064] At the end of the pipeline, advanced neuroscience experiments often require generating continuous, smooth 3D surface polygonal mesh models. Classical reconstruction algorithms inevitably rely on a decompressed, continuous, dense 3D mesh. To resolve the contradiction between this requirement and the block-level anti-expansion architecture, SABER introduces a "compromise based on fine-grained layer slicing" for the compressed domain contour extraction module. The execution logic is as follows: During initialization, memory is not allocated for the entire target data field. Instead, a high-aligned blank physical reuse buffer is precisely allocated based on the physical size dimension of a single original compressed block of data on the vertical axis. When performing contour extraction, it proceeds layer by layer from bottom to top according to the slice coordinates. When processing a single slice level, non-zero flags are activated to ignore the original compressed block data of all missed objects, and only the essential candidate original compressed block data (i.e., non-empty original compressed block data containing the target neuron flag) is allowed to be decompressed and injected into the specified region of the reuse buffer as needed. After completing the surface mesh generation and geometric extraction calculations for this level of data, the algorithm immediately issues a memory reset command (memset) to the operating system to forcibly flush all blank physical reuse caches to zero, restoring them to blank state to accommodate the data deluge of the next level. This extreme cyclic reuse perfectly and thoroughly avoids the exponential data expansion delay and memory surge crash (OOM) that inevitably occur when decompressing the entire volume.

[0065] like Figure 7 As shown, the compression domain contour extraction execution process mainly includes: ① Initializing or obtaining a blank physical multiplexing cache, which is used to carry the decompressed data of the current slice or the current local block (target neuron connected branch block). ② Traversing the main data structure array / zero-copy proxy array in the block set manager according to the slice level or block order; of course, since the target neuron connected branch block set was determined in the previous seed-driven in-situ isolation process, the main data structure array considered here corresponds to the target neuron connected branch block set, and the global main data structure in the target neuron connected branch block set is a target neuron connected branch block; at the same time, the zero-copy proxy array is the zero-copy proxy data structure corresponding to the target neuron connected branch block set. ③ Performing intersection judgment on the current target neuron connected branch block: accessing the corresponding zero-copy proxy data structure, judging whether it intersects with the current slice level according to the coordinate range; if it does not intersect with the current level, it is skipped directly; if it intersects, the corresponding global main data structure is accessed. ④ Determining the write position of the corresponding global main data structure in the current blank physical multiplexing cache. ⑤ Perform on-demand decompression and write the corresponding global master data structure to the blank physical reuse cache. ⑥ Perform contour extraction on the local dense data in the blank physical reuse cache to generate the surface contour of the target neuron or point cloud boundary information. ⑦ After the current slice is processed, perform memory zeroing / reset on the blank physical reuse cache so that the cache can be reused by the next layer slice or the next batch of candidate blocks, thereby avoiding memory bloat caused by full decompression of the entire volume of data.

[0066] Based on the above introduction, the hardware-software co-optimization details and register-level communication principles of the system provided by this invention are as follows: When processing data transmitted from a high-speed NVMe (Non-Volatile Memory Interface) solid-state drive, the traditional data copying between user mode and kernel mode becomes a new bottleneck. In this embodiment, the traditional operating system kernel VFS (Virtual File System) layer is bypassed through the SPDK (Storage Performance Development Kit), realizing direct DMA (Direct Memory Access) copying from the solid-state drive flash memory to the C++ execution layer StoredBlock memory space. In order to fully utilize the CPU's high-speed L1 / L2 cache (which belong to the CPU's internal cache), the zero-copy proxy data structure is strictly limited to 64 bytes during memory allocation, which is exactly aligned with the CPU cache line size of modern x86 architecture (Complex Instruction Set Architecture) and ARM architecture (Reduced Instruction Set Architecture), enabling the sparse-aware masking module to trigger the hardware prefetching mechanism with zero latency when performing palette scanning.

[0067] In summary, the system and its core architecture provided by this invention not only possess underlying technological innovations such as compressed domain block-level sparse filtering, on-demand decompression, and collaborative execution of sparse-aware operators, but also have good compatibility and modularity at the peripheral interface level, enabling them to be integrated into existing connectionomics point cloud extraction and processing workflows in the form of plug-in or alternative interfaces. Experimental evaluations show that on the FAFB-FFN dataset (Full Adult Fly Brain Flood-Filling Network dataset, a large-scale connectivityomics dataset based on whole-brain electron microscopy reconstruction data of adult fruit flies and the segmentation results of flood-filling networks), targeting 86,819 pairs of real neuron breakage candidate pairs, and under the condition of 52 physical CPU cores concurrently, the system of this invention reduces the average end-to-end latency of a single pair point cloud extraction task from 8.07 seconds in the traditional dense baseline to 1.75 seconds, achieving an overall speedup of approximately 4.62 times; the system throughput increases from 6.45 tasks / second to 29.78 tasks / second, achieving a throughput improvement of approximately 4.6 times; at the same time, the memory bus bandwidth usage decreases from 211.89 GB / s to 77.97 GB / s, a reduction of approximately 63%. At the operator level, the on-demand sparse loading module achieves approximately 11.55 times speedup compared to traditional crawling processes, the sparse-aware masking module achieves approximately 3.36 times speedup, and the seed-driven in-situ isolation / filtering module achieves approximately 8.95 times speedup. Regarding data ingestion scale, the system reduces the amount of data processed from approximately 4,979.39 GB to approximately 3.03 GB through block-level alignment, sparse filtering, and compressed state preservation, resulting in a data ingestion reduction of approximately 1,645 times. Furthermore, on the independent MICrONS dataset (Machine Intelligence from Cortical Networks dataset, a large-scale connectomics dataset for reconstructing neural circuits in the mouse visual cortex), when validating on 81,825 candidate location requests, this invention reduces the single-task extraction latency from 12.42 seconds to 3.74 seconds under 104 concurrent processes, achieving an end-to-end speedup of approximately 3.32 times, and reducing memory bus bandwidth usage from 217.48 GB / s to 77.70 GB / s. The experimental results above demonstrate that the present invention can stably reduce request amplification and memory bandwidth pressure on different types of large-scale connectionomics datasets, and significantly improve the concurrent throughput and end-to-end processing efficiency of the neuron pair point cloud extraction task.

[0068] III. Example Introduction.

[0069] This example provides a system deployment architecture for an adaptive distributed cluster environment.

[0070] The system provided by this invention can be deployed in large-scale supercomputing center environments with hundreds of CPU and GPU nodes. Each node initializes an independent block central manager structure in its local DRAM (Dynamic Random Access Memory). Nodes communicate with each other via a high-speed RDMA (Remote Direct Memory Access) network, sharing zero-copy proxy data structures in distributed memory. The system introduces a consistent hashing algorithm to ensure that point cloud extraction requests for the same neuron fall into the node with most of the decompression cache for that neuron, thereby maximizing the global hit rate of the local dynamic decompression cache pool. For extremely large neurons, when the blank physical reuse cache (Buffer) of a single machine is insufficient to support contour extraction at the whole brain domain level, the system adopts a distributed slicing scheme, allocating slices to multiple nodes on the Z-axis (vertical axis), each using memset for cyclic reuse before merging the results.

[0071] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0073] Example 2 This invention also provides an accelerated method for extracting paired point clouds of neurons based on sparse sensing, which is mainly based on the system implementation provided in the foregoing embodiments. The method includes: The data layer stores the raw compressed block data involved in the request bounding box read from the storage cluster. The raw compressed block data is a compressed representation of a three-dimensional connected omics voxel segmentation image block. By employing a block-level memory architecture in the C++ execution layer, and utilizing a pre-defined structure object to preserve the original compressed block data format, empty block identifiers and non-zero identifiers are introduced. Simultaneously, a sparse perception operator suite is configured to complete the point cloud extraction task under the call of the Python interface layer. Among them, the empty block identifier is used to mark whether all three-dimensional connectomics voxels inside the block are background mask values, and the non-zero identifier is used to mark whether the block contains neuron identifiers that meet the extraction conditions. The point cloud extraction task from the upper application is processed by calling the C++ execution layer through the Python interface layer. This includes calling the sparse perception operator suite of the C++ execution layer, which uses empty block identifiers and non-zero identifiers to skip the original compressed block data that is irrelevant to the target neuron identifier, and directly performing the matching and comparison of the target neuron identifier in the palette dictionary array contained in the non-empty original compressed block data containing the target neuron identifier, thereby completing the spatial localization and feature extraction of the point cloud. Both C++ and Python are computer programming languages.

[0074] Since the specific technical details involved in this method have been described in detail in the previous embodiments, they will not be repeated here.

[0075] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A sparse-aware based neuron pairwise point cloud extraction acceleration system, characterized in that, include: The data layer is used to store the raw compressed block data involved in the request bounding box read from the storage cluster, wherein the raw compressed block data is a compressed representation of a three-dimensional connected omics voxel segmentation image block; The C++ execution layer adopts a block-level memory architecture. It uses a preset structure object to retain the original compressed block data format and introduces empty block identifiers and non-zero identifiers. At the same time, it configures a sparse perception operator suite to complete the point cloud extraction task under the call of the Python interface layer. Among them, the empty block identifier is used to mark whether all three-dimensional connectivity voxels inside the block are background mask values, and the non-zero identifier is used to mark whether the block contains neuron identifiers that meet the extraction conditions. The Python interface layer processes point cloud extraction tasks from upper-layer applications by calling the C++ execution layer. This includes calling the sparse perception operator suite of the C++ execution layer, which uses empty block identifiers and non-zero identifiers to skip raw compressed block data that is irrelevant to the target neuron identifier. For non-empty raw compressed block data containing the target neuron identifier, the matching and comparison of the target neuron identifier is performed directly in the palette dictionary array contained therein, thereby completing the spatial localization and feature extraction of the point cloud. Both C++ and Python are computer programming languages.

2. The neuron pairwise point cloud extraction acceleration system based on sparse perception according to claim 1, wherein, The original compressed block data is encoded using a compressed segmented format and includes bit width information, a palette dictionary array, and a compressed bit stream.

3. The neuron pairwise point cloud extraction acceleration system based on sparse perception according to claim 1, wherein, The system also includes: implementing a three-tiered avoidance framework when processing point cloud extraction tasks. The first layer is a block-granular zero decompression layer, which is implemented through the sparse-aware operator suite of the C++ execution layer. It retains the original compressed block data read from the third layer as the basic storage unit. Through the three-dimensional geometric bounding box mapping calculation, it directly skips the original compressed block data that is not intersecting outside the requested bounding box specified in the point cloud extraction task, maintains the data compression representation and does not trigger decompression. The second layer is the sparse perception skip layer, which uses empty block identifiers and non-zero identifiers to determine whether the original compressed block data contains the target neuron identifier. For the non-empty original compressed block data containing the target neuron identifier, the matching and comparison of the target neuron identifier is directly performed in the palette dictionary array contained therein, and finally the spatial localization and feature extraction of the point cloud are completed. The third layer is a single-threaded I / O read optimization layer, configured in the data layer. It is configured with a deterministic single-threaded cache reader, which reads the original compressed block data involved in the request bounding box sequentially according to the spatial physical storage order in the storage cluster; where I / O refers to input / output.

4. The accelerated system for extracting paired point clouds of neurons based on sparse perception according to claim 1, characterized in that, The preset structure object includes a global master data structure, which, while retaining the original compressed block data format, additionally introduces empty block identifiers and non-zero identifiers.

5. The accelerated system for extracting paired point clouds of neurons based on sparse perception according to claim 4, characterized in that, The sparse-aware operator suite includes an on-demand sparse loading module, the operation of which includes: The process involves acquiring the original compressed block data related to the request bounding box read by a deterministic single-threaded cache reader; calculating the physical coordinates corresponding to the original compressed block data and performing intersection calculations with the request bounding box; for original compressed block data with overlapping boundaries, applying a clipping mask operation to erase the 3D connectomics voxels extending beyond the request bounding box boundary on the palette dictionary array and forcibly overwriting them with background mask values; then, directly reading the palette dictionary array of the original compressed block data and matching it with the target neuron identifiers; determining the original compressed block data containing the target neuron identifiers based on the matching results; if an original compressed block data does not contain two target neuron identifiers... If any one of the following is a target neuron identifier, it belongs to the original compressed block data without the target neuron. If it contains any target neuron identifier, it belongs to the non-empty original compressed block data containing the target neuron. To construct a global master data structure in the block collection manager of the C++ execution layer for the original compressed block data that overlaps with the request bounding box, configure the non-zero identifier of the corresponding global master data structure so that the original compressed block data containing the target neuron identifier and the original compressed block data without the target neuron identifier have different non-zero identifiers. And add the corresponding zero-copy proxy data structure for the synchronous atomication of the original compressed block data containing the target neuron identifier to form a zero-copy proxy array.

6. The accelerated system for extracting paired point clouds of neurons based on sparse perception according to claim 5, characterized in that, The sparse-aware operator suite includes a sparse-aware mask module, the operation of which includes: The system reads the global master data structure from the master data structure array as the data for the current operation; calculates the memory index of the corresponding original compressed state block data based on the data coordinate mapping of the current operation; queries the non-zero identifier of the original compressed state block data corresponding to the memory index, and if it does not contain the target neuron identifier, it exits the processing of the corresponding original compressed state block data; for non-empty original compressed state block data containing the target neuron identifier, it obtains the pointer to the palette dictionary array through the zero-copy proxy data structure, initiates a sequential scan of the palette dictionary array, and overwrites the non-target neuron identifiers in the data of the current operation with the background mask values, completing the in-situ logical removal in the compressed domain state; after processing all the global master data structures in the master data structure array, it obtains the compressed domain target mask block set.

7. The accelerated system for extracting paired point clouds of neurons based on sparse perception according to claim 6, characterized in that, The sparse sensing operator suite includes: a seed-driven in-situ isolation module, whose working process is divided into a sparse anchor point localization stage and a sparse-guided breadth-first search stage; The sparse anchor point localization stage includes: starting a block-by-block traversal, traversing each global master data structure in the target mask block set of the compressed domain, calling the empty block identifier and non-zero identifier to filter the global master data structure of the original compressed block data that does not contain the target neuron identifier; for the global master data structure corresponding to the original compressed block data containing the target neuron identifier, calculating the distance between the internal voxel coordinates and the geometric center of the target neuron pair and continuously updating it, and outputting the effective neuron closest to the center as the starting anchor point; wherein, the geometric center of the target neuron pair is the input information of the point cloud extraction task; The sparse-guided breadth-first search phase includes: expanding the search from the starting anchor point, calculating the non-zero identifier of the global master data structure to which the original compressed block data corresponding to the voxel coordinates of the expanded neighboring points belongs; if the non-zero identifier determines that the target neuron identifier is not present, the response search branch is truncated; if the target neuron identifier is present, the corresponding global master data structure is further verified and assigned to the target neuron connected branch block set, wherein the data in the target neuron connected branch block set are all global master data structures corresponding to the original compressed block data containing the target neuron.

8. The accelerated system for extracting paired point clouds of neurons based on sparse perception according to claim 7, characterized in that, Also includes: The C++ execution layer is configured with a local dynamic decompression cache pool. During the sparse-guided breadth-first search phase, when the feature values ​​of neighboring points are obtained, i.e., the coordinates of neighboring voxels to be expanded generated during the sparse-guided breadth-first search, the local dynamic decompression cache pool is queried first. If a match is found, the voxel identifier value stored at the coordinates of the voxel to be expanded is extracted from it. If no match is found, the corresponding original compressed block data is decompressed in reverse to generate the original array and written to the local dynamic decompression cache pool. When the local dynamic decompression cache pool triggers a threshold, the least recently used strategy is used to evict redundant original arrays.

9. The accelerated system for extracting paired point clouds of neurons based on sparse perception according to claim 7, characterized in that, The sparse sensing operator suite includes: a compressed domain contour extraction module, which adopts a compromise execution scheme based on fine-grained layer slicing; during system initialization, based on the size of a single original compressed block data on a specific axis, a blank physical reuse cache aligned with the operating system is requested; each target neuron connected branch block in the target neuron connected branch block set corresponds to a global master data structure of an original compressed block data, and contour extraction is performed layer by layer according to the slice level, that is, the three-dimensional space covered by the target neuron connected branch block set is divided into multiple slice levels, and the target neuron connected branch block set is used to extract the contour. The zero-copy proxy data structure corresponding to the connected branch block of the intermediate neuron is used to determine the intersection status. Specifically, when processing the current slice level, the zero-copy proxy data structure corresponding to the connected branch block of the target neuron is used to determine whether it intersects with the current slice level by the coordinate range. If it is determined to be intersecting, the original compressed block data of the corresponding connected branch block of the target neuron is decompressed into the blank physical multiplexing cache, and the surface mesh generation and extraction calculation of the current slice level are performed. After that, the memory reset instruction is called to flush the blank physical multiplexing cache to zero in order to prepare for the decompression and multiplexing of the next slice level.

10. A method for accelerating the extraction of paired point clouds of neurons based on sparse perception, characterized in that, Based on the system implementation according to any one of claims 1 to 9, the method includes: The data layer stores the original compressed block data involved in the request bounding box read from the storage cluster. The original compressed block data is a compressed representation of a three-dimensional connected omics voxel segmentation image block. By employing a block-level memory architecture in the C++ execution layer, and utilizing a pre-defined structure object to preserve the original compressed block data format, empty block identifiers and non-zero identifiers are introduced. Simultaneously, a sparse perception operator suite is configured to complete the point cloud extraction task under the call of the Python interface layer. Among them, the empty block identifier is used to mark whether all three-dimensional connectomics voxels inside the block are background mask values, and the non-zero identifier is used to mark whether the block contains neuron identifiers that meet the extraction conditions. The point cloud extraction task from the upper application is processed by calling the C++ execution layer through the Python interface layer. This includes calling the sparse perception operator suite of the C++ execution layer, which uses empty block identifiers and non-zero identifiers to skip the original compressed block data that is irrelevant to the target neuron identifier, and directly performing the matching and comparison of the target neuron identifier in the palette dictionary array contained in the non-empty original compressed block data containing the target neuron identifier, thereby completing the spatial localization and feature extraction of the point cloud. Both C++ and Python are computer programming languages.

Images