Multimodal data management system, method, electronic device and storage medium

Abstract

The present invention relates to the technical field of flow cell data management, and specifically discloses a multimodal data management system, method, electronic device and storage medium, wherein the multimodal data management system comprises a first cache module, a synchronization alignment module, a second cache module, a flow storage module and a third cache module. The multimodal data management system caches collected multimodal data, synchronously aligned multimodal data and converted data after format conversion based on the first cache module, the second cache module and the third cache module to realize three-level data cache, and schedules cached data based on the synchronization alignment module and the flow storage module to integrate, organize and store multimodal data about flow cells in a storage hard disk, effectively matching the data flow rate difference in each link of data processing, and realizing real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams.

Description

Technical Field

[0001] The present application relates to the technical field of flow cytometry data management, and in particular to a multimodal data management system, method, electronic device and storage medium. Background Art

[0002] Flow cytometry is a cutting-edge technology for high-speed, quantitative analysis and sorting of suspended cells. This technology uses optical and electronic detectors to continuously analyze multiple parameters of each cell passing through, greatly improving the efficiency and accuracy of cell analysis.

[0003] Flow cytometry imaging technology is a perfect combination of flow cytometry and cell microscopy. It can not only obtain statistical data of cell populations, but also provide bright field or fluorescent images of single cells, providing researchers with rich information about cell morphology, cell structure and subcellular signal distribution.

[0004] In the field of biomedical research, the demand for cell analysis is becoming increasingly complex, requiring multi-parameter, high-throughput and subcellular characteristics to be met at the same time. To achieve this goal, researchers use cell feature analysis methods that combine multi-spectral and imaging data to perform cell analysis and sorting, so as to conduct a more comprehensive and in-depth analysis of cells.

[0005] The acquisition stage of flow cytometry data must meet the high-throughput characteristics while having a certain degree of real-time performance. In related technologies, the introduction of high-speed flow cell imaging paradigms has increased the amount of data carried by individual cells. The acquisition equipment for flow cytometry data can only meet this demand by increasing the number of acquisition threads, but it is difficult to ensure the synchronous alignment of the acquired image data and spectral data, and it is also impossible to store these data efficiently and orderly for subsequent offline analysis and backtracking. This results in deficiencies in real-time data acquisition, synchronous alignment, and orderly storage in the multimodal flow cytometric analysis and sorting process.

[0006] There is currently no effective technical solution to the above problems. Summary of the invention

[0007] The purpose of the present application is to provide a multimodal data management system, method, electronic device and storage medium to achieve real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams.

[0008] In a first aspect, the present application provides a multimodal data management system for arranging multimodal data of flow cells based on time series, the multimodal data management system comprising:

[0009] A first cache module, configured to cache received first modal data and second modal data;

[0010] a synchronization alignment module, connected to the first cache module, configured to extract the first modality data and the second modality data from the first cache module, and store the first modality data and the second modality data in a second cache module in a synchronously aligned manner;

[0011] The second cache module is connected to the synchronization alignment module and is used to cache the first modality data and the second modality data that are synchronously aligned;

[0012] a flow storage module connected to the second cache module, configured to extract the first modal data and the second modal data after synchronization alignment from the second cache module, and convert the first modal data and the second modal data after synchronization alignment into conversion data of a target format;

[0013] A third cache module, connected to the flow storage module, for storing the conversion data in the target format;

[0014] The flow storage module is also used to extract conversion data in a target format from the third cache module, and when the extracted conversion data reaches a preset amount, merge the preset amount of conversion data and store it in a storage hard disk.

[0015] The multimodal data management system of the present application caches the collected multimodal data, the synchronously aligned multimodal data, and the converted data after format conversion based on the first cache module, the second cache module, and the third cache module respectively to realize three-level data cache, and schedules the cached data based on the synchronous alignment module and the flow storage module to integrate and store the multimodal data about flow cells in the storage hard disk. This processing process effectively matches the data flow rate difference in each link of data processing, realizes the real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams, and can meet the transmission and storage requirements of flow cell multimodal data.

[0016] The multimodal data management system, wherein the multimodal data management system is applied in a flow cytometer, the first modal data is image data, and the second modal data is spectral data.

[0017] Since the image data and the spectral data are temporally matchable during the acquisition process, in this example, the first modality data and the second modality data are preferably image data and spectral data, respectively.

[0018] In the multimodal data management system, the synchronous alignment module transmits the image data based on multi-threading and transmits the spectral data based on a single thread.

[0019] In this example, for the image data stream, the synchronization alignment module uses a multi-threaded working method to transfer image data to expand its data flow; while for the spectral data stream, the amount of spectral data is small, and its acquisition rate is usually higher than the acquisition rate of image data, so the synchronization alignment module uses a single-threaded working method to transfer the spectral data.

[0020] The multimodal data management system, wherein the first cache module is further used to allocate first metadata to the image data based on the time sequence when the image data is stored therein; the second cache module includes a plurality of data memory blocks, and the process of the synchronization alignment module synchronously aligning the image data and the spectral data includes:

[0021] When a frame of image data is extracted from the first cache module, the image data is cached in a data memory block in the second cache module based on the first meta information;

[0022] When a piece of spectral data is extracted from the first cache module, second meta information of the spectral data is obtained, and the spectral data is cached in a corresponding data memory block based on a matching relationship between the first meta information and the second meta information.

[0023] The multimodal data management system, wherein the second cache module includes a first cache queue and a second cache queue, the first cache queue and the second cache queue both include a number of data memory blocks, the first cache queue is used to cooperate with the synchronization alignment module to synchronously align the received first modality data and the second modality data, and the second cache queue is used to cache the first modality data and the second modality data after synchronization alignment.

[0024] The multimodal data management system, wherein the flow storage module comprises:

[0025] A producer thread, configured to extract the first modal data and the second modal data after synchronization alignment from the second cache module, and convert the first modal data and the second modal data after synchronization alignment into conversion data in a target format, so as to store the conversion data in the third cache module;

[0026] The consumer thread is used to extract the conversion data in the target format from the third cache module, and when the extracted conversion data reaches a preset amount, merge the preset amount of conversion data to obtain a storage file, and store the storage file in the storage hard disk.

[0027] The multimodal data management system, wherein the producer thread and the consumer thread are both multiple, and the number of the consumer threads is greater than the number of the producer threads.

[0028] In a second aspect, the present application further provides a multimodal data management method, which is applied in a multimodal data management system, wherein the multimodal data management system comprises:

[0029] A first cache module, configured to cache received first modal data and second modal data;

[0030] A second cache module, used for caching the synchronously aligned first modality data and second modality data;

[0031] A third cache module, used for storing the converted data in the target format;

[0032] The multimodal data management method comprises the following steps:

[0033] S1. extracting the first modal data and the second modal data from the first cache module, and storing the first modal data and the second modal data in a synchronous and aligned manner into a second cache module;

[0034] S2, extracting the first modality data and the second modality data after synchronization alignment from the second cache module, and converting the first modality data and the second modality data after synchronization alignment into conversion data in a target format;

[0035] S3. Extracting conversion data in a target format from the third cache module, and when the extracted conversion data reaches a preset amount, merging the preset amount of conversion data and storing them in a storage hard disk.

[0036] The multimodal data management method of the embodiment of the present application is based on a first cache module, a second cache module and a third cache module to cache the collected multimodal data, the synchronously aligned multimodal data and the converted data after format conversion respectively to achieve three-level data cache, and the cached data is scheduled to integrate and organize the multimodal data about flow cells and store them in a storage hard disk. This processing process effectively matches the data flow rate difference in each link of data processing, realizes the real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams, and can meet the transmission and storage requirements of flow cell multimodal data.

[0037] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the second aspect are executed.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps in the method provided in the second aspect are executed.

[0039] As can be seen from the above, the embodiments of the present application provide a multimodal data management system, method, electronic device and storage medium, wherein the multimodal data management system caches the collected multimodal data, synchronously aligned multimodal data and converted data after format conversion based on the first cache module, the second cache module and the third cache module respectively to realize three-level data cache, and schedules the cached data based on the synchronous alignment module and the flow storage module to integrate and store the multimodal data about flow cells in the storage hard disk. This processing process effectively matches the data flow rate difference in each link of data processing, realizes the real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams, and can meet the transmission and storage requirements of flow cell multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the structure of a multimodal data management system provided in an embodiment of the present application.

[0041] Figure 2 It is a schematic diagram of transferring the data memory blocks in the first cache queue in the second cache module to the second cache queue.

[0042] Figure 3 It is a schematic diagram of transferring the data memory blocks in the second cache queue in the second cache module to the first cache queue.

[0043] Figure 4 It is a schematic diagram of the connection structure of the second cache module, the circulation storage module and the third cache module.

[0044] Figure 5 A flowchart of a multimodal data management method provided in an embodiment of the present application.

[0045] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] Figure numerals: 101, first cache module; 102, synchronization alignment module; 103, second cache module; 104, flow storage module; 105, third cache module; 106, storage hard disk; 1031, first cache queue; 1032, second cache queue; 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0048] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0049] First, please refer to Figure 1-Figure 4 Some embodiments of the present application provide a multimodal data management system for arranging multimodal data of flow cells based on time series, and the multimodal data management system includes:

[0050] A first cache module 101, configured to cache received first modality data and second modality data;

[0051] The synchronization alignment module 102 is connected to the first cache module 101, and is used to extract the first modality data and the second modality data from the first cache module 101, and store the first modality data and the second modality data in the second cache module 103 in a synchronous and aligned manner;

[0052] A second cache module 103, connected to the synchronization alignment module 102, for caching the first modality data and the second modality data that are synchronously aligned;

[0053] The flow storage module 104 is connected to the second cache module 103, and is used to extract the first modality data and the second modality data after synchronization alignment from the second cache module 103, and convert the first modality data and the second modality data after synchronization alignment into conversion data of a target format;

[0054] A third cache module 105, connected to the flow storage module 104, is used to store the conversion data in the target format;

[0055] The flow storage module 104 is further used to extract the conversion data in the target format from the third cache module 105 , and when the extracted conversion data reaches a preset amount, merge the preset amount of conversion data and store it in the storage hard disk 106 .

[0056] Specifically, the multimodal data management system of the embodiment of the present application manages the multimodal data of flow cells with high-throughput characteristics, and can be used with a multimodal flow analysis sorter. It aims to provide a real-time and efficient multimodal data integration, organization and storage method suitable for high-throughput data acquisition scenarios, so as to facilitate subsequent online or offline analysis of flow cell data for classification or retrieval.

[0057] It should be noted that the multimodal data management system of the embodiment of the present application is also applicable to the integration, organization and storage of a larger number of modal data, and is not limited to two types of modal data.

[0058] More specifically, the multimodal data management system of the embodiment of the present application sets three levels of cache modules and two levels of scheduling modules, wherein the three levels of cache modules include a first cache module 101, a second cache module 103 and a third cache module 105, the first cache module 101 is used to cache the first modality data and the second modality data about the flow cell acquired based on time series acquisition, the second cache module 103 is used to cache the first modality data and the second modality data that are synchronously aligned, and the third cache module 105 is used to cache the converted data with the target format converted from the first modality data and the second modality data based on the synchronous alignment, and the two levels of scheduling modules include a synchronous alignment module 102 and a flow storage module 104, and the synchronous alignment module 102 is used to synchronously align the first modality data and the second modality data cached by the first cache module 101 The first modal data and the second modal data stored in the second cache module 103 are dispatched to the second cache module 103 for caching. The flow storage module 104 has two functions. One is to convert the synchronously aligned first modal data and the second modal data stored in the second cache module 103 into conversion data of the target format and then dispatch them to the third cache module 105 for caching. The second is to extract and merge the preset number of conversion data cached in the third cache module 105, and dispatch them to the storage hard disk 106. The storage hard disk 106 is a non-volatile storage device that the multimodal data management system ultimately uses to store relevant data of flow cells. Based on the above-mentioned data flow method, the multimodal data management system of the embodiment of the present application effectively matches the data flow rate difference in each link of data processing, realizes the real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams, and can meet the transmission and storage requirements of multimodal data of flow cells.

[0059] It should be noted that the multimodal data management system of the embodiment of the present application is based on the first cache module 101 to cache the received first modality data and second modality data in real time and continuously, thereby realizing the continuous transmission and storage of multimodal data of flow cells.

[0060] More specifically, the first cache module 101, the second cache module 103 and the third cache module 105 can be independent cache devices, or can be cache areas created by dividing storage areas based on a single storage device. In the embodiment of the present application, the latter is preferred.

[0061] The multimodal data management system of the embodiment of the present application is based on the first cache module 101, the second cache module 103 and the third cache module 105 to cache the collected multimodal data, the synchronously aligned multimodal data and the converted data after format conversion to achieve three-level data cache, and based on the synchronous alignment module 102 and the flow storage module 104, the cache data is scheduled to integrate and store the multimodal data about flow cells in the storage hard disk 106. This processing process effectively matches the data flow rate difference in each link of data processing, realizes the real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams, and can meet the transmission and storage requirements of flow cell multimodal data.

[0062] In some preferred embodiments, the multimodal data management system is applied in a flow cytometer, the first modality data is image data, and the second modality data is spectral data.

[0063] Specifically, the first modality data and the second modality data are two types of modality data about flow cytometry, such as image data, spectral data, high-dimensional data, cell cycle data, etc. Since image data and spectral data are time-series matchable during the acquisition process, in an embodiment of the present application, the first modality data and the second modality data are preferably image data and spectral data, respectively.

[0064] In some preferred embodiments, the first cache module 101 includes two memory spaces with consecutive addresses, which are respectively used to store the received first modality data and the second modality data.

[0065] Specifically, both memory spaces adopt a first-in-first-out ring structure. The first modal data and the second modal data received by the first cache module 101 are stored in the two memory spaces respectively, and the first modal data are stored in the corresponding memory space in sequence according to the ring structure, and the second modal data are stored in the corresponding memory space in sequence according to the ring structure. That is, after the data is full, the newly received modal data will overwrite the earliest received modal data in the current memory space to achieve continuous reception of modal data.

[0066] It should be noted that, in this embodiment, the synchronization alignment module 102 extracts and copies the data in the first cache module 101 and transfers it to the second cache module 103. The data stored in the first cache module 101 is overwritten by a first-in-first-out method to achieve continuous data reception and update.

[0067] More specifically, in order to meet the real-time requirements of high-throughput data, the first cache module 101 is created before the multimodal data management system of the embodiment of the present application is connected to the multimodal data acquisition device.

[0068] More specifically, the two memory spaces are both continuous memory spaces with fixed capacity.

[0069] In some preferred embodiments, the synchronization and alignment module 102 transmits image data based on multiple threads and transmits spectral data based on a single thread.

[0070] Specifically, in this embodiment, the synchronization alignment module 102 mainly functions to schedule and align the data in the first cache module 101 so as to transfer it to the second cache module 103. During the transfer process, the high-throughput characteristics and real-time requirements of the data flow need to be considered. Therefore, for the image data flow, the synchronization alignment module 102 adopts a multi-threaded working method to transfer the image data to expand its data flow rate; and for the spectral data flow, the amount of spectral data is small, and its acquisition rate is usually higher than that of the image data, so the synchronization alignment module 102 adopts a single-threaded working method to transfer the spectral data.

[0071] More specifically, the goal of the synchronous alignment module 102 is to associate the image data and spectral data acquired under the triggering of the synchronization signal, and provide multimodal data for subsequent analysis and sorting. The multi-threaded transmission of image data and the single-threaded transmission of spectral data can ensure that the two types of data can be transmitted in real time and synchronously while meeting high-throughput requirements.

[0072] In some preferred embodiments, the first cache module 101 is further used to allocate first metadata to the image data based on the time sequence when the image data is stored therein; the second cache module 103 includes a plurality of data memory blocks, and the process of synchronous alignment module 102 synchronously aligning the image data and the spectral data includes:

[0073] When a frame of image data is extracted from the first buffer module 101, the image data is buffered in a data memory block in the second buffer module 103 based on the first meta information;

[0074] When a piece of spectrum data is extracted from the first cache module 101 , the second metadata of the spectrum data is obtained, and the spectrum data is cached in a corresponding data memory block based on a matching relationship between the first metadata and the second metadata.

[0075] Specifically, when the spectral acquisition device collects spectral data, it generally automatically generates corresponding second metadata. Therefore, the multimodal data management system of the embodiment of the present application performs synchronous alignment of image data and spectral data based on metadata synchronization to reduce the time complexity of the synchronous alignment process and improve the processing efficiency of the synchronous alignment.

[0076] More specifically, since the image data still retains its timing when stored in the first cache module 101, the first cache module 101 can assign first metadata to the image data based on the timing (generally set to the image ID, and accumulated according to the timing), wherein the initial value of the first metadata is the same as the second metadata, so that the subsequently acquired image data and spectral data can be synchronously aligned based on the first metadata and the second metadata. This processing method makes full use of the timing of the data stream itself, and can avoid using sorting, searching and other methods to locate the target ID during the synchronization alignment process, thereby reducing time complexity, improving the efficiency of data synchronization alignment, and maintaining the real-time nature of data stream processing.

[0077] More specifically, the process of caching the image data into the data memory block in the second cache module 103 based on the first metadata is actually the process of filling the first metadata as the data ID of several blocks of synchronization into the data memory block and then storing the corresponding image data; caching the spectral data into the corresponding data memory block based on the matching relationship between the first metadata and the second metadata is actually analyzing whether the first metadata and the second metadata in the data block are consistent or corresponding, and storing the corresponding spectral data into the data memory block if they are consistent or corresponding, so as to realize the synchronous alignment processing of the spectral data and the image data.

[0078] More specifically, image data is transferred based on a hierarchical strategy of "meta information first, image later", which can effectively maintain the temporal sequence of image data in a multi-threaded working environment.

[0079] More specifically, the first meta information includes an image ID, and the second meta information includes a spectrum ID, wherein the image ID and the spectrum ID are data IDs used to synchronously align image data and spectrum data.

[0080] In some preferred embodiments, Figure 2 and Figure 3 As shown, the second cache module 103 includes a first cache queue 1031 and a second cache queue 1032. The first cache queue 1031 and the second cache queue 1032 both include a number of data memory blocks. The first cache queue 1031 is used to cooperate with the synchronization alignment module 102 to synchronize and align the received first modal data and the second modal data, and the second cache queue 1032 is used to cache the first modal data and the second modal data after synchronization alignment.

[0081] Specifically, in this embodiment, the data memory blocks in the first cache queue 1031 and the second cache queue 1032 can be converted into each other, that is, the data memory blocks in the second cache queue 1032 will return to the tail of the first cache queue 1031 after the data output is emptied to receive data again, and the data memory blocks in the first cache queue 1031 will be transferred to the tail of the second cache queue 1032 after storing the first modal data and the second modal data, thereby realizing the recycling of data memory blocks.

[0082] More specifically, each of the first cache queue 1031 and the second cache queue 1032 is provided with a mutex lock to ensure thread safety.

[0083] More specifically, the data (synchronous data) stored in the data memory block in the second cache queue 1032 includes at least data ID (first metadata or second metadata, preferably first metadata), image data and spectral data. In some other embodiments, it also includes an image flag bit, which is used to indicate whether the image data in the corresponding data memory block is empty.

[0084] More specifically, in this implementation, the multimodal data management system of the embodiment of the present application can also determine whether the spectral data is missing by comparing the first metadata and the second metadata.

[0085] In some preferred implementations, the first cache queue 1031 automatically creates a new data memory block when the data memory block therein is empty.

[0086] Specifically, in this embodiment, the queue group consisting of the first cache queue 1031 and the second cache queue 1032 in the second cache module 103 supports dynamic expansion. When the number of data memory blocks is insufficient, the first cache queue 1031 will create a new data memory block to meet data transmission requirements.

[0087] More specifically, the second cache queue 1032 sets the first cache queue 1031 and the second cache queue 1032 to implement a memory recovery function, and sets the first cache queue 1031 as a memory space with a dynamic expansion function, which can help it to achieve long-term work under high-throughput data flow, reduce the time required to open up additional memory, and help improve data flow efficiency.

[0088] In some preferred implementations, the multimodal data management system of the embodiment of the present application further includes: a data acquisition module (not shown in the figure), which is connected to the multimodal data acquisition device.

[0089] Specifically, the data acquisition module is connected to the acquisition device based on the interface protocol of the acquisition device, and is used to acquire the multimodal data of the data acquisition device in real time and store it in the first cache module 101 .

[0090] More specifically, the data acquisition module preferably includes: an image data acquisition module and a spectral data acquisition module, wherein the image data acquisition module establishes a connection with the image acquisition device based on the interface protocol of the image acquisition device, is used to receive image data and corresponding acquisition timestamps, and stores the image data and corresponding acquisition timestamps into the first cache module 101 in the order of receipt, the first cache module 101 allocates first metadata according to the stored image data and corresponding acquisition timestamps based on timing, and the spectral data acquisition module establishes a connection with the spectral acquisition device based on the interface protocol of the spectral acquisition device, is used to receive and parse spectral data and corresponding second metadata, and stores the spectral data and corresponding second metadata into the first cache module 101 in the order of receipt.

[0091] It should be noted that the image acquisition device and the spectrum acquisition device are triggered based on the synchronization signal, so that the image data and the spectrum data have a corresponding relationship and can be synchronously aligned; the spectrum information collected by the spectrum acquisition device includes spectrum data and spectrum ID, and the spectrum ID is accumulated in time sequence.

[0092] In some preferred embodiments, the third cache module 105 is composed of multiple cache queue groups, each cache queue group includes two cache queues, and its structure is consistent with the second cache module 103, except that its two cache queues are only used to store conversion data, and there is no need to consider data alignment processing, that is, the two cache queues each contain a number of data memory blocks, and the data memory block of the first queue is transferred to the end of the second queue after storing the conversion data, and the conversion data stored in the data memory block in the second queue is returned to the first queue again after being removed based on the flow storage module 104, and the first queue also has a dynamic expansion function.

[0093] Specifically, in this embodiment, the number of cache queue groups in the third cache module 105 is determined based on the difference between the rate at which data is written to the hard disk and the rate at which the first modal data and the second modal data are acquired, thereby ensuring that the amount of converted data stored in the third cache module 105 does not overflow.

[0094] In some preferred embodiments, Figure 4 As shown, the flow storage module 104 includes:

[0095] A producer thread is used to extract the first modal data and the second modal data after synchronization alignment from the second cache module 103, and convert the first modal data and the second modal data after synchronization alignment into conversion data of a target format, so as to store the conversion data into the third cache module 105;

[0096] The consumer thread is used to extract the conversion data in the target format from the third cache module 105 , and when the extracted conversion data reaches a preset amount, merge the preset amount of conversion data to obtain a storage file, and store the storage file into the storage hard disk 106 .

[0097] Specifically, as a second-level scheduling module, the flow storage module 104 needs to store the obtained synchronized and aligned data in a certain format to the computer's storage hard disk 106. Therefore, this module mainly involves scheduling and storage based on image data and spectral data.

[0098] More specifically, since the rate at which data is written to the storage hard disk 106 is lower than the rate at which data is acquired and synchronized, in an embodiment of the present application, the flow storage module 104 adopts a "consumer-producer" model to improve the rate of data flow and storage; wherein the producer thread is used to complete the transfer of the aligned and synchronized data from the second cache module 103 to the third cache module 105, and to realize the timely recovery of the data memory blocks in the second cache queue 1032 in the second cache module 103, and the aligned and synchronized data is converted into conversion data with a target format during the transfer process; the recovery of the data memory blocks can reduce the probability of creating additional new data memory blocks to avoid possible traffic bottlenecks as much as possible, and maintain the circulation efficiency of the data memory blocks to support long-term work; the consumer thread is used to complete the consumption of the conversion data of the third cache module 105 (merged into a storage file to be moved to the storage hard disk 106), and to realize the timely recovery of the cache queue group in the third cache module 105. In this embodiment, multiple data memory blocks storing a preset number of conversion data in the third cache module 105 can be temporarily placed in a consumer queue, and batch recovery can be realized after the conversion data is consumed.

[0099] More specifically, the preset number is set based on the size of the spectral data group collected by the spectral collection device, so that the flow storage module 104 can merge the converted data from the same spectral data group into one storage data to be written into the storage hard disk 106 as a storage file.

[0100] More specifically, overly dispersed data will cause the hard disk to create dispersed data without a clear starting point based on the process of randomly writing to the memory, resulting in the inability to utilize the full capacity of the hard disk and increasing the addressing time from obtaining the write command to finding the address. Therefore, the processing method of merging multiple converted data into a larger storage file and writing it to the storage hard disk 106 can avoid the written data being too dispersed, thereby improving the write performance of the hard disk.

[0101] More specifically, in this embodiment, each producer thread is preferably used to process the synchronized and aligned data corresponding to the spectral data of the same source, which can be triggered and controlled based on the data ID in the second cache module 103 .

[0102] In some preferred implementations, there are multiple producer threads and multiple consumer threads, and the number of consumer threads is greater than the number of producer threads.

[0103] Specifically, in an embodiment of the present application, the flow storage module 104 adopts a "multi-consumer-multi-producer" model to further improve the rate of data flow and storage, wherein the ratio of the number of producer threads to consumer threads is not less than 2:1, so as to expand the caliber of data outflow and improve the concurrency of data writing to the hard disk.

[0104] Second, please refer to Figure 5 Some embodiments of the present application also provide a multimodal data management method, which is applied in a multimodal data management system. The multimodal data management system includes:

[0105] A first cache module 101, configured to cache received first modality data and second modality data;

[0106] A second cache module 103, configured to cache the synchronously aligned first modality data and second modality data;

[0107] A third cache module 105, used to store the converted data in the target format;

[0108] The multimodal data management method includes the following steps:

[0109] S1, extracting first modal data and second modal data from the first cache module 101, and storing the first modal data and the second modal data in a synchronously aligned manner into the second cache module 103;

[0110] S2, extracting the first modality data and the second modality data after synchronization alignment from the second cache module 103, and converting the first modality data and the second modality data after synchronization alignment into conversion data in a target format;

[0111] S3, extracting the conversion data in the target format from the third cache module 105, and when the extracted conversion data reaches a preset amount, merging the preset amount of conversion data and storing them in the storage hard disk 106.

[0112] The multimodal data management method of the embodiment of the present application is based on the first cache module 101, the second cache module 103 and the third cache module 105 to cache the collected multimodal data, the synchronously aligned multimodal data and the converted data after format conversion to achieve three-level data cache, and the multimodal data about flow cells is integrated, organized and stored in the storage hard disk 106 by scheduling the cached data. This processing process effectively matches the data flow rate difference in each link of data processing, realizes the real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams, and can meet the transmission and storage requirements of flow cell multimodal data.

[0113] Based on the same concept as the system in the above embodiment, the multimodal data management method provided in the present application is implemented by the system in the above embodiment. The specific limitations of each step can refer to the description of the system and will not be repeated here.

[0114] Third, please refer to Figure 6 Some embodiments of the present application also provide a structural diagram of an electronic device. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiments.

[0115] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method in any optional implementation of the above embodiment is executed. The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0116] In summary, the embodiments of the present application provide a multimodal data management system, method, electronic device and storage medium, wherein the multimodal data management system caches the collected multimodal data, the synchronously aligned multimodal data and the converted data after format conversion based on the first cache module 101, the second cache module 103 and the third cache module 105 respectively to realize three-level data cache, and schedules the cached data based on the synchronous alignment module 102 and the flow storage module 104 to integrate and organize the multimodal data about flow cells and store them in the storage hard disk 106. This processing process effectively matches the data flow rate difference in each link of data processing, realizes the real-time acquisition, synchronous alignment and orderly storage of high-throughput multimodal data streams, and can meet the transmission and storage requirements of flow cell multimodal data.

[0117] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0118] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0120] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0121] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal data management system for arranging multimodal data of flow cells based on time series, characterized in that: The multimodal data management system comprises: A first cache module, configured to cache received first modal data and second modal data; a synchronization alignment module, connected to the first cache module, configured to extract the first modality data and the second modality data from the first cache module, and store the first modality data and the second modality data in a second cache module in a synchronously aligned manner; The second cache module is connected to the synchronization alignment module and is used to cache the first modality data and the second modality data that are synchronously aligned; a flow storage module connected to the second cache module, configured to extract the first modal data and the second modal data after synchronization alignment from the second cache module, and convert the first modal data and the second modal data after synchronization alignment into conversion data of a target format; A third cache module, connected to the flow storage module, for storing the conversion data in the target format; The flow storage module is further used to extract the conversion data of the target format from the third cache module, and when the extracted conversion data reaches a preset amount, merge the preset amount of conversion data and store it in the storage hard disk; The circulation storage module includes: A producer thread, configured to extract the first modal data and the second modal data after synchronization alignment from the second cache module, and convert the first modal data and the second modal data after synchronization alignment into conversion data in a target format, so as to store the conversion data in the third cache module; The consumer thread is used to extract the conversion data in the target format from the third cache module, and when the extracted conversion data reaches a preset amount, merge the preset amount of conversion data to obtain a storage file, and store the storage file in the storage hard disk.

2. The multimodal data management system according to claim 1, characterized in that: The multimodal data management system is applied in a flow cytometer, wherein the first modal data is image data and the second modal data is spectral data.

3. The multimodal data management system according to claim 2, characterized in that: The synchronization alignment module transmits the image data based on multiple threads and transmits the spectral data based on a single thread.

4. The multimodal data management system according to claim 2, characterized in that: The first cache module is further configured to allocate first meta information to the image data based on a time sequence when the image data is stored therein; The second cache module includes a plurality of data memory blocks, and the process of the synchronization alignment module synchronously aligning the image data and the spectral data includes: When a frame of image data is extracted from the first cache module, the image data is cached in a data memory block in the second cache module based on the first meta information; When a piece of spectral data is extracted from the first cache module, second meta information of the spectral data is obtained, and the spectral data is cached in a corresponding data memory block based on a matching relationship between the first meta information and the second meta information.

5. The multimodal data management system according to claim 1, characterized in that: The second cache module includes a first cache queue and a second cache queue, each of which includes a plurality of data memory blocks. The first cache queue is used to cooperate with the synchronization alignment module to synchronously align and receive the first modality data and the second modality data, and the second cache queue is used to cache the first modality data and the second modality data after synchronization alignment.

6. The multimodal data management system according to claim 1, characterized in that: There are multiple producer threads and multiple consumer threads, and the number of consumer threads is greater than the number of producer threads.

7. A multimodal data management method, applied in the multimodal data management system according to any one of claims 1 to 6, characterized in that: The multimodal data management system comprises: A first cache module, configured to cache received first modal data and second modal data; A second cache module, used for caching the synchronously aligned first modality data and second modality data; A third cache module, used for storing the converted data in the target format; The multimodal data management method comprises the following steps: S1. extracting the first modal data and the second modal data from the first cache module, and storing the first modal data and the second modal data in a synchronous and aligned manner into a second cache module; S2, extracting the first modality data and the second modality data after synchronization alignment from the second cache module, and converting the first modality data and the second modality data after synchronization alignment into conversion data in a target format; S3. Extracting conversion data in a target format from the third cache module, and when the extracted conversion data reaches a preset amount, merging the preset amount of conversion data and storing them in a storage hard disk.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method as claimed in claim 7 are executed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 7 are executed.

Images