Data storage method and device and data reading method and device

By extracting and merging the original data of biological data files and classifying and storing and reading them in preset formats, the problem of slow analysis speed and high cost caused by disordered information of gene expression matrix text files is solved, and efficient biological data storage is achieved and read.

CN119028444APending Publication Date: 2024-11-26SHENZHEN HUADA SANJIAN QIFA TECHNOLOGY CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202310604448.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing biological data storage formats, especially the gene expression matrix text files, have disordered information, resulting in slow downstream analysis speed and increased analysis costs.

Method used

Extract the original data of biological data files, merge the gene data set through multi-thread parallel processing, and classify the merged data and read it in a preset format. Use the API interface to write the data to the file to achieve orderly storage and Quick read.

Benefits of technology

Converting disordered biological data into ordered data reduces data redundancy, reduces storage and analysis costs, and accelerates data reading and analysis query speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028444A_ABST
    Figure CN119028444A_ABST
Patent Text Reader

Abstract

The invention provides a data storage method and device and a data reading method and device, and relates to the technical field of computers. The method comprises the following steps: acquiring a biological data file, and extracting original biological data of the biological data file; carrying out data merging on the original biological data; and classifying and storing the merged biological data according to a preset format. Classified storage according to the preset format is achieved, disordered data are converted into ordered data, data redundancy is reduced, and the storage cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data storage method, a data reading method and a device. Background Art

[0002] With the development of spatiotemporal omics and sequencing technologies, more and more biological data files will be generated, especially data files for quantitative analysis of gene expression. These files record the name, coordinate points and corresponding expression of genes, and will be repeatedly used by downstream analysis processes. The storage and analysis costs of biological data are constantly increasing, so a new biological data format is needed to store and manage these data for downstream analysis.

[0003] However, the current biological data storage format is mostly gene expression matrix text files. The information stored in gene expression matrix text files is relatively disordered, and downstream analysis requires traversing the entire file. The entire analysis process is very slow, which increases the analysis cost. Summary of the invention

[0004] The present disclosure provides a data storage method, a data reading method, a device, an electronic device and a storage medium.

[0005] According to a first aspect of the present disclosure, a data storage method is provided, the method comprising: acquiring a biological data file and extracting original biological data from the biological data file; merging the original biological data; and classifying and storing the merged biological data in a preset format.

[0006] In some embodiments, obtaining a biological data file and extracting raw biological data from the biological data file includes: preprocessing the biological data file to obtain multiple gene data sets, each gene data set including a gene name, gene coordinates, and gene expression; and extracting corresponding raw biological data based on the gene name, wherein the raw biological data includes a gene offset and a gene number.

[0007] In some embodiments, preprocessing the biological data file to obtain multiple gene data sets includes: reading the biological data file in parallel through multiple threads; merging gene coordinates and gene expression levels with the same gene name into one gene data set to obtain multiple gene data sets.

[0008] In some embodiments, data merging of raw biological data includes: traversing multiple gene data sets, and processing data in the multiple gene data sets in parallel through multiple threads; each thread delivers the processed data to a data queue; the main thread extracts data from the data queue, and merges the data of each gene into a first genome, wherein the first genome includes a first array and a second array, the gene name, gene offset and number of genes of each gene are merged into the first array, and the gene coordinates and gene expression of each gene are merged into the second array.

[0009] In some embodiments, the first genome also includes a third array, and the main thread extracts data from the data queue, and merging the data of each gene into the first genome includes: merging the number of exons of each gene into the third array, wherein the third array corresponds one-to-one to the second array.

[0010] In some embodiments, data merging of the original biological data further includes: the main thread merging the number of genes and the number of gene expression levels contained in the DNA nanoball block into a second genome, the second genome including a first matrix, wherein each item of the first matrix corresponds to a DNA nanoball block.

[0011] In some embodiments, the main thread merges the number of genes and the number of gene expression levels contained in the DNA nanoball block into the second genome, including: merging genes with the same gene name in the same DNA nanoball block into the same gene name; and accumulating the number of gene expression levels corresponding to the genes.

[0012] In some embodiments, the main thread merging the number of genes and the number of gene expression levels contained in the DNA nanoball block into the second genome includes: merging the data of the DNA nanoball blocks contained in different first matrices in the second genome.

[0013] In some embodiments, data merging of the original biological data also includes: the main thread merges the number of exons contained in the DNA nanoball block into a third genome, the third genome includes a second matrix, wherein each item of the second matrix corresponds to a DNA nanoball block, and the coordinate points of the second matrix correspond one-to-one to the coordinate points of the first matrix.

[0014] In some embodiments, data merging of the original biological data further includes: the main thread merging the gene name, the number of gene expression levels, and the gene enrichment information of each gene into a fourth gene set.

[0015] In some embodiments, classifying and storing the merged biological data according to a preset format includes: calling an API to write the merged biological data into a biological data file, wherein the preset format is a classification storage table of gene number, gene name, gene offset and gene number, and different genomes are written into any position of the biological data file in any order.

[0016] According to the embodiments of the present disclosure, by acquiring a biological data file and extracting the original biological data of the biological data file; merging the original biological data; and classifying and storing the merged biological data according to a preset format, classification and storage according to a preset format are implemented, and disordered data is converted into ordered data, thereby reducing data redundancy and lowering storage costs.

[0017] According to a second aspect of the present disclosure, a data reading method is provided, the method comprising: acquiring biological data information, the biological data information comprising gene names and / or gene numbers; and classifying and reading biological data corresponding to the gene names and / or gene numbers according to the biological data information.

[0018] In some embodiments, reading the biological data corresponding to the gene name and / or the gene number includes: reading the biological data corresponding to the gene name and / or the gene number in parallel through multiple threads.

[0019] In some embodiments, reading biological data corresponding to a gene name and / or a gene number includes: obtaining a gene coordinate range; and reading biological data corresponding to a gene name and / or a gene number within the gene coordinate range.

[0020] In some embodiments, the gene coordinate range is a preset shape, or the method further includes: receiving a read range instruction, wherein the read range instruction includes the gene coordinate range.

[0021] In some embodiments, according to the biological data information, classifying and reading biological data corresponding to the gene name and / or gene number from the biological data file includes: reading the gene offset and the number of genes corresponding to the gene name and / or gene number from a classification storage table in a preset format according to the gene name and / or gene number; determining the reading starting position according to the gene offset; and reading the gene data corresponding to the number of genes from the first genome.

[0022] According to an embodiment of the present disclosure, biological data information is obtained, and the biological data information includes gene names and / or gene numbers; based on the biological data information, biological data corresponding to the gene names and / or gene numbers are classified and read, so as to realize reading of orderly biological data information, accelerate the speed of analysis query, and reduce analysis costs.

[0023] According to a third aspect of the present disclosure, a data storage device is provided, which includes: an acquisition unit for acquiring a biological data file and extracting original biological data from the biological data file; a merging unit for merging the original biological data; and a storage unit for classifying and storing the merged biological data in a preset format.

[0024] According to the embodiments of the present disclosure, a biological data file is obtained through a data storage device, and the original biological data of the biological data file is extracted; the original biological data is merged; the merged biological data is classified and stored according to a preset format, thereby realizing classified storage according to a preset format, converting disordered data into ordered data, reducing data redundancy, and lowering storage costs.

[0025] According to a fourth aspect of the present disclosure, a data reading device is provided, which includes: an acquisition unit for acquiring biological data information, the biological data information including gene names and / or gene numbers; and a reading unit for classifying and reading biological data corresponding to the gene names and / or gene numbers according to the biological data information.

[0026] According to an embodiment of the present disclosure, biological data information is acquired through a data reading device, the biological data information including gene names and / or gene numbers; based on the biological data information, biological data corresponding to the gene names and / or gene numbers are classified and read, thereby realizing orderly reading of biological data information, accelerating the speed of analysis query, and reducing analysis costs.

[0027] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0028] at least one processor; and

[0029] a memory communicatively connected to at least one processor; wherein,

[0030] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of the first aspect or the second aspect mentioned above.

[0031] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method of the aforementioned first aspect or second aspect.

[0032] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method of the first aspect or the second aspect as described above.

[0033] According to the embodiments of the present disclosure, by acquiring a biological data file and extracting the original biological data of the biological data file; merging the original biological data; and classifying and storing the merged biological data according to a preset format, classification and storage according to a preset format are implemented, and disordered data is converted into ordered data, thereby reducing data redundancy and lowering storage costs.

[0034] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0036] Figure 1 A schematic diagram of a data storage method provided by an embodiment of the present disclosure;

[0037] Figure 2 A schematic diagram of a data storage method provided by an embodiment of the present disclosure;

[0038] Figure 3 A flowchart of a data reading method provided by an embodiment of the present disclosure;

[0039] Figure 4 A schematic diagram of obtaining expression and exon data of a gene provided in an embodiment of the present disclosure;

[0040] Figure 5 A schematic diagram of a BGEF data storage structure provided in an embodiment of the present disclosure;

[0041] Figure 6 A structural schematic diagram of a data storage device provided in an embodiment of the present disclosure;

[0042] Figure 7 A structural schematic diagram of a data reading device provided in an embodiment of the present disclosure;

[0043] Figure 8 A schematic block diagram of an exemplary electronic device 800 is provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] The following describes a data storage method, a data reading method, an apparatus, an electronic device, and a storage medium according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0046] With the development of spatiotemporal omics and sequencing technologies, more and more biological data files will be generated, especially data files for quantitative analysis of gene expression. These files record the name, coordinate points and corresponding expression of genes, and will be repeatedly used by downstream analysis processes. The storage and analysis costs of biological data are constantly increasing, so a new biological data format is needed to store and manage these data for downstream analysis.

[0047] However, the data storage format of gene expression quantitative analysis is mostly gene expression matrix text file, the content is as follows:

[0048]

[0049] Among them, the gene expression amount can also be UMI count, molecular identifier (MID, Molecular Identifiers), unique molecular identifier (UMI, unique molecular identifiers). The above gene expression matrix text file is only an example of the present disclosure and is not limited in the present disclosure.

[0050] It can be seen that the information stored in the gene expression matrix text file is relatively disordered, and downstream analysis requires traversing the entire file. The entire analysis process is very slow, which increases the analysis cost.

[0051] To this end, in response to the above problems, the present disclosure proposes a data storage method, defines a new storage format BGEF (Bin Gene Expression File) for quantitative analysis data of gene expression, pre-analyzes and processes the text file, extracts part or all of the key information, and then stores the processed data in a classified manner, thereby converting disordered data into ordered data, reducing information redundancy and storage costs. At the same time, it also speeds up the speed of downstream data reading and reduces analysis costs. At the same time, it can pre-calculate and store gene expression data in various preset file formats, and can calculate and store various statistical information in advance, which is convenient for subsequent information query and analysis statistics.

[0052] The data storage method proposed in the present disclosure can be applied to a device with a data analysis function, and the application scenario is not limited in the present disclosure. The present disclosure mainly takes the data file scenario of quantitative analysis of gene expression as an example.

[0053] The following describes in detail with reference to the accompanying drawings a data storage method, a data reading method, a device, an electronic device, a storage medium, and a program product proposed in the present disclosure.

[0054] Figure 1 A schematic diagram of a data storage method provided by an embodiment of the present disclosure. Figure 1 As shown, the method comprises the following steps:

[0055] Step 101: Obtain a biological data file and extract original biological data from the biological data file.

[0056] In one implementation of the present disclosure, an input biological data file is read in parallel, and raw data information of the file is extracted.

[0057] The original biological data includes gene offset and gene number, which are part or all of the gene names, gene coordinates and gene expression in the biological data file obtained through pre-analysis and processing. The original biological data is extracted from the processed biological data file according to the gene name.

[0058] It should be noted that there may be multiple biological data files or one biological data file, which is not limited in the embodiments of the present disclosure.

[0059] Step 102: merge the original biological data.

[0060] In one embodiment of the present disclosure, data merging is performed based on the gene names obtained by preprocessing the biological data file to obtain multiple gene data sets. Then, the multiple gene data sets obtained are traversed, and each gene data set is processed in parallel using a thread pool, and each thread delivers the processed data to a data queue; the main thread then extracts data from the data queue, and merges the data of each gene into a first genome, wherein the first genome includes a first array and a second array, and the gene name, gene offset, and gene number of each gene are merged into the first array, and the gene coordinates and gene expression of each gene are merged into the second array.

[0061] Step 103, classify and store the merged biological data according to a preset format.

[0062] In one embodiment of the present disclosure, the biological data acquired according to the above steps is classified and stored according to a preset format.

[0063] In the present disclosure, the merged biological data can be written into a biological data file according to a preset format by calling an application program interface (eg, an API interface).

[0064] Among them, API interface (Application Program Interface) is a set of predefined functions, through which computer software can communicate with each other. Developers can develop applications through API interface programs, which can reduce the writing of useless programs and ease programming tasks.

[0065] The preset format is a classified storage table of gene number, gene name, gene offset and gene number, and different gene sets are written into any position of the biological data file in any order, that is, there is no restriction on the storage location and writing order of biological data in the present disclosure.

[0066] Therefore, according to the embodiments of the present disclosure, by acquiring a biological data file and extracting the original biological data of the biological data file; merging the original biological data; and classifying and storing the merged biological data according to a preset format, classification and storage according to a preset format are implemented, and disordered data is converted into ordered data, thereby reducing data redundancy and lowering storage costs.

[0067] Figure 2 A flowchart of a data storage method provided in an embodiment of the present disclosure. Figure 2 based on Figure 1 In the embodiment shown, step 101 and step 103 are further defined. Figure 2 In the embodiment shown, step 101 includes step 201 and step 202, and step 103 includes step 204. Figure 2 As shown, the method includes the following steps.

[0068] Step 201 , preprocessing the biological data file to obtain a plurality of gene data sets, each gene data set including a gene name, gene coordinates and gene expression level.

[0069] In some embodiments of the present disclosure, multiple threads are used to read input files in parallel, and data of biological data files are merged according to gene names; according to gene offsets and gene numbers, gene data sets are obtained, and the gene data sets include gene names and gene coordinates and gene expression values ​​corresponding to the gene names. In other words, gene coordinates and gene expression values ​​with the same gene name are merged into one gene data set to obtain multiple gene data sets. An example of a gene data set for bin1 is as follows:

[0070] Gene 1{(x,y,Mid),(x,y,mid),(x,y,mid),…}

[0071] Gene 2{(x,y,Mid),(x,y,mid),(x,y,mid),…}

[0072] …

[0073] Gene N{(x,y,Mid),(x,y,mid),(x,y,mid),…}

[0074] Among them, x, y are the coordinates of the gene, and mid is the expression level of the gene.

[0075] It can be understood that preprocessing of biological data files is to extract some attributes that have an important impact on the target output from a large number of data attributes in the biological data files, that is, to reduce the dimension of the original data, so as to achieve the purpose of improving the quality of biological data and increasing the speed of biological data mining.

[0076] Step 202: extract the corresponding original biological data according to the gene name, wherein the original biological data includes the gene offset and the number of genes.

[0077] In some embodiments of the present disclosure, Extract key biological data from gene datasets based on gene names Data, i.e., the original biological data in this disclosure.

[0078] Step 203: merge the original biological data.

[0079] In some embodiments of the present disclosure, multiple gene data sets are traversed, and data in the multiple gene data sets are processed in parallel through multiple threads; each thread delivers the processed data to a data queue; the main thread extracts data from the data queue, and merges the data of each gene into a first genome, wherein the first genome includes a first array and a second array, the gene name, gene offset and number of genes of each gene are merged into the first array, and the gene coordinates and gene expression of each gene are merged into the second array.

[0080] The first genome can be a geneExp group, which can contain several binsize information, each binsize is a dataset. The first array can be a gene dataset, which is a one-dimensional array. Each item of data is a combination, recording the information of each gene (gene name, gene offset and gene number).

[0081] The second array may be expression, which is a one-dimensional array. Each item in the array is a combination that records part or all of the coordinate information (x, y) and gene expression level of each gene.

[0082] In an optional embodiment of the present disclosure, the first genome also includes a third array, and the main thread extracts data from the data queue and merges the number of exons of each gene into the third array, wherein the third array corresponds one-to-one to the second array.

[0083] In the present disclosure, the third genome may be an exon, which is a one-dimensional array that records the number of part or all of the exons of each gene and has a one-to-one correspondence with expression.

[0084] It is important to understand that exons (expressed regions) are part of eukaryotic genes. They are preserved after splicing and can be expressed as proteins during protein biosynthesis. Exons are gene sequences that appear last in mature RNA, also known as expressed sequences.

[0085] In an optional embodiment of the present disclosure, the main thread merges the number of genes and the number of gene expression levels contained in the DNA nanoball block into a second genome, and the second genome includes a first matrix, wherein each item of the first matrix corresponds to a DNA nanoball block.

[0086] The second genome may be a wholeExp group, and the wholeExp group may contain several binsize information. The first matrix may be a two-dimensional matrix maintained in each binsize in the wholeExp group, that is, each binsize is a dataset, which stores a two-dimensional matrix data.

[0087] Each entry in the matrix corresponds to a dnd, recording the midcount (number of expression levels) and genecount (number of genes) contained in this dnb.

[0088] Specifically, genes with the same gene name in the same DNA nanoball block are merged under the same gene name; the number of gene expression levels corresponding to the gene is accumulated. At the same time, the data of the DNA nanoball blocks contained in different first matrices in the second genome are merged. In other words, the dnb information is counted, that is, the number of genes contained in each dnb (DNA nanoball) block and the corresponding number of gene expression levels. Each bin will save part or all of the gene data. Due to the setting of binsize, genes with the same gene name in the same dnb will be regarded as one gene, and the gene expression levels will be accumulated. The types of genes will not decrease, but the corresponding expression array will decrease (gene data will be merged).

[0089] It can be understood that the data merging of dnb is the data merging of dnb between different bins, that is, merging is performed on the basis of bin1 to form bin10, bin20, etc., so as to realize data storage of different dimensions.

[0090] Taking bin10 as an example, the entire area will be divided into several small blocks of 10*10, each small block is regarded as a dnb block, and the coordinates of the genes in it are adjusted according to the following formula: x'=x / 10*10, y'=y / 10*10. If the same gene name exists in the same small block, mid is accumulated. For example:

[0091] gene1(102,102,1)gene1(105,105,1)→gene1(100,100,2)

[0092] gene2(107,107,2)gene2(109,109,2)→gene2(100,100,4)

[0093] The statistical information of the dnb block is: the number of genes is 2, and the number of mids is 6.

[0094] In an optional embodiment of the present disclosure, the main thread merges the number of exons contained in the DNA nanoball block into a third genome, and the third genome includes a second matrix, wherein each item of the second matrix corresponds to a DNA nanoball block, and the coordinate points of the second matrix correspond one-to-one to the coordinate points of the first matrix.

[0095] The third genome can be a wholeExpExon group, which can contain several binsize information. The second matrix can be a two-dimensional matrix stored in the binsize in the wholeExpExon group, that is, each binsize is a dataset, which stores a two-dimensional matrix data. Each item of the two-dimensional matrix corresponds to a dnb, which records the number of exons contained in this dnb. The wholeExpExon group and the wholeExp coordinate point are one-to-one corresponding.

[0096] In an optional embodiment of the present disclosure, the main thread merges the gene name, the number of gene expression levels, and the gene enrichment information of each gene into the fourth gene set.

[0097] The fourth gene set may be a stat group, which includes the name of each gene, the corresponding number of gene expression levels, and the corresponding E10 score (i.e., gene enrichment information) at bin100 resolution.

[0098] Step 204: Calling an API to write the merged biological data into a biological data file according to a preset format.

[0099] In some embodiments of the present disclosure, each thread delivers its processed data to the data queue, the main thread extracts the corresponding data from the data queue, generates the corresponding classification storage table according to the preset format, and finally calls the HDF5 API to write the biological data to the file. It should be noted that different genomes are written to any position of the biological data file in any order, that is, the storage location and writing order of the biological data are not limited in the present disclosure.

[0100] The preset format is a classified storage table of gene number, gene name, gene offset and gene number.

[0101] In summary, according to the embodiments of the present disclosure, the original data information is retained and classified, data redundancy is reduced, and other preset data can be counted as needed, which accelerates downstream analysis and query. Compared with the gene expression matrix text file, the reading and query speed is improved.

[0102] Figure 3 The following is a flow chart of a data reading method provided by an embodiment of the present disclosure. Figure 3 As shown, the method comprises the following steps:

[0103] Step 301, obtaining biological data information, the biological data information including gene name and / or gene number.

[0104] In some embodiments of the present disclosure, the gene name and / or gene number to be queried in the biological data file is obtained.

[0105] Step 302: According to the biological data information, classify and read the biological data corresponding to the gene name and / or gene number.

[0106] In some embodiments of the present disclosure, a gene coordinate range is obtained; and biological data corresponding to a gene name and / or a gene number is read in parallel through multiple threads. The gene coordinate range is a preset shape, or, in the present disclosure, a read range instruction may be received, and the read range instruction includes the gene coordinate range.

[0107] In other words, biological data information is read in parallel through multiple threads; it can be filtered according to a rectangular area (or gene coordinate range), such as [100,200][600,700], or it can be filtered according to an irregular area selected by the user, and then the biological data in the area can be filtered out.

[0108] According to the gene name and / or gene number, the gene offset and the number of genes corresponding to the gene name and / or gene number are read from a classification storage table in a preset format; the reading start position is determined according to the gene offset; and the gene data corresponding to the number of genes are read from the first genome.

[0109] Specifically, the first array, the second array, and the third array of the first genome cooperate with each other to obtain the corresponding biological data. According to the classification storage table in the preset format, and the obtained gene name, offset, and number, the data starting position of the current gene can be obtained from the second array and the third array respectively according to the offset, and the data of the specified size can be read according to the number of genes. These data are the corresponding biological data of the gene.

[0110] For example, the BGEF data example:

[0111] id gene offset count 0 Fth1 0 5 1 Hnrnpdl 5 4 2 Ntrk2 9 10 3 Rnf187 19 6

[0112] According to the gene information, the expression information of the gene can be queried in the expression array. If you want to find the information of Rnf187 No. 3, you need to start from the 19th position of the expression array and read 6 data. These 6 data are the expression information contained in Rnf187. The same is true for exon information acquisition.

[0113] like Figure 4 The figure shows a schematic diagram of obtaining the expression and exon data of a gene. According to the name Hnrnpal, (Hnrnpal, 5, 4) can be matched from the gene dataset, which means that the data of Hnrnpal is stored from position 5 in the expression and exon arrays, and there are 4 data in total. Similarly, (Rnf87, 19, 6) means that the data of Rnf87 is stored from position 19 in the expression and exon arrays, and there are 6 data in total.

[0114] In summary, according to the embodiments of the present disclosure, by acquiring biological data information, the biological data information includes gene names and / or gene numbers; according to the biological data information, the biological data corresponding to the gene names and / or gene numbers are classified and read, so as to realize the reading of orderly biological data information, accelerate the speed of analysis query, and reduce the analysis cost.

[0115] For ease of understanding, based on Figure 1 , Figure 2 In the embodiment shown, the present disclosure provides a schematic diagram of a BGEF data storage structure, such as Figure 5 shown.

[0116] In some embodiments of the present disclosure, the BGEF file structure is similar to the hierarchical structure of a directory, which is divided into groups and datasets. Groups are similar to directories, and datasets are similar to files. BGEF contains a required gene Exp group and multiple optional groups (wholeExp, wholeExpExon, stat). The format of each group is as follows:

[0117] (1) Gene Exp group

[0118] Gene Exp group can contain several binsize information. Each binsize is a dataset, which contains three datasets: gene, expression, exon (exon, optional dataset).

[0119] The gene dataset is a one-dimensional array, and each item of data is a combination that records the information of each gene (gene name, offset and number).

[0120] expression is a one-dimensional array. Each item in the array is a combination that records part or all of the coordinate information (x, y) and mid of each gene.

[0121] Exon is a one-dimensional array that records the number of some or all exons of each gene, and has a one-to-one correspondence with expression.

[0122] These three datasets need to work together to obtain specific data. According to the above format definition, after obtaining the name, offset and number of a gene, the data starting position of this gene can be obtained from the expression and exon arrays according to the offset, and the data of the specified size can be read according to the number. These data are the valid data of the gene. The expression and exon arrays are of equal length and share the offset and number of the same gene.

[0123] (2) wholeExp group (optional)

[0124] The wholeExp group can contain several binsize information. Each binsize is a dataset that stores a two-dimensional matrix. Each entry in the matrix corresponds to a dnd, which records the midcount (number of expression levels) and gene count contained in this dnb.

[0125] (3)wholeExpExon group (optional)

[0126] The wholeExpExon group can contain several binsize information. Each binsize is a dataset that stores a two-dimensional matrix. Each entry in the matrix corresponds to a dnd, which records the number of exons contained in this dnb. This group corresponds one-to-one to the wholeExp coordinate point.

[0127] (4) stat group (optional)

[0128] The stat group contains the name of each gene at bin100 resolution, the corresponding number of mids, and the corresponding E10 score (gene enrichment information).

[0129] In summary, the present disclosure defines a new format for storing gene expression data, which sorts out disordered data and stores them by data type, thereby speeding up analysis and query.

[0130] Corresponding to the above-mentioned data storage method, the present disclosure also proposes a data storage device. Figure 6 FIG. 6 is a schematic diagram of a data storage device 600 provided in an embodiment of the present disclosure. Figure 6 As shown, including:

[0131] An acquisition unit 610, configured to acquire a biological data file and extract original biological data from the biological data file;

[0132] A merging unit 620, configured to perform data merging on the original biological data;

[0133] The storage unit 630 is used to classify and store the merged biological data according to a preset format.

[0134] In some embodiments, the acquisition unit 610 is used to: pre-process the biological data file to obtain multiple gene data sets, each gene data set includes a gene name, gene coordinates and gene expression; according to the gene name, extract the corresponding original biological data, wherein the original biological data includes a gene offset and a gene number.

[0135] In some embodiments, the acquisition unit 610 is used to: read biological data files in parallel through multiple threads; merge gene coordinates and gene expression levels with the same gene name into one gene data set to obtain multiple gene data sets.

[0136] In some embodiments, the merging unit 620 is used to: traverse multiple gene data sets, and process the data in the multiple gene data sets in parallel through multiple threads; each thread delivers the processed data to the data queue; the main thread extracts data from the data queue, and merges the data of each gene into a first genome, wherein the first genome includes a first array and a second array, the gene name, gene offset and number of genes of each gene are merged into the first array, and the gene coordinates and gene expression of each gene are merged into the second array.

[0137] In some embodiments, the merging unit 620 is used for: the first genome also includes a third array, the main thread extracts data from the data queue, and merges the number of exons of each gene into the third array, wherein the third array corresponds one-to-one to the second array.

[0138] In some embodiments, the merging unit 620 is used for: the main thread merges the number of genes and the number of gene expression levels contained in the DNA nanoball block into the second genome, and the second genome includes a first matrix, wherein each item of the first matrix corresponds to a DNA nanoball block.

[0139] In some embodiments, the merging unit 620 is used to: merge genes with the same gene name in the same DNA nanoball block into the same gene name; and accumulate the number of gene expression levels corresponding to the genes.

[0140] In some embodiments, the merging unit 620 is used to merge the data of the DNA nanoball blocks contained in different first matrices in the second genome.

[0141] In some embodiments, the merging unit 620 is used for: the main thread merges the number of exons contained in the DNA nanoball block into the third genome, the third genome includes a second matrix, wherein each item of the second matrix corresponds to a DNA nanoball block, and the coordinate points of the second matrix correspond one-to-one to the coordinate points of the first matrix.

[0142] In some embodiments, the merging unit 620 is used for: the main thread merges the gene name, the number of gene expression levels and the gene enrichment information of each gene into the fourth gene set.

[0143] In some embodiments, the storage unit 630 is used to call an API to write the merged biological data into a biological data file, wherein the preset format is a classified storage table of gene number, gene name, gene offset and gene number, and different genomes are written into any position of the biological data file in any order.

[0144] In summary, according to the embodiments of the present disclosure, a biological data file is obtained through a data storage device, and the original biological data of the biological data file is extracted; the original biological data is merged; the merged biological data is classified and stored according to a preset format, thereby realizing classified storage according to a preset format, converting disordered data into ordered data, reducing data redundancy, and reducing storage costs.

[0145] It should be noted that since the device embodiment of the present disclosure corresponds to the above-mentioned method embodiment, the above-mentioned explanation of the method embodiment is also applicable to the device of the present embodiment, and the principle is the same. For details not disclosed in the device embodiment, reference can be made to the above-mentioned method embodiment, and they will not be repeated in this disclosure.

[0146] Corresponding to the above-mentioned data reading method, the present disclosure also provides a data reading device. Figure 7 FIG. 7 is a schematic diagram of a data reading device 700 provided in an embodiment of the present disclosure. Figure 7 As shown, including:

[0147] An acquisition unit 710 is used to acquire biological data information, where the biological data information includes a gene name and / or a gene number;

[0148] The reading unit 720 is used to classify and read the biological data corresponding to the gene name and / or gene number according to the biological data information.

[0149] In some embodiments, the reading unit 720 is used to read biological data corresponding to gene names and / or gene numbers in parallel through multiple threads.

[0150] In some embodiments, the reading unit 720 is used to: obtain a gene coordinate range; and read biological data corresponding to a gene name and / or gene number within the gene coordinate range.

[0151] In some embodiments, the gene coordinate range is a preset shape, or the reading unit 720 is further used to: receive a reading range instruction, wherein the reading range instruction includes the gene coordinate range.

[0152] In some embodiments, the reading unit 720 is used to: read the gene offset and the number of genes corresponding to the gene name and / or the gene number from a classification storage table in a preset format according to the gene name and / or the gene number; determine the reading start position according to the gene offset; and read the gene data corresponding to the number of genes from the first genome.

[0153] According to an embodiment of the present disclosure, biological data information is acquired through a data reading device, the biological data information including gene names and / or gene numbers; based on the biological data information, biological data corresponding to the gene names and / or gene numbers are classified and read, thereby realizing orderly reading of biological data information, accelerating the speed of analysis query, and reducing analysis costs.

[0154] It should be noted that since the device embodiment of the present disclosure corresponds to the above-mentioned method embodiment, the above-mentioned explanation of the method embodiment is also applicable to the device of the present embodiment, and the principle is the same. For details not disclosed in the device embodiment, reference can be made to the above-mentioned method embodiment, and they will not be repeated in this disclosure.

[0155] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0156] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0157] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 802 or a computer program loaded from a storage unit 808 to a RAM (Random Access Memory) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An I / O (Input / Output) interface 805 is also connected to the bus 804.

[0158] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0159] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPU (Central Processing Unit), GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as methods for data storage and / or data reading. For example, in some embodiments, the methods for data storage and / or data reading may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the aforementioned data storage and / or data reading method in any other appropriate manner (eg, by means of firmware).

[0160] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0162] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0164] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0165] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0166] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0167] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data storage method, It is characterized in that The method comprises: Acquire a biological data file, and extract original biological data from the biological data file; Performing data merging on the original biological data; The merged biological data is classified and stored according to a preset format.

2. The method according to claim 1, It is characterized in that The step of obtaining a biological data file and extracting original biological data from the biological data file includes: Preprocessing the biological data file to obtain multiple gene data sets, each gene data set including a gene name, a gene coordinate, and a gene expression level; According to the gene name, the corresponding original biological data is extracted, wherein the original biological data includes a gene offset and a gene number.

3. The method according to claim 2, It is characterized in that The preprocessing of the biological data file to obtain a plurality of gene data sets includes: Reading the biological data file in parallel through multiple threads; The gene coordinates and gene expression levels with the same gene name are merged into one gene data set to obtain the multiple gene data sets.

4. The method according to claim 3, It is characterized in that The data merging of the original biological data comprises: Traversing the multiple gene data sets, and processing the data in the multiple gene data sets in parallel through multiple threads; Each thread delivers the processed data to the data queue; The main thread extracts data from the data queue and merges the data of each gene into a first genome, wherein the first genome includes a first array and a second array, the gene name, gene offset and number of genes of each gene are merged into the first array, and the gene coordinates and gene expression of each gene are merged into the second array.

5. The method according to claim 4, It is characterized in that The first genome also includes a third array, and the main thread extracts data from the data queue, and merges the data of each gene into the first genome, including: The number of exons of each gene is merged into the third array, The third array corresponds to the second array one by one.

6. The method according to claim 4, It is characterized in that The data merging of the original biological data further includes: the main thread merging the number of genes and the number of gene expression levels contained in the DNA nanoball block into a second genome, the second genome including the first matrix, Each item of the first matrix corresponds to a DNA nanoball block.

7. The method according to claim 6, It is characterized in that The main thread merges the number of genes and the number of gene expression levels contained in the DNA nanoball block into the second genome, including: Genes with the same gene name in the same DNA nanoball block were grouped under the same gene name; The number of gene expression levels corresponding to the genes is accumulated.

8. The method according to claim 6, It is characterized in that The main thread merges the number of genes and the number of gene expression levels contained in the DNA nanoball block into the second genome, including: The data of the DNA nanoball blocks contained in the different first matrices in the second genome are merged.

9. The method according to claim 6, It is characterized in that The data merging of the original biological data further includes: The main thread merges the number of exons contained in the DNA nanoball block into a third genome, and the third genome includes a second matrix, Each item of the second matrix corresponds to a DNA nanoball block, and the coordinate points of the second matrix correspond one-to-one to the coordinate points of the first matrix.

10. The method according to any one of claims 4 to 9, It is characterized in that The data merging of the original biological data further includes: The main thread merges the gene name, the number of gene expression levels and the gene enrichment information of each gene into the fourth gene set.

11. The method according to claim 10, It is characterized in that The step of classifying and storing the merged biological data in a preset format includes: Calling API to write the merged biological data into the biological data file, The preset format is a classified storage table of gene number, gene name, gene offset and gene number, and different gene sets are written into any position of the biological data file in any order.

12. A data reading method, It is characterized in that The method comprises: Acquiring biological data information, wherein the biological data information includes a gene name and / or a gene number; According to the biological data information, the biological data corresponding to the gene name and / or gene number are classified and read.

13. The method according to claim 12, It is characterized in that The step of reading the biological data corresponding to the gene name and / or gene number includes: The biological data corresponding to the gene name and / or gene number are read in parallel through multiple threads.

14. The method according to claim 12, It is characterized in that The step of reading the biological data corresponding to the gene name and / or gene number includes: Get the gene coordinate range; Read the biological data corresponding to the gene name and / or gene number within the gene coordinate range.

15. The method according to claim 14, It is characterized in that The gene coordinate range is a preset shape, or the method further includes: A read range instruction is received, wherein the read range instruction includes the gene coordinate range.

16. The method according to any one of claims 12 to 15, It is characterized in that The step of classifying and reading the biological data corresponding to the gene name and / or gene number from the biological data file according to the biological data information includes: According to the gene name and / or gene number, read the gene offset and the number of genes corresponding to the gene name and / or gene number from a classification storage table in a preset format; Determine the read start position according to the gene offset; Gene data corresponding to the number of genes is read from the first genome.

17. A data storage device, It is characterized in that include: An acquisition unit, used for acquiring a biological data file and extracting original biological data from the biological data file; A merging unit, used for merging the original biological data; The storage unit is used to classify and store the merged biological data according to a preset format.

18. A data reading device, It is characterized in that include: An acquisition unit, used to acquire biological data information, wherein the biological data information includes a gene name and / or a gene number; The reading unit is used to classify and read the biological data corresponding to the gene name and / or gene number according to the biological data information.

19. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11 or claims 12-16.

20. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11 or claims 12-16.

21. A computer program product, It is characterized in that A computer program is included which, when executed by a processor, implements the method according to any one of claims 1-11 or claims 12-16.

Citation Information

Patent Citations

  • Big data processing method and system

    CN107346312A

  • Data real-time storage method and device, data query method and device, equipment and medium

    CN111339078A

  • System and method for storage and analysis of gene expression data

    US20040234995A1

  • Method for storage of gene expression results

    US20080015113A1