Processing and storage read-write method for sequencing data

Abstract

This invention relates to a method for processing, storing, reading, and writing sequencing data, which defines a unique writing structure and reading method. It solves the problems of excessive memory consumption and heavy computer load caused by loading large-scale h5ad files of single-cell RNA sequencing data, while also improving the analysis difficulties caused by high noise and large batch effects in single-cell RNA sequencing data. The method includes the following steps: First, the single-cell RNA sequencing data file is read using the h5py method to access the underlying data structure; the file content is presented in a hierarchical HDF5 structure. Then, the required groups and datasets are extracted and compiled into a database for appropriate processing. Next, gene names are standardized according to the correspondence between gene names in different formats. Finally, sample screening, data dimension unification, and standardization are performed. Simultaneously, to adapt to this new data structure, a custom dataset object with parallel data reading capabilities is designed.

Description

Technical Field

[0001] This invention relates to the fields of data processing and bioinformatics, and particularly to the processing, storage, and retrieval of single-cell RNA sequencing data in genomics. Background Technology

[0002] With the rapid development of biotechnology, the scale and complexity of biological data have increased significantly, bringing unprecedented challenges to bioinformatics data processing. Taking single-cell RNA sequencing data (scRNAseq) as an example, this technology can generate massive amounts of gene expression data, reaching hundreds of millions or even billions of reads, and is characterized by high dimensionality, high noise, and significant batch effects. These characteristics impose significant limitations on traditional methods for analyzing single-cell data. For instance, when using the scanpy method for reading data, even if only a small amount of data in the h5ad file needs to be processed, several gigabytes or even hundreds of gigabytes of data still need to be directly loaded into memory, placing a huge burden on computer equipment. Simultaneously, the high dimensionality, high noise, and large batch effects of the data result in significant differences in sequencing data from different donors, causing considerable difficulties for subsequent analysis. Therefore, a new method for processing, storing, and reading single-cell RNA sequencing data is urgently needed to improve these problems. Summary of the Invention

[0003] The purpose of this invention is to improve the problems of difficult reading, high noise, and significant batch effects in single-cell RNA sequencing data, and therefore proposes a method for processing, storing, reading, and writing sequencing data. The technical solution is as follows: A method for processing, storing, reading, and writing sequencing data: (1) First, use the h5py method to read the file containing single-cell RNA sequencing data and then reconstruct the Groups and Datasets required for subsequent analysis in the file into a new dataset; (2) Standardize gene names. Since gene names exist in various formats in different single-cell RNA sequencing data files, gene names need to be standardized according to the correspondence between gene names in different formats. (3) Screen out unwanted cell data, including cell gene expression data and corresponding tags; (4) Take the intersection of genes to unify the data dimensions of different samples; (5) Standardization processing: Logarithmic transformation is performed on gene expression data to achieve standardization processing, thereby eliminating the differences between different gene expression levels.

[0004] (6) To adapt to the new data structure, a dataset object that can perform parallel data reading is defined.

[0005] The beneficial effects of this invention are: This invention proposes a method for processing, storing, reading, and writing sequencing data, which can effectively save memory usage and computer load, and improve the analysis interference caused by high noise and batch effects in single-cell RNA sequencing data. It also includes a custom dataset object with parallel data reading capabilities to adapt to new data structures. Attached Figure Description

[0006] Figure 1 This is an overall structural diagram of the present invention; Detailed Implementation

[0007] To make the technical solution of the present invention clearer, the present invention will be further described below with reference to the accompanying drawings. The present invention is implemented in specific steps: (1) First, the file containing single-cell RNA sequencing data is read using the h5py method, which allows direct access to the underlying data structure. After reading, the file content will be presented in a hierarchical structure of HDF5. The file contains several important parts, such as the X group, obs group, var group, obsm group, varm group, and obsp group, which respectively store gene expression information, cell metadata, gene metadata, cell-related multidimensional annotation data, gene-related multidimensional annotation data, and inter-cell relationship matrix; reconstruct the dataset, after reading the data, extract the required groups and datasets, and process them, for example: extract the dataset obs / cell_type corresponding to the label such as cell type. This group stores the cell type label data corresponding to each cell, which can be split into two parts. The cell type coding data and the cell type category data are stored in the obs / cell_type / codes and obs / cell_type / categories datasets, respectively. Alternatively, gene expression data can be stored as sparse matrices in raw / X / data, raw / X / indices, and raw / X / indptr respectively, or the X group can be divided into X / data, X / indptr, and X / indices, storing the non-zero values ​​of the sparse matrix and their position information respectively for subsequent analysis. (2) Standardize gene names. Since gene names in different single-cell RNA sequencing files exist in multiple formats, gene names need to be standardized according to the correspondence between gene names in different formats. (3) Screening out unwanted cells, removing those that do not meet specific criteria or are outside the scope of research interest, thereby improving the accuracy and efficiency of data analysis. (4) Gene intersection: In order to unify the data dimensions, it is necessary to take the intersection of genes of all dataset samples so that all cell samples have common gene expression. in, It is the first The gene set of a sample It is the intersection of genes from all samples; (5) Single-cell sequencing data typically exhibit a long-tailed distribution, with most genes showing low expression levels and a few showing high expression levels. To reduce the impact of extreme values, a logarithmic transformation is used to compress this long-tailed distribution into a range closer to a normal distribution. The calculation formula is as follows: Wherein, it is assumed that the original data is , Indicates the first In the nth sample The expression value of each gene, This represents the numerical value after logarithmic transformation.

[0008] (6) To adapt to the new data structure, a dataset object that can perform parallel data reading is defined.

[0009] First, a serial dataset class, `SerialDataset`, is defined. `SerialDataset` concatenates multiple datasets to form a unified dataset. The parameter passed to `__init__` is a list containing multiple `Dataset` objects. `SerialDataset` assigns two indices to samples in each `Dataset` object: `dataset_indexes` (representing the index of the dataset object to which the sample belongs) and `inner_indexes` (representing the index of the sample within its dataset object). In `__getitem__`, the corresponding sample is retrieved from the concatenated dataset based on the given index `select_index`.

[0010] Then a sparse value dataset object, H5SparseSample, is defined. The function of the H5SparseSample object is to extract the gene expression value data raw / X / data, raw / X / indices and raw / X / indptr of the specified index of the dataset reconstructed in step (1) and convert them into dense data.

[0011] Then a dense value dataset object is defined: H5DenseSample. The function of the H5DenseSample object is to extract the labels of the specified index of the reconstructed dataset in step (1), such as cell type label obs / cell_type / codes and organ label obs / cell_types / codes.

[0012] Finally, an integrated dataset object, AlignDataset, is constructed. AlignDataset reads data from multiple H5ad files and concatenates the datasets into a unified dataset. The AlignDataset object inherits from SerialDataset and takes H5SparseSample (which retrieves gene expression values, sparse data) and H5DenseSample (which retrieves label data, dense data) as input dataset lists to form a custom dataset object with parallel data reading capabilities for subsequent data reading.

Claims

1. A method for processing, storing, reading, and writing sequencing data, characterized in that, The specific steps include: (1) First, use the h5py method to read the file containing single-cell RNA sequencing data and then reconstruct the Groups and Datasets required for subsequent analysis in the file into a new dataset; (2) Standardize gene names. Since gene names exist in various formats in different single-cell RNA sequencing data files, gene names need to be standardized according to the correspondence between gene names in different formats. (3) Screen out unwanted cell data, including cell gene expression data and corresponding tags; (4) Take the intersection of genes to unify the data dimensions of different samples; (5) Standardization processing: Logarithmic transformation is performed on gene expression data to achieve standardization processing, thereby eliminating the differences between different gene expression levels; (6) To adapt to this new data structure, a dataset object that can perform parallel data reading is defined.

2. The method for processing, storing, reading, and writing sequencing data as described in claim 1, characterized in that: Steps (1), (2), (3), (4), and (5) define a method and process for writing single-cell RNA sequencing data based on h5py: The h5py method is used to read the file containing single-cell RNA sequencing data in the manner of reading HDF5 files, allowing direct access to the underlying data structure; after reading, the file content will be presented in the hierarchical structure of HDF5; the file contains several important parts, such as the X group, obs group, var group, obsm group, varm group, and obsp group, which respectively store gene expression information, cell metadata, gene metadata, cell-related multidimensional annotation data, gene-related multidimensional annotation data, and inter-cell relationship matrix; the dataset is reconstructed, and after reading the data, the required groups and datasets are extracted and processed. For example, the dataset obs / cell_type corresponding to the cell type is extracted. This group stores the cell type label data corresponding to each cell, which can be split into two parts: cell type encoding data and cell type class. Different data are stored in the obs / cell_type / codes and obs / cell_type / categories datasets respectively; or gene expression value data are stored in raw / X / data, raw / X / indices and raw / X / indptr respectively in the form of sparse matrix storage, or the X group is divided into X / data, X / indptr and X / indices, storing the non-zero values ​​of the sparse matrix and their position information respectively. Then, the data is standardized by unifying gene names, screening cells, taking the intersection of genes, and standardizing to form a new data structure for subsequent analysis.

3. The method for processing, storing, reading, and writing sequencing data as described in claim 1, characterized in that: The custom dataset object in step (6): First, a serial dataset class is defined: SerialDataset. SerialDataset concatenates multiple datasets to form a unified dataset. The parameter passed to __init__ is a list containing multiple Dataset objects. SerialDataset assigns two indices to samples in each dataset object: dataset_indexes (representing the index of the dataset object to which the sample belongs) and inner_indexes (representing the index of the sample within the dataset object). In __getitem__, the corresponding sample is retrieved from the concatenated dataset based on the given index select_index. Then a sparse value dataset object is defined: H5SparseSample. The function of the H5SparseSample object is to extract the gene expression value data raw / X / data, raw / X / indices and raw / X / indptr of the specified index of the dataset reconstructed in step (1) and convert them into dense data. Then a dense value dataset object is defined: H5DenseSample. The function of the H5DenseSample object is to extract the labels of the specified index of the dataset reconstructed in step (1), such as cell type label obs / cell_type / codes and organ label obs / cell_types / codes. Finally, an integrated dataset object, AlignDataset, is constructed. AlignDataset reads data from multiple H5ad files and concatenates the datasets into a unified dataset. AlignDataset inherits from SerialDataset and takes H5SparseSample (which retrieves gene expression values, sparse data) and H5DenseSample (which retrieves label data, dense data) as input dataset lists to form a custom dataset object with parallel data reading capabilities for data reading.

Images