A method for visual modeling of genomic sequences based on optical character recognition document understanding in a breeding application

Abstract

The application discloses a kind of genomic sequence visual modeling methods based on optical character recognition type document understanding in breeding application.The method first obtains the genomic sequence to be processed and carries out standardization processing;Subsequently, the normalized genomic sequence is rendered into a one-page or multi-page structured DNA document image according to the preset page layout, and a page annotation set is generated;Then, a bidirectional mapping relationship between the genomic interval and the DNA document page area is established;Then, the DNA document image is converted into a document-level visual token sequence using a visual encoder, a projection module and a multi-page fusion module;Next, an OCR type prompt task set is constructed and the corresponding supervision result is generated;Subsequently, the genomic visual modeling model is supervised trained based on the document-level visual token sequence and the prompt task set;Finally, the trained model is used to complete genomic representation extraction and downstream task prediction, and output functional prediction, region positioning, subsequence retrieval, missing fragment completion or phenotype prediction result.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary fields of bioinformatics, genome sequence modeling, computer vision, and artificial intelligence for breeding applications. More specifically, it relates to a genome sequence visual modeling method based on optical character recognition document understanding for breeding applications. Background Technology

[0002] Genomic sequence visual modeling refers to transforming a one-dimensional DNA sequence into a two-dimensional or multi-page image object that can be processed by a visual model or a visual-language model, while preserving the correspondence between nucleotide characters, sequence coordinates, and spatial regions within this object. In breeding applications, researchers need to identify regulatory elements, splicing sites, variable functional regions, and candidate regions related to traits across long fragments or the entire genome, and further complete subspecies classification, phenotypic prediction, or deletion inference. Because genomic functional signals are typically sparse, discontinuous, and distributed over long distances, simply reading the one-dimensional sequence position by position is insufficient to balance computational efficiency, regional localization, and interpretability. Therefore, it is necessary to organize genomic sequences into visual documents with page, line, character, and regional structures.

[0003] Existing methods related to visual modeling of genome sequences mainly include the following: The first is a sequence modeling method based on one-dimensional tokens or k-mers, which encodes DNA sequences by bases or k-mers and inputs them into convolutional neural networks, Transformers, state-space models, or long-context language models, and reduces the computational cost of long sequences through local convolution, sparse attention, linear attention, etc. The second is a two-dimensional matrix or simple image method, which arranges sequence fragments into two-dimensional matrices or pseudo-images, and then uses two-dimensional convolutional networks, visual Transformers, or image patch encoders to extract local patterns. The third is a window aggregation, hierarchical pooling, or low-dimensional compression method for long sequences, which reduces the number of effective tokens and computational overhead when inputting long sequences by aggregating local fragments, pooling hierarchical representations, or compressing the original base / k-mer tokens into fewer contextual representations.

[0004] However, the three existing visual modeling methods for genome sequences mentioned above all have some significant drawbacks: First, the one-dimensional token-based or k-mer-based sequence modeling methods still require approximately uniform processing of a large number of low-information-density background regions, making it difficult to perform structure-aware reading of sparse and discontinuously distributed functional regions, resulting in high computational resource consumption under long sequence conditions; Second, the two-dimensional matrixing or simple image-based methods typically only convert local sequence patterns into visual features, lacking a reversible annotation mechanism that simultaneously records nucleotide characters, global sequence coordinates, and page pixel regions during rendering, leading to insufficiently direct region localization, subsequence retrieval, and evidence interpretation; Third, the window aggregation, hierarchical pooling, or low-dimensional compression methods focus primarily on reducing the number of tokens, lacking reconfigurable visual compression and multi-page fusion mechanisms combined with document understanding, making it difficult to retain fine-grained nucleotide information and cross-page context with a limited effective token budget; Fourth, all three types of methods require the design of task-specific modules for reading, localization, transcription, completion, retrieval, and classification, making it difficult to simultaneously support DNA reading, region of interest transcription, masked region completion, query subsequence localization, and downstream function prediction within a unified framework. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a genome sequence visual modeling method based on optical character recognition document understanding for breeding applications. Its purpose is to solve the technical problems of existing one-dimensional token-based or k-mer-based sequence modeling methods, which still require approximately uniform processing of large amounts of low-information-density background regions, making it difficult to perform structure-aware reading of sparse and discontinuously distributed functional regions, resulting in high computational resource consumption under long sequence conditions. It also addresses the technical problems of existing two-dimensional matrixing or simple image-based methods, which typically only convert local sequence patterns into visual features, lacking a reversible annotation mechanism that simultaneously records nucleotide characters, global sequence coordinates, and page pixel regions during rendering, leading to insufficiently direct region localization, subsequence retrieval, and evidence interpretation. Furthermore, it addresses the technical problems of existing window aggregation, hierarchical pooling, or low-dimensional compression methods, which focus on reducing the number of tokens and lack reconfigurable visual compression and multi-page fusion mechanisms combined with document understanding, making it difficult to retain fine-grained nucleotide information and cross-page context with a limited effective token budget. Finally, it addresses the technical problem that the above three methods cannot simultaneously support DNA reading, region of interest transcription, masked region completion, query subsequence localization, and downstream functional prediction within a unified framework.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for visual modeling of genome sequences based on optical character recognition document understanding is provided for breeding applications, comprising the following steps: (1) Obtain the genome sequence to be processed and perform normalization processing on the genome sequence to obtain a normalized genome sequence; (2) Based on the preset document rendering parameters, the standardized genome sequence obtained in step (1) is subjected to document rendering processing to obtain a set of DNA document images and its corresponding page annotation set; (3) Based on the standardized genome sequence obtained in step (1) and the page annotation set obtained in step (2), establish the mapping relationship between the genome region and the DNA document page region; (4) Input the DNA document image set obtained in step (2) into the visual encoder to obtain the page-level visual feature sequence, and perform feature projection processing and multi-page feature fusion processing on the page-level visual feature sequence in sequence to obtain the document-level visual token sequence. (5) Based on the DNA document image set obtained in step (2) and the mapping relationship between the genomic region and the DNA document page region obtained in step (3), construct an optical character recognition prompt task set and generate corresponding supervision results for each optical character recognition prompt task in the optical character recognition prompt task set; (6) Based on the document-level visual token sequence obtained in step (4) and the optical character recognition prompt task set obtained in step (5), supervised training is performed on the genome visual modeling model including the visual encoder, projection module, multi-page fusion module and document decoder to obtain the trained genome visual modeling model. (7) Obtain the genome sequence to be predicted from the sequence text file in FASTA format, the sequence text file in FASTQ format with base quality values, the reference genome database, the sequencing read splicing results, the individual sequence after mutation substitution, or the nucleotide sequence input by the user. Use the genome visual modeling model obtained in step (6) to process the genome sequence to be predicted in sequence to obtain the corresponding genome function prediction results, region location results, subsequence retrieval results, missing fragment completion results, or phenotype prediction results.

[0007] Preferably, step (1) includes the following sub-steps: (1-1) Obtain the genome sequence to be processed, read the genome sequence to be processed to obtain the nucleotide characters in the genome sequence to be processed, and record one or more of the following: sample number, species information, chromosome number, original sequence start coordinates, original sequence end coordinates and data source corresponding to the genome sequence to be processed; (1-2) Convert the lowercase nucleotide characters in the genome sequence to be processed to uppercase nucleotide characters to obtain the genome sequence after one update, and delete the nucleotide characters in the genome sequence after one update that are not included in the preset nucleotide character set {A,C,G,T,N} to obtain the genome sequence after two updates; where A represents adenine base, C represents cytosine base, G represents guanine base, T represents thymine base, and N represents unknown base or undetermined base; (1-3) Assign global sequence coordinates to each nucleotide character in the second-updated genome sequence obtained in step (1-2) to obtain the coordinate-assigned genome sequence as the standardized genome sequence.

[0008] Preferably, step (2) includes the following sub-steps: (2-1) Based on the preset document rendering parameters and the length Len of the standardized genome sequence obtained in step (1-3), determine the number of text lines M that each page can accommodate, the number of nucleotide characters Q that each line in each page can accommodate, and the total number of nucleotide characters C that a single page can accommodate. Based on the length Len of the standardized genome sequence and the total number of nucleotide characters C, obtain the total number of pages P according to the formula P=⌈Len / C⌉ (where ⌈·⌉ means rounding up). Set the page counter p=0 and initialize the DNA document image set and page annotation set to empty. (2-2) Determine whether the page counter p is greater than or equal to the total number of pages P. If yes, proceed to step (2-12); otherwise, proceed to step (2-3). (2-3) Create the p-th page (initially a blank page) according to the preset document rendering parameters, create the record item corresponding to the p-th page in the page annotation set, and set the line counter m=1; (2-4) Determine whether the line counter m is greater than the number of text lines M that a single page can hold. If yes, proceed to step (2-10); otherwise, proceed to step (2-5). (2-5) Set the column counter q=1; (2-6) Calculate the current plotted sequence coordinates s = p × C + (m-1) × Q + (q-1) based on the page counter p, row counter m and column counter q, and determine whether the column counter q is greater than the number of nucleotide characters Q that each row can hold, or whether the current plotted sequence coordinates s are greater than or equal to the length Len of the normalized genome sequence obtained in step (1-3). If so, proceed to step (2-9); otherwise, proceed to step (2-7). (2-7) Obtain a nucleotide character whose global sequence coordinates are equal to the current drawing sequence coordinates s from the normalized genome sequence obtained in step (1-3). Based on the row counter m, column counter q and preset document rendering parameters, obtain the drawing position of the nucleotide character on the p-th page and place the nucleotide character at the drawing position. (2-8) Add the character content c, global sequence coordinates g=s, and pixel-level bounding box r=(x1,y1,x2,y2) corresponding to the nucleotide character obtained in step (2-7) to the record item corresponding to the p-th page in the page annotation set as nucleotide annotation items, set the column counter q=q+1, and return to step (2-6); where x1 and y1 represent the horizontal and vertical coordinates of the upper left corner of the pixel-level bounding box corresponding to the nucleotide character, respectively, and x2 and y2 represent the horizontal and vertical coordinates of the lower right corner of the pixel-level bounding box corresponding to the nucleotide character, respectively; (2-9) Set the row counter m=m+1 and return to step (2-4). (2-10) Save the p-th page as a DNA document image page_p, and add the DNA document image page_p to the DNA document image set; (2-11) Set the page counter p = p + 1 and return to step (2-2); (2-12) Output the DNA document image set D={page_0,page_1,...,page_{P-1}} and its corresponding page annotation set.

[0009] Preferably, the genome sequence to be processed comes from a FASTA format sequence text file, a FASTQ format sequence text file with base quality values, a reference genome database, sequencing read splicing results, a genome fragment set, an individual sequence after mutation substitution, or a nucleotide sequence input by the user. The genome sequence to be processed is a DNA sequence of a human, animal, plant, microorganism or other species, or a chromosome fragment, regulatory region, candidate variant region or whole genome sequence slice; The preset document rendering parameters include one or more of the following: page resolution, page width, page height, font type, font size, character spacing, line spacing, page margins, number of characters per line, maximum character capacity per page, background color, text color, page orientation, and rendering start position. Each DNA document image in the DNA document image set D corresponds to a record in the page annotation set. This record includes multiple nucleotide annotations in the corresponding page. Each nucleotide annotation includes a nucleotide character, global coordinates, and a pixel-level bounding box.

[0010] Preferably, step (3) includes the following sub-steps: (3-1) Select any continuous interval [i,j) from the normalized genome sequence obtained in step (1). Based on the global sequence coordinates of all nucleotide annotations in the page annotation set obtained in step (2), retrieve all nucleotide annotations covered by the interval. Here, i represents the global sequence coordinate of the first nucleotide character covered by the interval, j represents the endpoint covered by the interval, i.e., the global sequence coordinate of the position after the last nucleotide character, and 0≤i<j≤lengthLen of the normalized genome sequence. (3-2) Determine whether all nucleotide annotations covered by the interval [i,j) are completely located in the same DNA document image in the DNA document image set D; if so, generate the minimum bounding rectangle covering the interval as the corresponding interval bounding box based on the pixel-level bounding boxes of all nucleotide annotations in the interval, and proceed to step (3-3); otherwise, generate the corresponding page bounding box in each DNA document image where the interval [i,j) is located, and all page bounding boxes constitute the interval bounding box set corresponding to the interval, and proceed to step (3-3). (3-3) Associate the interval [i,j) with the interval bounding box or set of interval bounding boxes obtained in step (3-2) to form a mapping relationship from interval to region; (3-4) Obtain the DNA document page region to be reverse mapped, its page index, and pixel coordinate range from the given region information. In the page annotation set obtained in step (2), retrieve all nucleotide annotation items whose corresponding pixel-level bounding boxes intersect with or are covered by the DNA document page region. Sort all nucleotide annotation items in descending order according to the global sequence coordinates saved by each nucleotide annotation item to obtain the sorting result. Set the global sequence coordinates corresponding to the first nucleotide annotation item in the sorting result as the starting global sequence coordinates and set the global sequence coordinates corresponding to the last nucleotide annotation item in the sorting result as the ending global sequence coordinates to obtain the sequence interval corresponding to the DNA document page region. Associate the DNA document page region with the sequence interval to form a reverse mapping relationship from region to interval. (3-5) Based on the nucleotide annotations retrieved in step (3-1), the interval bounding boxes or sets of interval bounding boxes generated in step (3-2), the interval-to-region mapping relationship formed in step (3-3), and the region-to-interval inverse mapping relationship formed in step (3-4), construct the mapping relationship between genomic intervals and DNA document page regions.

[0011] Preferably, step (3-5) specifically involves: first, using the interval [i,j) as the index key, and saving the interval bounding box or set of interval bounding boxes obtained in step (3-2) as the index value to obtain a forward index record from interval to region; then, based on the forward index record and the page annotation set obtained in step (2), recording the page bounding box generated for each DNA document image in step (3-2) with the page index of the DNA document image and the starting and ending global sequence coordinates covered by the page bounding box in a mapping manner to obtain a page bounding box index record; subsequently... Then, based on the region-to-interval inverse mapping results obtained in step (3-4), the page bounding box index record is processed to establish a region-to-interval inverse index to obtain a region-to-interval inverse index record. Subsequently, the interval bounding box set obtained in step (3-2) is sorted according to the page index and global sequence coordinates to obtain an ordered mapping record of the interval bounding box set. Finally, the interval-to-region forward index record, page bounding box index record, region-to-interval inverse index record, and ordered mapping record of the interval bounding box set are summarized into a mapping relationship between genomic intervals and DNA document page regions.

[0012] Preferably, step (4) includes the following sub-steps: (4-1) Adjust each page of DNA document image in the DNA document image set obtained in step (2) to the preset input size, and divide the DNA document image after adjustment into multiple image blocks; (4-2) Encode each image block corresponding to each page of DNA document image obtained in step (4-1) using a visual front-end network to obtain the image block feature corresponding to the image block, and arrange the image block features corresponding to all image blocks of the DNA document image of the page in the page spatial order to form the page-level visual feature sequence corresponding to the DNA document image of the page. (4-3) Perform linear projection, nonlinear mapping or multilayer perceptron mapping on the page-level visual feature sequence corresponding to each page of DNA document image obtained in step (4-2) to obtain the projected page-level visual feature sequence corresponding to the page of DNA document image. (4-4) Perform multi-page feature fusion processing on the projected page-level visual feature sequences corresponding to all DNA document images in the DNA document image set obtained in step (4-3) to obtain the fused document-level visual features; wherein the multi-page feature fusion processing adopts any one or more combinations of self-attention, cross-attention, gated aggregation, mean pooling, max pooling, and learnable query aggregation. (4-5) The fused document-level visual features obtained in step (4-4) are successively normalized, positionally encoded, and represented with a fixed length to obtain a document-level visual token sequence.

[0013] Preferably, step (5) includes the following sub-steps: (5-1) Configure DNA sequence reading prompts for the DNA document image set obtained in step (2) to obtain an updated DNA document image set. Use the standardized genome sequence obtained in step (1-3) as the supervision result, and combine the DNA sequence reading prompts, the updated DNA document image set and the supervision result into a DNA sequence reading task sample. (5-2) Based on the mapping relationship between the genomic regions and the DNA document page regions obtained in step (3), select multiple text lines, text blocks or continuous intervals from the DNA document image set and its corresponding page annotation set obtained in step (2), and configure region joint localization prompts for the selected text lines, text blocks or continuous intervals; obtain the nucleotide text fragments covered by each selected text line, text block or continuous interval, and obtain the page bounding box corresponding to the nucleotide text fragment according to the mapping relationship obtained in step (3). All the obtained nucleotide text fragments constitute DNA text, and all the page bounding boxes corresponding to all nucleotide text fragments constitute the supervision result of the region joint localization task. Combine the region joint localization prompts, the DNA document image set and the supervision result into a region joint localization task sample. (5-3) Based on the mapping relationship between the genomic regions and DNA document page regions obtained in step (3), multiple regions of interest bounding boxes are obtained through user input, random sampling, or preset task configuration, and a region of interest transcription prompt is configured for each region of interest bounding box; all nucleotide characters covered by each region of interest bounding box are retrieved in the page annotation set obtained in step (2), and all nucleotide characters are arranged in the order of their corresponding global sequence coordinates to form the nucleotide sequence text corresponding to the region of interest bounding box. All regions of interest bounding boxes and their corresponding nucleotide sequence text constitute the supervision result of the region of interest transcription task, and the region of interest transcription prompt, DNA document image set, region of interest bounding box and the supervision result are combined into a region of interest transcription task sample; (5-4) Based on the mapping relationship between the genomic regions and DNA document page regions obtained in step (3), obtain multiple continuous nucleotide regions from the standardized genomic sequence or preset task configuration obtained in step (1-3), and configure mask region completion prompts for each continuous nucleotide region; perform mask processing on the page regions corresponding to all the obtained continuous nucleotide regions in the DNA document image set obtained in step (2) to obtain the masked DNA document image set, use the nucleotide sequence text corresponding to all continuous nucleotide regions in the standardized genomic sequence obtained in step (1-3) as the supervision result, and combine the mask region completion prompts, all the masked DNA document image sets and their corresponding page region bounding boxes, and all supervision results together as the mask region completion task sample; (5-5) Obtain the query nucleotide subsequence from the standardized genome sequence obtained in step (1-3), and retrieve all continuous nucleotide intervals corresponding to the query nucleotide subsequence from the standardized genome sequence obtained in step (1-3); input all continuous nucleotide intervals into the mapping relationship between the genome interval and the DNA document page area obtained in step (3) to obtain the pixel-level bounding box corresponding to each continuous nucleotide interval, and configure the query subsequence location prompt for the continuous nucleotide interval; combine the query nucleotide subsequence, all continuous nucleotide intervals, their corresponding pixel-level bounding boxes and the query subsequence location prompt into a query subsequence location task sample; (5-6) From the sample number, species information, chromosome number, and data source recorded in step (1-1), obtain multiple document-level tags corresponding to the DNA document image set obtained in step (2), perform category encoding processing on all document-level tags to obtain multiple document-level category supervision results, and configure global document category recognition prompts for multiple document-level category supervision results; combine the global document category recognition prompts, DNA document image set, and document-level category supervision results into multiple global document category recognition task samples; wherein, the document-level tag includes one or more of chromosome number, species category, subspecies category, tissue category, functional category, sample category, phenotypic category, or breeding trait category; (5-7) Combine the DNA sequence reading task samples obtained in step (5-1), the region joint localization task samples obtained in step (5-2), the region of interest transcription task samples obtained in step (5-3), the mask region completion task samples obtained in step (5-4), the query subsequence localization task samples obtained in step (5-5), and the global document category recognition task samples obtained in step (5-6) into an optical character recognition prompt task set.

[0014] Preferably, step (6) includes the following sub-steps: (6-1) Obtain multiple task samples from the optical character recognition prompt task set obtained in step (5), as well as the DNA sequence reading prompt, region joint localization prompt, region of interest transcription prompt, mask region completion prompt, query subsequence localization prompt, or global document category recognition prompt and the supervision result corresponding to each task sample; (6-2) Input all the task samples obtained in step (6-1) into the visual encoder, projection module and multi-page fusion module in the genome visual modeling model in sequence to obtain the document-level visual token sequence; (6-3) Input the DNA sequence reading prompts, region joint localization prompts, region of interest transcription prompts, mask region completion prompts, query subsequence localization prompts, or global document category recognition prompts and the document-level visual token sequence obtained in step (6-2) into the document decoder so that it generates the task output sequence in an autoregressive manner; (6-4) Based on the supervision results corresponding to all task samples obtained in step (6-1) and the task output sequence obtained in step (6-3), obtain the autoregressive cross-entropy loss value of the task output sequence relative to the supervision results; (6-5) Update the parameters of the genome visual modeling model based on the autoregressive cross-entropy loss value obtained in step (6-4); (6-6) Repeat steps (6-1) to (6-5) until the preset number of training steps is reached or the genome visual model converges, thereby obtaining the trained genome visual model.

[0015] Preferably, the formula for calculating the autoregressive cross-entropy loss is as follows: in, This represents the t-th token in the output sequence of the supervision results corresponding to all task samples. Z represents all tokens before the t-th token in the output sequence, Z represents the document-level visual token sequence obtained in step (6-2), q represents the task cues corresponding to all task samples, and T represents the token length of the supervision result output sequence corresponding to all task samples. During the update process in step (6-5), the visual front-end network in the visual encoder is frozen, and at least one of the projection module, multi-page fusion module and document decoder is fine-tuned with full parameters or low-rank adaptation.

[0016] In summary, the technical solution conceived in this invention achieves the following beneficial effects compared to the prior art: (1) Since the present invention adopts steps (1) to (2), the genome sequence to be processed is first standardized, and then the standardized genome sequence is rendered into a DNA document image with page, line, character and region structure, so that sparse and discontinuous functional signals can be organized and read in visual space. Therefore, it can solve the technical problem that the existing one-dimensional token or k-mer-based methods require approximately uniform scanning of low information density background regions and large computational resource consumption under long sequence conditions. (2) Since the present invention adopts steps (2) to (3), it records nucleotide characters, global sequence coordinates and pixel-level bounding boxes while rendering DNA document images, and further establishes a bidirectional mapping relationship between genomic regions and DNA document page regions. Therefore, it can solve the technical problems of existing two-dimensional matrix or simple image methods lacking reversible annotation mechanisms, region localization, subsequence retrieval and evidence interpretation that are not direct enough. (3) Since the present invention adopts step (4), it performs visual encoding, feature projection and multi-page feature fusion on the DNA document image set in sequence, and compresses the multi-page DNA document image into a document-level visual token sequence. Therefore, it can solve the technical problem that existing window aggregation, hierarchical pooling or low-dimensional compression methods are difficult to retain fine-grained nucleotide information and cross-page context at the same time with a small effective token budget. (4) Since the present invention adopts steps (5) to (6), DNA sequence reading, region joint localization, region of interest transcription, mask region completion, query subsequence localization and global document category recognition are constructed into an optical character recognition prompt task set, and the genome visual model is supervised and trained using this task set. Therefore, it can solve the technical problem that the existing three methods usually need to design task-specific modules separately when expanding tasks, and it is difficult to support reading, localization, transcription, completion, retrieval and classification in a unified framework at the same time. (5) Because of the use of step (7), the present invention can transfer the trained genome visual modeling model to breeding application-related tasks such as genome function prediction, region localization, subsequence retrieval, missing fragment completion and phenotypic prediction, thereby reducing the downstream adaptation complexity and improving deployment flexibility. (6) The present invention is simple to implement and can complete data reading based on FASTA, FASTQ, reference genome database, sequencing read splicing results or user input sequences. It can also achieve end-to-end processing through preset page parameters, visual encoder, projection module, multi-page fusion module and document decoder, which is convenient for engineering deployment. (7) The present invention has wide applicability and can be applied to genome sequences of different lengths, species, chromosomes and data sources. It can also be extended according to page resolution, font size, line spacing, single page capacity and multi-page fusion method. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall process of the genome sequence visual modeling method based on optical character recognition document understanding in the breeding application of this invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] The basic idea of ​​this invention is to provide a visual modeling method for genome sequences based on optical character recognition document understanding in breeding applications. This method converts DNA sequences into DNA document images according to a defined page layout and stores the positional annotations of each nucleotide character or continuous interval in the page coordinates. Subsequently, the visual model is used to encode the DNA document image, and the document decoder is used to generate text, regions, retrieval, or classification results based on prompts. After pre-training, the visual encoder can be used as a general genome characterization extractor for breeding trait prediction, subspecies classification, and other long-sequence downstream tasks.

[0020] The technical terms used in this invention will be explained and described below: DNA document image: refers to an image document formed by rendering a one-dimensional genome sequence according to preset page parameters. A DNA document image may include one or more pages, each page containing nucleotide characters arranged in reading order.

[0021] Nucleotide characters in a DNA document image correspond to the text to be recognized, page pixel regions correspond to the regions to be located, and task prompts describe the operations that the model needs to perform. The document decoder generates corresponding outputs based on visual tokens and task prompts. The task prompts include DNA sequence reading prompts, region joint localization prompts, region of interest transcription prompts, masked region completion prompts, query subsequence localization prompts, and global document category recognition prompts.

[0022] A pixel-level bounding box is a rectangular region used to represent the pixel positions of nucleotide characters, text lines, continuous fragments, or regions of interest in a DNA document image. A pixel-level bounding box can be represented as [img_id, x1, y1, x2, y2], where img_id represents the page index, and x1, y1, x2, and y2 represent the pixel coordinates of the top-left and bottom-right corners of the rectangular region.

[0023] Interval-to-region mapping refers to the mapping relationship between continuous intervals in a genome sequence and one or more pixel-level bounding boxes in a DNA document image. Region-to-interval mapping involves determining the nucleotide characters and global sequence coordinates covered by a page region in reverse order.

[0024] Visual tokens are vector representations obtained by the visual encoder after performing image patch encoding, feature projection, and fusion on the DNA document image. Visual tokens are used to provide the document decoder with information about the visual content and spatial layout of the DNA document.

[0025] OCR-based prompting tasks refer to prompting conditions built by drawing on optical character recognition and document understanding. The model outputs nucleotide text, bounding boxes, completion results, search results, or category labels based on the input image and prompting instructions.

[0026] like Figure 1 As shown, this invention provides a method for visual modeling of genome sequences based on optical character recognition document understanding in breeding applications, comprising the following steps: (1) Obtain the genome sequence to be processed and perform normalization processing on the genome sequence to obtain a normalized genome sequence.

[0027] In this step, the genome sequence to be processed can come from a FASTA format sequence text file (FASTA for short), a FASTQ format sequence text file with base quality values ​​(FASTQ for short), a reference genome database, sequencing read splicing results, a collection of genome fragments, an individual sequence after mutation substitution, or a nucleotide sequence input by the user. The genome sequence to be processed can be a DNA sequence of humans, animals, plants, microorganisms or other species, or it can be a chromosome fragment, a regulatory region, a candidate variant region or a whole genome sequence slice.

[0028] This step includes the following sub-steps: (1-1) Obtain the genome sequence to be processed, read the genome sequence to obtain the nucleotide characters in the genome sequence to be processed, and record one or more of the following: sample number, species information, chromosome number, original sequence start coordinates, original sequence end coordinates and data source corresponding to the genome sequence to be processed.

[0029] (1-2) Convert the lowercase nucleotide characters in the genome sequence to be processed to uppercase nucleotide characters to obtain the genome sequence after one update, and delete the nucleotide characters in the genome sequence after one update that are not included in the preset nucleotide character set {A,C,G,T,N} to obtain the genome sequence after two updates; where A represents adenine base, C represents cytosine base, G represents guanine base, T represents thymine base, and N represents unknown base or undetermined base.

[0030] (1-3) Assign global sequence coordinates to each nucleotide character in the second-updated genome sequence obtained in step (1-2) to obtain the coordinate-assigned genome sequence as the standardized genome sequence.

[0031] The advantage of this step (1) is that by unifying uppercase and lowercase, filtering illegal characters, and allocating global coordinates, a standardized genome sequence with a consistent character set and a clear coordinate system can be obtained, providing a stable input basis for subsequent DNA document generation, pixel-level annotation, and coordinate mapping.

[0032] (2) Based on the preset document rendering parameters, the standardized genome sequence obtained in step (1) is processed for document rendering to obtain a set of DNA document images and its corresponding page annotation set.

[0033] In this step, the preset document rendering parameters include one or more of the following: page resolution, page width, page height, font type, font size, character spacing, line spacing, page margins, number of characters per line, maximum character capacity per page, background color, text color, page orientation, and rendering start position. A monospace font is used to ensure a stable positional relationship between adjacent nucleotide characters.

[0034] This step includes the following sub-steps: (2-1) Based on the preset document rendering parameters and the length Len of the standardized genome sequence obtained in step (1-3), determine the number of text lines M that each page can accommodate, the number of nucleotide characters Q that each line in each page can accommodate, and the total number of nucleotide characters C that a single page can accommodate. Based on the length Len of the standardized genome sequence and the total number of nucleotide characters C, obtain the total number of pages P according to the formula P=⌈Len / C⌉ (where ⌈·⌉ means rounding up). Set the page counter p=0 and initialize the DNA document image set and page annotation set to empty. (2-2) Determine whether the page counter p is greater than or equal to the total number of pages P. If yes, proceed to step (2-12); otherwise, proceed to step (2-3). (2-3) Create the p-th page (initially a blank page) according to the preset document rendering parameters, create the record item corresponding to the p-th page in the page annotation set, and set the line counter m=1; (2-4) Determine whether the line counter m is greater than the number of text lines M that a single page can hold. If yes, proceed to step (2-10); otherwise, proceed to step (2-5). (2-5) Set the column counter q=1; (2-6) Calculate the current plotted sequence coordinates s = p × C + (m-1) × Q + (q-1) based on the page counter p, row counter m and column counter q, and determine whether the column counter q is greater than the number of nucleotide characters Q that each row can hold, or whether the current plotted sequence coordinates s are greater than or equal to the length Len of the normalized genome sequence obtained in step (1-3). If so, proceed to step (2-9); otherwise, proceed to step (2-7). (2-7) Obtain a nucleotide character whose global sequence coordinates are equal to the current drawing sequence coordinates s from the normalized genome sequence obtained in step (1-3). Based on the row counter m, column counter q and preset document rendering parameters, obtain the drawing position of the nucleotide character on the p-th page and place the nucleotide character at the drawing position. (2-8) Add the character content c, global sequence coordinates g=s, and pixel-level bounding box r=(x1,y1,x2,y2) corresponding to the nucleotide character obtained in step (2-7) to the record item corresponding to the p-th page in the page annotation set as nucleotide annotation items, set the column counter q=q+1, and return to step (2-6); where x1 and y1 represent the horizontal and vertical coordinates of the upper left corner of the pixel-level bounding box corresponding to the nucleotide character, respectively, and x2 and y2 represent the horizontal and vertical coordinates of the lower right corner of the pixel-level bounding box corresponding to the nucleotide character, respectively.

[0035] (2-9) Set the row counter m=m+1 and return to step (2-4). (2-10) Save the p-th page as a DNA document image page_p, and add the DNA document image page_p to the DNA document image set; (2-11) Set the page counter p = p + 1 and return to step (2-2); (2-12) Output the DNA document image set D={page_0,page_1,...,page_{P-1}} and its corresponding page annotation set; Specifically, each DNA document image in the DNA document image set D corresponds to a record in the page annotation set. This record includes multiple nucleotide annotations in the corresponding page, and each nucleotide annotation includes a nucleotide character, global coordinates, and a pixel-level bounding box.

[0036] The advantage of this step (2) is that it can save nucleotide characters, global sequence coordinates and pixel-level bounding boxes while generating DNA document images, so that the bidirectional mapping in the subsequent step (3) and the region-based prompting task in step (5) have traceable page annotation basis.

[0037] (3) Based on the standardized genome sequence obtained in step (1) and the page annotation set obtained in step (2), establish the mapping relationship between the genome region and the DNA document page region.

[0038] This step includes the following sub-steps: (3-1) Select any continuous interval [i,j) from the normalized genome sequence obtained in step (1). Based on the global sequence coordinates of all nucleotide annotations in the page annotation set obtained in step (2), retrieve all nucleotide annotations covered by the interval. Here, i represents the global sequence coordinate of the first nucleotide character covered by the interval, j represents the endpoint covered by the interval, i.e., the global sequence coordinate of the position after the last nucleotide character, and 0≤i<j≤length Len of the normalized genome sequence.

[0039] (3-2) Determine whether all nucleotide annotations covered by the interval [i,j) are completely located in the same DNA document image in the DNA document image set D; if so, generate the minimum bounding rectangle covering the interval as the corresponding interval bounding box based on the pixel-level bounding boxes of all nucleotide annotations in the interval, and proceed to step (3-3); otherwise, generate the corresponding page bounding box in each DNA document image where the interval [i,j) is located, and all page bounding boxes constitute the interval bounding box set corresponding to the interval, and proceed to step (3-3). (3-3) Associate the interval [i,j) with the interval bounding box or set of interval bounding boxes obtained in step (3-2) to form a mapping relationship from interval to region; (3-4) Obtain the DNA document page region to be reverse mapped, its page index, and pixel coordinate range from the given region information. In the page annotation set obtained in step (2), retrieve all nucleotide annotation items whose corresponding pixel-level bounding boxes intersect with or are covered by the DNA document page region. Sort all nucleotide annotation items in descending order according to the global sequence coordinates saved by each nucleotide annotation item to obtain the sorting result. Set the global sequence coordinates corresponding to the first nucleotide annotation item in the sorting result as the starting global sequence coordinates and set the global sequence coordinates corresponding to the last nucleotide annotation item in the sorting result as the ending global sequence coordinates to obtain the sequence interval corresponding to the DNA document page region. Associate the DNA document page region with the sequence interval to form a reverse mapping relationship from region to interval. (3-5) Based on the nucleotide annotations retrieved in step (3-1), the interval bounding boxes or sets of interval bounding boxes generated in step (3-2), the interval-to-region mapping relationship formed in step (3-3), and the region-to-interval inverse mapping relationship formed in step (3-4), construct the mapping relationship between genomic intervals and DNA document page regions.

[0040] Specifically, this step involves first using the interval [i,j) as the index key and saving the interval bounding box or set of interval bounding boxes obtained in step (3-2) as the index value to obtain a forward index record from interval to region; then, based on this forward index record and the page annotation set obtained in step (2), the page bounding box generated for each DNA document image in step (3-2) is mapped to the page index of the DNA document image and the starting and ending global sequence coordinates covered by the page bounding box to obtain a page bounding box index record; subsequently, based on the inverse mapping result from region to interval obtained in step (3-4), the page bounding box index record is processed to establish a reverse index from region to interval to obtain a region The process begins by reversing the interval index record (enabling reverse lookup of corresponding sequence intervals within page regions); then, the interval bounding box set obtained in step (3-2) is sorted according to the page index and global sequence coordinates to obtain an ordered mapping record of the interval bounding box set (the purpose of which is to maintain a continuous correspondence between multiple page regions and the original genome intervals); finally, the forward index record from interval to region, the page bounding box index record, the reverse index record from region to interval, and the ordered mapping record of the interval bounding box set are summarized into a mapping relationship between genome intervals and DNA document page regions (which is used for region localization, transcription of regions of interest (ROI), query retrieval, mask completion, and result interpretation).

[0041] Through the above steps, the present invention can bidirectionally associate any functional region, variant site neighborhood, query subsequence location, or model interest region in a genome sequence with a page region in a DNA document image.

[0042] The advantage of this step (3) is that it can establish a bidirectional index between the genomic region and the page region, so that the mapping relationship can be directly called for region localization, transcription of region of interest (ROI), query retrieval, mask completion and result interpretation.

[0043] (4) Input the DNA document image set obtained in step (2) into the visual encoder to obtain the page-level visual feature sequence, and perform feature projection processing and multi-page feature fusion processing on the page-level visual feature sequence in sequence to obtain the document-level visual token sequence.

[0044] In this step, the visual encoder can be a convolutional neural network, a visual Transformer, a hybrid convolutional-transformer network, a document image encoding network, an optical character recognition (OCR) visual encoding network, or other visual front-ends capable of processing image patches. Feature projection processing is used to map the features output by the visual front-end to the latent space dimension required by the document decoder, and multi-page feature fusion processing is used to aggregate multi-page image features into a document-level representation.

[0045] This step includes the following sub-steps: (4-1) Adjust each page of DNA document image in the DNA document image set obtained in step (2) to the preset input size (specifically 640×640, or set according to the input requirements of the visual front-end network in the visual encoder), and divide the DNA document image after adjustment into multiple image blocks (the image blocks can be non-overlapping image blocks or image blocks with a certain overlap ratio).

[0046] (4-2) Encode each image block corresponding to each page of DNA document image obtained in step (4-1) using a visual front-end network to obtain the image block feature corresponding to the image block, and arrange the image block features corresponding to all image blocks of the DNA document image of the page in the page spatial order to form the page-level visual feature sequence corresponding to the DNA document image of the page.

[0047] (4-3) Perform linear projection, nonlinear mapping or multilayer perceptron mapping on the page-level visual feature sequence corresponding to each page of DNA document image obtained in step (4-2) (so that the page-level visual feature sequence matches the input dimension of the document decoder) to obtain the projected page-level visual feature sequence corresponding to the page of DNA document image.

[0048] (4-4) Perform multi-page feature fusion processing on the projected page-level visual feature sequences corresponding to all DNA document images in the DNA document image set obtained in step (4-3) to obtain the fused document-level visual features.

[0049] Specifically, multi-page feature fusion processing can employ any one or more combinations of self-attention, cross-attention, gated aggregation, mean pooling, max pooling, and learnable query aggregation.

[0050] (4-5) The fused document-level visual features obtained in step (4-4) are successively normalized, positionally encoded, and represented with a fixed length to obtain a document-level visual token sequence.

[0051] Specifically, the document-level visual token sequence obtained in this step can represent the content, layout, and cross-page context of the input DNA document.

[0052] The advantage of this step (4) is that it can convert multi-page DNA document images into fixed-length or compact document-level visual token sequences, while preserving page layout, local nucleotide patterns and cross-page context information while reducing the effective token budget.

[0053] (5) Based on the DNA document image set obtained in step (2) and the mapping relationship between the genomic region and the DNA document page region obtained in step (3), construct an optical character recognition prompt task set and generate corresponding supervision results for each optical character recognition prompt task in the optical character recognition prompt task set.

[0054] This invention abstracts the genome sequence understanding task into an OCR-style document understanding task.

[0055] The task prompts in this step include the DNA sequence reading prompt in step (5-1), the region joint localization prompt in step (5-2), the region of interest transcription prompt in step (5-3), the masked region completion prompt in step (5-4), the query subsequence localization prompt in step (5-5), and the global document category recognition prompt in step (5-6). These task prompts, together with the DNA document image set, bounding boxes, or labels, constitute the input for subsequent supervised training.

[0056] This step includes the following sub-steps: (5-1) Configure DNA sequence reading prompts for the DNA document image set obtained in step (2) to obtain an updated DNA document image set. Use the standardized genome sequence obtained in step (1-3) as the supervision result, and combine the DNA sequence reading prompts, the updated DNA document image set and the supervision result into a DNA sequence reading task sample.

[0057] (5-2) Based on the mapping relationship between the genomic regions and the DNA document page regions obtained in step (3), select multiple text lines, text blocks or continuous intervals from the DNA document image set and its corresponding page annotation set obtained in step (2), and configure region joint localization prompts for the selected text lines, text blocks or continuous intervals; obtain the nucleotide text fragments covered by each selected text line, text block or continuous interval, and obtain the page bounding box corresponding to the nucleotide text fragment according to the mapping relationship obtained in step (3). All the obtained nucleotide text fragments constitute DNA text, and all the page bounding boxes corresponding to all nucleotide text fragments constitute the supervision result of the region joint localization task. Combine the region joint localization prompts, the DNA document image set and the supervision result into a region joint localization task sample.

[0058] (5-3) Based on the mapping relationship between the genomic regions and DNA document page regions obtained in step (3), multiple regions of interest bounding boxes are obtained through user input, random sampling, or preset task configuration, and a region of interest transcription prompt is configured for each region of interest bounding box; all nucleotide characters covered by each region of interest bounding box are retrieved in the page annotation set obtained in step (2), and all nucleotide characters are arranged in the order of their corresponding global sequence coordinates to form the nucleotide sequence text corresponding to the region of interest bounding box. All regions of interest bounding boxes and their corresponding nucleotide sequence text constitute the supervision result of the region of interest transcription task, and the region of interest transcription prompt, DNA document image set, region of interest bounding box, and supervision result are combined into a region of interest transcription task sample.

[0059] (5-4) Based on the mapping relationship between the genomic intervals and DNA document page regions obtained in step (3), obtain multiple continuous nucleotide intervals from the standardized genomic sequence or preset task configuration obtained in step (1-3), and configure mask region completion prompts for each continuous nucleotide interval; perform mask processing on the page regions corresponding to all the obtained continuous nucleotide intervals in the DNA document image set obtained in step (2) to obtain the masked DNA document image set, use the nucleotide sequence text corresponding to all continuous nucleotide intervals in the standardized genomic sequence obtained in step (1-3) as the supervision result, and combine the mask region completion prompts, all the masked DNA document image sets and their corresponding page region bounding boxes, and all supervision results together as the mask region completion task sample.

[0060] (5-5) Obtain the query nucleotide subsequence from the standardized genome sequence obtained in step (1-3), and retrieve all continuous nucleotide intervals corresponding to the query nucleotide subsequence from the standardized genome sequence obtained in step (1-3); input all continuous nucleotide intervals into the mapping relationship between the genome interval and the DNA document page area obtained in step (3) to obtain the pixel-level bounding box corresponding to each continuous nucleotide interval, and configure the query subsequence location prompt for the continuous nucleotide interval. Combine the query nucleotide subsequence, all continuous nucleotide intervals, their corresponding pixel-level bounding boxes and the query subsequence location prompt into a query subsequence location task sample.

[0061] (5-6) From the sample number, species information, chromosome number and data source recorded in step (1-1), obtain multiple document-level tags corresponding to the DNA document image set obtained in step (2), perform category encoding processing on all document-level tags to obtain multiple document-level category supervision results, and configure global document category recognition prompts for multiple document-level category supervision results; combine the global document category recognition prompts, DNA document image set and document-level category supervision results into multiple global document category recognition task samples; wherein, the document-level tag includes one or more of chromosome number, species category, subspecies category, tissue category, functional category, sample category, phenotypic category or breeding trait category.

[0062] (5-7) Combine the DNA sequence reading task samples obtained in step (5-1), the region joint localization task samples obtained in step (5-2), the region of interest transcription task samples obtained in step (5-3), the mask region completion task samples obtained in step (5-4), the query subsequence localization task samples obtained in step (5-5), and the global document category recognition task samples obtained in step (5-6) into an optical character recognition prompt task set.

[0063] Specifically, for DNA sequence reading task samples, the optical character recognition prompt task set contains nucleotide sequence text, and the corresponding supervision result is the nucleotide sequence text of the normalized genome sequence, which is then used to recover the complete nucleotide character sequence in the DNA document according to the page reading order constraint model. For region joint localization task samples, region of interest transcription task samples, and query subsequence localization task samples, the optical character recognition prompt task set contains nucleotide sequence text and bounding box fields, and the corresponding supervision results are text-region pairs composed of nucleotide text fragments and their corresponding pixel-level bounding boxes, nucleotide sequence text covered by the given region of interest bounding box, and pixel-level bounding boxes corresponding to all occurrences of the query nucleotide subsequence in the DNA document image set, respectively. For masked region completion task samples, the optical character recognition prompt task set contains the completed nucleotide sequence text and the corresponding region bounding box, and the corresponding supervision result is the nucleotide sequence text that should be completed within the masked region. For global document category recognition task, the optical character recognition prompt task set contains category labels, and the corresponding supervision result is the document-level category label corresponding to the DNA document image set.

[0064] The format of each optical character recognition (OCR) cue task in the OCR task set can be a natural language template, key-value pairs, JavaScript Object Notation (JSON), Extensible Markup Language (XML), or other structured text formats.

[0065] The advantage of this step (5) is that it unifies DNA reading, region joint localization, region of interest transcription, masked region completion, query subsequence localization and global document category recognition into a prompt task sample, so that the subsequent step (6) can learn multiple genome document understanding capabilities under the same supervised training framework.

[0066] (6) Based on the document-level visual token sequence obtained in step (4) and the optical character recognition prompt task set obtained in step (5), supervised training is performed on the genome visual model including the visual encoder, projection module, multi-page fusion module and document decoder to obtain the trained genome visual model.

[0067] The document decoder in this step can be an autoregressive language model, a sequence-to-sequence decoder, a visual language model decoder, a hybrid expert decoder, or other models capable of generating output sequences based on visual tokens and textual cues. This document decoder receives task cues and document-level visual token sequences and generates nucleotide text, bounding boxes, category labels, or structured prediction results according to a preset output format.

[0068] This step includes the following sub-steps: (6-1) Obtain multiple task samples from the optical character recognition prompt task set obtained in step (5), as well as the DNA sequence reading prompt, region joint localization prompt, region of interest transcription prompt, mask region completion prompt, query subsequence localization prompt, or global document category recognition prompt and the supervision result corresponding to each task sample.

[0069] (6-2) Input all the task samples obtained in step (6-1) into the visual encoder, projection module and multi-page fusion module in the genome visual modeling model in sequence to obtain the document-level visual token sequence.

[0070] (6-3) Input the DNA sequence reading prompts, region joint localization prompts, region of interest transcription prompts, mask region completion prompts, query subsequence localization prompts, or global document category recognition prompts and the document-level visual token sequence obtained in step (6-2) into the document decoder so that it generates the task output sequence in an autoregressive manner.

[0071] (6-4) Based on the supervision results corresponding to all task samples obtained in step (6-1) and the task output sequence obtained in step (6-3), obtain the autoregressive cross-entropy loss value of the task output sequence relative to the supervision results; Specifically, the formula for calculating the autoregressive cross-entropy loss is as follows: in, This represents the t-th token in the output sequence of the supervision results corresponding to all task samples. Z represents all tokens before the t-th token in the output sequence, Z represents the document-level visual token sequence obtained in step (6-2), q represents the task cues corresponding to all task samples, and T represents the token length of the supervision result output sequence corresponding to all task samples.

[0072] (6-5) Update the parameters of the genome visual modeling model based on the autoregressive cross-entropy loss value obtained in step (6-4); during the update process, the visual front-end network in the visual encoder is frozen, and at least one of the projection module, multi-page fusion module and document decoder is fine-tuned with full parameters or low-rank adaptation. (6-6) Repeat steps (6-1) to (6-5) until the preset number of training steps (200,000 in this invention) is reached or the genome visual modeling model converges, thereby obtaining the trained genome visual modeling model.

[0073] Furthermore, during training, one or more strategies can be employed, such as task-balanced sampling, random sampling of regions of interest, random sampling of subsequence lengths, random truncation at the end of the page, random setting of mask span, font perturbation, line spacing perturbation, page resolution perturbation, and start position perturbation, to improve the robustness of the model to different sequence lengths, different page layouts, and different rendering parameters.

[0074] The advantage of this step (6) is that by jointly training the visual encoder, projection module, multi-page fusion module and document decoder through task prompts and supervision results, the model can simultaneously learn nucleotide recognition, region localization, sequence retrieval, fragment completion and category prediction capabilities, and the training cost can be reduced by freezing the visual front end and low-rank adaptation.

[0075] (7) Obtain the genome sequence to be predicted from the FASTA format sequence text file, the FASTQ format sequence text file with base quality value, the reference genome database, the sequencing read splicing results, the individual sequence after mutation substitution, or the nucleotide sequence input by the user. Using the genome visual modeling model obtained in step (6), perform characterization extraction and downstream task prediction on the genome sequence to be predicted in sequence to obtain the corresponding genome function prediction results, region location results, subsequence retrieval results, missing fragment completion results, or phenotype prediction results.

[0076] The advantage of this step (7) is that it enables the trained genome visual modeling model to be transferred to tasks such as functional prediction, regional localization, subsequence retrieval, missing fragment completion and phenotypic prediction in breeding applications, thereby improving the flexibility of downstream deployment.

[0077] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for visual modeling of genome sequences based on optical character recognition document understanding in breeding applications, characterized in that, Includes the following steps: (1) Obtain the genome sequence to be processed and perform normalization processing on the genome sequence to obtain a normalized genome sequence; (2) Based on the preset document rendering parameters, the standardized genome sequence obtained in step (1) is subjected to document rendering processing to obtain a set of DNA document images and its corresponding page annotation set; (3) Based on the standardized genome sequence obtained in step (1) and the page annotation set obtained in step (2), establish the mapping relationship between the genome region and the DNA document page region; (4) Input the DNA document image set obtained in step (2) into the visual encoder to obtain the page-level visual feature sequence, and perform feature projection processing and multi-page feature fusion processing on the page-level visual feature sequence in sequence to obtain the document-level visual token sequence. (5) Based on the DNA document image set obtained in step (2) and the mapping relationship between the genomic region and the DNA document page region obtained in step (3), construct an optical character recognition prompt task set and generate corresponding supervision results for each optical character recognition prompt task in the optical character recognition prompt task set; (6) Based on the document-level visual token sequence obtained in step (4) and the optical character recognition prompt task set obtained in step (5), supervised training is performed on the genome visual modeling model including the visual encoder, projection module, multi-page fusion module and document decoder to obtain the trained genome visual modeling model. (7) Obtain the genome sequence to be predicted from the sequence text file in FASTA format, the sequence text file in FASTQ format with base quality values, the reference genome database, the sequencing read splicing results, the individual sequence after mutation substitution, or the nucleotide sequence input by the user. Use the genome visual modeling model obtained in step (6) to process the genome sequence to be predicted in sequence to obtain the corresponding genome function prediction results, region location results, subsequence retrieval results, missing fragment completion results, or phenotype prediction results.

2. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to claim 1, characterized in that, Step (1) includes the following sub-steps: (1-1) Obtain the genome sequence to be processed, read the genome sequence to be processed to obtain the nucleotide characters in the genome sequence to be processed, and record one or more of the following: sample number, species information, chromosome number, original sequence start coordinates, original sequence end coordinates and data source corresponding to the genome sequence to be processed; (1-2) Convert the lowercase nucleotide characters in the genome sequence to be processed to uppercase nucleotide characters to obtain the genome sequence after one update, and delete the nucleotide characters in the genome sequence after one update that are not included in the preset nucleotide character set {A,C,G,T,N} to obtain the genome sequence after two updates; where A represents adenine base, C represents cytosine base, G represents guanine base, T represents thymine base, and N represents unknown base or undetermined base; (1-3) Assign global sequence coordinates to each nucleotide character in the second-updated genome sequence obtained in step (1-2) to obtain the coordinate-assigned genome sequence as the standardized genome sequence.

3. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to claim 1 or 2, characterized in that, Step (2) includes the following sub-steps: (2-1) Based on the preset document rendering parameters and the length Len of the standardized genome sequence obtained in step (1-3), determine the number of text lines M that each page can accommodate, the number of nucleotide characters Q that each line in each page can accommodate, and the total number of nucleotide characters C that a single page can accommodate. Based on the length Len of the standardized genome sequence and the total number of nucleotide characters C, obtain the total number of pages P according to the formula P=⌈Len / C⌉ (where ⌈·⌉ means rounding up). Set the page counter p=0 and initialize the DNA document image set and page annotation set to empty. (2-2) Determine whether the page counter p is greater than or equal to the total number of pages P. If yes, proceed to step (2-12); otherwise, proceed to step (2-3). (2-3) Create the p-th page (initially a blank page) according to the preset document rendering parameters, create the record item corresponding to the p-th page in the page annotation set, and set the line counter m=1; (2-4) Determine whether the line counter m is greater than the number of text lines M that a single page can hold. If yes, proceed to step (2-10); otherwise, proceed to step (2-5). (2-5) Set the column counter q=1; (2-6) Calculate the current plotted sequence coordinates s = p × C + (m-1) × Q + (q-1) based on the page counter p, row counter m and column counter q, and determine whether the column counter q is greater than the number of nucleotide characters Q that each row can hold, or whether the current plotted sequence coordinates s are greater than or equal to the length Len of the normalized genome sequence obtained in step (1-3). If so, proceed to step (2-9); otherwise, proceed to step (2-7). (2-7) Obtain a nucleotide character whose global sequence coordinates are equal to the current drawing sequence coordinates s from the normalized genome sequence obtained in step (1-3). Based on the row counter m, column counter q and preset document rendering parameters, obtain the drawing position of the nucleotide character on the p-th page and place the nucleotide character at the drawing position. (2-8) Add the character content c, global sequence coordinates g=s, and pixel-level bounding box r=(x1,y1,x2,y2) corresponding to the nucleotide character obtained in step (2-7) to the record item corresponding to the p-th page in the page annotation set as nucleotide annotation items, set the column counter q=q+1, and return to step (2-6); where x1 and y1 represent the horizontal and vertical coordinates of the upper left corner of the pixel-level bounding box corresponding to the nucleotide character, respectively, and x2 and y2 represent the horizontal and vertical coordinates of the lower right corner of the pixel-level bounding box corresponding to the nucleotide character, respectively; (2-9) Set the row counter m=m+1 and return to step (2-4). (2-10) Save the p-th page as a DNA document image page_p, and add the DNA document image page_p to the DNA document image set; (2-11) Set the page counter p = p + 1 and return to step (2-2); (2-12) Output the DNA document image set D={page_0,page_1,...,page_{P-1}} and its corresponding page annotation set.

4. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to any one of claims 1 to 3, characterized in that, The genome sequences to be processed come from FASTA format sequence text files, FASTQ format sequence text files with base quality values, reference genome databases, sequencing read splicing results, genome fragment sets, individual sequences after mutation substitutions, or user-input nucleotide sequences; The genome sequence to be processed is a DNA sequence of a human, animal, plant, microorganism or other species, or a chromosome fragment, regulatory region, candidate variant region or whole genome sequence slice; The preset document rendering parameters include one or more of the following: page resolution, page width, page height, font type, font size, character spacing, line spacing, page margins, number of characters per line, maximum character capacity per page, background color, text color, page orientation, and rendering start position. Each DNA document image in the DNA document image set D corresponds to a record in the page annotation set. This record includes multiple nucleotide annotations in the corresponding page. Each nucleotide annotation includes a nucleotide character, global coordinates, and a pixel-level bounding box.

5. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to claim 4, characterized in that, Step (3) includes the following sub-steps: (3-1) Select any continuous interval [i,j) from the normalized genome sequence obtained in step (1). Based on the global sequence coordinates of all nucleotide annotations in the page annotation set obtained in step (2), retrieve all nucleotide annotations covered by the interval. Here, i represents the global sequence coordinate of the first nucleotide character covered by the interval, j represents the endpoint covered by the interval, i.e., the global sequence coordinate of the position after the last nucleotide character, and 0≤i<j≤lengthLen of the normalized genome sequence. (3-2) Determine whether all nucleotide annotations covered by the interval [i,j) are completely located in the same DNA document image in the DNA document image set D; if so, generate the minimum bounding rectangle covering the interval as the corresponding interval bounding box based on the pixel-level bounding boxes of all nucleotide annotations in the interval, and proceed to step (3-3); otherwise, generate the corresponding page bounding box in each DNA document image where the interval [i,j) is located, and all page bounding boxes constitute the interval bounding box set corresponding to the interval, and proceed to step (3-3). (3-3) Associate the interval [i,j) with the interval bounding box or set of interval bounding boxes obtained in step (3-2) to form a mapping relationship from interval to region; (3-4) Obtain the DNA document page region to be reverse mapped, its page index, and pixel coordinate range from the given region information. In the page annotation set obtained in step (2), retrieve all nucleotide annotation items whose corresponding pixel-level bounding boxes intersect with or are covered by the DNA document page region. Sort all nucleotide annotation items in descending order according to the global sequence coordinates saved by each nucleotide annotation item to obtain the sorting result. Set the global sequence coordinates corresponding to the first nucleotide annotation item in the sorting result as the starting global sequence coordinates and set the global sequence coordinates corresponding to the last nucleotide annotation item in the sorting result as the ending global sequence coordinates to obtain the sequence interval corresponding to the DNA document page region. Associate the DNA document page region with the sequence interval to form a reverse mapping relationship from region to interval. (3-5) Based on the nucleotide annotations retrieved in step (3-1), the interval bounding boxes or sets of interval bounding boxes generated in step (3-2), the interval-to-region mapping relationship formed in step (3-3), and the region-to-interval inverse mapping relationship formed in step (3-4), construct the mapping relationship between genomic intervals and DNA document page regions.

6. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to claim 5, characterized in that, Step (3-5) specifically involves the following steps: First, the interval [i,j) is used as the index key, and the interval bounding box or set of interval bounding boxes obtained in step (3-2) is saved as the index value to obtain a forward index record from the interval to the region; then, based on this forward index record and the page annotation set obtained in step (2), the page bounding box generated for each DNA document image in step (3-2) is mapped and recorded with the page index of the DNA document image and the starting and ending global sequence coordinates covered by the page bounding box to obtain a page bounding box index record; subsequently, Based on the inverse mapping results from region to interval obtained in step (3-4), the page bounding box index record is processed to establish a reverse index from region to interval to obtain a reverse index record from region to interval. Then, the set of interval bounding boxes obtained in step (3-2) is sorted according to the page index and global sequence coordinates to obtain an ordered mapping record of the set of interval bounding boxes. Finally, the forward index record from interval to region, the page bounding box index record, the reverse index record from region to interval, and the ordered mapping record of the set of interval bounding boxes are summarized into a mapping relationship between genomic intervals and DNA document page regions.

7. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to claim 6, characterized in that, Step (4) includes the following sub-steps: (4-1) Adjust each page of DNA document image in the DNA document image set obtained in step (2) to the preset input size, and divide the DNA document image after adjustment into multiple image blocks; (4-2) Encode each image block corresponding to each page of DNA document image obtained in step (4-1) using a visual front-end network to obtain the image block feature corresponding to the image block, and arrange the image block features corresponding to all image blocks of the DNA document image of the page in the page spatial order to form the page-level visual feature sequence corresponding to the DNA document image of the page. (4-3) Perform linear projection, nonlinear mapping or multilayer perceptron mapping on the page-level visual feature sequence corresponding to each page of DNA document image obtained in step (4-2) to obtain the projected page-level visual feature sequence corresponding to the page of DNA document image. (4-4) Perform multi-page feature fusion processing on the projected page-level visual feature sequences corresponding to all DNA document images in the DNA document image set obtained in step (4-3) to obtain the fused document-level visual features; in Multi-page feature fusion processing employs any one or more combinations of self-attention, cross-attention, gated aggregation, mean pooling, max pooling, and learnable query aggregation; (4-5) The fused document-level visual features obtained in step (4-4) are successively normalized, positionally encoded, and represented with a fixed length to obtain a document-level visual token sequence.

8. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to claim 7, characterized in that, Step (5) includes the following sub-steps: (5-1) Configure DNA sequence reading prompts for the DNA document image set obtained in step (2) to obtain an updated DNA document image set. Use the standardized genome sequence obtained in step (1-3) as the supervision result, and combine the DNA sequence reading prompts, the updated DNA document image set and the supervision result into a DNA sequence reading task sample. (5-2) Based on the mapping relationship between the genomic regions and the DNA document page regions obtained in step (3), select multiple text lines, text blocks or continuous intervals from the DNA document image set and its corresponding page annotation set obtained in step (2), and configure region joint localization prompts for the selected text lines, text blocks or continuous intervals; obtain the nucleotide text fragments covered by each selected text line, text block or continuous interval, and obtain the page bounding box corresponding to the nucleotide text fragment according to the mapping relationship obtained in step (3). All the obtained nucleotide text fragments constitute DNA text, and all the page bounding boxes corresponding to all nucleotide text fragments constitute the supervision result of the region joint localization task. Combine the region joint localization prompts, the DNA document image set and the supervision result into a region joint localization task sample. (5-3) Based on the mapping relationship between the genomic regions and DNA document page regions obtained in step (3), multiple regions of interest bounding boxes are obtained through user input, random sampling, or preset task configuration, and a region of interest transcription prompt is configured for each region of interest bounding box; all nucleotide characters covered by each region of interest bounding box are retrieved in the page annotation set obtained in step (2), and all nucleotide characters are arranged in the order of their corresponding global sequence coordinates to form the nucleotide sequence text corresponding to the region of interest bounding box. All regions of interest bounding boxes and their corresponding nucleotide sequence text constitute the supervision result of the region of interest transcription task, and the region of interest transcription prompt, DNA document image set, region of interest bounding box and the supervision result are combined into a region of interest transcription task sample; (5-4) Based on the mapping relationship between the genomic regions and DNA document page regions obtained in step (3), obtain multiple continuous nucleotide regions from the standardized genomic sequence or preset task configuration obtained in step (1-3), and configure mask region completion prompts for each continuous nucleotide region; perform mask processing on the page regions corresponding to all the obtained continuous nucleotide regions in the DNA document image set obtained in step (2) to obtain the masked DNA document image set, use the nucleotide sequence text corresponding to all continuous nucleotide regions in the standardized genomic sequence obtained in step (1-3) as the supervision result, and combine the mask region completion prompts, all the masked DNA document image sets and their corresponding page region bounding boxes, and all supervision results together as the mask region completion task sample; (5-5) Obtain the query nucleotide subsequence from the standardized genome sequence obtained in step (1-3), and retrieve all continuous nucleotide intervals corresponding to the query nucleotide subsequence from the standardized genome sequence obtained in step (1-3); input all continuous nucleotide intervals into the mapping relationship between the genome interval and the DNA document page area obtained in step (3) to obtain the pixel-level bounding box corresponding to each continuous nucleotide interval, and configure the query subsequence location prompt for the continuous nucleotide interval; combine the query nucleotide subsequence, all continuous nucleotide intervals, their corresponding pixel-level bounding boxes and the query subsequence location prompt into a query subsequence location task sample; (5-6) From the sample number, species information, chromosome number and data source recorded in step (1-1), obtain multiple document-level tags corresponding to the DNA document image set obtained in step (2), perform category encoding on all document-level tags to obtain multiple document-level category supervision results, and configure global document category recognition prompts for multiple document-level category supervision results; The global document category recognition prompts, DNA document image sets, and document-level category supervision results are combined into multiple global document category recognition task samples; wherein, the document-level labels include one or more of the following: chromosome number, species category, subspecies category, tissue category, functional category, sample category, phenotypic category, or breeding trait category; (5-7) Combine the DNA sequence reading task samples obtained in step (5-1), the region joint localization task samples obtained in step (5-2), the region of interest transcription task samples obtained in step (5-3), the mask region completion task samples obtained in step (5-4), the query subsequence localization task samples obtained in step (5-5), and the global document category recognition task samples obtained in step (5-6) into an optical character recognition prompt task set.

9. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to claim 8, characterized in that, Step (6) includes the following sub-steps: (6-1) Obtain multiple task samples from the optical character recognition prompt task set obtained in step (5), as well as the DNA sequence reading prompt, region joint localization prompt, region of interest transcription prompt, mask region completion prompt, query subsequence localization prompt, or global document category recognition prompt and the supervision result corresponding to each task sample; (6-2) Input all the task samples obtained in step (6-1) into the visual encoder, projection module and multi-page fusion module in the genome visual modeling model in sequence to obtain the document-level visual token sequence; (6-3) Input the DNA sequence reading prompts, region joint localization prompts, region of interest transcription prompts, mask region completion prompts, query subsequence localization prompts, or global document category recognition prompts and the document-level visual token sequence obtained in step (6-2) into the document decoder so that it generates the task output sequence in an autoregressive manner; (6-4) Based on the supervision results corresponding to all task samples obtained in step (6-1) and the task output sequence obtained in step (6-3), obtain the autoregressive cross-entropy loss value of the task output sequence relative to the supervision results; (6-5) Update the parameters of the genome visual modeling model based on the autoregressive cross-entropy loss value obtained in step (6-4); (6-6) Repeat steps (6-1) to (6-5) until the preset number of training steps is reached or the genome visual model converges, thereby obtaining the trained genome visual model.

10. The genome sequence visual modeling method based on optical character recognition document understanding in breeding applications according to claim 9, characterized in that, The formula for calculating the autoregressive cross-entropy loss is: ； in, This represents the t-th token in the output sequence of the supervision results corresponding to all task samples. Z represents all tokens before the t-th token in the output sequence, Z represents the document-level visual token sequence obtained in step (6-2), q represents the task cues corresponding to all task samples, and T represents the token length of the supervision result output sequence corresponding to all task samples. During the update process in step (6-5), the visual front-end network in the visual encoder is frozen, and at least one of the projection module, multi-page fusion module and document decoder is fine-tuned with full parameters or low-rank adaptation.

Images