High-resolution large field-of-view imaging system based on multi-image stitching and fusion

Abstract

A high-resolution large-field imaging system based on multi-image stitching and fusion first obtains multiple high-resolution local pathological section images and introduces a light compensation technology to improve the image quality and ensure that details are clearly visible. Then, a dynamic convolution labeling model is used to perform semantic feature encoding on the light-compensated pathological section images to capture fine local features such as cell textures and edges, and at the same time, a selection method based on the activity of the feature neighborhood is introduced to highlight important pathological features. Finally, the sparsified semantic features of the pathological section images are cascaded and input into an image fusion engine based on the AIGC model to generate a global pathological section fusion image that covers a larger tissue range and retains high-resolution details. In this way, not only the dual advantages of high resolution and large field of view are achieved, but also clearer and more targeted image support is provided for pathological diagnosis, significantly improving the accuracy and efficiency of diagnosis.

Description

Technical Field

[0001] This application relates to the field of intelligent image processing technology, and more specifically, to a high-resolution large field of view imaging system based on multi-image stitching and fusion. Background Art

[0002] Pathological section analysis is an indispensable part of medical diagnosis, especially for the early detection and treatment of serious diseases such as cancer.

[0003] In traditional pathological imaging technology, the microscope is the most commonly used tool. However, traditional microscope imaging faces the dilemma of being difficult to balance resolution and field of view. Although high-resolution imaging can clearly present the microscopic structure of cells, such as the morphology of cell nuclei and the details of organelles, it can often only observe a very small local area. If the field of view is to be expanded to observe a larger range of pathological sections, the resolution will decrease accordingly, resulting in blurred images and many key pathological features being difficult to distinguish. But in tumor pathological diagnosis, doctors not only need to understand the microscopic features of tumor cells, but also need to know information such as the distribution range of tumors in the whole tissue and the boundary with surrounding normal tissues. Traditional imaging technology is difficult to meet these two aspects of requirements simultaneously.

[0004] Therefore, a high-resolution large field of view imaging solution based on multi-image stitching and fusion is desired. Summary of the Invention

[0005] This Summary of the Invention section is provided to introduce concepts in a brief form, which will be described in detail in the following Detailed Description section. This Summary of the Invention section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

[0006] In a first aspect, a high-resolution large field of view imaging system based on multi-image stitching and fusion is provided, which includes:

[0007] A pathological section image acquisition module for acquiring a plurality of pathological section images;

[0008] A pathological section image compensation and extraction module for performing illumination compensation and feature extraction on each of the plurality of pathological section images to obtain a plurality of semantic feature encoded maps of pathological section images;

[0009] The pathological section image semantic feature selection module is used to perform pathological section image semantic feature selection on each of the pathological section image semantic feature encoding maps in the multiple pathological section image semantic feature encoding maps to obtain multiple sparse pathological section image semantic feature encoding maps. Among them, the pathological section image semantic feature selection module includes: an image semantic feature local decomposition unit, which is used to perform feature decoupling and feature flattening on the features of the pathological section image semantic feature encoding map to obtain a set of pathological section image semantic local feature vectors; an image semantic feature local selection and aggregation unit, which is used to perform feature neighborhood activity selection and shape reshaping on the set of pathological section image semantic local feature vectors to obtain the sparse pathological section image semantic feature encoding map;

[0010] The global pathological section image generation module is used to concatenate the multiple sparse pathological section image semantic feature encoding maps into a global pathological section image semantic feature splicing encoding map, and based on the global pathological section image semantic feature splicing encoding map, obtain a global pathological section fusion image.

[0011] Optionally, the pathological section image compensation and extraction module includes: a pathological section compensation unit, which is used to perform illumination compensation on each of the multiple pathological section images to obtain multiple illumination-compensated pathological section images; a pathological section feature extraction unit, which is used to input each of the multiple illumination-compensated pathological section images into a pathological section image feature extractor to obtain the multiple pathological section image semantic feature encoding maps.

[0012] Optionally, the pathological section image feature extractor is a pathological section image feature extractor based on a dynamic convolution labeling model.

[0013] Optionally, the image semantic feature local decomposition unit is used to: perform feature decoupling on the pathological section image semantic feature encoding map along the channel dimension to obtain a set of pathological section image semantic feature encoding matrices; perform feature flattening on each of the pathological section image semantic feature encoding matrices in the set of pathological section image semantic feature encoding matrices to obtain the set of pathological section image semantic local feature vectors.

[0014] Optionally, the local selection and aggregation unit of the image semantic features includes: an image semantic local feature descending sorting subunit, configured to perform a descending sort on the set of pathological slice image semantic local feature vectors to obtain a descending sequence of pathological slice image semantic local feature vectors; an image semantic local feature selection subunit, configured to perform feature selection on the descending sequence of pathological slice image semantic local feature vectors to obtain a descending sequence of selected pathological slice image semantic local feature vectors; and an image semantic feature shape reshaping subunit, configured to reshape the feature shape of the descending sequence of selected pathological slice image semantic local feature vectors to obtain the sparse pathological slice image semantic feature encoded map.

[0015] Optionally, the image semantic local feature descending sorting subunit is configured to: input each pathological slice image semantic local feature vector in the set of pathological slice image semantic local feature vectors into an importance measurement module to obtain a set of pathological slice image semantic local feature importance score values; and based on the set of pathological slice image semantic local feature importance score values, perform a descending sort on the set of pathological slice image semantic local feature vectors to obtain the descending sequence of pathological slice image semantic local feature vectors.

[0016] Optionally, the image semantic local feature selection subunit is configured to: calculate the feature neighborhood activity of each pathological slice image semantic local feature vector based on the neighborhood features of each pathological slice image semantic local feature vector in the descending sequence of pathological slice image semantic local feature vectors to obtain a sequence of pathological slice image semantic local feature neighborhood activities; and based on the sequence of pathological slice image semantic local feature neighborhood activities, perform feature selection on the descending sequence of pathological slice image semantic local feature vectors to obtain the descending sequence of selected pathological slice image semantic local feature vectors.

[0017] Optionally, the global pathological slice image generation module is configured to: after concatenating the multiple sparse pathological slice image semantic feature encoded maps into the global pathological slice image semantic feature splicing encoded map, input it into an image fusion engine based on the AIGC model to obtain the global pathological slice fusion image.

[0018] Optionally, the image fusion engine is an image fusion engine based on the AIGC model.

[0019] With the above technical solution, first, multiple high-resolution local pathological section images are obtained, and a light compensation technique is introduced to improve the image quality and ensure that details are clearly visible. Then, a dynamic convolution labeling model is used to perform semantic feature encoding on the light-compensated pathological section images, capturing fine local features such as cell textures and edges. At the same time, a selection method based on feature neighborhood activity is introduced to highlight important pathological features. Finally, the sparsified semantic features of the pathological section images are cascaded and input into an image fusion engine based on the AIGC model to generate a global pathological section fusion image that covers a larger tissue range and retains high-resolution details. In this way, not only the dual advantages of high resolution and large field of view are achieved, but also clearer and more targeted image support is provided for pathological diagnosis, significantly improving the accuracy and efficiency of diagnosis.

[0020] Other features and advantages of the present application will be described in detail in the following detailed implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 FIG. is a block diagram of a high-resolution large-field-of-view imaging system based on multi-image stitching and fusion according to an embodiment of the present application.

[0023] Figure 2 FIG. is a block diagram of the pathological section image compensation and extraction module in the high-resolution large-field-of-view imaging system based on multi-image stitching and fusion according to an embodiment of the present application.

[0024] Figure 3 FIG. is a block diagram of the pathological section image semantic feature selection module in the high-resolution large-field-of-view imaging system based on multi-image stitching and fusion according to an embodiment of the present application.

[0025] Figure 4 FIG. is a block diagram of the local selection and aggregation unit of the image semantic features in the high-resolution large-field-of-view imaging system based on multi-image stitching and fusion according to an embodiment of the present application.

[0026] Figure 5 FIG. is a flowchart of a high-resolution large-field-of-view imaging method based on multi-image stitching and fusion according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not used to limit the protection scope of the present application.

[0028] It should be understood that the various steps recited in the method embodiments of the present application can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this regard.

[0029] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0030] It should be noted that the concepts such as "first", "second", etc. mentioned in the present application are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependence.

[0031] It should be noted that the modifications of "one" and "plural" mentioned in the present application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] The following will describe the specific embodiments of the present application in detail with reference to the accompanying drawings.

[0034] It should be understood that resolution refers to the ability of an imaging system to resolve the smallest details. High-resolution imaging means that the system can obtain images with richer and more accurate details. In the context of pathological section imaging, high resolution allows pathologists to observe the fine structure of cells, such as the morphology of the cell nucleus, the details of organelles within the cytoplasm, and the integrity of the cell membrane. The field of view refers to the range that an imaging system can observe. Large-field imaging means that the system can obtain images over a wider range. For pathological section examination, a large field of view means that more tissue areas can be seen in a single fused image, providing more comprehensive tissue information for pathological analysis.

[0035] Based on this, the present application proposes a high-resolution large-field imaging system 100 based on multi-image stitching and fusion. Its technical concept is to obtain multiple pathological section images, use machine learning-based image processing and imaging techniques to perform illumination compensation on each of the pathological section images, then perform semantic feature encoding on each of the illumination-compensated pathological section images, and then perform neighborhood activity semantic feature selection on the semantic features of each pathological section image, so as to intelligently generate a global pathological section fusion image according to the global stitching representation of the sparse semantic features of each pathological section image. This method of the present application not only retains the high-resolution details of each local image but also covers a larger tissue range, achieving the dual advantages of high resolution and large field of view.

[0036] In the present application, Figure 1 As shown in the block diagram of the high-resolution large-field imaging system based on multi-image stitching and fusion according to an embodiment of the present application, Figure 1 As shown, the high-resolution large-field imaging system 100 based on multi-image stitching and fusion includes:

[0037] A pathological section image acquisition module 110, configured to acquire multiple pathological section images;

[0038] A pathological section image compensation and extraction module 120, configured to perform illumination compensation and feature extraction on each of the multiple pathological section images to obtain multiple semantic feature encoded maps of pathological section images;

[0039] A pathological section image semantic feature selection module 130, configured to perform pathological section image semantic feature selection on each of the multiple semantic feature encoded maps of pathological section images respectively to obtain multiple sparse semantic feature encoded maps of pathological section images;

[0040] The global pathological section image generation module 140 is configured to concatenate the semantic feature encoding maps of the multiple sparsified pathological section images into a global pathological section image semantic feature splicing encoding map, and obtain a global pathological section fusion image based on the global pathological section image semantic feature splicing encoding map.

[0041] In the above high-resolution large field-of-view imaging system 100 based on multi-image splicing and fusion, the pathological section image acquisition module 110 is configured to acquire multiple pathological section images. Specifically, first, during the acquisition of pathological section images, sample preparation is the first and crucial step. This stage includes sampling from patients, usually obtaining tissue samples during surgical resection or biopsy. These samples will then go through a series of processes such as fixation (e.g., using formalin), dehydration, clearing, infiltration with paraffin, etc., and then be cut into extremely thin sections with a microtome and attached to glass slides. To enhance the visibility of cell structures, the pathological sections are also stained, and the most commonly used method is hematoxylin and eosin staining (H&E staining). In addition, according to research needs, special immunohistochemical staining or other techniques may also be used to label specific proteins or molecules to more clearly observe the target structures.

[0042] Next, it enters the microscope imaging stage. Here, it is crucial to select the appropriate microscope equipment. Based on the required resolution and field of view size, an appropriate microscope is selected, such as a fully automated high-resolution scanning microscope. This type of microscope can automatically scan the entire glass slide and generate high-resolution digital images. After placing the prepared glass slide in the digital scanner, the instrument will automatically adjust the focal length and translation platform according to the preset program to cover the entire surface of the glass slide. Each movement will capture a high-definition image of a small area, namely the so-called "tile". For samples with special staining or fluorescent labeling, multi-wavelength light sources and corresponding filters may also be required to separately capture information in different color channels, which helps to more comprehensively understand the tissue structure and function.

[0043] As the scanning process progresses, each tile is continuously photographed, and the software will seamlessly stitch these individual tiles into a complete panoramic image in real time. This process relies on precise coordinate mapping and image registration algorithms to ensure seamless connection and no distortion. At the same time, important information about each image, such as magnification, exposure time, scanning date, etc., which are metadata, will also be recorded. These information are very important for subsequent analysis. After the image acquisition is completed, there is a quality inspection step to ensure that all images are clearly distinguishable without obvious artifacts or defects. If there are problems, the corresponding area needs to be rescanned to ensure the consistency and reliability of the image quality.

[0044] After high-quality pathological section images are successfully obtained, the next step is data management. All pathological section images and their associated metadata are properly stored for long-term archiving and future retrieval. The pathological section images obtained in this way not only provide more detailed and intuitive diagnostic basis for pathologists, but also provide strong support for medical research and development, promoting the understanding of disease mechanisms and the development of treatment methods.

[0045] In the above high-resolution large-field imaging system 100 based on multi-image stitching and fusion, the pathological section image compensation and extraction module 120 is used to perform illumination compensation and feature extraction on each of the multiple pathological section images to obtain multiple semantic feature encoded maps of pathological section images. That is, during the processing, first, illumination compensation is performed on each pathological section image, which helps to improve the image quality and retain more detailed information. Then, semantic feature encoding and selection are carried out. One of the purposes of this series of operations is also to maintain the high-resolution characteristics after stitching and fusion. By precisely processing the features of each section image, the final generated global image can inherit the high-resolution details of each section, thus providing clearer image information for pathological diagnosis. And by seamlessly stitching these "puzzle pieces" together, a global pathological section fusion image is generated. This method realizes large-field imaging because it successfully combines multiple individual, high-resolution section images, enabling the final image to cover a larger tissue area and providing a more complete view of the tissue structure.

[0046] In a specific embodiment of the present application, Figure 2 As shown in the block diagram of the pathological section image compensation and extraction module in the high-resolution large-field imaging system based on multi-image stitching and fusion according to the embodiments of the present application, Figure 2 As shown, the pathological section image compensation and extraction module 120 includes: a pathological section compensation unit 121 for performing illumination compensation on each of the multiple pathological section images to obtain multiple illuminated compensated pathological section images; a pathological section feature extraction unit 122 for respectively inputting each of the multiple illuminated compensated pathological section images into a pathological section image feature extractor to obtain the multiple semantic feature encoded maps of pathological section images.

[0047] Among them, the pathological section image feature extractor is a pathological section image feature extractor based on a dynamic convolution labeling model.

[0048] Specifically, considering that during the process of obtaining pathological section images, due to factors such as uneven light source distribution of imaging devices, differences in the placement angles of sections, or interference from ambient light, it is very easy to result in inconsistent light intensities in different regions. Moreover, unsatisfactory lighting conditions such as uneven lighting, over-brightness, or over-darkness will reduce the overall quality of the image, making the image appear blurred, with details lost, or noise generated, etc. For example, some fine structures of cells may be hidden in the darkness and difficult to distinguish in overly dark regions, while overly bright regions may cause color distortion and blurred cell boundaries. Therefore, in order to correct this phenomenon of uneven lighting caused by external factors and restore the true appearance of pathological sections, this application performs light compensation on each of the multiple pathological section images to enhance the clarity, contrast, etc. of the images, improve the image quality, and obtain multiple light-compensated pathological section images.

[0049] During the implementation of light compensation, it is first necessary to conduct a detailed analysis of the originally obtained pathological section images to identify the specific manifestation forms of uneven lighting. This can be achieved by calculating the brightness values of each pixel point in the image and plotting a brightness histogram. By observing the brightness distribution, it is possible to determine which regions are over-bright or over-dark. In addition, methods such as spatial frequency analysis can also be used to detect whether there are periodic brightness changes in the image, which may be caused by stripes or other non-uniformities of the light source itself. For example, if the problem is a gradual uneven lighting caused by the strong center and weak edge of the microscope light source, then a radially symmetric lighting model can be selected for fitting.

[0050] Once the specific manifestation forms of uneven lighting are determined, the next step is to construct a lighting model. The lighting model is a mathematical description of the actual lighting conditions, which helps to understand and predict how lighting affects the image. For pathological section images, a model suitable for a specific application scenario is usually selected. For example, in the case of the above-mentioned radial uneven lighting, adopting a radially symmetric lighting model can better simulate the influence of the light source. Subsequently, a preprocessing algorithm is applied to estimate and remove the influence of lighting. Commonly used methods include global gray stretching, which is simple but may amplify noise and cannot well solve local uneven lighting; contrast-limited adaptive histogram equalization (CLAHE), which dynamically adjusts the brightness according to the characteristics of local regions of the image, can improve the visibility of local details while maintaining the overall contrast of the image, and is especially suitable for pathological sections with complex lighting patterns; multi-scale decomposition and reconstruction, which decomposes the image into sub-bands at multiple different scales, then performs light correction on each sub-band separately, and finally recombines them into a complete image, effectively removing the uneven lighting caused by different frequency bands while retaining important texture information; and deep learning methods based on convolutional neural networks (CNNs), which automatically capture lighting characteristics through learning a large amount of labeled data and generate more natural and realistic correction results.

[0051] After selecting an appropriate preprocessing algorithm, the correction of uneven illumination can be started. The specific approach is to calculate the ideal brightness value that each pixel should have according to the illumination model, and then adjust the brightness at the corresponding position in the original image so that the finally output image is as close as possible to the ideal state. In this process, some boundary conditions also need to be considered, such as avoiding excessive amplification of noise or introducing new artifacts. After completing the illumination compensation, the result must be verified to ensure that the image quality has indeed been improved. This can be carried out through two methods: visual inspection and quantitative evaluation. Visual inspection relies on the experience judgment of professionals to see whether the image becomes clearer, whether the contrast is appropriate, and whether the details are more obvious; while quantitative evaluation involves using objective metrics such as signal-to-noise ratio (SNR), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), etc. to measure the change in image quality before and after correction. Only when all tests indicate that the illumination compensation has achieved the expected goal can this process be considered successful.

[0052] In a specific embodiment of the present application, assume there is a pathological section image. The central part of it appears abnormally bright due to the concentration of the microscope light source, while the surrounding area is relatively dark. In this case, first use image processing software to load this pathological section image and conduct a preliminary analysis on it, and it is found that the brightness distribution shows an obvious radial gradient. Based on this observation, it is decided to adopt a simple radially symmetric illumination model to simulate this uneven illumination phenomenon and design a corresponding correction scheme accordingly. Next, apply the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm to preprocess the image to optimize the local contrast without changing the global brightness distribution. Then, calculate the ideal brightness value of each pixel according to the selected illumination model and adjust the original image according to this standard to ensure that the brightness of the central and edge regions tends to be consistent. Finally, check the processed image again to confirm that it is not only visually smoother and has no obvious transition traces, but also has significant improvements in quantitative metrics, such as a higher SNR and a better SSIM score. Through the above steps, the pathological section image with serious uneven illumination problems has successfully restored its due details and contrast after illumination compensation processing, providing high-quality basic data for subsequent pathological research.

[0053] Then, considering that each pathological section image contains many local details of great diagnostic value, such as cell texture, edges and other detailed information, and there is also a global semantic correlation relationship among the local texture and edge detailed information. Based on this, the present application uses a pathological section image feature extractor based on a dynamic convolution labeling model to respectively perform feature extraction on each of the multiple light-compensated pathological section images to obtain multiple semantic feature encoding maps of pathological section images. It should be understood that the dynamic convolution labeling model is a model that combines the advantages of a convolutional neural network (CNN) and a Transformer architecture. Specifically, first, in the local feature extraction stage, the model divides the input image into multiple local blocks. For an image as complex and detail-rich as the light-compensated pathological section image, CNN can efficiently process each local block. The CNN slides a convolutional kernel over the local block to detect the change patterns between pixels, thereby being able to accurately capture fine local features such as texture and edges. For example, in a pathological section, the CNN can keenly detect the texture changes of cells, such as the differences in texture between cancer cells and normal cells, and the edge features of tissue boundaries, which are crucial for pathological diagnosis. Then, it enters the global feature integration stage. The model stitches together the feature maps obtained by processing each local block through the CNN and uses them as the input to the Transformer model. The Transformer model uses its self-attention mechanism to calculate the correlation and importance of each element in the feature map in the global context. In this way, the model can comprehensively understand the semantics of the entire image and grasp the associations between local features in the pathological section image as a whole, such as the spatial relationships between different cell populations and the mutual influence between the diseased area and the normal tissue area. In particular, the "dynamic" in the dynamic convolution labeling model means that the model can flexibly adjust its own processing method according to the characteristics of the input data. This includes dynamically adjusting the weight distribution of attention through the self-attention mechanism or adaptively determining which local features need to be analyzed more deeply. This ability enables the model to more flexibly adapt to different complexity scenarios, thereby improving its adaptability and processing efficiency.

[0054] In the above high-resolution large field-of-view imaging system 100 based on multi-image stitching and fusion, the pathological section image semantic feature selection module 130 is used to respectively perform pathological section image semantic feature selection on each of the multiple semantic feature encoding maps of pathological section images to obtain multiple sparsified semantic feature encoding maps of pathological section images. Among them, Figure 3 is a block diagram of the pathological section image semantic feature selection module in the high-resolution large field-of-view imaging system based on multi-image stitching and fusion according to an embodiment of the present application, as Figure 3As shown, the pathological section image semantic feature selection module 130 includes: an image semantic feature local decomposition unit 131, which is used to perform feature decoupling and feature flattening on the pathological section image semantic feature encoded map features to obtain a set of pathological section image semantic local feature vectors; an image semantic feature local selection and aggregation unit 132, which is used to perform feature neighborhood activity selection and shape reshaping on the set of pathological section image semantic local feature vectors to obtain the sparse pathological section image semantic feature encoded map.

[0055] Furthermore, considering that the values of different pathological features in diagnosis are different. Some features may have a stronger direct association with the lesion, such as the specific morphological features of the diseased cells or the inflammatory reaction features around the diseased tissue. And, although each pathological section image semantic feature encoded map contains rich semantic information, there may be a large amount of redundancy in it. For example, the cell feature encodings in different regions may have similar parts, or the importance of some local features in the overall pathological judgment is relatively low. Therefore, in order to screen out the most representative and key features and reduce this redundant information, the present application introduces a method for selecting pathological section image semantic features to process each of the pathological section image semantic feature encoded maps in the multiple pathological section image semantic feature encoded maps respectively to obtain multiple sparse pathological section image semantic feature encoded maps. That is, this method dynamically measures the importance and related interaction of a feature in its surrounding area (semantic neighborhood) through the feature neighborhood activity, thereby highlighting these key features and making the subsequent processing more efficient and accurate.

[0056] In a specific embodiment of the present application, the image semantic feature local decomposition unit 131 is used to: perform feature decoupling on the pathological section image semantic feature encoded map along the channel dimension to obtain a set of pathological section image semantic feature encoded matrices; perform feature flattening on each of the pathological section image semantic feature encoded matrices in the set of pathological section image semantic feature encoded matrices to obtain the set of pathological section image semantic local feature vectors.

[0057] First, perform feature decoupling and feature flattening on the pathological section image semantic feature encoded map to obtain a set of pathological section image semantic local feature vectors. Specifically, feature decoupling means decomposing the pathological section image semantic feature encoded map to obtain a set of pathological section image semantic feature encoded matrices.

[0058] This process can be expressed by the formula:

[0059]

[0060] Where is the pathological section image semantic feature encoded map, is the feature decoupling operation, , , and are the 1st, 2nd, th, and th pathological section image semantic feature encoding matrices in the set of pathological section image semantic feature encoding matrices.

[0061] Then, flatten the features of each pathological section image semantic feature encoding matrix in the set of pathological section image semantic feature encoding matrices to obtain a set of pathological section image semantic local feature vectors, so as to ensure that the elements in each pathological section image semantic local feature vector represent different but related feature dimensions. It should be understood that the generation of independent components is ensured through feature decoupling and feature flattening, thus providing materials for subsequent feature selection.

[0062] This process can be expressed by the formula:

[0063]

[0064] where, is the feature flattening operation, , , and are respectively the 1st, 2nd, th, and th pathological section image semantic local feature vectors in the set of pathological section image semantic local feature vectors, , , and are the 1st, 2nd, th, and th pathological section image semantic feature encoding matrices in the set of pathological section image semantic feature encoding matrices.

[0065] In a specific embodiment of the present application, Figure 4 is the block diagram of the image semantic feature local selection and aggregation unit in the high-resolution large field of view imaging system based on multi-image stitching and fusion according to the embodiment of the present application, as shown in Figure 4As shown, the image semantic feature local selection and aggregation unit 132 includes: an image semantic local feature descending order sorting subunit 1321, configured to perform a descending order arrangement on the set of pathological slice image semantic local feature vectors to obtain a descending sequence of pathological slice image semantic local feature vectors; an image semantic local feature selection subunit 1322, configured to perform feature selection on the descending sequence of pathological slice image semantic local feature vectors to obtain a descending sequence of selected pathological slice image semantic local feature vectors; and an image semantic feature shape reshaping subunit 1323, configured to perform feature shape reshaping on the descending sequence of selected pathological slice image semantic local feature vectors to obtain the sparsified pathological slice image semantic feature encoded map.

[0066] Specifically, the image semantic local feature descending order sorting subunit 1321 is configured to: input each pathological slice image semantic local feature vector in the set of pathological slice image semantic local feature vectors into an importance measurement module to obtain a set of pathological slice image semantic local feature importance score values; and based on the set of pathological slice image semantic local feature importance score values, perform a descending order arrangement on the set of pathological slice image semantic local feature vectors to obtain the descending sequence of pathological slice image semantic local feature vectors.

[0067] That is, input each pathological slice image semantic local feature vector in the set of pathological slice image semantic local feature vectors into an importance measurement module to obtain a set of pathological slice image semantic local feature importance score values. The importance measurement module calculates the importance score of each pathological slice image semantic local feature according to a specific algorithm, and these scores reflect the degree of contribution of each pathological slice image semantic local feature importance score value to the overall task (such as classification or regression).

[0068] This process can be expressed by the formula:

[0069]

[0070] Where and are respectively the corresponding weight matrix and bias vector, is matrix multiplication, is the weight scoring vector, is the set of pathological slice image semantic local feature importance score values, , , and are respectively the 1st, 2nd, th and The importance score value of the semantic local features of a pathological section image is the th semantic local feature vector of the pathological section image in the set of semantic local feature vectors of the pathological section image.

[0071] Furthermore, based on the set of importance score values of the semantic local features of the pathological section image, the set of semantic local feature vectors of the pathological section image is sorted in descending order to obtain the descending sequence of the semantic local feature vectors of the pathological section image. That is, according to the obtained importance score values of the semantic local features of the pathological section image, all the importance score values of the semantic local features of the pathological section image are sorted from high to low in importance to form a descending sequence. This sorting mechanism can not only identify those semantic local features of the pathological section image that are most critical to the task objective, but also retain a certain proportion or number of high-score semantic local features of the pathological section image according to actual needs, so as to achieve the purpose of sparsification. The sorting itself does not change the content of the semantic local features of the pathological section image, but provides a clear view of the relative importance of the semantic local features of the pathological section image. This step not only simplifies the process of selecting the semantic local features of the pathological section image, but also enables subsequent computing resources to be concentrated on the most important semantic local features of the pathological section image.

[0072] This process can be expressed by the formula:

[0073]

[0074] where is to sort the set of semantic local feature vectors of the pathological section image in descending order based on , , , and are the 1st, 2nd, th and th semantic local feature vectors of the pathological section image in the descending sequence of the semantic local feature vectors of the pathological section image respectively, , , and are the 1st, 2nd, th and th semantic local feature vectors of the pathological section image in the set of semantic local feature vectors of the pathological section image respectively.

[0075] Specifically, the image semantic local feature selection sub-unit 1322 is configured to: calculate the feature neighborhood activity of each pathological slice image semantic local feature vector based on the neighborhood features of each pathological slice image semantic local feature vector in the descending sequence of the pathological slice image semantic local feature vectors, so as to obtain a sequence of pathological slice image semantic local feature neighborhood activities; and perform feature selection on the descending sequence of the pathological slice image semantic local feature vectors based on the sequence of the pathological slice image semantic local feature neighborhood activities, so as to obtain a descending sequence of the selected pathological slice image semantic local feature vectors.

[0076] On this basis, calculate the feature neighborhood activity of each pathological slice image semantic local feature vector based on the neighborhood features of each pathological slice image semantic local feature vector in the descending sequence of the pathological slice image semantic local feature vectors, so as to obtain a sequence of pathological slice image semantic local feature neighborhood activities. That is, considering the relationship between the local features of the pathological slice image semantic local feature vectors and their neighboring features (i.e., the feature neighborhood), an index of the pathological slice image semantic local feature neighborhood activity can be obtained by calculating the interaction degree between the local features of each pathological slice image semantic local feature vector and the surrounding features. The pathological slice image semantic local feature neighborhood activity reflects the association strength between the local features of each pathological slice image semantic local feature vector and the surrounding features, emphasizes the interaction between features, rather than just the importance of individual features. By introducing the pathological slice image semantic local feature neighborhood activity, the spatial distribution characteristics of the neighborhood features of the pathological slice image semantic local feature vectors can be further considered, deepening the understanding of the feature space of the neighborhood features of the pathological slice image semantic local feature vectors, and providing a more comprehensive perspective for selecting the most effective neighborhood features of the pathological slice image semantic local feature vectors.

[0077] This process can be expressed by the formula:

[0078]

[0079] where is the th pathological slice image semantic local feature vector in the descending sequence of the pathological slice image semantic local feature vectors, is the th eigenvalue in , is the number of eigenvalues in and are the and corresponding pathological slice image semantic weighted average eigenvalues respectively, is The neighborhood activity of the semantic local features of the corresponding pathological section image.

[0080] Furthermore, based on the sequence of the neighborhood activities of the semantic local features of the pathological section image, feature selection is performed on the descending sequence of the semantic local feature vectors of the pathological section image to obtain the descending sequence of the selected semantic local feature vectors of the pathological section image. That is, using the sequence of the neighborhood activities of the semantic local features of the pathological section image, the most valuable descending sequence of the semantic local feature vectors of the pathological section image is further screened out. At this stage, the neighborhood feature selection of the semantic local feature vectors of the pathological section image no longer depends solely on the importance scores of individual features, but comprehensively considers the relationship between the neighborhood features of the semantic local feature vectors of the pathological section image and their neighbors. The neighborhood features of the semantic local feature vectors of the pathological section image selected in this way can not only well represent the original data, but also better capture the patterns and rules in the data. The descending sequence of the selected semantic local feature vectors of the pathological section image is expected to be more compact and more expressive, which is beneficial to improving the model performance and interpretability. In a specific implementation, the neighborhood feature selection strategy of the semantic local feature vectors of the pathological section image can be flexibly adjusted according to specific applications, such as setting a fixed activity threshold, dynamically selecting a certain percentage of the highest activity features, or adopting more complex combination rules.

[0081] This process can be expressed by the formula:

[0082]

[0083]

[0084] Wherein, , , and are the 1st, 2nd, th, and th semantic local feature vectors of the pathological section image in the descending sequence of the semantic local feature vectors of the pathological section image respectively, , , and are the 1st, 2nd, th, and th selected semantic local feature vectors of the pathological section image in the descending sequence of the selected semantic local feature vectors of the pathological section image respectively, is the neighborhood activity of the semantic local features of the corresponding pathological section image, is for feature selection, is the preset threshold.

[0085] Finally, reshape the descending sequence of the selected pathological slice image semantic local feature vectors to obtain the sparse pathological slice image semantic feature encoding map. That is, reorganize the selected pathological slice image semantic local feature vectors back into the spatial structure of the pathological slice image semantic feature encoding map to form the sparse pathological slice image semantic feature encoding map. The reshaping process involves reorganizing the one-dimensional selected pathological slice image semantic local feature vectors into the multi-dimensional form of the sparse pathological slice image semantic feature encoding map, ensuring consistency with the original input format, which is crucial for maintaining the consistency and compatibility of the model architecture. The reshaped sparse pathological slice image semantic feature encoding map can also be used as an intermediate representation for downstream tasks, such as feature visualization or auxiliary diagnostic tools.

[0086] This process can be represented by the formula:

[0087]

[0088] where , , and are the 1st, 2nd, th, and th selected pathological slice image semantic local feature vectors in the descending sequence of the selected pathological slice image semantic local feature vectors respectively, is the shape reshaping operation, is the sparse pathological slice image semantic feature encoding map.

[0089] In the above high-resolution large field of view imaging system 100 based on multi-image stitching and fusion, the global pathological slice image generation module 140 is used to: after concatenating the multiple sparse pathological slice image semantic feature encoding maps into the global pathological slice image semantic feature stitching encoding map, input it into an image fusion engine based on an AIGC model to obtain the global pathological slice fusion image.

[0090] where the image fusion engine is an image fusion engine based on an AIGC model.

[0091] Subsequently, after cascading the semantic feature encoding maps of the multiple sparsified pathological slice images into a global pathological slice image semantic feature splicing encoding map, it is input into an image fusion engine based on an AIGC model to obtain a global pathological slice fusion image. It should be understood that each semantic feature encoding map of the sparsified pathological slice image contains the key semantic features after screening of the corresponding pathological slice, and these features are local and represent the important information of each slice itself. Therefore, in order to integrate the scattered semantic features into a unified representation and form a more comprehensive semantic feature representation, in this application, the multiple semantic feature encoding maps of the sparsified pathological slice images are cascaded into a global pathological slice image semantic feature splicing encoding map. This helps to grasp the semantic information of all slices as a whole and form a "panoramic" perspective covering the content of all slices. For example, for a relatively large pathological tissue sample, different slices may contain different parts of the diseased tissue or normal tissue parts adjacent to the diseased tissue. After cascading, it can better understand the distribution and spread of the lesion in the whole tissue. Immediately afterwards, it is input into the image fusion engine based on the AIGC model. By virtue of the excellent capabilities demonstrated by the AIGC model in image generation and fusion, based on the input feature information, by learning a large number of data patterns and semantic relationships, it can intelligently generate high-quality image content to obtain a global pathological slice fusion image. In this way, the fused image can be constructed based on the most valuable features, so that while maintaining high-resolution details, it can also better integrate the semantic information of each slice and provide a clearer and more targeted image for pathological diagnosis.

[0092] Here, when each semantic feature encoding map of the multiple pathological slice images represents the dynamic local image semantic association features of the pathological slice image after light compensation in a single-sample domain, when performing feature selection based on the feature neighborhood activity, the complexity of the feature neighborhood activity mechanism in the global sample domain will lead to insufficient long-distance selective splicing representation of the global pathological slice image semantic feature splicing encoding map obtained by cascading the multiple sparsified pathological slice image semantic feature encoding maps, thereby reducing the expression effect of the global pathological slice image semantic feature splicing encoding map and affecting the image quality of the global pathological slice fusion image obtained by inputting it into the image fusion engine based on the AIGC model.

[0093] Preferably, in one example, when inputting the global pathological slice image semantic feature splicing encoding map into an image fusion engine based on an AIGC model to obtain a global pathological slice fusion image, optimizing the global pathological slice image semantic feature splicing encoding map includes the following steps:

[0094] Unfold the global pathological section image semantic feature splicing encoding map to obtain a global pathological section image semantic feature splicing encoding feature set composed of each feature value of the global pathological section image semantic feature splicing encoding map;

[0095] In response to the feature value and the feature value in the global pathological section image semantic feature splicing encoding feature set, the distance value is less than or equal to the distance difference hyperparameter , that is:

[0096]

[0097] wherein, and respectively represent the feature value and the feature value in the global pathological section image semantic feature splicing encoding feature set;

[0098] Multiply the feature value by a predetermined weight and then add it to the feature value to obtain an optimized feature value , represents the predetermined weight;

[0099] In response to the absolute value of the difference between the feature value and the feature value in the global pathological section image semantic feature splicing encoding feature set being greater than the distance difference hyperparameter , that is :

[0100] Subtract the feature value from the square of each feature value in the global pathological section image semantic feature splicing encoding feature set, and then sum all the feature values in the global pathological section image semantic feature splicing encoding feature set to obtain a medical image shallow-deep joint perception encoding heterogeneous projection value;

[0101]

[0102] wherein, the scale of the global pathological section image semantic feature splicing encoding map is equal to the width of the feature matrix of the global pathological section image semantic feature splicing encoding map multiplied by the height and then multiplied by the number of channels of the global pathological section image semantic feature splicing encoding map, represents the medical image shallow-deep joint perception encoding heterogeneous projection value;

[0103] Each feature value of the medical image shallow-deep joint perception coding heterogeneous projection value and the global pathological slice image semantic feature splicing coding map is After adding the product of the scale of the global pathological slice image semantic feature splicing coding map, it is divided by the square of the scale of the global pathological slice image semantic feature splicing coding map to obtain each feature value of the global pathological slice image semantic feature splicing coding map. The corresponding medical image shallow-deep joint perception encoding multi-dimensional latent primitive values:

[0104]

[0105] in, Represents each eigenvalue The corresponding medical image shallow-deep joint perception encodes multi-dimensional latent primitive values;

[0106] The multi-dimensional latent primitive value of the medical image shallow-deep joint perception coding is multiplied by the first After the eigenvalue, calculate the product with the first The weighted sum between the eigenvalues ​​is the optimized Eigenvalue ,in, represents the first weighted hyperparameter, represents the second weighted hyperparameter;

[0107] The optimization of combining the global pathological slice image semantic features and splicing encoding feature sets is Eigenvalue To obtain an optimized global pathology slice image semantic feature splicing coding map, wherein the minimum eigenvalue of the global pathology slice image semantic feature splicing coding feature set remains unchanged.

[0108] Finally, the optimized global pathology slice image semantic feature splicing coding map is input into the image fusion engine based on the AIGC model to obtain the global pathology slice fusion image.

[0109] That is, because the feature set of the global pathology slice image semantic feature splicing coding map obtained by cascading the multiple sparse pathology slice image semantic feature coding maps will produce a global feature collaboration efficiency attenuation phenomenon under the preset feature value arrangement rule when the non-local feature span breaks through the distribution constraint of adjacent elements. To solve this problem, a high-dimensional representation method of heterogeneous feature inner product projection of the global pathology slice image semantic feature splicing coding map is adopted to explicitly model the association network topology of the entire domain of feature values ​​of the global pathology slice image semantic feature splicing coding map.

[0110] Thus, by constructing a dynamic generation mechanism for multi-scale latent primitives of the global pathological slice image semantic feature stitching coding map, establishing a coupling conduction model between the feature dimensions of the global pathological slice image semantic feature stitching coding map, realizing the feature coding reconstruction of the global pathological slice image semantic feature stitching coding map under the condition of maintaining spatio-temporal continuity, finally improving the multi-level feature decoupling performance, enhancing the selective stitching expression effect of the global pathological slice image semantic feature stitching coding map, and improving the image quality of the global pathological slice fusion image obtained by inputting the image fusion engine based on the AIGC model.

[0111] Based on this, it first obtains multiple high-resolution local pathological slice images and introduces a light compensation technique to improve the image quality and ensure that details are clearly visible; then, it uses a dynamic convolution labeling model to perform semantic feature encoding on the light-compensated pathological slice images, capturing fine local features such as cell textures and edges, and at the same time introducing a selection method based on the activity of the feature neighborhood to highlight important pathological features; finally, it cascades the semantic features of the thinned pathological slice images and inputs them into the image fusion engine based on the AIGC model to generate a global pathological slice fusion image that covers a larger tissue range and retains high-resolution details. In this way, not only the dual advantages of high resolution and large field of view are realized, but also clearer and more targeted image support is provided for pathological diagnosis, significantly improving the accuracy and efficiency of diagnosis.

[0112] In one embodiment of the present application, Figure 5 is a flowchart of a high-resolution large-field imaging method based on multi-image stitching and fusion according to an embodiment of the present application, as Figure 5As shown, the high-resolution large field-of-view imaging method based on multi-image stitching and fusion according to an embodiment of the present application includes: S210, obtaining a plurality of pathological section images; S220, performing illumination compensation and feature extraction on each of the plurality of pathological section images to obtain a plurality of semantic feature encoding maps of pathological section images; S230, respectively performing semantic feature selection of pathological section images on each of the plurality of semantic feature encoding maps of pathological section images to obtain a plurality of sparsified semantic feature encoding maps of pathological section images, wherein S230, respectively performing semantic feature selection of pathological section images on each of the plurality of semantic feature encoding maps of pathological section images to obtain a plurality of sparsified semantic feature encoding maps of pathological section images includes: S231, performing feature decoupling and feature flattening on the features of the semantic feature encoding map of the pathological section image to obtain a set of semantic local feature vectors of the pathological section image; S232, performing feature neighborhood activity selection and shape reshaping on the set of semantic local feature vectors of the pathological section image to obtain the sparsified semantic feature encoding map of the pathological section image; S240, cascading the plurality of sparsified semantic feature encoding maps of pathological section images into a global semantic feature stitching encoding map of pathological section images, and based on the global semantic feature stitching encoding map of pathological section images, obtaining a global pathological section fusion image.

[0113] Those skilled in the art can understand that the specific operations of each step in the above high-resolution large field-of-view imaging method based on multi-image stitching and fusion have been described in detail above with reference to Figures 1 to 4 the description of the high-resolution large field-of-view imaging system 100 based on multi-image stitching and fusion, and therefore, its repeated description will be omitted.

[0114] The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features with similar functions disclosed in the present application.

[0115] Moreover, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the foregoing discussion, these should not be construed as limitations on the scope of the present application. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features that are described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0116] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the claimed subject matter is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementation. With regard to the apparatus in the foregoing embodiments, the specific manner in which each module performs an operation has been described in detail in the embodiments related to the method, and will not be elaborated herein.

Claims

1. A high-resolution and large-field-of-view imaging system based on multi-image stitching and fusion, characterized in that: include: A pathological slice image acquisition module, used for acquiring a plurality of pathological slice images; A pathological slice image compensation and extraction module, used for performing illumination compensation and feature extraction on each of the plurality of pathological slice images to obtain a plurality of pathological slice image semantic feature coding maps; A pathological slice image semantic feature selection module is used to perform pathological slice image semantic feature selection on each pathological slice image semantic feature coding map in the multiple pathological slice image semantic feature coding maps to obtain multiple sparse pathological slice image semantic feature coding maps, wherein the pathological slice image semantic feature selection module includes: an image semantic feature local decomposition unit, which is used to perform feature decoupling and feature flattening on the features of the pathological slice image semantic feature coding map to obtain a set of pathological slice image semantic local feature vectors; an image semantic feature local selection aggregation unit, which is used to perform feature neighborhood activity selection and shape reshaping on the set of pathological slice image semantic local feature vectors to obtain the sparse pathological slice image semantic feature coding map; The global pathology slice image generation module is used to cascade the multiple sparse pathology slice image semantic feature coding maps into a global pathology slice image semantic feature splicing coding map, and input the global pathology slice image semantic feature splicing coding map into an image fusion engine based on the AIGC model to obtain a global pathology slice fusion image.

2. The high-resolution and large-field-of-view imaging system based on multi-image stitching and fusion according to claim 1, characterized in that: The pathological slice image compensation and extraction module comprises: A pathological slice compensation unit, used for performing illumination compensation on each of the plurality of pathological slice images to obtain a plurality of illumination-compensated pathological slice images; The pathological slice feature extraction unit is used to input each of the illumination compensated pathological slice images in the multiple illumination compensated pathological slice images into the pathological slice image feature extractor to obtain the semantic feature coding images of the multiple pathological slice images.

3. The high-resolution and large-field-of-view imaging system based on multi-image stitching and fusion according to claim 2, characterized in that: The pathological slice image feature extractor is a pathological slice image feature extractor based on a dynamic convolutional labeling model.

4. The high-resolution and large-field-of-view imaging system based on multi-image stitching and fusion according to claim 3, characterized in that: The image semantic feature local decomposition unit is used to: Decoupling the pathological slice image semantic feature coding map along the channel dimension to obtain a set of pathological slice image semantic feature coding matrices; Feature flattening is performed on each pathological slice image semantic feature coding matrix in the set of pathological slice image semantic feature coding matrices to obtain a set of pathological slice image semantic local feature vectors.

5. The high-resolution and large-field-of-view imaging system based on multi-image stitching and fusion according to claim 4, characterized in that: The image semantic feature local selection aggregation unit includes: An image semantic local feature descending order sorting subunit is used to sort the set of pathological slice image semantic local feature vectors in descending order to obtain a descending sequence of pathological slice image semantic local feature vectors; An image semantic local feature selection subunit, used for performing feature selection on the descending sequence of the semantic local feature vectors of the pathological slice image to obtain a descending sequence of the semantic local feature vectors of the pathological slice image after selection; The image semantic feature shape reshaping subunit is used to perform feature shape reshaping on the descending sequence of the semantic local feature vectors of the selected pathological slice image to obtain the sparse pathological slice image semantic feature coding map.

6. The high-resolution and large-field-of-view imaging system based on multi-image stitching and fusion according to claim 5, characterized in that: The image semantic local feature descending order sorting subunit is used to: Inputting each pathological slice image semantic local feature vector in the set of pathological slice image semantic local feature vectors into an importance measurement module to obtain a set of pathological slice image semantic local feature importance score values; Based on the set of importance score values ​​of the semantic local features of the pathological slice images, the set of semantic local feature vectors of the pathological slice images is arranged in descending order to obtain a descending sequence of the semantic local feature vectors of the pathological slice images.

7. The high-resolution and large-field-of-view imaging system based on multi-image stitching and fusion according to claim 6, characterized in that: The image semantic local feature selection subunit is used to: Based on the neighborhood features of each pathological slice image semantic local feature vector in the descending sequence of the pathological slice image semantic local feature vectors, calculating the feature neighborhood activity of each pathological slice image semantic local feature vector to obtain a sequence of pathological slice image semantic local feature neighborhood activity; Based on the sequence of neighborhood activities of the semantic local features of the pathological slice images, feature selection is performed on the descending sequence of the semantic local feature vectors of the pathological slice images to obtain the descending sequence of the semantic local feature vectors of the selected pathological slice images.

Images