Data processing method and apparatus for image registration, and electronic device

By employing supervised machine learning methods and histogram equalization, combined with a neural network optimized by a smoothing layer, the accuracy problem of multimodal image registration was solved, achieving efficient registration of both monomodal and multimodal images, reducing costs and improving accuracy.

WO2024239287A1PCT designated stage expired Publication Date: 2024-11-28SHENZHEN HUADA GENE INST +1

Patent Information

Application Number
PCT/CN2023/096114
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing multimodal elastic registration schemes are difficult to implement, or direct implementation may result in poor image registration accuracy, especially since the modal differences between tissue images and expression maps are huge, making it difficult to find a suitable loss function to measure the gradient corresponding to the model's results.

Method used

A supervised machine learning method is adopted. After histogram equalization of the image to be registered, the deformation field information is obtained by using a pre-trained machine learning model. Based on the deformation field information, the coordinate information of the image after registration is determined. The smoothing layer module of the neural network in the model is optimized to improve the registration accuracy.

Benefits of technology

It improves the elastic registration accuracy of multimodal images, solves the elastic registration problem of homomodal and multimodal images, reduces the algorithm development cost, and enhances the amount of training data through automated data synthesis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023096114_28112024_PF_FP_ABST
    Figure CN2023096114_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to a data processing method and apparatus for image registration, and an electronic device, and relates to the technical field of data processing. The method comprises: firstly acquiring a first image and a second image to be registered; then registering the first image and the second image; then inputting the registered first image and second image into a machine learning model for calculation to obtain deformation field information, wherein the machine learning model is obtained by performing model training in advance according to image registration data of a sample image, and the image registration data comprises: the sample image and a deformed and corrected sample image; and determining corresponding coordinate information after image registration according to the deformation field information. The technical solutions of the present application solve the problem of elastic registration of multi-modal images, are easy to implement, and can improve the accuracy of image registration.
Need to check novelty before this filing date? Find Prior Art

Description

Data processing method, device and electronic equipment for image registration Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and electronic equipment for image registration. Background Art

[0002] Image registration involves aligning the position and shape of one or more images. It is a crucial task in image-related fields, with widespread applications in biology, medicine, and other engineering fields. Elastic registration of multimodal images has long been a challenging problem in image registration. This problem, elastic registration of expression maps based on tissue photographs, falls into this category.

[0003] Currently, existing multimodal elastic registration solutions use unsupervised deep learning methods for pairwise registration. This approach requires registering images that belong to the same modality, or two modalities with minimal differences, to calculate the loss and corresponding gradients for the output results in each training round, thereby starting to optimize the model parameters.

[0004] However, the inherent modalities of tissue images and expression maps differ significantly, making it difficult to find a suitable loss function to directly measure the gradient of the model's results. Consequently, existing technologies often struggle with implementation or suffer from poor image registration accuracy.

[0005] Summary of the Invention

[0006] In view of this, the present application provides a data processing method, device and electronic device for image registration, the main purpose of which is to improve the technical problems that the current existing multimodal elastic registration scheme is difficult to implement or directly implements it, resulting in poor accuracy of image registration.

[0007] In a first aspect, the present application provides a data processing method for image registration, comprising:

[0008] Acquire a first image and a second image to be registered;

[0009] performing registration based on the first image and the second image;

[0010] Inputting the registered first image and the second image into a machine learning model for calculation to obtain deformation field information, wherein the machine learning model is pre-trained based on registration data of a sample image, and the registration data includes: the sample image and an image after deformation correction of the sample image;

[0011] According to the deformation field information, corresponding coordinate information after image registration is determined.

[0012] In a second aspect, the present application provides a data processing device for image registration, comprising:

[0013] An acquisition module, configured to acquire a first image and a second image to be registered;

[0014] a processing module, configured to perform registration based on the first image and the second image;

[0015] The acquisition module is further configured to input the registered first image and the second image into a machine learning model for calculation to obtain deformation field information, wherein the machine learning model is pre-trained based on registration data of a sample image, and the registration data includes: the sample image and the image after deformation correction of the sample image;

[0016] The determination module is configured to determine corresponding coordinate information after image registration based on the deformation field information.

[0017] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method for image registration described in the first aspect.

[0018] In a fourth aspect, the present application provides an electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the data processing method for image registration described in the first aspect when executing the computer program.

[0019] By means of the above-mentioned technical solution, the present application provides a data processing method, device and electronic device for image registration. Compared with the currently existing multimodal elastic registration solution, the present application provides a supervised machine learning method, which pre-trains the model according to the registration data of the sample image to obtain a machine learning model. The registration data includes: the sample image and the image after deformation correction of the sample image. When in use, the first image and the second image to be registered are first registered; then the registered first image and the second image are input into the machine learning model for calculation to obtain deformation field information, and then the corresponding coordinate information after image registration is determined based on the deformation field information. By applying the technical solution of the present application, the problem of elastic registration of multimodal images is solved, it is easy to implement, and the accuracy of image registration can be improved.

[0020] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] FIG1 is a schematic diagram showing a flow chart of a data processing method for image registration provided in an embodiment of the present application;

[0024] FIG2 is a schematic flow chart of another data processing method for image registration provided in an embodiment of the present application;

[0025] FIG3 shows an example diagram of an expression level sample diagram provided in an embodiment of the present application;

[0026] FIG4 shows an example diagram of histogram equalization processing of an expression quantity sample graph provided in an embodiment of the present application;

[0027] FIG5 shows an example diagram of histogram equalization processing of a tissue photograph sample image provided by an embodiment of the present application;

[0028] FIG6 is a schematic diagram showing a storage format of a transformation relationship of rigid registration provided in an embodiment of the present application;

[0029] FIG7 is a schematic diagram showing a storage format of a transformation relationship of elastic registration provided by an embodiment of the present application;

[0030] FIG8 is a schematic diagram showing a sample graph of expression values ​​provided in an embodiment of the present application after rigid transformation and elastic transformation;

[0031] FIG9 is a schematic diagram showing the effect of elastic deformation image synthesis provided by an embodiment of the present application;

[0032] FIG10 is a schematic diagram showing an example of a model structure provided in an embodiment of the present application;

[0033] FIG11 is a schematic diagram showing the effect of the optimized model structure provided in an embodiment of the present application;

[0034] FIG12 is a schematic diagram showing an overall example process provided by an embodiment of the present application;

[0035] FIG13 shows a schematic structural diagram of a data processing device for image registration provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned objectives, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0037] In order to improve the technical problems that the existing multimodal elastic registration scheme is difficult to implement or directly implements, resulting in poor accuracy of image registration, this embodiment provides a data processing method for image registration, as shown in Figure 1, the method includes:

[0038] Step 101: Acquire a first image and a second image to be registered.

[0039] In practical applications, image registration can be categorized into rigid and elastic registration problems based on the transformation required for alignment. It can also be categorized into single-modality and multi-modality registration problems based on the modal differences of the input images. In this embodiment, the first and second images to be registered can be of the same or different modalities.

[0040] Step 102: perform registration based on the first image and the second image.

[0041] Specifically, rigid registration may be performed based on the first image and the second image.

[0042] In some examples, step 102 may specifically include: performing histogram equalization processing on the first image and the second image respectively; and registering the first image and the second image after the histogram equalization processing.

[0043] Histogram equalization is used to enhance the contrast of an image. It mainly transforms the histogram distribution of an image into a nearly uniform distribution, thereby increasing the difference between pixel values ​​and creating a contrast enhancement effect. This allows for better image registration in the future.

[0044] In practical applications, the first image and the second image may not correspond in position or size. Therefore, to ensure the computational efficiency and accuracy of the subsequent machine learning model, this embodiment requires registration of the first and second images after histogram equalization. For example, the first image is used as a reference image, and the second image is registered, such as by flipping, translating, and rotating it. Optionally, this registration may include modification of position and / or size, but does not include twisting or changing the shape.

[0045] Step 103: Input the registered first image and second image into a machine learning model for calculation to obtain deformation field information.

[0046] The machine learning model may be pre-trained based on the registration data of the sample image, and the registration data of the sample image may include: the sample image and the image after deformation correction of the sample image. The deformation field information may include elastic transformation (deformation) information of the image registration.

[0047] Step 104: Determine the corresponding coordinate information after image registration based on the deformation field information.

[0048] In the process of performing image registration on the first image and the second image in this embodiment, the corresponding coordinate information after image registration obtained in step 104 can be used to obtain image information after deformation field correction.

[0049] At present, the problem of multimodal elastic registration, especially the elastic registration problem with large modal differences, has always been a difficult problem in registration. Traditional feature point extraction algorithms have difficulty finding corresponding feature points on multimodal images, and it is difficult to find all feature points, so it is difficult to play a role in the problem of multimodal elastic registration. Compared with the existing multimodal elastic registration scheme (using an unsupervised deep learning scheme), this embodiment provides a supervised machine learning method, which pre-trains the model based on the registration data of the sample image to obtain a machine learning model. When used, the first image and the second image to be registered are first subjected to histogram equalization processing to enhance the contrast of the image and facilitate model prediction; then, the first image and the second image after the histogram equalization processing are registered; then, the registered first image and the second image are input into the machine learning model for calculation to obtain deformation field information, and then, based on the deformation field information, the corresponding coordinate information after image registration is determined. By applying the technical solution of this embodiment, the elastic registration problem of multimodal images can be solved, and in addition, the elastic registration problem of homomodal images can also be solved. In general, the technical solution of this embodiment can be used for both homomodal and multimodal elastic registration, such as elastic registration of medical images and elastic registration of biological experiment images, to improve the accuracy of image registration.

[0050] Furthermore, as a refinement and extension of the above embodiment, in order to fully illustrate the specific implementation process of the method of this embodiment, taking the example of the first image in the method of FIG1 being a tissue photograph and the second image being an expression level map based on the tissue photograph (e.g., a case of registering expression level data of a monkey brain with a tissue photograph as an example), this embodiment provides a specific method as shown in FIG2, which includes:

[0051] Step 201: Pre-train to obtain a machine learning model.

[0052] The machine learning model may be pre-trained based on registration data of sample images, where the registration data includes the sample image and a deformation-corrected image of the sample image. The sample image may specifically include a tissue photograph sample image and an expression quantity sample image based on the tissue photograph sample image. The deformation-corrected image of the sample image is an image after the expression quantity sample image has been deformed.

[0053] Optionally, the training process of the machine learning model includes: first, performing histogram equalization processing on the tissue photography sample image and the expression sample image respectively; then, using the tissue photography sample image after the histogram equalization processing as the reference image, performing rigid alignment and elastic alignment on the expression sample image after the histogram equalization processing, to obtain a data file containing the alignment relationship and the aligned expression sample image; then, using the tissue photography sample image after the histogram equalization processing and the expression sample image as feature data, and using the data file and the aligned expression sample image as label data, to train a machine learning model.

[0054] In this embodiment, in order to accurately train and obtain an effective machine learning model, data preparation is required, that is, preparing the data required for model training. First, an expression sample graph is generated based on the abundance statistics of the expression sample data. The expression sample data is a table that records the spatiotemporal transcriptome expression sample data, each row of which represents the situation of a certain gene captured by a sequencing site; each column represents the attributes such as the gene name, site coordinates, and capture amount in the captured gene situation. As shown in Table 1 below, the first 8 rows of an expression data example are shown.

[0055] Table 1

[0056] The expression sample map is generated by abundance statistics of the expression sample data. The statistical process includes the following steps: first, the minimum square area covered by the entire tissue is divided into several small, adjacent square areas based on the coordinates of the expression sample data. Each small square area is also called a "bin set." Then, the total captured amount of genes represented by the expression data is calculated within each bin set. Finally, an image is drawn based on the gene statistics of these bin sets. The pixel position of this image corresponds to the relative position of each bin set within the minimum square area covered by the tissue (for example, the pixel corresponding to the bin set in the first row and first column on the map is also located in the first row and first column on the map), and the pixel value is the total captured amount of genes in that bin set. An example of an expression sample map is shown in Figure 3.

[0057] After obtaining the tissue photography sample image and the expression quantity sample image, histogram equalization processing is performed. Among them, histogram equalization is to enhance the contrast of the image and facilitate manual alignment and model training. It mainly converts the histogram distribution of the image into an approximately uniform distribution, so that the difference between the pixel values ​​of the image is strengthened, showing a contrast enhancement effect. However, since the color information of the tissue photography sample image does not play a decisive role in the alignment of the tissue photography sample image and the expression quantity sample image, it is converted into a grayscale image for the purpose of saving storage and memory space. Therefore, it is optional to perform histogram equalization processing on the tissue photography sample image, which can specifically include: processing the tissue photography sample image into a grayscale image and then performing histogram equalization processing.

[0058] For example, as shown in Figure 4, the expression level sample image is histogram equalized. As shown in Figure 5, the real tissue photograph sample image is converted into a grayscale image and then histogram equalized.

[0059] After obtaining the histogram-equalized tissue image and expression level sample images, for images used for model training, if there is any mismatch in position or size between the two images, use the ImageJ plugin TrakEM2 to first perform rigid registration using the tissue image as the reference image. Alternatively, other methods, such as automated algorithms, can also be used for rigid registration. Then, elastic registration is performed. A data file containing the registration relationship (i.e., the TrakEM2 project file) and the registered expression level sample images are output. The TrakEM2 project file stores the transformation relationships recorded by the plugin during manual rigid and elastic registration.

[0060] The storage form of the transformation relationship of the rigid registration is a rigid transformation matrix, which describes the flipping, rotation angle, and horizontal and vertical translation of the image being registered, as shown in Figure 6. The storage form of the transformation relationship of the elastic registration is the coordinates of the manually specified control points before and after the transformation. Based on these coordinates, the deformation field of the entire image can be calculated. This coordinate example is shown in Figure 7. The expression sample image after registration refers to the expression sample image after the Blockface image is used as the reference image, and rigid transformation (flip, translation, and rotation) and elastic transformation (deformation) are performed. As shown in Figure 8, the results of the expression sample image before registration, after rigid transformation, and after elastic transformation are shown.

[0061] During the model training process, the parameter fitting of the multi-layer neural network is achieved based on the data drive of thousands of pairs of registered images and reference images; however, for the elastic registration task of the expression map based on the tissue photography, it is very difficult for R&D personnel to obtain large-scale data due to the limitations of its experimental cost and sequencing time. Therefore, there is a problem of high cost. For this reason, this embodiment can automatically synthesize elastic deformation images based on the collected experimental data, thereby reducing the cost of algorithm development. Correspondingly, optionally, the above-mentioned histogram equalization processing of the expression sample map includes: first, generating several pairs of control points based on the expression sample map; using the control points to calculate the sample deformation field, and applying the sample deformation field to the expression sample map to generate an elastically deformed synthesized expression sample map; and then performing histogram equalization processing on the elastically deformed synthesized expression sample map.

[0062] For example, for images used for model training, the computer first randomly generates several pairs of control points, including their original positions and their new positions. At the same time, the control points manually anchored during manual registration are added (their new positions are no longer randomly generated but remain consistent with their original positions). Using the Moving Least Squares Deformation theory, a deformation field is calculated and applied to the expression sample image before registration, forming a randomly deformed expression sample map.

[0063] Figure 9 shows the original image of the expression map and the composite image of the elastic deformation. The left image is the original image, and the right image is the composite image. As can be seen, the composite image further adds elastic deformation to the original image while retaining its rigid properties, such as position and size. This composite method amplifies the training data by a factor of 10 for training machine learning models.

[0064] Current technical models lack any mechanism to constrain the smoothness of the given registration field, which contradicts the common understanding that deformations between tissue images and expression maps are relatively smooth. This can lead to poor accuracy during image registration. To this end, this embodiment optimizes the smoothness of the output registration field, improving the accuracy of the registration results. The machine learning model employed in this embodiment includes a smoothing layer module after the activation function layer at a preset position in the neural network. This smoothing layer module is a sliding convolution kernel of a preset size. Alternatively, the machine learning model may be trained using the histogram-equalized tissue image sample and expression sample image as feature data, and the data file and the registered expression sample image as label data. Specifically, this may include: creating a smoothing layer module, which may be a sliding convolution kernel of a preset size (e.g., a kernel size of 15*15); adding the smoothing layer module to a preset image registration model (e.g., a VoxelMorph model) after the activation function layer at a preset position; then, training the preset image registration model based on the feature data and label data, and using the trained preset image registration model as the machine learning model.

[0065] For example, building a BlurConv module (smoothing layer module) essentially involves applying a two-dimensional sliding convolution kernel with a size of 15*15 (other sizes can be used based on actual needs, such as any kernel size larger than 3*3). The kernel calculates the median of the elements at different positions covered by the kernel when it slides to different positions in the input feature map (average kernel, Gaussian kernel, maximum kernel, etc. can also be used based on actual needs). The BlurConv module is added after the last activation function layer and the second-to-last activation function layer of the VoxelMorph model (the preset image registration model). The model structure is shown in Figure 10.

[0066] The input is a pair of tissue images and expression maps. The tissue images are all real data, while the expression maps are either real data or randomly synthesized deformed images based on real data. The manually deformed expression maps and data files corresponding to the input data are used as labels. The loss function combines the difference between the labeled image and the predicted image, as well as the gradients of the deformation field in the x and y directions. At the beginning of training, the loss function of the training set decreases continuously, while the loss function of the validation set converges and slowly increases after a certain number of steps. Training ends when the validation set loss function begins to increase, and the model is saved.

[0067] As shown in Figure 10, the structure of the machine learning model used in the present invention is shown. The input of the model is a tissue photograph sample image (alignment reference image) and an expression sample image, and the intermediate output (marked with the 'Local field' position) is the deformation field predicted by the model. The model structure adjacent to the input part is based on the transformation of the UNet and SpatialTransformer models. The specific changes are shown by the shaded arrows in Figure 10: After the information passes through the decoder and finally becomes a feature map of the same size as the input image, after each convolution, it undergoes smoothing processing of the BlurConv layer.

[0068] In order to illustrate the effect of the structural optimization design of this machine learning model, the model effects without optimization and after optimization were tested on different samples, as shown in Figure 11. It can be seen that the registration field output by the optimized model in this embodiment is smoother and more consistent with the data distribution of the registration field obtained by manual registration. In order to reveal the principle of how the smoothing layer in this embodiment improves the accuracy of the model output, the relevant research process is disclosed here:

[0069] The effects of four schemes were tested: (1) using the model in the existing technical solution; (2) not changing the existing model but only adding image smoothing processing after it; (3) adding a smoothing layer (BlurConv) after the last activation layer and then training the model; and (4) using a smoothing layer after both the penultimate and last activation layers and then training the model. The accuracy of the registration field derived by the model in the tissue region was calculated using the manually obtained registration field. On four of the test data, the mean squared error (MSE) and mean absolute error (MAE) were calculated. It can be seen that although directly applying smoothing processing to the model output data can optimize the accuracy of the model output, adding a smoothing layer to the model and then training the model makes the output results more accurate. Adding a smoothing layer after both the last and penultimate activation function layers will further improve the accuracy of the output compared to adding a smoothing layer only after the last activation function layer.

[0070] The above research results explain how innovation can significantly improve accuracy:

[0071] A. The models used in existing technologies primarily learn the semantic information of images at a low-resolution level. In this task, this can be understood as the basic units used for registration tasks in expression maps and tissue photographs (such as the line shapes of brain white matter). At a high-resolution level, they learn how to register each part of the image. While the low-resolution layer extracts the characteristic shapes of the image, it does not connect neighboring regions in Euclidean space. Therefore, adding a smoothing layer to the model can fuse information from each 15*15 neighboring region, thus establishing connections between neighboring regions. This allows regions without obvious features (low brightness) to be influenced by neighboring regions with obvious features, resulting in a smoother registration field output by the model.

[0072] B. In the last four layers of the model, the feature maps have been restored to the same size as the input image, thus belonging to the aforementioned high-resolution layer learning. Within these four layers, the convolutional layers continuously fuse features from different channels, ultimately outputting a result with only two channels (representing the horizontal and vertical registration fields, respectively). Adding smoothing layers not only to the output layer of these four layers, but also further away from the output layer, allows the encoder and decoder at lower levels to be influenced by the fusion of information from nearby regions during training, optimizing the degree of fit and ultimately showing improved accuracy.

[0073] Step 202: When image registration is required, a tissue photograph and an expression quantity map based on the tissue photograph are obtained.

[0074] The tissue photograph and the expression level map based on the tissue photograph may be two images to be registered. Optionally, the process of acquiring the expression level map includes: generating the expression level map based on abundance statistics of the expression level data.

[0075] Exemplarily, the expression map is generated based on the abundance statistics of the expression data, including: dividing the minimum square range covered by the target tissue into several adjacent square areas according to the coordinates of the expression data; counting the total captured amount of captured genes in each square area; and drawing an expression map based on the total captured amount of genes counted in each square area, wherein the pixel position in the expression map corresponds to the relative position of each square area in the minimum square range covered by the target tissue, and the pixel value of the pixel position is the total captured amount of genes counted in each square area.

[0076] For example, first, based on the coordinates of the expression data, the minimum square range covered by the entire tissue is divided into several small, adjacent square ranges, each of which is called a "Bin set"; then, the total captured genes represented by the above expression data are counted in each Binset range; finally, an image is drawn based on the gene statistics of these Binsets, and the pixel position of the image corresponds to the relative position of each Binset in the minimum square area covered by the above tissue (for example, the pixel corresponding to the Binset in the first row and first column is also in the first row and first column on the image), and the pixel value is the total captured gene of the Binset.

[0077] Step 203: Perform histogram equalization processing on the tissue photograph and the expression level graph respectively.

[0078] In order to save storage and memory space and correspond to the training process of the machine learning model, the tissue photograph is subjected to histogram equalization processing, which may specifically include: processing the tissue photograph into a grayscale image and then performing histogram equalization processing.

[0079] Step 204: Using the tissue photograph after the histogram equalization process as a reference image, register the expression level map after the histogram equalization process.

[0080] Step 205: Input the registered expression image and the tissue photograph image after histogram equalization processing into a preset machine learning model for calculation to obtain deformation field information corresponding to the expression image.

[0081] Step 206: Perform linear interpolation calculation on the deformation field information corresponding to the expression amount graph to obtain updated deformation field information.

[0082] Step 207: The horizontal and vertical coordinates in the spatiotemporal group expression data corresponding to the expression map are brought into the updated deformation field information to calculate the coordinate information of the expression map after registration.

[0083] For example, a. First, generate an expression map based on the abundance statistics of the expression data. b. Process the tissue photograph into a grayscale image, then perform histogram equalization, and then perform histogram equalization on the expression map; use the tissue photograph as a reference and rigidly align the expression map. c. Input the rigidly aligned image pair into the machine learning model (obtained in step 201), and perform linear interpolation on the intermediate result output by the model: the deformation field, to obtain a more precise deformation prediction result. d. Substitute the horizontal and vertical coordinates in the spatiotemporal group expression data corresponding to the expression map into the deformation field output by c, and calculate its new coordinates after alignment.

[0084] Step 208: Using the coordinate information of the expression map after registration, obtain the expression map after deformation correction.

[0085] For example, Figure 12 illustrates the overall technical process of the embodiment of Figure 2. The left half of Figure 12 illustrates the preparation of training data and the training of the deep learning model. The right half illustrates the preprocessing of data in a real-world problem, the use of the model, and the post-processing of the output results when using the model for inference. Finally, the output is an expression map after deformation field correction.

[0086] Compared to existing technologies, this embodiment's method, based on supervised machine learning and manually labeling the model, improves the feasibility of multimodal elastic registration. Based on collected experimental data, this embodiment's method automatically synthesizes elastically deformed images, reducing algorithm development costs. This method also optimizes the smoothness of the output registration field, improving the accuracy of the registration results.

[0087] Furthermore, as a specific implementation of the method shown in Figures 1 and 2, this embodiment provides a data processing device for image registration. As shown in Figure 13, the device includes: an acquisition module 31, a processing module 32, and a determination module 33.

[0088] An acquisition module 31 is configured to acquire a first image and a second image to be registered;

[0089] a processing module 32 configured to perform registration based on the first image and the second image;

[0090] The acquisition module 31 is further configured to input the registered first image and the second image into a machine learning model for calculation to obtain deformation field information, wherein the machine learning model is pre-trained based on registration data of a sample image, and the registration data includes: the sample image and the image after deformation correction of the sample image;

[0091] The determination module 33 is configured to determine corresponding coordinate information after image registration based on the deformation field information.

[0092] In a specific application scenario, optionally, the first image is a tissue photograph image, and the second image is an expression quantity image based on the tissue photograph image; the sample image includes: a tissue photograph sample image and an expression quantity sample image based on the tissue photograph sample image, and the image after deformation correction of the sample image is the image after deformation correction of the expression quantity sample image.

[0093] In a specific application scenario, the processing module 32 is further configured to perform histogram equalization processing on the tissue photograph sample image and the expression sample image respectively; using the tissue photograph sample image after the histogram equalization processing as a reference image, the expression sample image after the histogram equalization processing is aligned and elastically aligned to obtain a data file containing the alignment relationship and the aligned expression sample image; using the tissue photograph sample image after the histogram equalization processing and the expression sample image as feature data, and using the data file and the aligned expression sample image as label data to train and obtain the machine learning model.

[0094] In a specific application scenario, the processing module 32 is further configured to generate several pairs of control points based on the expression sample graph; use the control points to calculate the sample deformation field, and apply the sample deformation field to the expression sample graph to generate an elastic deformation synthesized expression sample graph; and perform histogram equalization processing on the elastic deformation synthesized expression sample graph.

[0095] In a specific application scenario, a smoothing layer module is provided after the activation function layer at a preset position of the neural network in the machine learning model, and the smoothing layer module is a sliding convolution kernel of a preset size.

[0096] In a specific application scenario, the processing module 32 is further configured to process the tissue photograph sample image into a grayscale image and then perform histogram equalization processing.

[0097] The processing module 32 is further configured to process the tissue photograph into a grayscale image and then perform histogram equalization processing.

[0098] In a specific application scenario, optionally, the deformation field information is the deformation field information corresponding to the expression quantity map;

[0099] Correspondingly, the determination module 33 is specifically configured to perform linear interpolation calculation on the deformation field information corresponding to the expression map to obtain updated deformation field information; the horizontal and vertical coordinates in the spatiotemporal group expression data corresponding to the expression map are brought into the updated deformation field information to calculate the coordinate information of the expression map after registration.

[0100] In a specific application scenario, the acquisition module 31 is further configured to, after determining the corresponding coordinate information after image registration based on the deformation field information, use the coordinate information to obtain the expression quantity map after deformation correction.

[0101] In a specific application scenario, the processing module 32 is specifically configured to perform histogram equalization processing on the first image and the second image respectively; and perform registration based on the first image and the second image after the histogram equalization processing.

[0102] In a specific application scenario, the processing module 32 is further configured to use the tissue photograph after the histogram equalization process as a reference image to perform registration on the expression quantity map after the histogram equalization process;

[0103] The processing module 32 is further configured to input the expression image after registration and the tissue photograph image after histogram equalization processing into a preset machine learning model for calculation.

[0104] In a specific application scenario, the acquisition module 31 is specifically configured to generate the expression level graph according to the abundance statistics of the expression level data.

[0105] In a specific application scenario, the acquisition module 31 is further configured to divide the minimum square range covered by the target tissue into several adjacent square areas according to the coordinates of the expression data; count the total captured amount of the captured genes in each square area; and draw the expression map based on the total captured amount of the corresponding counted genes in each square area, wherein the pixel position in the expression map corresponds to the relative position of each square area in the minimum square range covered by the target tissue, and the pixel value of the pixel position is the total captured amount of the corresponding counted genes in each square area.

[0106] It should be noted that for other corresponding descriptions of the functional units involved in the data processing device for image registration provided in this embodiment, reference can be made to the corresponding descriptions in FIG. 1 and FIG. 2 , which will not be repeated here.

[0107] Based on the above methods shown in FIG. 1 and FIG. 2 , this embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above methods shown in FIG. 1 and FIG. 2 are implemented.

[0108] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0109] Based on the above-mentioned method shown in Figures 1 and 2, and the virtual device embodiment shown in Figure 13, in order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, which can be specifically a personal computer, a laptop computer, etc., and the device includes a storage medium and a processor; the storage medium is used to store computer programs; the processor is used to execute computer programs to implement the above-mentioned method shown in Figures 1 and 2.

[0110] Optionally, the physical device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a Wi-Fi module, and the like. The user interface may include a display, an input unit such as a keyboard, and the like. Optional user interfaces may also include a USB interface, a card reader interface, and the like. The network interface may optionally include a standard wired interface, a wireless interface (such as a Wi-Fi interface), and the like.

[0111] Those skilled in the art will understand that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0112] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device, supporting the execution of information processing programs and other software and / or programs. The network communication module is used to enable communication between components within the storage medium, as well as with other hardware and software within the physical information processing device.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the solution of this embodiment, compared with the current existing technology, the method of this embodiment is based on a supervised machine learning method, and manually provides labels to the model, thereby improving the feasibility of multimodal elastic registration. The method of this embodiment automatically synthesizes elastic deformation images based on the collected experimental data, reducing the cost of algorithm development. The method of this embodiment optimizes the smoothness of the output registration field and improves the accuracy of the registration results.

[0114] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0115] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand and implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments described herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A data processing method for image registration, characterized in that: include: Acquire a first image and a second image to be registered; Performing registration according to the first image and the second image; Inputting the registered first image and the second image into a machine learning model for calculation to obtain deformation field information, wherein the machine learning model is obtained by pre-training the model according to the registration data of the sample image, and the registration data includes: the sample image and the image after deformation correction of the sample image; According to the deformation field information, corresponding coordinate information after image registration is determined.

2. The method according to claim 1, characterized in that The first image is a tissue photograph, and the second image is an expression quantity map based on the tissue photograph; the sample image includes: a tissue photograph sample image and an expression quantity sample map based on the tissue photograph sample image, and the image after deformation correction of the sample image is the image after deformation correction of the expression quantity sample image.

3. The method according to claim 2, characterized in that The training process of the machine learning model includes: Performing histogram equalization processing on the tissue photography sample graph and the expression level sample graph respectively; Taking the tissue photographed sample image after histogram equalization as a reference image, rigid registration and elastic registration are performed on the expression quantity sample image after histogram equalization to obtain a data file containing a registration relationship and the expression quantity sample image after registration; The tissue photography sample image and the expression sample image after histogram equalization processing are used as feature data, and the data file and the expression sample image after alignment are used as label data to train and obtain the machine learning model.

4. The method according to claim 3, characterized in that The histogram equalization processing is performed on the expression sample graph, including: Generate a number of pairs of control points according to the expression sample graph; Calculating a sample deformation field using the control points, and applying the sample deformation field to the expression quantity sample graph to generate an expression quantity sample graph synthesized by elastic deformation; A histogram equalization process is performed on the elastic deformation synthesized expression quantity sample graph.

5. The method according to claim 3, characterized in that: A smoothing layer module is provided after the activation function layer at a preset position of the neural network in the machine learning model, and the smoothing layer module is a sliding convolution kernel of a preset size.

6. The method according to claim 3, characterized in that The histogram equalization process is performed on the tissue photograph sample image, including: After the tissue photograph sample image is processed into a grayscale image, a histogram equalization process is performed. Performing histogram equalization processing on the first image includes: After the tissue photograph is processed into a grayscale image, a histogram equalization process is performed.

7. The method according to claim 2, characterized in that The deformation field information is the deformation field information corresponding to the expression amount graph; Determining corresponding coordinate information after image registration based on the deformation field information includes: Performing linear interpolation calculation on the deformation field information corresponding to the expression amount graph to obtain updated deformation field information; The horizontal and vertical coordinates in the spatiotemporal group expression data corresponding to the expression map are brought into the updated deformation field information to calculate the coordinate information of the expression map after registration.

8. The method according to claim 7, characterized in that After determining the corresponding coordinate information after image registration based on the deformation field information, the method further includes: The coordinate information is used to obtain an expression quantity map after deformation correction.

9. The method according to claim 2, characterized in that: The registering according to the first image and the second image comprises: Performing histogram equalization processing on the first image and the second image respectively; The first image and the second image after the histogram equalization processing are registered.

10. The method according to claim 9, characterized in that The registering the first image and the second image after the histogram equalization processing includes: Using the tissue photograph after histogram equalization as a reference image, registering the expression amount map after histogram equalization; The step of inputting the registered first image and the second image into a machine learning model for calculation includes: The expression level image after registration and the tissue photograph image after histogram equalization processing are input into a preset machine learning model for calculation.

11. The method according to any one of claims 2 to 10, characterized in that The process of obtaining the expression level map includes: The expression level graph is generated according to the abundance statistics of the expression level data.

12. The method according to claim 11, characterized in that Generating the expression level graph according to the abundance statistics of the expression level data includes: According to the coordinates of the expression data, the minimum square range covered by the target tissue is divided into a number of adjacent square areas; The total capture of captured genes was counted in each square area; Based on the total captured amount of genes counted in each square area, the expression level map is drawn, wherein the pixel position in the expression level map corresponds to the relative position of each square area in the minimum square range covered by the target tissue, and the pixel value of the pixel position is the total captured amount of genes counted in each square area.

13. A data processing device for image registration, characterized in that: include: An acquisition module, configured to acquire a first image and a second image to be registered; a processing module, configured to perform registration according to the first image and the second image; The acquisition module is further configured to input the registered first image and the second image into a machine learning model for calculation to obtain deformation field information, wherein the machine learning model is obtained by pre-training the model according to the registration data of the sample image, and the registration data includes: the sample image and the image after deformation correction of the sample image; The determination module is configured to determine corresponding coordinate information after image registration based on the deformation field information.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

Citation Information

Patent Citations

  • Image registration method and device, storage medium and computer equipment

    CN113989338A

  • Medical image registration model training method and equipment

    CN115830016A

  • Medical image processing method and device, storage medium and electronic equipment

    CN115861394A

  • Image Processing Method and Apparatus, and Computer Readable Storage Medium

    US20210209775A1

Cited By

  • Training method of ovarian cancer lesion segmentation model

    CN120374647A