Real-time molecular image intelligent fusion method and system

Abstract

The invention provides a real-time molecular image intelligent fusion method and system. The method comprises the following steps: acquiring a dimension matching feature set of molecular images of different modalities; based on the dimension matching feature set, obtaining fusion features of the molecular images of different modalities; dividing the molecular regions with similar fusion features together, identifying different regions of the fusion features, and obtaining the fusion features subjected to region division; and constructing a real-time three-dimensional model based on a preset three-dimensional reconstruction model and the fusion features subjected to region division. According to the method, feature fusion is carried out on molecular images of different modalities, limitation of single-modal image information is avoided, and compared with single-modal features, fusion features can capture differences of molecules in different aspects, so that similar molecules are better distinguished, the precision and reliability of molecular division are improved, and when three-dimensional reconstruction is carried out, the accuracy and reliability of molecular division are improved. And accurate space positioning can be realized by combining the information of molecular images in different modes.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent fusion of medical images, and specifically to a real-time intelligent fusion method and system for molecular images. Background Art

[0002] In the field of real-time molecular imaging intelligent fusion, there are currently several key challenges. The first is the compatibility of multimodal data. Various molecular imaging modalities such as PET, MRI, and CT are widely used, but the data of each modality are significantly different. For example, the metabolic function data of PET and the anatomical structure data of MRI are completely different in feature dimensions, data types, and physical meanings. Traditional fusion methods are unable to handle these differences. Simple normalization makes it difficult to explore potential associations, often resulting in information loss or incorrect fusion. Secondly, traditional fusion algorithms such as fixed weighted average or simple splicing lack the ability to dynamically evaluate the importance of features and cannot flexibly adjust strategies according to actual conditions. The fused features are difficult to balance information retention and consistency, authenticity, and lack accuracy and reliability, which limits clinical and scientific research applications. Furthermore, it is not easy to identify tissue types and lesion areas in fused features. Traditional methods that rely on manual experience or simple threshold segmentation are difficult to handle complex feature relationships and subtle differences, and are prone to misjudgment and omission.

[0003] Therefore, how to overcome the above-mentioned technical problems and defects becomes a key issue that needs to be addressed. Summary of the invention

[0004] In order to overcome the above problems existing in the prior art, the present application provides a real-time molecular imaging intelligent fusion method and system, which adopts the following technical solutions:

[0005] In a first aspect, the present application provides a real-time molecular imaging intelligent fusion method, comprising:

[0006] Obtain dimensional matching feature sets for molecular images of different modalities.

[0007] Based on the dimension matching feature set, the fusion features of molecular images of different modalities are obtained.

[0008] The molecular regions with similar characteristics of the fusion features are divided together, the different regions of the fusion features are identified, and the fusion features for regional division are obtained.

[0009] Based on the preset 3D reconstruction model, a real-time 3D model is constructed based on the fusion features of regional division.

[0010] Furthermore, the step of obtaining the dimension matching feature set of molecular images of different modalities includes:

[0011] Extract single-modal features from the obtained different-modal molecular images to obtain single-modal features, and perform dimension matching on different single-modal features to obtain a dimension-matching feature set.

[0012] Further, obtaining the fusion features of different-modal molecular images based on the dimension-matching feature set includes:

[0013] Perform a correlation analysis on the dimension-matching feature set to obtain the main feature combinations in the dimension-matching feature set, and fuse the main feature combinations through a generative adversarial network to obtain fusion features.

[0014] Further, perform single-modal feature extraction on the preprocessed different-modal molecular images through the encoder part of the convolutional autoencoder.

[0015] Further, performing single-modal feature extraction on the preprocessed different-modal molecular images through the encoder part of the convolutional autoencoder includes:

[0016] Based on the preprocessed different-modal molecular images and the convolutional layer of the encoder, obtain convolutional feature maps.

[0017] The convolutional feature maps are processed by an activation function. Based on the non-linear factors of the activation function, enhance the expression ability of the features to obtain a complex feature representation of the molecular images.

[0018] Input the convolutional feature maps processed by the activation function into the pooling layer. By downsampling the convolutional feature maps, screen the convolutional feature maps to obtain the key feature information of the molecular images.

[0019] Through the multi-layer stacking of the encoder, the encoder obtains a hierarchical feature representation of the molecular images, and gradually transforms the input original molecular images into low-dimensional features, that is, the single-modal features of different-modal molecular images.

[0020] Further, perform dimension matching on different single-modal features to obtain a dimension-matching feature set, including:

[0021] Based on the shared encoding layer, map the single-modal features of different-modal molecular images to a common low-dimensional space, realize the dimension matching of the single-modal features of different-modal molecular images, and obtain a dimension-matching feature set.

[0022] Further, perform a correlation analysis on the dimension-matching feature set to obtain the main feature combinations in the dimension-matching feature set, including:

[0023] Obtain the correlation coefficients between the features in the dimension-matching feature set;

[0024] Convert the correlation coefficients into a correlation matrix, where the rows and columns of the matrix respectively correspond to the features in the dimension feature matching set;

[0025] Traverse the correlation matrix based on a preset threshold, filter out the features that exceed the preset threshold, and combine the filtered features to obtain the main feature combination.

[0026] Furthermore, fuse the main feature combination through a generative adversarial network to obtain the fused features, including:

[0027] The input layer of the generative adversarial network generator obtains the main feature combination, and extracts the deep features of the main feature combination through a dense convolutional network;

[0028] Introduce an attention mechanism to obtain the fusion weights of the deep features, and perform weighted fusion on the deep features based on the fusion weights to obtain the attention map of the deep features;

[0029] Perform weighted fusion on the attention map and the main feature combination to generate the fused features;

[0030] The discriminator receives the fused features, and judges the fused features through the multi-layer convolutional network of the discriminator. When the distance difference between the generated fused features and the real features meets the preset threshold, the generated fused features are used as the final result.

[0031] In a second aspect, the present application also provides a real-time molecular imaging intelligent fusion system, including:

[0032] A dimension matching feature set acquisition module, configured to acquire a dimension matching feature set of different modality molecular images;

[0033] A fused feature acquisition module, configured to acquire the fused features of different modality molecular images based on the dimension matching feature set;

[0034] Regional division fused feature acquisition, configured to divide the molecular regions with similar features of the fused features together, identify different regions of the fused features, and obtain the fused features for regional division;

[0035] A three-dimensional model reconstruction module, configured to construct a real-time three-dimensional model based on a preset three-dimensional reconstruction model and the fused features for regional division.

[0036] In a third aspect, the present application provides an electronic device, including:

[0037] One or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to execute the method described in the first aspect.

[0038] Fourthly, the present application provides a computer-readable storage medium storing a computer program which, when running on a computer, causes the computer to execute the method described in the first aspect.

[0039] Fifthly, the present application provides a computer program which, when executed by a computer, is used to execute the method described in the first aspect.

[0040] In a possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor.

[0041] The present application has the following beneficial effects:

[0042] 1. The present application obtains a dimension matching feature set of different modality molecular images; based on the dimension matching feature set, it obtains the fusion features of different modality molecular images. By performing feature fusion on different modality molecular images, the present application comprehensively describes the molecular activities and anatomical structures in the organism, avoids the limitations of single-modality image information, and can dynamically evaluate the important capabilities of features.

[0043] 2. The present application divides molecular regions with similar features in the fusion features, identifies different regions of the fusion features, and obtains the fusion features for region division. After obtaining the fusion features, the present application can divide similar molecules based on multiple features, rather than relying only on the features of a single-modality image. The features provided by different modality molecular images reflect the characteristics of molecules from different angles. By comprehensively judging these features, it is possible to more accurately distinguish different types of molecules and reduce the possibility of misjudgment and missed judgment. Compared with the features of a single modality, the fusion features can capture the differences of molecules in different aspects, thus better distinguishing similar molecules and improving the accuracy and reliability of molecular division.

[0044] 3. The present application constructs a real-time three-dimensional model based on the fusion features for region division based on a preset three-dimensional reconstruction model. When performing three-dimensional reconstruction, combining the information of different modality molecular images can achieve accurate spatial positioning, where the multi-modality molecular images provide rich details and constraint conditions for three-dimensional reconstruction. Description of the Drawings

[0045] Figure 1 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0046] Figure 2 is a flowchart of the real-time molecular image intelligent fusion method according to the embodiments of the present application;

[0047] Figure 3Flow chart of single-modal feature extraction according to an embodiment of the present application;

[0048] Figure 4 Flow chart of obtaining main feature combinations according to an embodiment of the present application;

[0049] Figure 5 Flow chart of obtaining fused features according to an embodiment of the present application;

[0050] Figure 6 System flow chart according to an embodiment of the present application;

[0051] Figure 7 Schematic diagram of a computer device according to an embodiment of the present application. Detailed implementation manners

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0053] Reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0054] In order to enable those skilled in the art of this technology to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the drawings.

[0055] As Figure 1 shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0056] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0057] Terminal devices 101, 102, and 103 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, and so on.

[0058] Server 105 can be a server providing various services, such as a background server that supports the pages displayed on terminal devices 101, 102, and 103.

[0059] It should be noted that the real-time molecular imaging intelligent fusion method provided by the embodiments of the present application is generally executed by the server / terminal device. Correspondingly, the real-time molecular imaging intelligent fusion system is generally set in the server / terminal device.

[0060] It should be understood that Figure 1 the numbers of terminal devices, networks, and servers in

[0061] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers.

[0062] Molecular imaging: Molecular imaging combines molecular biology techniques and medical imaging. By using molecular probes to label specific molecules or cells in the living body and performing labeling and quantitative analysis on the labeled molecules, molecular imaging breaks through the limitation of traditional medical imaging that only shows anatomical structures, directly reflects the biological behaviors at the cellular and molecular levels, and is an important tool for precision medicine. Molecular imaging includes positron emission tomography (PET), magnetic resonance imaging (MRI), computed tomography (CT), optical imaging, etc.

[0063] Continuing to refer to Figure 2 , the flowchart of a real-time molecular imaging intelligent fusion method of the present application is shown in the figure. The method includes the following steps:

[0064] Step 201, obtain a dimension matching feature set of different modality molecular images.

[0065] In a possible implementation, the obtaining of the dimension matching feature set of different modality molecular images includes:

[0066] Performing single-modal feature extraction on the obtained different modality molecular images to obtain single-modal features, and performing dimension matching on different single-modal features to obtain a dimension matching feature set.

[0067] In the embodiments of the present application, performing dimension matching on different single-modal features includes performing dimension processing on different single-modal features to obtain single-modal features with comparable dimensions. Since molecular images of different modalities have different dimensions, dimension matching of molecular image features of different modalities can make different modality molecular images consistent in space.

[0068] Before performing single-modal feature extraction on different modality molecular images, noise removal and image correction processing are performed on different modality molecular images, and then normalization processing is performed to adjust the gray values of different modality molecular image data to a unified scale range. For example, normalizing the pixel values of each modality molecular image to between 0 and 1 to obtain preprocessed different modality molecular images.

[0069] In the embodiments of the present application, performing dimension processing on different single-modal features includes performing dimension elevation or dimension reduction processing.

[0070] A convolutional autoencoder is an autoencoder model that combines a convolutional neural network and can be used for feature extraction of images. In the process of performing feature extraction of images, by replacing the fully connected layer of the original autoencoder with a convolutional layer, image data can be processed efficiently. A convolutional autoencoder includes an encoder and a decoder part; for the encoder part, through multiple convolutional layers and pooling layers, the input image is compressed into a low-dimensional spatial representation. The convolutional layer extracts local features of the image through filters, and the pooling layer further reduces the computational complexity of the image by reducing the spatial dimension of the image.

[0071] The encoder convolutional layer of the convolutional autoencoder contains multiple convolutional kernels. In the process of performing feature extraction, through the operations of multiple convolutional layers and pooling layers step by step, single-modal features of different modality molecular images are obtained.

[0072] In the embodiments of the present application, each modality molecular image corresponds to an encoder for extracting the single-modal features of that modality molecular image.

[0073] In the embodiments of the present application, single-modal feature extraction is performed on preprocessed different modality molecular images through the encoder part of the convolutional autoencoder. Please refer to Figure 3 , and the specific implementation steps include:

[0074] Step 31: Based on the preprocessed multi-modal molecular images and the convolutional layers of the encoder, obtain convolutional feature maps.

[0075] In a possible implementation, obtaining convolutional feature maps based on the preprocessed multi-modal molecular images and the convolutional layers of the encoder includes:

[0076] Use the preprocessed multi-modal molecular images as the input of the encoder. The convolutional layers of the encoder scan the input data based on the convolutional kernel and the preset stride. During the scanning process, the convolutional kernel multiplies and sums with the local area of the molecular image element by element to obtain convolutional feature maps. Here, the convolutional feature maps are local features in the molecular image. The convolutional kernel can extract local feature information from the molecular image and then generate corresponding feature maps. Multiple convolutions generate multiple convolutional feature maps, and multiple convolutional feature maps are stacked together to form the output of the encoder convolutional layer. In the embodiments of the present application, when after multiple iterations, the change in the unimodal feature distribution of different molecular images is less than the preset threshold, the convolution operation is stopped. The encoder mentioned here is the encoder part in the convolutional autoencoder.

[0077] Step 32: Process the convolutional feature maps through an activation function. Based on the non-linear factors of the activation function, enhance the expression ability of the features to obtain complex feature representations of the molecular images. In the embodiments of the present application, the ReLu function can be used to set the feature values less than 0 to 0 and keep the feature values greater than 0 unchanged.

[0078] Step 33: Input the convolutional feature maps processed by the activation function into the pooling layer. By downsampling the convolutional feature maps, screen the convolutional feature maps to obtain key feature information of the molecular images. In the embodiments of the present application, a max pooling layer or an average pooling layer can be used to process the convolutional feature maps to obtain the key feature information of the convolutional feature maps and enhance the robustness of feature extraction.

[0079] Step 34: Through the multi-layer stacking of the encoder, the encoder obtains hierarchical feature representations of the molecular images, gradually transforming the input original molecular images into low-dimensional features, that is, the unimodal features of different multi-modal molecular images. These low-dimensional features contain the main feature information of the molecular images and represent the latent space feature representations of the molecular images. The latent space feature representations are smaller than the dimensions of the input molecular images, realizing the compression and feature extraction of the molecular images.

[0080] In the embodiments of the present application, perform dimension matching on different unimodal features to obtain a dimension-matched feature set. The specific implementation steps include:

[0081] Based on a shared coding layer, the unimodal features of different-modal molecular images are mapped to a common low-dimensional space, achieving dimensional matching of the unimodal features of different-modal molecular images and obtaining a dimensionally matched feature set. Through the shared coding layer, this application can achieve feature matching of the unimodal features of different-modal molecular images, learn a common mapping function through the unimodal features of different-modal molecular images, project the features with different dimensions and feature distributions in the unimodal features of different-modal molecular images into a unified low-dimensional space, and thus obtain a dimensionally matched feature set. Through the shared coding layer, this application obtains the commonalities and correlations between different-modal data in the unimodal features of different-modal molecular images, and thus represents features that are representative of the overall multimodal molecular images, completing the alignment of different-modal features in terms of dimension and semantics. During the process of dimensional alignment, it includes unified alignment of coordinate positions. For example, when performing intelligent fusion of PET images and MRI images, this application can obtain the internal connection between the metabolic changes in PET images and the anatomical structure changes in MRI images through the shared coding layer. This application can reduce the overfitting of the model to single-modal data through the shared coding layer and improve the generalization ability of different-modal data in actual application scenarios.

[0082] The shared coding layer is a key component of the autoencoder. The unimodal features obtained by the encoder corresponding to each modality are used as the input of the shared encoder, and the unimodal features of different-modal molecular images are mapped to a common feature space, thereby enabling dimensional matching of the unimodal features of different-modal molecular images.

[0083] Step 202: Based on the dimensionally matched feature set, obtain the fusion features of different-modal molecular images.

[0084] In a possible implementation manner, the obtaining of the fusion features of different-modal molecular images based on the dimensionally matched feature set includes:

[0085] Perform correlation analysis on the dimensionally matched feature set to obtain the main feature combinations in the dimensionally matched feature set, and fuse the main feature combinations through a generative adversarial network to obtain the fusion features.

[0086] In the embodiments of this application, for the correlation analysis of the dimensionally matched feature set to obtain the main feature combinations in the dimensionally matched feature set, please refer to Figure 4 and the specific content includes:

[0087] Step 41: Obtain the correlation coefficients between the features of the dimensionally matched feature set.

[0088] In the embodiments of the present application, obtaining the correlation coefficients between the features of the dimension matching feature set includes performing linear correlation analysis and non-linear correlation analysis on the features of the dimension matching feature set. For linear correlation analysis, the Pearson correlation coefficient can be used to obtain the degree of linear relationship between the features of the dimension matching feature set, and the Spearman correlation coefficient can be used to obtain the degree of non-linear relationship between the features of the dimension matching feature set.

[0089] Step 42: Convert the correlation coefficients into a correlation matrix, where the rows and columns of the matrix respectively correspond to the features in the dimension feature matching set. The elements in the correlation matrix represent the degree of correlation between the corresponding two features, and the value range is usually between [-1, 1]. The closer the absolute value is to 1, the stronger the correlation, and the closer to 0, the weaker the correlation.

[0090] Step 43: Traverse the correlation matrix based on a preset threshold, filter out the features that exceed the preset threshold, and combine the filtered features to obtain the main feature combination.

[0091] In the embodiments of the present application, the main feature combination is fused through a generative adversarial network to obtain the fused features. Please refer to Figure 5 , and the specific content includes:

[0092] Step 51: The input layer of the generative adversarial network generator obtains the main feature combination, and extracts the deep features of the main feature combination through a dense convolutional network. In the embodiments of the present application, the dense convolutional network is characterized in that each layer is connected to all subsequent layers, which can promote the propagation of features.

[0093] Step 52: Introduce an attention mechanism to obtain the fusion weights of the deep features, and perform weighted fusion on the deep features based on the fusion weights to obtain the attention map of the deep features. The attention map in the present application is a feature weight map, which represents the importance of features.

[0094] Step 53: Perform weighted fusion on the attention map and the main feature combination to generate the fused features. By performing weighted fusion on the attention map and the main feature combination in the present application, while retaining the main features, the combination of features can be further optimized, more effective features can be obtained, and the effect of feature fusion can be further improved.

[0095] Step 54: The discriminator receives the fused features and judges the fused features through the multi-layer convolutional network of the discriminator. When the distance difference between the generated fused features and the real features meets the preset threshold, the generated fused features are used as the final result.

[0096] Step 203: Divide the molecular regions with similar features of the fused features into groups, identify different regions of the fused features, and obtain the fused features for region division.

[0097] In the embodiments of the present application, different regions for identifying fusion features include different tissue regions or lesion regions.

[0098] In the embodiments of the present application, molecular regions with similar features of the fusion features are grouped together, different regions of the fusion features are identified, and the fusion features for region division are obtained, including:

[0099] Randomly initialize K clustering centers.

[0100] Calculate the distance between each feature of the fusion feature and all cluster centers, and assign each feature to the nearest cluster; calculate the average value of all features within each cluster as the new cluster center. Repeat this step until the cluster centers no longer change or reach the preset number of iterations.

[0101] According to the assignment results of the clusters, divide the molecular regions of the fusion feature, and each cluster corresponds to a region with similar features.

[0102] Step 204: Based on a preset three-dimensional reconstruction model, construct a real-time three-dimensional model based on the fusion feature for region division.

[0103] In the embodiments of the present application, based on a preset three-dimensional reconstruction model, constructing a real-time three-dimensional model based on the fusion feature for region division includes:

[0104] Divide the three-dimensional space into regular voxel grids, map the fusion features of each region to the corresponding voxels, assign corresponding values to each voxel, and form a preliminary three-dimensional data representation.

[0105] Generate a three-dimensional surface model based on the three-dimensional data representation; by traversing the voxel grid, judge the intersection with the object surface according to the feature values of the voxels, and generate a triangular mesh to approximately represent the surface of the object. When the fusion feature changes, update the three-dimensional model.

[0106] In the embodiments of the present application, different molecular information is distinguished and represented in the three-dimensional model based on color coding. For example, regions with different metabolic activity levels or different tissue types are represented by different colors, enabling the observer to quickly understand the key information in the image and improving the efficiency and accuracy of interpreting the results of molecular image fusion.

[0107] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM), etc.

[0108] It should be understood that although the steps in the flowchart of the accompanying drawings are shown sequentially according to the indication of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0109] Continuing to refer to Figure 6 , the real-time molecular imaging intelligent fusion system described in this embodiment includes:

[0110] A dimension matching feature set acquisition module 601, configured to acquire a dimension matching feature set of molecular images of different modalities;

[0111] A fusion feature acquisition module 602, configured to acquire fusion features of molecular images of different modalities based on the dimension matching feature set;

[0112] A region division fusion feature acquisition 603, configured to divide molecular regions with similar features of the fusion features together, identify different regions of the fusion features, and acquire the fusion features for region division;

[0113] A three-dimensional model reconstruction module 604, configured to construct a real-time three-dimensional model based on a preset three-dimensional reconstruction model and the fusion features for region division.

[0114] To solve the above technical problems, the embodiments of the present application also provide a computer device. Specifically, please refer to Figure 7 , Figure 7 This is the basic structural block diagram of the computer device in this embodiment.

[0115] The computer device 7 includes a memory 7a, a processor 7b, and a network interface 7c that are communicatively connected to each other via a system bus. It should be noted that only the computer device 7 with components 7a - 7c is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of the present technology can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0116] The computer device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can perform human-computer interaction with the user through means such as a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0117] The memory 7a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 7a can be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 7a can also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 7. Of course, the memory 7a can also include both the internal storage unit and the external storage device of the computer device 7. In this embodiment, the memory 7a is generally used to store the operating system and various application software installed on the computer device 7, such as the program code of the real-time molecular imaging intelligent fusion method. In addition, the memory 7a can also be used to temporarily store various data that have been output or will be output.

[0118] In some embodiments, the processor 7b may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 7b is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 7b is used to run the program code stored in the memory 7a or process data, such as running the program code of the real-time molecular imaging intelligent fusion method.

[0119] The network interface 7c may include a wireless network interface or a wired network interface. The network interface 7c is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0120] The present application also provides another implementation manner, that is, to provide a non-volatile computer-readable storage medium storing a program of the real-time molecular imaging intelligent fusion method, and the real-time molecular imaging intelligent fusion can be executed by at least one processor, so that the at least one processor executes the steps of the real-time molecular imaging intelligent fusion method as described above.

[0121] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0122] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The preferred embodiments of the present application are shown in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. Any equivalent structure made by using the specification and drawings of the present application, directly or indirectly applied in other related technical fields, shall be equally within the scope of the patent protection of the present application.

Claims

1. A real-time molecular imaging intelligent fusion method, characterized in that Including: Obtain the dimension matching feature sets of different modality molecular images; Based on the dimension matching feature sets, obtain the fusion features of different modality molecular images; Divide the molecular regions with similar features in the fusion features together, identify different regions of the fusion features, and obtain the fusion features for region division; Construct a real-time three-dimensional model based on the fusion features for region division.

2. The real-time molecular imaging intelligent fusion method according to claim 1, wherein The obtaining of the dimension matching feature sets of different modality molecular images includes: Perform single-modal feature extraction on the obtained different modality molecular images to obtain single-modal features, and perform dimension matching on different single-modal features to obtain dimension matching feature sets.

3. The real-time molecular imaging intelligent fusion method according to claim 1, characterized in that The obtaining of the fusion features of different modality molecular images based on the dimension matching feature sets includes: Perform correlation analysis on the dimension matching feature sets to obtain the main feature combinations in the dimension matching feature sets, and fuse the main feature combinations through a generative adversarial network to obtain fusion features.

4. The real-time molecular imaging intelligent fusion method according to claim 2, wherein Perform single-modal feature extraction on the preprocessed different modality molecular images through the encoder part of a convolutional autoencoder.

5. The real-time molecular imaging intelligent fusion method according to claim 4, wherein Performing single-modal feature extraction on the preprocessed different modality molecular images through the encoder part of a convolutional autoencoder includes: Based on the preprocessed different modality molecular images and the convolutional layers of the encoder, obtain convolutional feature maps; The convolutional feature maps are processed by an activation function. Based on the non-linear factors of the activation function, enhance the expression ability of the features to obtain the complex feature representation of the molecular images; Input the convolutional feature maps processed by the activation function into a pooling layer. By performing downsampling on the convolutional feature maps, screen the convolutional feature maps to obtain the key feature information of the molecular images; Through the multi-layer stacking of the encoder, the encoder obtains the hierarchical feature representation of the molecular images, and gradually transforms the input original molecular images into low-dimensional features, that is, the single-modal features of different modality molecular images.

6. The real-time molecular imaging intelligent fusion method according to claim 2, wherein Performing dimension matching on different single-modal features to obtain dimension matching feature sets includes: Based on a shared coding layer, map the single-modal features of different modality molecular images to a common low-dimensional space, realize the dimension matching of the single-modal features of different modality molecular images, and obtain dimension matching feature sets.

7. The real-time molecular imaging intelligent fusion method according to claim 3, wherein Perform correlation analysis on the dimension matching feature sets to obtain the main feature combinations in the dimension matching feature sets, including: Obtain the correlation coefficients between the features in the dimension matching feature sets; Convert the correlation coefficients into a correlation matrix, where the rows and columns of the matrix respectively correspond to the features in the dimension feature matching sets; Traverse the correlation matrix based on a preset threshold, screen out the features that exceed the preset threshold, and combine the screened features to obtain the main feature combinations.

8. The real-time molecular imaging intelligent fusion method according to claim 3, wherein Fusing the main feature combinations through a generative adversarial network to obtain fusion features includes: The input layer of the generative adversarial network generator obtains the main feature combinations, and extracts the deep features of the main feature combinations through a dense convolutional network; Introduce an attention mechanism to obtain the fusion weights of the deep features, and perform weighted fusion on the deep features based on the fusion weights to obtain the attention map of the deep features; Perform weighted fusion on the attention map and the main feature combinations to generate fusion features; The discriminator receives the fused features and judges the fused features through the multi-layer convolutional network of the discriminator. When the distance difference between the generated fused features and the real features meets the preset threshold, the generated fused features are taken as the final result.

9. A real-time molecular imaging intelligent fusion system for implementing the real-time molecular imaging intelligent fusion method of claims 1-8, characterized in that, It includes: A dimension matching feature set acquisition module for acquiring the dimension matching feature sets of different modality molecular images; A fused feature acquisition module for acquiring the fused features of different modality molecular images based on the dimension matching feature sets; Region division fused feature acquisition for dividing molecular regions with similar features of the fused features together, identifying different regions of the fused features, and acquiring the fused features for region division; A three-dimensional model reconstruction module for constructing a real-time three-dimensional model based on the fused features for region division.

10. An electronic device, characterized in that, It includes: One or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-8.

Images