Method and device for determining organ change information based on images and genes

Abstract

The present application discloses a method and device for determining organ change information based on images and genes, and relates to the field of medical technology. The device includes: an acquisition unit for acquiring a target gene sequence and X medical images of a target object in a preset time period from a target data source; a determination unit for determining a model input feature based on X medical images and target gene sequences through a trunk neural network; a prediction unit for inputting the model input feature into a target branch network connected to the trunk neural network, and predicting the change information of the target organ of the target object through the target branch network based on the model input feature. The present application solves the technical problem in the prior art that the accuracy of predicting the radiotherapy effect of a patient is poor due to the lack of technology for deep integration of the patient's image features and genetic information.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and more specifically, to a method and device for determining organ change information based on images and genes. Background Art

[0002] In the prior art, changes in a patient's tumor are usually determined based on image change information from the patient's medical images. For example, CT images or MR images are taken during regular physical examinations to detect whether the patient's tumor has shrunk or expanded.

[0003] However, in the medical field, there is a potential connection between a patient's tumor changes and their gene mutations. Therefore, in the prior art, only using the image change information of medical images to judge or predict the patient's tumor change trend has the problem of too single a criterion dimension, which leads to a technical problem of poor accuracy in predicting the patient's radiotherapy effect.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a method and device for determining organ change information based on images and genes, so as to at least solve the technical problem in the prior art of poor accuracy in predicting the radiotherapy effect on patients due to the lack of technology for deep integration of the patient's image features and genetic information.

[0006] According to one aspect of the present application, a device for determining organ change information based on images and genes is provided, comprising: an acquisition unit, used to acquire a target gene sequence and X medical images of a target object in a preset time period from a target data source, wherein X is an integer greater than 1; a determination unit, used to determine a model input feature based on the X medical images and the target gene sequence through a trunk neural network, wherein the model input feature is used to characterize the image changes between the X medical images and the gene mutation of the target object within the preset time period; a prediction unit, used to input the model input feature into a target branch network connected to the trunk neural network, and predict the change information of the target organ of the target object through the target branch network based on the model input feature, wherein different data sources correspond to different branch networks, and the target organ is an organ at risk and / or a target organ involved in the radiotherapy process of the target object.

[0007] Optionally, in the image- and gene-based organ change information determination device provided in the present application, the determination unit includes: a first acquisition subunit, used to acquire a reference gene sequence, wherein the reference gene sequence is a gene sequence of an object that does not suffer from cancer; a first determination subunit, used to determine a target gene feature based on the reference gene sequence and a target gene sequence of the target object, wherein the target gene feature is used to characterize the gene variation information of the target gene sequence compared to the reference gene sequence; a second determination subunit, used to determine a target image feature based on X medical images, wherein the target image feature is used to characterize the image changes of the X medical images within a preset time period; a feature fusion subunit, used to fuse the target image feature and the target gene feature into a model input feature through a backbone neural network.

[0008] Optionally, in the image and gene-based organ change information determination device provided in the present application, the first determination subunit includes: a first determination module, used to determine first-category genes and second-category genes from a target gene sequence, wherein the first-category genes are known genes involved in causing cancer, and the second-category genes are genes that undergo genetic mutations at a single nucleotide position; a first calculation module, used to calculate the ratio of the number of bases of the first-category genes to the number of bases of the third-category genes in the reference gene sequence, to obtain first-category gene characteristics, wherein the position of the third-category genes in the reference gene sequence is the same as the position of the first-category genes in the target gene sequence; a second calculation module, used to calculate the ratio of the number of bases of the second-category genes to the number of bases of the fourth-category genes in the reference gene sequence, to obtain second-category gene characteristics, wherein the position of the fourth-category genes in the reference gene sequence is the same as the position of the second-category genes in the target gene sequence; and a first processing module, used to use the first-category gene characteristics and the second-category gene characteristics as target gene characteristics.

[0009] Optionally, in the image and gene-based organ change information determination device provided in the present application, the feature fusion subunit includes: a second processing module, used to represent the target gene feature as a gene pathway; a third processing module, used to normalize the gene pathway to obtain the target gene pathway, wherein the specification of the target gene pathway is the same as the pathway specification of the backbone neural network; a fourth processing module, used to fuse the target gene pathway and the target image feature into a model input feature by feature splicing or feature dot multiplication based on the time characteristics of a preset time period.

[0010] Optionally, the image- and gene-based organ change information determination device also includes: a first acquisition unit, used to acquire historical medical images and historical gene sequences; a setting unit, used to set a first encoder and a first decoder, wherein the first encoder is used to extract image features of historical medical images and gene features of historical gene sequences; the first decoder is used to generate images to be evaluated based on the image features of historical medical images, and to generate gene sequences to be evaluated based on the gene features of historical gene sequences; and a first training unit, which trains a backbone neural network based on historical medical images, historical gene sequences, the first encoder and the first decoder.

[0011] Optionally, in the image and gene-based organ change information determination device provided in the present application, the first training unit includes: a training subunit, which is used to iteratively train the first encoder based on historical medical images and historical gene sequences, until the similarity between the image to be evaluated generated by the first decoder based on the image features extracted by the first encoder and the historical medical images is greater than a first threshold, and the similarity between the gene sequence to be evaluated generated by the first decoder based on the gene features extracted by the first encoder and the historical gene sequences is greater than a second threshold; a backbone neural network determination subunit, which is used to combine the trained first encoder and the second encoder into a backbone neural network, wherein the second encoder is used to perform feature fusion on the image features and gene features output by the first encoder.

[0012] Optionally, the image- and gene-based organ change information determination device also includes: a second acquisition unit, used to acquire medical images, gene sequences and actual organ change information of each reference object among N reference objects from the target data source, wherein N is an integer greater than 1; a feature extraction unit, used to extract image features of the medical image of the i-th reference object and gene features of the gene sequence of the i-th reference object through a trunk neural network, wherein the i-th reference object is any one of the N reference objects; a first processing unit, used to use the image features of the medical image of the i-th reference object as a first reference feature; a second processing unit, used to use the gene features of the gene sequence of the i-th reference object as a second reference feature; and a second training unit, used to train a target branch network based on the first reference feature, the second reference feature and a second decoder corresponding to the target data source.

[0013] Optionally, in the image and gene-based organ change information determination device provided in the present application, the second training unit includes: a target reference feature determination subunit, used to fuse the first reference feature and the second reference feature into a target reference feature through a trunk neural network; a second decoder training subunit, used to iteratively train the second decoder based on the target reference feature until the similarity between the organ change information generated by the second decoder based on the target reference feature and the actual organ change information of the i-th reference object is greater than a preset similarity; a target branch network determination subunit, used to use the trained second decoder as the target branch network.

[0014] Optionally, in the image and gene-based organ change information determination device provided in the present application, the target reference feature determination subunit includes: a collection interval duration acquisition module, used to obtain the collection interval duration of the historical gene sequence of each reference object in N reference objects, and obtain N collection interval durations, wherein the N collection interval durations correspond one-to-one to the N reference objects; a third calculation module, used to calculate the difference between the maximum and minimum values ​​of the N collection interval durations; a fourth calculation module, used to calculate the ratio of the collection interval duration of the historical gene sequence of the i-th reference object to the difference, and obtain the time variation parameter of the i-th reference object; a target reference feature determination module, used to fuse the first reference feature and the second reference feature into a target reference feature according to the time variation parameter.

[0015] According to another aspect of the present application, a method for determining organ change information based on images and genes is also provided, including: obtaining a target gene sequence and X medical images of a target object in a preset time period from a target data source, wherein X is an integer greater than 1; determining a model input feature based on the X medical images and the target gene sequence through a trunk neural network, wherein the model input feature is used to characterize the image changes between the X medical images and the gene mutations of the target object within the preset time period; inputting the model input feature into a target branch network connected to the trunk neural network, and predicting the change information of the target organ of the target object through the target branch network based on the model input feature, wherein different data sources correspond to different branch networks, and the target organ is an organ at risk and / or a target organ involved in the radiotherapy process of the target object.

[0016] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors control the operation of any one of the above-mentioned imaging and gene-based organ change information determination devices.

[0017] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located controls the operation of any one of the above-mentioned image- and gene-based organ change information determination devices.

[0018] In the present application, the target gene sequence and X medical images of the target object in a preset time period are first obtained from the target data source, where X is an integer greater than 1. Then, the model input features are determined based on the X medical images and the target gene sequence through the backbone neural network, where the model input features are used to characterize the image changes between the X medical images and the gene mutations of the target object in the preset time period. Finally, the model input features are input into the target branch network connected to the backbone neural network, and the target branch network predicts the change information of the target organ of the target object based on the model input features, where different data sources correspond to different branch networks, and the target organ is the endangered organ and / or target organ involved in the radiotherapy process of the target object.

[0019] It can be seen from the above content that compared with the prior art, the present application first determines the model input features based on X medical images and target gene sequences, and then uses the model input features to predict the change information of the target organ of the target object, thereby achieving the technical effect of comprehensively determining the change information of the target organ based on the gene variation and image variation of the target object. In addition, the present application also takes into account the time factor, that is, the model input features are used to characterize the image changes between X medical images and the gene variation of the target object within a preset time period, indicating that the model input features in the present application are time series features, and the change information of the target organ predicted thereby is more in line with the time change trend. Finally, the present application also designs different branch networks corresponding to different data sources, that is, each branch network is trained by the data of the corresponding data source, thereby improving the privacy security of patient data and ensuring that the patient data of each data source is only used locally and will not be leaked.

[0020] It can be seen that through the technical solution of the present application, the purpose of deeply fusing the patient's imaging features and genetic features is achieved, thereby predicting the patient's organ change information, thereby solving the technical problem in the prior art of poor accuracy in predicting the patient's radiotherapy effect due to the lack of technology for deeply fusing the patient's imaging features and genetic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 is a schematic diagram of an optional apparatus for determining organ change information based on images and genes according to an embodiment of the present application;

[0023] Figure 2 is an optional processing flow chart of medical images and genetic information according to an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of an optional feature fusion method according to an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of an optional model architecture according to an embodiment of the present application;

[0026] Figure 5 is an optional training flow chart of a target branch network according to an embodiment of the present application;

[0027] Figure 6 is an optional training flow chart of an overall neural network according to an embodiment of the present application;

[0028] Figure 7 It is a schematic diagram of an optional gene biology pathway according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0032] According to an embodiment of the present application, an embodiment of a device for determining organ change information based on images and genes is provided, wherein: Figure 1 is a schematic diagram of an optional apparatus for determining organ change information based on images and genes according to an embodiment of the present application, such as Figure 1 As shown, the device for determining organ change information based on images and genes includes the following units:

[0033] The acquisition unit 101 is used to acquire the target gene sequence and X medical images of the target object in a preset time period from the target data source, where X is an integer greater than 1.

[0034] Optionally, in an embodiment of the present application, different medical institutions may be used as different data sources, for example, N medical institutions correspond to N data sources, wherein the patient data of each medical institution is only stored in the database of the medical institution.

[0035] It should be noted that the N data sources are independent of each other. In other words, the N data sources are prohibited from accessing each other without permission, wherein the target data source is any one of the N data sources.

[0036] Optionally, the target object is a patient receiving medical treatment at a medical institution corresponding to the target data source. Therefore, the organ change information determination device can obtain the target gene sequence and X medical images of the target object in a preset time period from the target data source.

[0037] Optionally, medical images include but are not limited to various types of images such as MR images, CT images, and PET images; target gene sequences of target subjects include but are not limited to tumor drug-related genes, reported proto-oncogenes, tumor suppressor genes, hot spot exon regions of genes with high mutation frequencies in the population, etc. Among them, reported proto-oncogenes refer to genes known to cause cancer.

[0038] The determination unit 102 is used to determine the model input features according to the X medical images and the target gene sequence through the backbone neural network. The model input features are used to characterize the image changes between the X medical images and the gene variation of the target object within a preset time period.

[0039] Optionally, the model input features include at least target image features and target gene features. The target image features are used to characterize image changes of X medical images within a preset time period; the target gene features are used to characterize gene variation information of the target gene sequence compared to the reference gene sequence.

[0040] In an optional embodiment, Figure 2 is an optional processing flow chart of medical images and gene information according to an embodiment of the present application, such as Figure 2 As shown, the organ change information determination device can pre-process the X medical images of the target object, and the pre-processing includes but is not limited to performing noise reduction and registration processing on the X medical images. The pre-processing step can also be omitted.

[0041] Alternatively, if Figure 2 As shown, the organ change information determination device can input the preprocessed X medical images into the backbone neural network, and then the backbone neural network extracts the target image features based on the preprocessed X medical images. The organ change information determination device can also input the target gene sequence of the target object into the backbone neural network, and extract the target gene features of the target gene sequence through the backbone neural network. Finally, the organ change information determination device fuses the target gene features and the target image features into model input features through the backbone neural network.

[0042] The prediction unit 103 is used to input the model input features into the target branch network connected to the trunk neural network, and predict the change information of the target organ of the target object according to the model input features through the target branch network. Different data sources correspond to different branch networks, and the target organ is the endangered organ and / or target organ involved in the radiotherapy process of the target object.

[0043] Alternatively, if Figure 2 As shown, if the medical institution where the target object is treated is the first medical institution, the data source corresponding to the target object is the database of the first medical institution, and the branch network corresponding to the target object is the first branch network corresponding to the first medical institution. On this basis, the organ change information determination device can input the model input features corresponding to the target object into the first branch network, and the first branch network predicts the change information of the target organ of the target object based on the model input features. The target organ is the endangered organ and / or target organ involved in the radiotherapy process of the target object.

[0044] It should be noted that the technical solution of the present application can be applied to the diagnosis stage before radiotherapy of the target object, can also be applied to the side effect prediction stage after radiotherapy of the target object, and can also be applied to the observation stage during radiotherapy of the target object. Regardless of the stage, the technical solution of the present application can be used to predict the changes in the target object's organs at risk and / or target area. After predicting the changes in the target object's organs at risk and / or target area, the survival time and survival rate of the target object and the survival time and survival rate of cancer cells can be further predicted.

[0045] In addition, Figure 2 The second branch network corresponds to the second medical institution, and the third branch network corresponds to the third medical institution.

[0046] It can be seen from the above content that compared with the prior art, the present application first determines the model input features based on X medical images and target gene sequences, and then uses the model input features to predict the change information of the target organ of the target object, thereby achieving the technical effect of comprehensively determining the change information of the target organ based on the gene variation and image variation of the target object. In addition, the present application also takes into account the time factor, that is, the model input features are used to characterize the image changes between X medical images and the gene variation of the target object within a preset time period, indicating that the model input features in the present application are time series features, and the change information of the target organ predicted thereby is more in line with the time change trend. Finally, the present application also designs different branch networks corresponding to different data sources, that is, each branch network is trained by the data of the corresponding data source, thereby improving the privacy security of patient data and ensuring that the patient data of each data source is only used locally and will not be leaked.

[0047] It can be seen that through the technical solution of the present application, the purpose of deeply fusing the patient's imaging features and genetic features is achieved, thereby predicting the patient's organ change information, thereby solving the technical problem in the prior art of poor accuracy in predicting the patient's radiotherapy effect due to the lack of technology for deeply fusing the patient's imaging features and genetic information.

[0048] In an optional embodiment, the determination unit 102 includes: a first acquisition subunit, a first determination subunit, a second determination subunit, and a feature fusion subunit.

[0049] The first acquisition subunit is used for the organ change information determination device to obtain a reference gene sequence in the process of determining the model input features according to X medical images and the target gene sequence through the backbone neural network, wherein the reference gene sequence is a gene sequence of a subject who does not suffer from cancer.

[0050] The first determination subunit is used to determine the target gene feature according to the reference gene sequence and the target gene sequence of the target object, wherein the target gene feature is used to characterize the gene variation information of the target gene sequence compared with the reference gene sequence.

[0051] The second determining subunit is used to determine target image features according to the X medical images, wherein the target image features are used to characterize image changes of the X medical images within a preset time period.

[0052] The feature fusion subunit is used for the organ change information determination device to fuse the target image features and the target gene features into model input features through the backbone neural network.

[0053] Optionally, the reference gene sequence is a gene sequence of a subject known not to suffer from cancer. In the field of gene testing, the reference gene sequence can be understood as a "gold standard" gene sequence.

[0054] Optionally, the organ change information determination device can compare the reference gene sequence and the target gene sequence, for example, first eliminate low-quality sequences and connector sequences in the sequencing data of the target gene sequence, and then perform mutation detection through an autonomous analysis process, including detecting SNP (single nucleotide polymorphism) and CNV (copy number variations), so as to detect the gene variation information of the target gene sequence compared with the reference gene sequence.

[0055] It should be noted that by determining the target gene characteristics based on the reference gene sequence and the target gene sequence of the target object, the technical effect of detecting cancer-related mutant genes in the target object is achieved.

[0056] In an optional embodiment, the first determining subunit includes: a first determining module, a first calculating module, a second calculating module and a first processing module.

[0057] Among them, the first determination module is used to determine the first category genes and the second category genes from the target gene sequence, wherein the first category genes are known genes involved in causing cancer, and the second category genes are genes that have genetic mutations at a single nucleotide position; the first calculation module is used to calculate the ratio of the number of bases of the first category genes to the number of bases of the third category genes in the reference gene sequence, to obtain the first category gene characteristics, wherein the position of the third category genes in the reference gene sequence is the same as the position of the first category genes in the target gene sequence; the second calculation module is used to calculate the ratio of the number of bases of the second category genes to the number of bases of the fourth category genes in the reference gene sequence, to obtain the second category gene characteristics, wherein the position of the fourth category genes in the reference gene sequence is the same as the position of the second category genes in the target gene sequence; the first processing module is used to use the first category gene characteristics and the second category gene characteristics as target gene characteristics.

[0058] Optionally, the organ change information determination device can use tumor-related genes on the chromosome of the target object as the first type of genes, and use newly discovered SNP genes on the chromosome of the target object as the second type of genes, wherein the SNP gene is a gene with genetic variation at a single nucleotide position.

[0059] Optionally, the first type of gene feature Value1 can be calculated using the following formula (1):

[0060] ,

[0061] In formula (1), F1 represents the number of bases of the tumor-related gene on the jth chromosome of the target object; F2 represents the number of bases in the reference gene sequence corresponding to the jth chromosome at the same position as the tumor-related gene. Where j is a positive integer.

[0062] Optionally, the second type of gene feature Value2 can be calculated by the following formula (2):

[0063] ,

[0064] In formula (2), R1 represents the number of bases of the newly discovered SNP gene on the jth chromosome of the target object; R2 represents the number of bases in the reference gene sequence corresponding to the jth chromosome at the same position as the newly discovered SNP gene.

[0065] From the above content, it can be seen that the present application can determine the base mutation of the specific type of gene of the target object by comparing the number of bases of the specific type of gene of the target object with the number of bases of the specific type of gene at the same position in the reference gene sequence, thereby determining the mutation information of the target object's cancer-related genes.

[0066] In an optional embodiment, the feature fusion subunit includes: a second processing module, a third processing module, and a fourth processing module.

[0067] Among them, the second processing module is used to represent the target gene features as gene pathways; the third processing module is used to normalize the gene pathways to obtain the target gene pathways, wherein the specifications of the target gene pathways are the same as the pathway specifications of the backbone neural network; the fourth processing module is used to fuse the target gene pathways and the target image features into model input features based on the time characteristics of the preset time period by feature splicing or feature dot multiplication.

[0068] Optionally, methods of representing target gene features as gene pathways include but are not limited to various enrichment analysis methods.

[0069] Optionally, in order to enable the backbone neural network to process the target gene features, the gene pathway may be subjected to dimensionality reduction and normalization processing, the purpose of which is to make the specification of the target gene pathway the same as the pathway specification of the backbone neural network.

[0070] Finally, the organ change information determination device introduces the time features of the preset time period, and fuses the target gene pathway and image features into model input features through feature splicing or feature point multiplication.

[0071] Optionally, Figure 3 is a schematic diagram of an optional feature fusion method according to an embodiment of the present application, such as Figure 3 As shown, the first method is to splice the target image features and the target gene features according to the channel direction of the backbone neural network; the second method is to perform dot product calculation on the target image features and the target gene features.

[0072] In an optional embodiment, the apparatus for determining organ change information based on images and genes further includes: a first acquisition unit, a setting unit, and a first training unit. The first acquisition unit is used to acquire historical medical images and historical gene sequences; the setting unit is used to set a first encoder and a first decoder, wherein the first encoder is used to extract image features of historical medical images and gene features of historical gene sequences; the first decoder is used to generate images to be evaluated based on image features of historical medical images, and to generate gene sequences to be evaluated based on gene features of historical gene sequences; the first training unit is used to train a backbone neural network based on historical medical images, historical gene sequences, the first encoder, and the first decoder.

[0073] Optionally, Figure 4 is a schematic diagram of an optional model architecture according to an embodiment of the present application, such as Figure 4As shown, the first encoder is connected to the first decoder. During the model training stage, the function of the first encoder is to extract image features of historical medical images and gene features of historical gene sequences, and input the extracted image features and gene features into the first decoder; the function of the first decoder is to generate images to be evaluated based on the image features of historical medical images, and to generate gene sequences to be evaluated based on the gene features of historical gene sequences.

[0074] Optionally, the first training unit includes: a training subunit and a backbone neural network determination unit. The training subunit is used to iteratively train the first encoder based on historical medical images and historical gene sequences until the similarity between the image to be evaluated generated by the first decoder based on the image features extracted by the first encoder and the historical medical images is greater than a first threshold, and the similarity between the gene sequence to be evaluated generated by the first decoder based on the gene features extracted by the first encoder and the historical gene sequence is greater than a second threshold; the backbone neural network determination subunit is used to form a backbone neural network with the trained first encoder and the second encoder, wherein the second encoder is used to perform feature fusion on the image features and gene features output by the first encoder.

[0075] Optionally, the process of iteratively training the first encoder based on historical medical images and historical gene sequences includes the following steps:

[0076] Step 1, extracting image features of historical medical images and gene features of historical gene sequences through a first encoder;

[0077] Step 2, generating an image to be evaluated based on the image features of the historical medical image, and generating a gene sequence to be evaluated based on the gene features of the historical gene sequence by using a first decoder;

[0078] Step 3, detecting whether the similarity between the image to be evaluated and the historical medical image is greater than a first threshold, and detecting whether the similarity between the gene sequence to be evaluated and the historical gene sequence is greater than a second threshold, wherein the first threshold and the second threshold may be the same or different;

[0079] Step 4, when it is detected that the similarity between the image to be evaluated and the historical medical image is greater than the first threshold, and the similarity between the gene sequence to be evaluated and the historical gene sequence is greater than the second threshold, the first encoder and the second encoder are combined into a backbone neural network, wherein the second encoder is used to perform feature fusion on the image features and gene features output by the first encoder;

[0080] Step 5. When it is detected that the similarity between the image to be evaluated and the historical medical image is less than or equal to the first threshold, and / or the similarity between the gene sequence to be evaluated and the historical gene sequence is less than or equal to the second threshold, the parameters of the first encoder are updated, and steps 1 to 5 are repeated until the similarity between the image to be evaluated generated by the first decoder based on the image features extracted by the first encoder and the historical medical image is greater than the first threshold, and the similarity between the gene sequence to be evaluated generated by the first decoder based on the gene features extracted by the first encoder and the historical gene sequence is greater than the second threshold, and the first encoder and the second encoder updated for the last time form a backbone neural network.

[0081] In an optional embodiment, the apparatus for determining organ change information based on images and genes further includes: a second acquisition unit, a feature extraction unit, a first processing unit, a second processing unit, and a second training unit, which respectively perform Figure 5 The relevant steps in Figure 5 is an optional training flow chart of a target branch network according to an embodiment of the present application, comprising the following steps:

[0082] Step S501 : obtaining medical images, gene sequences, and actual organ change information of each of the N reference objects from the target data source.

[0083] In step S501, N is an integer greater than 1.

[0084] Optionally, in order to protect the patient data in each data source from being leaked, the patient data (such as medical images, gene sequences, and actual organ change information) in each data source is only used locally.

[0085] Step S502 , extracting the image features of the medical image of the i-th reference object and the gene features of the gene sequence of the i-th reference object through the backbone neural network.

[0086] In step S502 , the i-th reference object is any reference object among the N reference objects.

[0087] Optionally, a trained backbone neural network is used to perform feature extraction, that is, the backbone neural network is used to extract image features of the medical image of the i-th reference object and gene features of the gene sequence of the i-th reference object.

[0088] Step S503, taking the image feature of the medical image of the i-th reference object as the first reference feature;

[0089] Step S504: taking the gene feature of the gene sequence of the i-th reference object as the second reference feature.

[0090] Step S505: training a target branch network based on the first reference feature, the second reference feature, and a second decoder corresponding to the target data source.

[0091] Optionally, each data source has a corresponding second decoder, that is, N data sources correspond to N second encoders, and different second decoders are isolated from each other.

[0092] Optionally, the first reference feature and the second reference feature are used to train a second decoder corresponding to the target data source to obtain a target branch network corresponding to the target data source.

[0093] Optionally, the second training unit includes: a target reference feature determination subunit, a second decoder training subunit, and a target branch network determination subunit. The target reference feature determination subunit is used to fuse the first reference feature and the second reference feature into a target reference feature through a trunk neural network; the second decoder training subunit is used to iteratively train the second decoder based on the target reference feature until the similarity between the organ change information generated by the second decoder based on the target reference feature and the actual organ change information of the i-th reference object is greater than a preset similarity; the target branch network determination subunit is used to use the trained second decoder as the target branch network.

[0094] Optionally, the organ change information determination device can use feature point multiplication or feature concatenation through a backbone neural network to fuse the first reference feature and the second reference feature into a target reference feature, and then iteratively train the second decoder based on the target reference feature, wherein the training convergence condition of the second decoder is that the second decoder can generate organ change information that is sufficiently similar to the actual organ change information of the i-th reference object based on the target reference feature (that is, the similarity between the organ change information generated by the second decoder based on the target reference feature and the actual organ change information of the i-th reference object is greater than a preset similarity).

[0095] Optionally, the target reference feature determination subunit includes: a collection interval duration acquisition module, used to obtain the collection interval duration of the historical gene sequence of each reference object in N reference objects, and obtain N collection interval durations, wherein the N collection interval durations correspond one-to-one to the N reference objects; a third calculation module, used to calculate the difference between the maximum value and the minimum value of the N collection interval durations; a fourth calculation module, used to calculate the ratio of the collection interval duration of the historical gene sequence of the i-th reference object to the difference, and obtain the time variation parameter of the i-th reference object; a target reference feature determination module, used to merge the first reference feature and the second reference feature into a target reference feature according to the time variation parameter.

[0096] Optionally, the time variation parameter T can be calculated using the following formula (3):

[0097] ,

[0098] In formula (3), Characterizes the collection interval of the historical gene sequence of the i-th reference object, Represents the maximum value among N collection intervals. Indicates the minimum value among N collection intervals.

[0099] In an optional embodiment, Figure 6 is an optional overall neural network training flow chart according to an embodiment of the present application, such as Figure 6 As shown, historical medical images are collected to establish a backbone neural network for pre-training; the relevant tumor gene information and SNP gene information reported by the reference object are obtained, the obtained gene information is expressed, and the gene biological pathway is obtained. The gene biological pathway is expanded to obtain a matrix, the rows of the matrix represent the number of genes, the columns are the number of reference objects, and a column of time variation parameters is added.

[0100] Among them, Figure 6 As shown in the figure, the gene feature corresponding to the relevant tumor gene information is the ratio of the number of bases in the tumor-related gene to the number of bases at the same position in the reference gene sequence (referring to the same position as the tumor-related gene); the gene feature corresponding to the SNP gene information is the ratio of the number of bases in the SNP gene of the reference object to the number of bases at the same position in the reference gene sequence (referring to the same position as the SNP gene). The time variation parameter refers to the above formula (3).

[0101] Optionally, the organ change information determination device can also calculate the area of ​​the gene biological pathway, and at the same time add an attention mechanism so that each pathway where genes communicate with each other has a weight distribution.

[0102] Optionally, Figure 7 This is a schematic diagram of an optional gene biology pathway according to an embodiment of the present application, wherein h represents the gene in each channel, w represents the patient, and on the basis of the gene biology pathway, a time variation parameter may also be added.

[0103] Optionally, the organ change information determination device can perform w and h, and normalize each pathway to maintain information exchange between each gene and form the same specifications as the backbone network.

[0104] Alternatively, if Figure 6 As shown in the figure, the preprocessing adaptation of the backbone neural network includes the steps of introducing the attention mechanism to increase the interactive information between genes, adding time variation parameters, and performing dimensionality reduction and normalization on the gene biological pathway.

[0105] Optionally, after increasing the information interaction between genes, subsequent branch network training and subsequent weight "pruning" of irrelevant genes can be performed to increase the weight of gene biological pathways that become more important over time.

[0106] Alternatively, if Figure 6 As shown, during the training of the overall neural network, the branch network can be decoded and expanded, and the fully connected layer after the expansion is used to predict the label information. Among them, the trained neural network can output interpretable information about the gene biological pathway.

[0107] It can be seen from the above content that compared with the prior art, the present application first determines the model input features based on X medical images and target gene sequences, and then uses the model input features to predict the change information of the target organ of the target object, thereby achieving the technical effect of comprehensively determining the change information of the target organ based on the gene variation and image variation of the target object. In addition, the present application also takes into account the time factor, that is, the model input features are used to characterize the image changes between X medical images and the gene variation of the target object within a preset time period, indicating that the model input features in the present application are time series features, and the change information of the target organ predicted thereby is more in line with the time change trend. Finally, the present application also designs different branch networks corresponding to different data sources, that is, each branch network is trained by the data of the corresponding data source, thereby improving the privacy security of patient data and ensuring that the patient data of each data source is only used locally and will not be leaked.

[0108] According to another aspect of the present application, a method for determining organ change information based on images and genes is also provided, including: obtaining a target gene sequence and X medical images of a target object in a preset time period from a target data source, wherein X is an integer greater than 1; determining a model input feature based on the X medical images and the target gene sequence through a trunk neural network, wherein the model input feature is used to characterize the image changes between the X medical images and the gene mutations of the target object within the preset time period; inputting the model input feature into a target branch network connected to the trunk neural network, and predicting the change information of the target organ of the target object through the target branch network based on the model input feature, wherein different data sources correspond to different branch networks, and the target organ is an organ at risk and / or a target organ involved in the radiotherapy process of the target object.

[0109] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors control the operation of any one of the above-mentioned imaging and gene-based organ change information determination devices.

[0110] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located controls the operation of any one of the above-mentioned image- and gene-based organ change information determination devices.

[0111] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0112] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0117] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A device for determining organ change information based on images and genes, characterized in that: include: An acquisition unit, used to acquire a target gene sequence and X medical images of a target object in a preset time period from a target data source, where X is an integer greater than 1; a determination unit, configured to determine a model input feature through a trunk neural network according to the X medical images and the target gene sequence; a prediction unit, configured to input the model input feature into a target branch network connected to the trunk neural network, and predict the change information of the target organ of the target object through the target branch network according to the model input feature, wherein different data sources correspond to different branch networks, and the target organ is an organ at risk and / or a target organ involved in the radiotherapy process of the target object; The determination unit includes: a first acquisition subunit, used to acquire a reference gene sequence, wherein the reference gene sequence is a gene sequence of an object that does not suffer from cancer; a first determination subunit, used to determine a target gene feature according to the reference gene sequence and a target gene sequence of the target object, wherein the target gene feature is used to characterize the gene variation information of the target gene sequence compared with the reference gene sequence; a second determination subunit, used to determine a target image feature according to the X medical images, wherein the target image feature is used to characterize the image changes of the X medical images within the preset time period; a feature fusion subunit, used to fuse the target image feature and the target gene feature into the model input feature through a backbone neural network; The apparatus for determining organ change information based on images and genes further includes: a first acquisition unit, used to acquire historical medical images and historical gene sequences; a setting unit, used to set a first encoder and a first decoder, wherein the first encoder is used to extract image features of the historical medical images and gene features of the historical gene sequences; the first decoder is used to generate images to be evaluated based on the image features of the historical medical images, and to generate gene sequences to be evaluated based on the gene features of the historical gene sequences; a first training unit, trained to obtain the backbone neural network based on the historical medical images, the historical gene sequences, the first encoder and the first decoder; The target branch network is trained based on the medical imaging features and gene features of the reference object extracted by the trunk neural network and the actual organ change information of the reference object.

2. The device for determining organ change information based on images and genes according to claim 1, characterized in that: The first determining subunit includes: A first determination module is used to determine a first type of gene and a second type of gene from the target gene sequence, wherein the first type of gene is a gene known to cause cancer, and the second type of gene is a gene with genetic variation at a single nucleotide position; A first calculation module is used to calculate the ratio of the number of bases of the first type of gene to the number of bases of the third type of gene in the reference gene sequence to obtain a first type of gene feature, wherein the position of the third type of gene in the reference gene sequence is the same as the position of the first type of gene in the target gene sequence; a second calculation module, configured to calculate a ratio of the number of bases of the second gene to the number of bases of the fourth gene in the reference gene sequence, to obtain a second gene feature, wherein the position of the fourth gene in the reference gene sequence is the same as the position of the second gene in the target gene sequence; The first processing module is used to use the first type of gene features and the second type of gene features as the target gene features.

3. The device for determining organ change information based on images and genes according to claim 1, characterized in that: The feature fusion subunit includes: A second processing module, for representing the target gene feature as a gene pathway; A third processing module is used to perform normalization processing on the gene pathway to obtain a target gene pathway, wherein the specification of the target gene pathway is the same as the pathway specification of the backbone neural network; The fourth processing module is used to fuse the target gene pathway and the target image feature into the model input feature through feature concatenation or feature point multiplication based on the time feature of the preset time period.

4. The device for determining organ change information based on images and genes according to claim 1, characterized in that: The first training unit comprises: a training subunit, configured to iteratively train the first encoder based on the historical medical image and the historical gene sequence until the similarity between the image to be evaluated generated by the first decoder based on the image features extracted by the first encoder and the historical medical image is greater than a first threshold, and the similarity between the gene sequence to be evaluated generated by the first decoder based on the gene features extracted by the first encoder and the historical gene sequence is greater than a second threshold; The backbone neural network determines a subunit, which is used to form the backbone neural network with the trained first encoder and the second encoder, wherein the second encoder is used to perform feature fusion on the image features and gene features output by the first encoder.

5. The device for determining organ change information based on images and genes according to claim 1, characterized in that: The device for determining organ change information based on images and genes also includes: A second acquisition unit is used to acquire medical images, gene sequences, and actual organ change information of each reference object among the N reference objects from the target data source, where N is an integer greater than 1; a feature extraction unit, configured to extract image features of a medical image of an i-th reference object and gene features of a gene sequence of the i-th reference object through the backbone neural network, wherein the i-th reference object is any one of the N reference objects; A first processing unit, configured to use the image feature of the medical image of the i-th reference object as a first reference feature; A second processing unit, configured to use the gene feature of the gene sequence of the i-th reference object as a second reference feature; The second training unit is used to train the target branch network according to the first reference feature, the second reference feature and a second decoder corresponding to the target data source.

6. The device for determining organ change information based on images and genes according to claim 5, characterized in that: The second training unit comprises: a target reference feature determination subunit, configured to fuse the first reference feature and the second reference feature into a target reference feature through the backbone neural network; A second decoder training subunit is used to iteratively train the second decoder according to the target reference feature until the similarity between the organ change information generated by the second decoder according to the target reference feature and the actual organ change information of the i-th reference object is greater than a preset similarity; The target branch network determination subunit is used to use the trained second decoder as the target branch network.

7. The device for determining organ change information based on images and genes according to claim 6, characterized in that: The target reference feature determination subunit includes: A collection interval duration acquisition module is used to obtain the collection interval duration of the historical gene sequence of each reference object in the N reference objects, and obtain N collection interval durations, wherein the N collection interval durations correspond to the N reference objects one by one; A third calculation module is used to calculate the difference between the maximum value and the minimum value of the N collection interval durations; A fourth calculation module is used to calculate the ratio of the collection interval of the historical gene sequence of the i-th reference object to the difference, so as to obtain the time variation parameter of the i-th reference object; A target reference feature determination module is used to fuse the first reference feature and the second reference feature into the target reference feature according to the time variation parameter.

8. A method for determining organ change information based on images and genes, characterized in that: include: Obtaining a target gene sequence and X medical images of a target subject in a preset time period from a target data source, where X is an integer greater than 1; Determining a model input feature based on the X medical images and the target gene sequence through a backbone neural network, wherein the model input feature is used to characterize image changes between the X medical images and gene mutations of the target object within the preset time period; Inputting the model input features into a target branch network connected to the trunk neural network, and predicting the change information of the target organ of the target object according to the model input features through the target branch network, wherein different data sources correspond to different branch networks, and the target organ is an organ at risk and / or a target organ involved in the radiotherapy process of the target object; Wherein, determining the model input features according to the X medical images and the target gene sequence through the backbone neural network includes: obtaining a reference gene sequence, wherein the reference gene sequence is a gene sequence of an object that does not suffer from cancer; determining a target gene feature according to the reference gene sequence and the target gene sequence of the target object, wherein the target gene feature is used to characterize the gene variation information of the target gene sequence compared with the reference gene sequence; determining a target image feature according to the X medical images, wherein the target image feature is used to characterize the image changes of the X medical images within the preset time period; fusing the target image feature and the target gene feature into the model input feature through the backbone neural network; The training process of the backbone neural network includes: obtaining historical medical images and historical gene sequences; setting a first encoder and a first decoder, wherein the first encoder is used to extract image features of the historical medical images and gene features of the historical gene sequences; the first decoder is used to generate images to be evaluated based on the image features of the historical medical images, and to generate gene sequences to be evaluated based on the gene features of the historical gene sequences; the backbone neural network is trained based on the historical medical images, the historical gene sequences, the first encoder and the first decoder; The target branch network is trained based on the medical imaging features and gene features of the reference object extracted by the trunk neural network and the actual organ change information of the reference object.

9. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors control the operation of the image and gene-based organ change information determination device as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program runs, the device where the computer-readable storage medium is located controls the operation of the image- and gene-based organ change information determination device as described in any one of claims 1 to 7.

Images