Abnormal tissue growth prediction method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides an abnormal tissue growth prediction method, device, electronic equipment and storage medium, the method comprising: obtaining an abnormal tissue growth prediction model; obtaining a historical sample sequence and a target growth duration, the historical sample sequence being arranged in chronological order by M historical samples obtained by examining abnormal tissues; performing a time sequence feature extraction operation; inputting the time sequence features of the historical sample sequence and the Mth historical sample with added noise data into a generative model, outputting post-growth image noise and post-growth abnormal tissue segmentation results; based on a preset denoising formula, using the Mth historical sample with added noise data and post-growth image noise to obtain post-growth image prediction results. In this way, the recurrent neural network for extracting time dimension information is embedded into the generative model for extracting spatial dimension information, improving the performance of the model and making the prediction results of abnormal tissue growth more accurate.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of image processing technology, and more specifically to methods, apparatus, electronic devices, and storage media for predicting abnormal tissue growth. Background Technology

[0002] In the early clinical screening of diseases caused by abnormalities in human tissues, regular follow-up of patients is of great significance for improving the accuracy of screening. In particular, by conducting in-depth analysis and calculation of medical imaging data (e.g., computed tomography scans) during the follow-up process, it is possible to automatically predict the growth patterns of individual abnormal tissues.

[0003] Currently, discriminative models are mostly used to predict abnormal tissue growth. These models can predict pixel-level segmentation results after the growth of a single abnormal tissue, as well as the resulting local image. However, in practical applications, discriminative models have the following problems: for clearly grown nodules, the predicted segmentation results are often too small; the predicted local image results are often too close to the image before growth, making them difficult to distinguish; furthermore, some areas in the local image results exhibit texture distortion. Summary of the Invention

[0004] Embodiments of this disclosure provide methods, apparatus, electronic devices, and storage media for predicting abnormal tissue growth.

[0005] In a first aspect, embodiments of this disclosure provide a method for predicting abnormal tissue growth, the method comprising:

[0006] A prediction model for abnormal tissue growth is obtained, which includes a temporal feature extraction network and a generative model.

[0007] The historical sample sequence and target growth duration are obtained. The historical sample sequence is composed of M historical samples obtained from the inspection of abnormal tissue, arranged in chronological order of inspection time. M is a natural number. The historical samples include the inspection images and the corresponding abnormal tissue segmentation results.

[0008] Perform the following temporal feature extraction operation: According to the order of historical samples in the above historical sample sequence, input the above M historical samples and the check time interval between the corresponding historical sample and the next historical sample into the above temporal feature extraction network in sequence, and output the temporal features of the above historical sample sequence, wherein the check time interval between the Mth historical sample and the next historical sample is the above target growth duration.

[0009] The temporal features of the above historical sample sequence and the Mth historical sample with added noise data are input into the above generative model, and the results of image noise and abnormal tissue segmentation after growth are output.

[0010] Based on the preset denoising formula, the Mth historical sample of the above-mentioned noise-added data and the above-mentioned noise in the grown image are used to obtain the prediction result of the grown image.

[0011] In some optional implementations, the temporal feature extraction network includes a temporal convolutional neural network and an attention extraction network. The network sequentially inputs the M historical samples and the check time interval between each historical sample and the next historical sample according to their order in the historical sample sequence, outputting the temporal features of the historical sample sequence, including:

[0012] The first historical sample in the above historical sample sequence and the inspection time interval between the first historical sample and the second historical sample are input into the above temporal feature extraction network. The temporal convolutional neural network and the attention extraction network in the above temporal feature extraction network output the corresponding temporal features respectively.

[0013] For a positive integer i between 2 and M, the following temporal feature update operations are performed sequentially from 2 to M: the i-th historical sample and the check time interval between the i-th historical sample and the (i+1)-th historical sample are input into the temporal feature extraction network. Each temporal convolutional neural network and attention extraction network are combined with the temporal features obtained after inputting the (i-1)-th historical sample into the corresponding network to update the temporal features of the corresponding network.

[0014] In some optional implementations, the generative model described above is a diffusion model, including a temporal convolutional neural network and an attention extraction network. The temporal features of the historical sample sequence and the Mth historical sample with added noise data are input into the generative model, and the output is the result of image noise and abnormal tissue segmentation after growth, including:

[0015] The Mth historical sample with added noise data is input into the input layer of the generative model.

[0016] For each temporal convolutional neural network and attention extraction network in the above temporal feature extraction network, the following corresponding input operation is performed: the temporal convolutional neural network or attention extraction network corresponding to the above generative model is determined as the matching neural network; the temporal features output by the temporal convolutional neural network or attention extraction network corresponding to the above historical sample sequence are input into the above matching neural network;

[0017] The output layer of the above generative model outputs the noise of the grown image and the segmentation results of abnormal tissues after growth.

[0018] In some optional implementations, the above method further includes:

[0019] Replace the examination image in the Mth historical sample with the above-mentioned post-growth image prediction results, and return to perform the above-mentioned temporal feature extraction operation.

[0020] In some optional implementations, the aforementioned Mth historical sample with added noise data includes the examination image with added noise data and the abnormal tissue segmentation result with added noise data; and

[0021] The Mth historical sample of the above-mentioned noise-added data is obtained through the following noise-adding operation: determine the current time step; add noise to the examination image and abnormal tissue segmentation result of the above-mentioned Mth historical sample according to the noise-adding formula of the current time step, so as to obtain the examination image of the above-mentioned noise-added data and the abnormal tissue segmentation result of the above-mentioned noise-added data.

[0022] In some optional implementations, the above-mentioned abnormal tissue growth prediction model is trained through the following training operations:

[0023] Multiple training sample sequences are obtained. The training sample sequence consists of N training samples obtained by inspecting the parts of the same object with abnormal tissue at different times, arranged in chronological order of inspection time. N is a positive integer not less than 3. The training samples include the inspection images and the corresponding abnormal tissue segmentation results.

[0024] Obtain the abnormal tissue growth prediction model before training. The above-mentioned abnormal tissue growth prediction model before training includes the temporal feature extraction network and the generative model before training.

[0025] For each training sample sequence, the following forward propagation operation is performed: the first to N-1 training samples in the training sample sequence and the check time interval between the corresponding training sample and the next training sample are sequentially input into the temporal feature extraction network before training to obtain the temporal features of the training sample sequence; the temporal features of the training samples and the N-1 training sample with added noise data are input into the generative model before training to obtain the image noise after training sample growth and the abnormal tissue segmentation result after training sample growth; based on the difference between the image noise after training sample growth and the noise added to the N-1 training sample, and the difference between the abnormal tissue segmentation result after training sample growth and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence, the forward propagation loss function of the training sample sequence is determined;

[0026] Based on the forward propagation loss function of the above training sample sequences, the parameters of the abnormal tissue growth prediction model before training are adjusted to obtain the abnormal tissue growth prediction model.

[0027] In some optional implementations, determining the forward propagation loss function of the training sample sequence based on the difference between the noise in the image after training growth and the noise added to the (N-1)th training sample, and the difference between the abnormal tissue segmentation result after training growth and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence, includes:

[0028] Using the mean squared error function, calculate the first loss function between the noise in the image after the above training growth and the noise added to the inspection image of the N-1th training sample.

[0029] Using the cross-entropy loss function and / or the Dice loss function, calculate the second loss function between the abnormal tissue segmentation result after the above training growth and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence;

[0030] Based on the first loss function and the second loss function described above, the forward propagation loss function of the training sample sequence is calculated.

[0031] In some optional implementations, the above-mentioned multiple training sample sequences are obtained through the following sample generation steps:

[0032] Obtain original examination information of at least one object, wherein each original examination information includes multiple sets of original examination data obtained from physical examinations of the same object at different times, and the original examination data includes original examination images, abnormal tissue segmentation results corresponding to the original examination images, and coordinates of at least one abnormal tissue center point.

[0033] For multiple sets of raw inspection data in each raw inspection information, perform the following training sample sequence generation operation:

[0034] The original examination images and corresponding abnormal tissue segmentation results in the multiple sets of original examination data are resampled respectively, so that the resampled original examination images and resampled abnormal tissue segmentation results in each set of original examination data have the same resolution, and the resampled abnormal tissue center point coordinates corresponding to each marked abnormal tissue center point coordinate in each set of original examination data are determined according to the resampled resolution.

[0035] Based on each resampled examination image and the corresponding resampled abnormal tissue segmentation results, the coordinates of the center points of each of the above-mentioned resampled abnormal tissues are globally rigidly registered to obtain the corresponding registered resampled abnormal tissue center point identifiers.

[0036] For each of the above-mentioned registration and resampling abnormal tissue center point identifiers, according to the inspection time sequence of each original inspection data, with the coordinates of the resampling abnormal tissue center point corresponding to the registration and resampling abnormal tissue center point identifier in each group of original inspection data as the center, the image and segmentation result of a preset size are extracted from the resampling image and resampling abnormal tissue segmentation result corresponding to each group of original inspection data, respectively, and used as the inspection image and the corresponding abnormal tissue segmentation result to generate training samples and add them to the training sample sequence corresponding to the registration and resampling abnormal tissue center point identifier.

[0037] Secondly, embodiments of this disclosure provide an abnormal tissue growth prediction device, the device comprising:

[0038] The model acquisition module is used to acquire an abnormal tissue growth prediction model, which includes a temporal feature extraction network and a generative model.

[0039] The input acquisition module is used to acquire historical sample sequences and target growth duration. The historical sample sequence is composed of M historical samples obtained from the inspection of abnormal tissues, arranged in chronological order of inspection time, where M is a natural number. The historical samples include inspection images and corresponding abnormal tissue segmentation results.

[0040] The temporal feature extraction module is used to perform the following temporal feature extraction operation: according to the order of the historical samples in the above historical sample sequence, the above M historical samples and the check time interval between the corresponding historical sample and the next historical sample are sequentially input into the above temporal feature extraction network, and the temporal features of the above historical sample sequence are output, wherein the check time interval between the Mth historical sample and the next historical sample is the above target growth duration.

[0041] The generative module is used to input the temporal features of the above historical sample sequence and the Mth historical sample with added noise data into the above generative model, and output the noise of the grown image and the segmentation results of abnormal tissue after growth.

[0042] The denoising module is used to obtain the prediction result of the grown image based on a preset denoising formula, using the Mth historical sample of the above-mentioned noise-added data and the noise of the grown image.

[0043] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect.

[0044] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any implementation of the first aspect.

[0045] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method described in any of the implementations of the first aspect.

[0046] To improve the accuracy of abnormal tissue growth prediction, the abnormal tissue growth prediction method, apparatus, electronic device, and storage medium provided in the embodiments of this disclosure first acquire an abnormal tissue growth prediction model, which includes a temporal feature extraction network and a generative model; then, acquire a historical sample sequence and a target growth duration, wherein the historical sample sequence is composed of M historical samples obtained from the inspection of abnormal tissue, arranged in chronological order of inspection time, where M is a natural number, and the historical samples include inspection images and corresponding abnormal tissue segmentation results; and then perform the following temporal feature extraction operation: according to the order of the historical samples in the above historical sample sequence, ... The M historical samples and the check interval between the corresponding historical sample and the next historical sample are input into the temporal feature extraction network to output the temporal features of the historical sample sequence, where the check interval between the Mth historical sample and the next historical sample is the target growth duration. Then, the temporal features of the historical sample sequence and the Mth historical sample with added noise data are input into the generative model to output the post-growth image noise and the post-growth abnormal tissue segmentation result. Finally, based on the preset denoising formula, the post-growth image prediction result is obtained using the Mth historical sample with added noise data and the post-growth image noise.

[0047] By embedding a temporal feature neural network that extracts time-dimensional information into a generative model that extracts spatial-dimensional information, the performance of the abnormal tissue growth prediction model is improved, making the prediction results for abnormal tissue growth more accurate. Attached Figure Description

[0048] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. In the drawings:

[0049] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0050] Figure 2This is a flowchart of an embodiment of the abnormal tissue growth prediction method according to the present disclosure;

[0051] Figure 3 This is a schematic diagram of a structure according to an embodiment of the abnormal tissue growth prediction model of this disclosure;

[0052] Figure 4A This is a flowchart of the training operation for training an abnormal tissue growth prediction model according to this disclosure;

[0053] Figure 4B This is a breakdown flowchart of step 401 of the abnormal tissue growth prediction method according to this disclosure;

[0054] Figure 4C This is a breakdown flowchart of step 403 of the abnormal tissue growth prediction method according to this disclosure;

[0055] Figure 5 This is a schematic diagram of a structure of an embodiment of the abnormal tissue growth prediction device according to the present disclosure;

[0056] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0057] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0058] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0059] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the abnormal tissue growth prediction methods, apparatus, electronic devices, and storage media of this disclosure can be applied.

[0060] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0061] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as image processing applications, voice recognition applications, short video social applications, audio and video conferencing applications, live video streaming applications, document editing applications, input method applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0062] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed on the terminal devices listed above. They can be implemented as multiple software programs or software modules (e.g., used to provide abnormal tissue growth prediction services) or as a single software program or software module. No specific limitations are imposed here.

[0063] In some cases, the abnormal tissue growth prediction method provided in this disclosure can be executed by terminal devices 101, 102, and 103, and correspondingly, the abnormal tissue growth prediction device can be set in terminal devices 101, 102, and 103. In this case, the system architecture 100 may not include server 105.

[0064] In some cases, the abnormal tissue growth prediction method provided in this disclosure can be jointly executed by terminal devices 101, 102, and 103 and server 105. For example, the step of "obtaining the abnormal tissue growth prediction model" can be executed by terminal devices 101, 102, and 103, while steps such as "obtaining historical sample sequences and target growth duration" can be executed by server 105. This disclosure does not limit this. Correspondingly, the abnormal tissue growth prediction device can also be respectively set in terminal devices 101, 102, and 103 and server 105.

[0065] In some cases, the abnormal tissue growth prediction method provided in this disclosure can be executed by server 105. Accordingly, the abnormal tissue growth prediction device can also be set in server 105. In this case, the system architecture 100 may not include terminal devices 101, 102, and 103.

[0066] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (for example, used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0067] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0068] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of an abnormal tissue growth prediction method according to the present disclosure, which includes the following steps:

[0069] Step 201: Obtain the abnormal tissue growth prediction model.

[0070] In this embodiment, the abnormal tissue growth prediction model is a pre-trained model, which may include a temporal feature extraction network and generative models. The abnormal tissue growth prediction model is used to predict image noise and abnormal tissue segmentation results after a certain growth duration, based on an examination information sequence consisting of examination images obtained from examining abnormal tissue at different times and corresponding abnormal tissue segmentation results arranged in order from earliest to latest examination time, and the desired growth duration.

[0071] The temporal feature extraction network is used to extract the temporal dimension information of the examination images and corresponding anomalous tissue segmentation results of the same anomalous tissue at different follow-up times. For example, it can be a recurrent convolutional neural network (R-CNN), a convolutional long short-term memory network (ConvLSTM), or a gated recurrent unit (GRU), which can simultaneously learn spatial and / or temporal features in the image sequence. The generative model is used to generate or replicate samples from the dataset, or even generate entirely new samples that are similar to but different from the training data, to extract the spatial dimension information of the examination images and corresponding anomalous tissue segmentation results. It can be a generative model such as a diffusion model.

[0072] Generative models, with their advantages of generating highly realistic images and the ability to generate corresponding images based on constraints, are more suitable for tasks involving the prediction of abnormal tissue growth. Combining these models with temporal feature extraction networks to fully exploit temporal and spatial information can further improve the predictive performance of abnormal tissue growth prediction tasks.

[0073] Here, obtaining the abnormal tissue growth prediction model may include, but is not limited to, obtaining the model structure, model parameters, and corresponding parameter values ​​of the abnormal tissue growth prediction model.

[0074] Step 202: Obtain historical sample sequences and target growth duration.

[0075] In this embodiment, the historical sample sequence is composed of M historical samples obtained from the inspection of abnormal tissue, arranged in chronological order of the inspection time, where M is a natural number.

[0076] For example, a historical sample is generated by performing medical imaging examinations on the same abnormal tissue (e.g., the same lung nodule) of object A (e.g., a patient or object) at different times, and obtaining the corresponding examination images and abnormal tissue segmentation results (e.g., automatically segmented by manual annotation or an abnormal tissue segmentation algorithm, such as various convolutional neural networks or deep neural networks). Arranging one or more historical samples of object A in chronological order of examination time yields the historical sample sequence of object A.

[0077] The examination images can be three-dimensional image data such as CT images, magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET-CT), etc.

[0078] The abnormal tissue segmentation result can be a pixel-level abnormal tissue segmentation result. For example, it can be a three-dimensional mask image, where each voxel in the three-dimensional mask image represents a region of abnormal tissue being examined, and the different values ​​corresponding to the voxel can be used to characterize whether the region is abnormal or not.

[0079] The target growth duration is the time interval between the prediction result generated by the abnormal tissue growth prediction model and the examination time of the latest historical sample in the historical sample sequence (i.e., the Mth historical sample in the historical sample sequence).

[0080] It should be noted that the value of M is not specifically limited here. That is, it can obtain a historical sample sequence including one historical sample, or it can obtain a historical sample sequence including two, three, or any other natural number of historical samples.

[0081] Step 203: Perform time series feature extraction.

[0082] The temporal feature extraction operation can be performed as follows: according to the order of historical samples in the above historical sample sequence, each historical sample in the historical sample sequence and the inspection time interval between the corresponding historical sample and the next historical sample are sequentially input into the temporal feature extraction network in the abnormal tissue growth prediction model obtained in step 201, so as to output the temporal features of the historical sample sequence.

[0083] Here, temporal features can be features in the form of feature maps.

[0084] Since the Mth historical sample in the historical sample sequence has no subsequent historical sample, the check interval between the Mth historical sample and the next historical sample is defined as the target growth duration. That is, according to the check time order, each input to the temporal feature extraction network is a historical sample and the time interval between the check time of that historical sample and the check time of the next historical sample. For the last historical sample, the target growth duration is input synchronously with it. Thus, after the above M inputs, the temporal feature extraction network finally extracts the temporal and spatial features corresponding to the M historical samples in its output temporal features, including the temporal features after the target growth duration has elapsed since the check time of the last historical sample.

[0085] For example, when the target growth period is 80 days, the historical sample sequence includes 3 historical samples, i.e., M=3, and these 3 historical samples are arranged in chronological order of inspection time. The specific inspection times are shown in Table 1 below:

[0086]

[0087] Table 1

[0088] The time interval between the first historical sample and the inspection time of the first historical sample (March 15, 2021) and the inspection time of the second historical sample (May 13, 2021) is 60 days. The time interval between the second historical sample and the inspection time of the second historical sample (May 13, 2021) and the inspection time of the third historical sample (August 10, 2021) is 90 days. The third historical sample and the target growth time of 80 days are input into the temporal feature extraction network to output the temporal features of the historical sample sequence.

[0089] For example, when the target growth period is 80 days and the historical sample sequence includes only one historical sample (i.e., M=1), it is only necessary to input this one historical sample and the target growth period of 80 days into the temporal feature extraction network to output the temporal features of the historical sample sequence.

[0090] As one possible implementation, when the time-series feature extraction network includes multiple convolutional neural networks, after inputting M historical samples and the inspection time interval between the corresponding historical sample and the next historical sample into the time-series feature extraction network, each convolutional neural network can extract the corresponding time-series features, and finally each convolutional neural network in the time-series feature extraction network can output the corresponding time-series features.

[0091] For details, see Figure 3 , Figure 3 This is a schematic diagram of a structure according to an embodiment of the abnormal tissue growth prediction model of this disclosure. Figure 3 As shown, the abnormal tissue growth prediction model 300 includes a temporal feature extraction network 301 and a generative model 302. The temporal feature extraction network 301 can include a temporal convolutional neural network and an attention extraction network, the number and arrangement of which can be set as needed. The temporal convolutional neural network can be a convolutional network with a recurrent neural network (RNN). The attention extraction network can be a QKV attention mechanism similar to the Transformer attention mechanism, used to extract global attention information.

[0092] When M > 1, the first historical sample and the check time interval between the first and second historical samples are first input into the temporal feature extraction network. Each temporal convolutional neural network or attention extraction network incorporates the historical samples and check time intervals into the network through an embedding layer, then extracts temporal features and passes the temporal features to the next historical sample in the time sequence. Each time subsequent historical samples and time intervals are input into the temporal feature extraction network, each temporal convolutional neural network and attention extraction network will combine the temporal features obtained from the previously input historical samples and corresponding time intervals to generate new temporal features, replacing the previous temporal features.

[0093] For example, the temporal feature extraction network includes four sequentially connected networks as shown in Table 2: temporal convolutional neural network N1, temporal convolutional neural network N2, attention extraction network N3, and temporal convolutional neural network N4. First, the first historical sample and the check time interval between the first and second historical samples are input into network N1, which outputs temporal feature a1. Temporal feature a1 is then input into network N2, which outputs temporal feature b1. Temporal feature b1 is input into network N3, which outputs temporal feature c1. Temporal feature c1 is then input into network N4, which outputs temporal feature d1. The final inputs and outputs of the four networks are shown in Table 2.

[0094]

[0095] Table 2

[0096] Then, the second historical sample and the inspection time interval between the second and third historical samples are input into network N1. Network N1 combines the temporal feature a1 output by network N1 last time and outputs a new temporal feature a. 1,2 Temporal characteristics a 1,2 The input is given to network N2, and network N2 combines the temporal feature b1 from its previous output with the new temporal feature b1. 1,2 Temporal characteristics b 12 The input is given to network N3, and network N3 combines the temporal feature c1 from its previous output with the new temporal feature c1. 1,2 Temporal features c 1,2 The input is given to network N4, and network N4 combines the temporal feature d1 from its previous output with the input of the input, and outputs a new temporal feature d. 1,2 The final temporal characteristics of the input data and output of the four networks are shown in Table 3 below:

[0097]

[0098] Table 3

[0099] Similarly, the input data and output data of the four networks are successively input into the temporal feature extraction network along with the third, fourth, and so on, up to the Mth historical sample. Table 4 below shows the results.

[0100]

[0101] Table 4

[0102] Step 204: Input the temporal features of the above historical sample sequence and the Mth historical sample with added noise data into the above generative model, and output the image noise and abnormal tissue segmentation results after growth.

[0103] The temporal features of the historical sample sequence finally obtained in step 203, along with the Mth historical sample with added noise data, are input into the generative model. The generative model can combine the temporal features to predict image noise and segmentation results. Compared with the traditional diffusion model, the generative model in the abnormal tissue growth prediction model has undergone innovative adjustments. It changes the prediction objective of UNet in the diffusion model, utilizing the correlation between the segmentation result prediction and the prediction of local images after growth. By integrating these two tasks, the generative model uses the same network to make joint predictions for them, thereby ensuring that the common information required for both tasks can be effectively extracted by the same network. This collaborative enhancement not only improves the model's ability to complete individual tasks but also optimizes the overall prediction accuracy and efficiency.

[0104] As one possible implementation, the generative model can be a diffusion model comprising a temporal convolutional neural network and an attention extraction network. In this model, the temporal convolutional neural network and attention extraction network correspond one-to-one with the networks in the temporal feature extraction network. See also Figure 3 The diffusion model consists of a downsampling stage (left side) and an upsampling stage (right side). The network structure in the downsampling stage corresponds one-to-one with the structure of the temporal feature extraction network.

[0105] Based on the above, the Mth historical sample with added noise data is first input into the input layer of the generative model. Then, for each temporal convolutional neural network and attention extraction network in the temporal feature extraction network: the temporal convolutional neural network or attention extraction network corresponding to that network in the generative model is first identified as a matching neural network, and then the temporal features output by that network corresponding to the historical sample sequence are input into the matching neural network. Finally, the output layer of the generative model outputs the segmentation results of the grown image noise and the grown abnormal tissue.

[0106] For example, see Figure 3When inputting temporal features into the generative model 302, the temporal features generated by the temporal convolutional neural network 301 in the temporal feature extraction network can be input into the temporal convolutional neural network corresponding to it; the temporal features generated by the attention extraction network in the temporal feature extraction network 301 can be input into the attention extraction network corresponding to it.

[0107] Specifically, in the downsampling stage of the generative model, the temporal convolutional neural network can use a convolutional neural network with an RNN module. It combines the temporal features of the connected network inputs, image information, and / or encoding results to encode the network's output, which is then fed into the next network. The attention extraction network can globally extract attention information, obtain its output, and feed it into the next network. In the upsampling stage of the generative model, each network can decode the input encoding and attention information to obtain its output, which is then fed into the next network.

[0108] Step 205: Based on the preset denoising formula, the Mth historical sample of the above-mentioned noise-added data and the above-mentioned noise in the grown image are used to obtain the prediction result of the grown image.

[0109] The core idea of ​​the generative model in the abnormal tissue growth prediction model is to transform the data into Gaussian noise by progressively adding noise, and then learn an inverse process to reverse this noise addition operation, thereby generating or restoring the data. Therefore, based on the generative model, the corresponding preset denoising formula can be derived. Based on the preset denoising formula, using the Mth historical sample of the noise-added data and the noise in the grown image, the prediction result of the grown image can be obtained.

[0110] For example, the preset denoising formula is:

[0111]

[0112] in, , and The value of can be set according to the preset denoising formula, and is not limited here.

[0113] Substituting the noise from the examination image and the post-growth image in the Mth historical sample into the above formula, the result can be calculated. .

[0114] As a possible implementation, similar to traditional diffusion models, the post-growth image prediction result obtained from a single time step iteration may not be very accurate. Therefore, multiple time step iterations are needed to obtain a more accurate post-growth image prediction result. That is, after step 205, the above method further includes the following step 206:

[0115] Step 206: Replace the examination image in the Mth historical sample with the above-mentioned post-growth image prediction results, and continue to perform the temporal feature extraction operation.

[0116] That is, the examination image in the Mth historical sample is replaced with the image prediction result after growth, and then the process returns to step 203.

[0117] For example, the historical sample sequence before replacement includes {the first historical sample, the second historical sample, and the third historical sample}. The examination image in the Mth historical sample is replaced with the prediction result of the grown image to obtain the 3'th historical sample. Then, the updated historical sample sequence becomes {the first historical sample, the second historical sample, and the 3'th historical sample}. Then, the updated historical sample sequence is used to return to step 203.

[0118] Executing step 206 once is equivalent to performing one time step iteration. Executing step 206 multiple times can improve the accuracy of the predicted results of the grown image. It should be noted that the iteration can be stopped after executing step 206 n times as needed, and no limit is imposed here.

[0119] In addition, the aforementioned Mth historical sample with added noise data includes the inspection image with added noise data and the abnormal tissue segmentation result with added noise data. When multiple time steps need to be iterated, the aforementioned Mth historical sample with added noise data is obtained through the following noise addition operation: determine the current time step; add noise to the inspection image and the abnormal tissue segmentation result of the aforementioned Mth historical sample according to the noise addition formula of the current time step, respectively, to obtain the aforementioned inspection image with added noise data and the aforementioned abnormal tissue segmentation result with added noise data.

[0120] The abnormal tissue growth prediction method provided in the above embodiments of this disclosure can achieve the following technical effects:

[0121] (1) The temporal feature extraction network for extracting time dimension information is embedded into the generative model for extracting spatial dimension information, which improves the performance of the abnormal tissue growth prediction model and makes the prediction results of abnormal tissue growth more accurate.

[0122] (2) The prediction objectives of the model were adjusted to make full use of the correlation between the segmentation result prediction and the local image prediction after growth, so that the generative model can use the same network to make joint predictions on them, thereby improving the prediction accuracy and efficiency of the model in completing the two tasks.

[0123] (3) By using temporal features to extract spatial and temporal features from the network learning image sequence, the learning efficiency is improved.

[0124] (4) Optionally, by introducing an attention mechanism, the model can capture the relationship between any two elements in the sequence and weight the key elements, which can effectively improve the accuracy of the model prediction.

[0125] (5) Optionally, the accuracy of abnormal tissue growth prediction can be improved by repeatedly iterating over multiple time steps using a generative model.

[0126] See also Figure 4A , Figure 4A The training operation flowchart for training an abnormal tissue growth prediction model according to this disclosure includes the following steps:

[0127] Step 401: Obtain multiple training sample sequences.

[0128] Each training sample sequence consists of N training samples obtained from examining the same object (e.g., a patient or subject) at different times, focusing on areas with abnormal tissue (e.g., a lung with pulmonary nodules), arranged chronologically according to the examination time. N is a positive integer not less than 3. The training samples include examination images and corresponding abnormal tissue segmentation results. Here, the abnormal tissue segmentation results can be obtained through manual annotation or an automatic abnormal tissue segmentation algorithm, or first segmented using an automatic algorithm and then manually annotated. The abnormal tissue segmentation results can be, for example, 3D mask images.

[0129] As one possible implementation, step 401 above may include the following: Figure 4B Steps 4011 to 4012 shown:

[0130] Step 4011: Obtain the original inspection information of at least one object.

[0131] Each medical imaging examination yields a set of raw examination data. Multiple sets of raw examination data from multiple examinations constitute the raw examination information for that object. In other words, each set of raw examination information can include multiple sets of raw examination data obtained from medical imaging examinations of the same object at different times.

[0132] The original examination data includes the original examination images, the corresponding abnormal tissue segmentation results, and the coordinates of at least one abnormal tissue center point.

[0133] The original examination images can be, for example, three-dimensional image data such as CT images, magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET-CT), etc.

[0134] The segmentation results of abnormal tissues corresponding to the original examination images can be pixel-level segmentation results.

[0135] In practice, a single original examination image may contain multiple abnormal tissues. The results of abnormal tissue segmentation can often only distinguish whether a tissue is abnormal but not different abnormal tissues. Therefore, different abnormal tissues can be distinguished by setting a center point for each abnormal tissue. The coordinates of the center point of the abnormal tissue can be calculated by an automated algorithm or manually labeled. That is, the coordinates of the center point of an abnormal tissue are used to represent or correspond to an abnormal tissue.

[0136] As one possible implementation, to improve training performance, the coordinates of the center points of abnormal tissues with excessively small (e.g., less than 4 mm) or excessively large (e.g., greater than 30 mm) 2D diameters in the original examination images can be removed. Alternatively, original examination data with excessively short (e.g., less than 30 days) or excessively long (e.g., greater than 1500 days) examination intervals between adjacent sets of original examination data can be removed to improve model learning performance.

[0137] Step 4012: For each set of original inspection data in the original inspection information, perform the training sample sequence generation operation.

[0138] The training sample sequence generation operation may include the following steps 40121 to 40123:

[0139] Step 40121: Resample the original examination images and corresponding abnormal tissue segmentation results in the multiple sets of original examination data, so that the resampled original examination images and resampled abnormal tissue segmentation results in each set of original examination data have the same resolution, and determine the resampled abnormal tissue center point coordinates corresponding to the coordinates of each marked abnormal tissue center point in each set of original examination data according to the resampled resolution.

[0140] The purpose of resampling is to ensure that all sets of original examination data have the same resolution for subsequent analysis and comparison. This step involves resampling not only the original examination images but also the segmentation results of abnormal tissues. Resampling ensures consistency in spatial scale between the images and segmentation results, facilitating subsequent analysis. Furthermore, based on the resampling resolution, the coordinates of the resampled abnormal tissue center points corresponding to the coordinates of each labeled abnormal tissue center point in each set of original examination data can be determined. This step improves the accuracy of subsequent analysis.

[0141] Step 40122: Based on each resampled examination image and the corresponding resampled abnormal tissue segmentation results, perform global rigid registration of the coordinates of the center points of each of the above-mentioned resampled abnormal tissues to obtain the corresponding registered resampled abnormal tissue center point identifiers.

[0142] Global rigid registration is an image registration method that aligns identical structures in different images by performing operations such as rotation, translation, and scaling. In this step, by performing global rigid registration on the resampled inspection images and anomalous tissue segmentation results, the corresponding registered resampled anomalous tissue center point identifier can be obtained for the coordinates of each resampled anomalous tissue center point. That is, anomalous tissues in inspection images and anomalous tissue segmentation results obtained at different times after resampling are matched to the same registered resampled anomalous tissue center point identifier. The purpose of this step is to ensure the spatial consistency of image data acquired at different time points, providing an accurate reference for subsequent analysis.

[0143] Step 40123: For each of the above-mentioned registration and resampling abnormal tissue center point identifiers, according to the inspection time order of each original inspection data, with the coordinates of the resampling abnormal tissue center point corresponding to the registration and resampling abnormal tissue center point identifier in each group of original inspection data as the center, a preset size image and segmentation result are extracted from the resampling image and resampling abnormal tissue segmentation result corresponding to each group of original inspection data, respectively, and used as the inspection image and the corresponding abnormal tissue segmentation result to generate training samples and add them to the training sample sequence corresponding to the registration and resampling abnormal tissue center point identifier.

[0144] The purpose of this step is to generate training samples. By cropping the image and segmentation results centered on the registered and resampled anomalous tissue center point identifier according to a preset size, it can be ensured that the training samples contain key information about the anomalous tissue. Simultaneously, adding these training samples to the training sample sequence corresponding to the registered and resampled anomalous tissue center point identifier provides sufficient data support for subsequent anomalous tissue growth prediction models. Optionally, considering the common size of anomalous tissue, the preset image size can be a 32*32*32 pixel image block.

[0145] Thus, through steps 4011 to 4012, multiple training sample sequences have been generated.

[0146] As one possible implementation, since the training sample sequence is similar to the historical sample sequence except that it includes no less than 3 samples, when obtaining the historical sample sequence in step 202, a method similar to step 4012 can be used to process multiple sets of original inspection data of the target object to obtain the historical sample sequence corresponding to one of the abnormal tissues. This will not be elaborated here.

[0147] Step 402: Obtain the abnormal tissue growth prediction model before training.

[0148] The structure of the abnormal tissue growth prediction model remains unchanged before and after training; only the model parameter values ​​differ. The abnormal tissue growth prediction model before training may also include a pre-training temporal feature extraction network and a pre-training generative model.

[0149] Step 403: For each training sample sequence, perform the forward propagation operation:

[0150] The forward propagation operation here can include, for example: Figure 4C Steps 4031 to 4033 are shown below:

[0151] Step 4031: The first to N-1 training samples in the training sample sequence and the check time interval between the corresponding training sample and the next training sample are sequentially input into the above-mentioned time-series feature extraction network before training to obtain the training sample time-series features of the training sample sequence.

[0152] For example, when the training sample sequence includes 4 training samples, i.e., N=4, and these 4 training samples are arranged in chronological order of inspection time, as shown in Table 5 below:

[0153]

[0154] Table 5

[0155] The time intervals between the first training sample and the inspection time of the first training sample (March 15, 2020) and the inspection time of the second training sample (May 13, 2020) (60 days), the time intervals between the second training sample and the inspection time of the second training sample (May 13, 2020) and the inspection time of the third training sample (August 10, 2020) (90 days), and the time interval between the third training sample and the inspection time of the third training sample (August 10, 2020) and the inspection time of the fourth training sample (October 5, 2020) (57 days) are sequentially input into the temporal feature extraction network before training, and the training sample temporal features of the training sample sequence are output.

[0156] Specifically, the structure and execution process of the temporal extraction network are similar to those in step 203 above, and will not be repeated here.

[0157] Step 4032: Input the temporal features of the above training samples and the N-1th training sample with added noise data into the generative model before training to obtain the image noise after training sample growth and the abnormal tissue segmentation results after training sample growth.

[0158] This step is similar to step 204, and will not be repeated here.

[0159] Step 4033: Based on the difference between the image noise after the training sample growth and the noise added to the (N-1)th training sample, and the difference between the abnormal tissue segmentation result after the training sample growth and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence, determine the forward propagation loss function of the training sample sequence.

[0160] Traditional diffusion models can only predict the generated image noise. For image noise, the mean squared error loss function is generally used to calculate the difference between the image noise after the training samples are grown and the noise added to the N-1th training sample. However, the mean squared error loss function is not suitable for learning segmentation tasks.

[0161] This embodiment modifies the prediction objective of the UNet in the diffusion model, enabling it to predict both the generated image noise and the segmentation result of the grown abnormal tissue. Furthermore, considering that the mean squared error loss function is unsuitable for segmentation tasks, when calculating the difference between the segmentation result of the grown abnormal tissue on the training sample and the segmentation result of the Nth training sample in the sequence, other loss functions such as cross-entropy, weighted cross-entropy, BCELOSS (Binary Cross Entropy), Focal Loss, Dice Loss, IoU Loss, Jaccard coefficients, Tversky coefficients, Lovasz-Softmax Loss, BCE Loss + Dice Loss, and Focal Loss + Dice Loss can be used.

[0162] As one possible implementation, step 4033 can first use the mean squared error function to calculate a first loss function between the noise in the trained grown image and the noise added to the inspection image of the (N-1)th training sample; then use the cross-entropy loss function and / or the Dice loss function to calculate a second loss function between the abnormal tissue segmentation result after the trained grown image and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence; finally, based on the first loss function and the second loss function, calculate the forward propagation loss function of the training sample sequence. For example, the sum of the first loss function and the second loss function can be used as the forward propagation loss function of the training sample sequence.

[0163] In this way, we can utilize the mean squared error loss function of the traditional diffusion model, and at the same time introduce other loss functions, making the training process more comprehensive and refined. It takes into account both the training effect of the noise prediction task and the training effect of the segmentation result prediction task, so that the common information that the network needs to extract for both tasks is extracted by the same network, thus synergistically enhancing the ability to complete both tasks.

[0164] Step 404: Based on the forward propagation loss function of the above multiple training sample sequences, adjust the parameters of the abnormal tissue growth prediction model before training to obtain the abnormal tissue growth prediction model.

[0165] Adjusting the parameters of the abnormal tissue growth prediction model before training for each training sample sequence's forward propagation loss function might reduce training efficiency. Instead, after calculating the forward propagation loss function for multiple training sample sequences, the parameters of the abnormal tissue growth prediction model before training can be adjusted once. After multiple adjustments, the final abnormal tissue growth prediction model can be obtained.

[0166] In summary, UNet is used to predict segmentation results after adding noise during traditional diffusion model training. During testing, Gaussian noise is started and gradually denoised using a specific formula. However, this method is limited by the mean squared error loss function and is not ideal for segmentation tasks.

[0167] To overcome this limitation, the abnormal tissue growth prediction model adds noise to the forward process of the diffusion model, allowing UNet to directly predict the original segmentation results. During the testing phase, the progressive denoising formula was re-derived based on the modifications made during training.

[0168] This innovation allows training for segmentation tasks to utilize loss functions better suited to segmentation, such as cross-entropy and Dice loss functions. This adjustment not only optimizes the training process but also improves performance on segmentation tasks. The abnormal tissue growth prediction model fully leverages the correlation between segmentation result prediction and post-growth local image prediction. By using the same generative model to jointly predict both tasks, the network ensures that it can extract the common information needed for both tasks. This collaborative enhancement not only improves the model's ability to complete individual tasks but also promotes mutual reinforcement between the two tasks, thereby achieving an overall performance improvement.

[0169] Through experiments, 1919 training sample sequences were first obtained from the original examination data of lung nodules. Then, the samples were divided into training set, validation set and test set in a ratio of 7:1:2 for model training, validation and testing.

[0170] The performance of the trained abnormal tissue growth prediction model was tested using two test sets (the first part of the test set was a test set proportionally divided from 1919 training sample sequences, and the second part was a growth dataset extracted from the first part of the test set based on the criterion that the abnormal tissue volume of the last training sample increased by more than 50% compared to the abnormal tissue volume of the penultimate training sample), to measure the network's predictive performance for nodule volume changes.

[0171] For the segmentation results of aberrant tissue after growth, the Dice metric (also known as the Dice coefficient or Sørensen-Dice coefficient, a similarity metric used to evaluate image segmentation quality) is used to measure the similarity between the predicted segmentation result of aberrant tissue after growth and the segmentation result of the actual grown tissue (the last training sample in the training sample sequence). The results of the tests on the first and second part test sets are denoted as dice and dice growth, respectively.

[0172] For the post-growth image prediction results, the union of the segmentation results of each training sample is first used as a mask for the region of interest. Then, the mask is applied to both the image prediction result and the actual post-growth nodule image, and the SSIM (Structural Similarity Index Measure, a metric used to measure the visual similarity between two images) values ​​are calculated. The results of testing on the first and second part test sets are denoted as SSIM and SSIM growth, respectively.

[0173] The specific results are shown in Table 6 below:

[0174]

[0175] Table 6

[0176] The test results in Table 6 show that the trained abnormal tissue growth prediction model performs well in both cases of abnormal tissue growth and obvious growth.

[0177] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an abnormal tissue growth prediction device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0178] like Figure 5 As shown, the abnormal tissue growth prediction device 500 of this embodiment includes: a model acquisition module 501, an input acquisition module 502, a temporal feature extraction module 503, a generative module 504, and a denoising module 505.

[0179] Among them, the model acquisition module 501 is used to acquire an abnormal tissue growth prediction model, which includes a temporal feature extraction network and a generative model.

[0180] The input acquisition module 502 is used to acquire historical sample sequences and target growth time. The historical sample sequence is composed of M historical samples obtained from the inspection of abnormal tissues, arranged in chronological order of inspection time. M is a natural number. The historical samples include inspection images and corresponding abnormal tissue segmentation results.

[0181] The temporal feature extraction module 503 is used to perform the following temporal feature extraction operation: according to the order of the historical samples in the above historical sample sequence, the above M historical samples and the check time interval between the corresponding historical sample and the next historical sample are sequentially input into the above temporal feature extraction network, and the temporal features of the above historical sample sequence are output, wherein the check time interval between the Mth historical sample and the next historical sample is the above target growth duration.

[0182] Generative module 504 is used to input the temporal features of the above historical sample sequence and the above Mth historical sample with added noise data into the above generative model, and output the noise of the grown image and the segmentation results of abnormal tissue after growth.

[0183] The denoising module 505 is used to obtain the prediction result of the grown image based on a preset denoising formula, using the Mth historical sample of the noise-added data and the noise of the grown image.

[0184] In this embodiment, the specific processing of the abnormal tissue growth prediction device 500 and its resulting technical effects can be referred to separately. Figure 2 The relevant descriptions of steps 201, 202, 203, 204 and 205 in the corresponding embodiments will not be repeated here.

[0185] In some optional implementations, the temporal feature extraction network includes a temporal convolutional neural network and an attention extraction network, and the temporal feature extraction module 503 includes:

[0186] The first extraction unit 5031 is used to input the first historical sample in the above-mentioned historical sample sequence and the inspection time interval between the first historical sample and the second historical sample into the above-mentioned temporal feature extraction network. The temporal convolutional neural network and the attention extraction network in the above-mentioned temporal feature extraction network respectively output the corresponding temporal features.

[0187] The subsequent extraction unit 5032 is used to perform the following temporal feature update operations sequentially for positive integers i between 2 and M, in order from 2 to M: input the i-th historical sample and the inspection time interval between the i-th historical sample and the (i+1)-th historical sample into the above temporal feature extraction network, and each temporal convolutional neural network and attention extraction network combine the temporal features obtained after inputting the (i-1)-th historical sample into the corresponding network to update the temporal features of the corresponding network.

[0188] In some optional implementations, the generative model described above is a diffusion model, including a temporal convolutional neural network and an attention extraction network. The generative module 504 includes:

[0189] The first input unit 5041 is used to input the Mth historical sample with added noise data into the input layer of the generative model.

[0190] The second input unit 5042 is used to perform the following corresponding input operations for each temporal convolutional neural network and attention extraction network in the above-mentioned temporal feature extraction network: determine the temporal convolutional neural network or attention extraction network in the above-mentioned generative model that corresponds to the temporal convolutional neural network or attention extraction network as the matching neural network; and input the temporal features output by the temporal convolutional neural network or attention extraction network that correspond to the above-mentioned historical sample sequence into the above-mentioned matching neural network.

[0191] The output unit 5043 is used to output the growth-grown image noise and the segmentation results of the abnormal tissue after growth in the output layer of the above generative model.

[0192] In some alternative embodiments, the device 400 further includes:

[0193] The iteration module 506 is used to replace the examination image in the Mth historical sample with the above-mentioned post-growth image prediction result, and return to perform the above-mentioned temporal feature extraction operation.

[0194] In some optional implementations, the aforementioned Mth historical sample with added noise data includes the examination image with added noise data and the abnormal tissue segmentation result with added noise data; and

[0195] The Mth historical sample of the above-mentioned noise-added data is obtained through the following noise-adding operation: determine the current time step; add noise to the examination image and abnormal tissue segmentation result of the above-mentioned Mth historical sample according to the noise-adding formula of the current time step, so as to obtain the examination image of the above-mentioned noise-added data and the abnormal tissue segmentation result of the above-mentioned noise-added data.

[0196] In some optional implementations, the above-mentioned abnormal tissue growth prediction model is trained through the following training operations:

[0197] Multiple training sample sequences are obtained. The training sample sequence consists of N training samples obtained by inspecting the parts of the same object with abnormal tissue at different times, arranged in chronological order of inspection time. N is a positive integer not less than 3. The training samples include the inspection images and the corresponding abnormal tissue segmentation results.

[0198] Obtain the abnormal tissue growth prediction model before training. The above-mentioned abnormal tissue growth prediction model before training includes the temporal feature extraction network and the generative model before training.

[0199] For each training sample sequence, the following forward propagation operation is performed: the first to N-1 training samples in the training sample sequence and the check time interval between the corresponding training sample and the next training sample are sequentially input into the temporal feature extraction network before training to obtain the temporal features of the training sample sequence; the temporal features of the training samples and the N-1 training sample with added noise data are input into the generative model before training to obtain the image noise after training sample growth and the abnormal tissue segmentation result after training sample growth; based on the difference between the image noise after training sample growth and the noise added to the N-1 training sample, and the difference between the abnormal tissue segmentation result after training sample growth and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence, the forward propagation loss function of the training sample sequence is determined;

[0200] Based on the forward propagation loss function of the above training sample sequences, the parameters of the abnormal tissue growth prediction model before training are adjusted to obtain the abnormal tissue growth prediction model.

[0201] In some optional implementations, determining the forward propagation loss function of the training sample sequence based on the difference between the noise in the image after training growth and the noise added to the (N-1)th training sample, and the difference between the abnormal tissue segmentation result after training growth and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence, includes:

[0202] Using the mean squared error function, calculate the first loss function between the noise in the image after the above training growth and the noise added to the inspection image of the N-1th training sample.

[0203] Using the cross-entropy loss function and / or the Dice loss function, calculate the second loss function between the abnormal tissue segmentation result after the above training growth and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence;

[0204] Based on the first loss function and the second loss function described above, the forward propagation loss function of the training sample sequence is calculated.

[0205] In some optional implementations, the above-mentioned multiple training sample sequences are obtained through the following sample generation steps:

[0206] Obtain original examination information of at least one object, wherein each original examination information includes multiple sets of original examination data obtained from physical examinations of the same object at different times, and the original examination data includes original examination images, abnormal tissue segmentation results corresponding to the original examination images, and coordinates of at least one abnormal tissue center point.

[0207] For multiple sets of raw inspection data in each raw inspection information, perform the following training sample sequence generation operation:

[0208] The original examination images and corresponding abnormal tissue segmentation results in the multiple sets of original examination data are resampled respectively, so that the resampled original examination images and resampled abnormal tissue segmentation results in each set of original examination data have the same resolution, and the resampled abnormal tissue center point coordinates corresponding to each marked abnormal tissue center point coordinate in each set of original examination data are determined according to the resampled resolution.

[0209] Based on each resampled examination image and the corresponding resampled abnormal tissue segmentation results, the coordinates of the center points of each of the above-mentioned resampled abnormal tissues are globally rigidly registered to obtain the corresponding registered resampled abnormal tissue center point identifiers.

[0210] For each of the above-mentioned registration and resampling abnormal tissue center point identifiers, according to the inspection time sequence of each original inspection data, with the coordinates of the resampling abnormal tissue center point corresponding to the registration and resampling abnormal tissue center point identifier in each group of original inspection data as the center, the image and segmentation result of a preset size are extracted from the resampling image and resampling abnormal tissue segmentation result corresponding to each group of original inspection data, respectively, and used as the inspection image and the corresponding abnormal tissue segmentation result to generate training samples and add them to the training sample sequence corresponding to the registration and resampling abnormal tissue center point identifier.

[0211] It should be noted that the implementation details and technical effects of each module and unit in the abnormal tissue growth prediction device provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0212] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing the electronic device of the present disclosure. Figure 6 The computer system 600 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0213] like Figure 6 As shown, the computer system 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0214] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows computer system 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 A computer system 600 with various electronic devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0215] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by a processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0216] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0217] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0218] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following functions: Figure 2 The embodiments shown and their alternative implementations illustrate a method for predicting abnormal tissue growth.

[0219] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0220] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0221] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily limit the specific unit; for example, a model acquisition module can also be described as "a module for inputting the temporal features of the historical sample sequence and the Mth historical sample with added noise data into the generative model, and outputting the grown image noise and the grown abnormal tissue segmentation result".

[0222] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method of abnormal tissue growth prediction, characterized by, The method comprises the following steps: obtaining an abnormal tissue growth prediction model, the abnormal tissue growth prediction model comprising a time sequence feature extraction network and a generative model, the abnormal tissue growth prediction model being used to predict image noise and abnormal tissue segmentation results after a growth time based on a check information sequence and the growth time to be predicted, the check information sequence being composed of check images and corresponding abnormal tissue segmentation results obtained by checking abnormal tissues at different times in a sequence from early to late according to check time, the generative model being a diffusion model comprising a time sequence convolutional neural network and an attention extraction network; obtaining a historical sample sequence and a target growth time, the historical sample sequence being arranged in a sequence according to check time by M historical samples obtained by checking abnormal tissues, M being a natural number, the historical sample comprising a check image and a corresponding abnormal tissue segmentation result; performing the following time sequence feature extraction operation: inputting the M historical samples and the check time interval between the corresponding historical sample and the next historical sample in a sequence according to the order of the historical samples in the historical sample sequence into the time sequence feature extraction network, and outputting time sequence features of the historical sample sequence, wherein the check time interval between the Mth historical sample and the next historical sample is the target growth time; inputting the time sequence features of the historical sample sequence and the Mth historical sample with added noise data into the generative model, and outputting post-growth image noise and post-growth abnormal tissue segmentation results; obtaining post-growth image prediction results based on a preset denoising formula using the Mth historical sample with added noise data and the post-growth image noise.

2. The method of claim 1, wherein, The time sequence feature extraction network comprises a time sequence convolutional neural network and an attention extraction network, and the inputting the M historical samples and the check time interval between the corresponding historical sample and the next historical sample in a sequence according to the order of the historical samples in the historical sample sequence into the time sequence feature extraction network and outputting the time sequence features of the historical sample sequence comprises: inputting the first historical sample in the historical sample sequence and the check time interval between the first historical sample and the second historical sample into the time sequence feature extraction network, and outputting corresponding time sequence features by the time sequence convolutional neural network and the attention extraction network in the time sequence feature extraction network; for a positive integer i between 2 and M, performing the following time sequence feature updating operation in a sequence from 2 to M: inputting the i-th historical sample and the check time interval between the i-th historical sample and the i+1-th historical sample into the time sequence feature extraction network, and updating the time sequence features of each time sequence convolutional neural network and attention extraction network in combination with the time sequence features obtained by inputting the i-1-th historical sample into the corresponding network.

3. The method of claim 2, wherein, The inputting the time sequence features of the historical sample sequence and the Mth historical sample with added noise data into the generative model and outputting post-growth image noise and post-growth abnormal tissue segmentation results comprises: inputting the Mth historical sample with added noise data into an input layer of the generative model. For each time series convolutional neural network and attention extraction network in the time series feature extraction network, the following corresponding input operation is performed: determining the time series convolutional neural network or attention extraction network corresponding to the time series convolutional neural network or attention extraction network in the generative model as a matching neural network; inputting the time series feature output by the time series convolutional neural network or attention extraction network and corresponding to the historical sample sequence into the matching neural network; The output layer of the generative model outputs a post-growth image noise and a post-growth abnormal tissue segmentation result.

4. The method of claim 1, wherein, The method further comprises: The method further comprises:

5. The method of claim 4, wherein, The Mth historical sample with added noise data comprises an examination image with added noise data and an abnormal tissue segmentation result with added noise data; and The Mth historical sample with added noise data is obtained through the following noise adding operation: determining a current time step; According to a noise adding formula of the current time step, noise is added to the examination image and the abnormal tissue segmentation result of the Mth historical sample respectively to obtain the examination image with added noise data and the abnormal tissue segmentation result with added noise data.

6. The method of claim 1, wherein, The abnormal tissue growth prediction model is obtained through the following training operation: a plurality of training sample sequences are obtained, wherein each training sample sequence is arranged in chronological order by N training samples obtained by examining the same object at different times and at different positions where abnormal tissues exist, N is a positive integer not less than 3, and each training sample comprises an examination image and a corresponding abnormal tissue segmentation result; an abnormal tissue growth prediction model before training is obtained, which comprises a time series feature extraction network before training and a generative model before training; for each training sample sequence, the following forward propagation operation is performed: inputting the first to the N-1th training samples in the training sample sequence and the examination time interval between the corresponding training sample and the next training sample into the time series feature extraction network before training to obtain training sample time series features of the training sample sequence; inputting the training sample time series features and the N-1th training sample with added noise data into the generative model before training to obtain training sample post-growth image noise and training sample post-growth abnormal tissue segmentation result; determining a forward propagation loss function of the training sample sequence according to the difference between the training sample post-growth image noise and the noise added to the N-1th training sample, and the difference between the training sample post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence; According to the forward propagation loss functions of the plurality of training sample sequences, the parameters of the abnormal tissue growth prediction model before training are adjusted to obtain the abnormal tissue growth prediction model.

7. The method of claim 6, wherein, The difference between the training post-growth image noise and the noise added to the N-1th training sample, and the difference between the training post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence, are used to determine the forward propagation loss function of the training sample sequence, including: The first loss function between the training post-growth image noise and the noise added to the N-1th training sample is calculated using the mean square error function; The second loss function between the training post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result of the Nth training sample in the training sample sequence is calculated using the cross-entropy loss function and / or the Dice loss function; The forward propagation loss function of the training sample sequence is calculated according to the first loss function and the second loss function.

8. The method of claim 6, wherein, The plurality of training sample sequences are obtained through the following sample generation steps: Obtain the original examination information of at least one object, wherein each original examination information includes a plurality of sets of original examination data obtained by physical examination of the same object at different times, and the original examination data includes original examination images and corresponding abnormal tissue segmentation results and at least one abnormal tissue center point coordinate; For each set of original examination data in each original examination information, the following training sample sequence generation operations are performed: Resample the original examination images and corresponding abnormal tissue segmentation results in the plurality of sets of original examination data respectively, so that the resampled original examination images and resampled abnormal tissue segmentation results in each set of resampled original examination data have the same resolution, and the resampled abnormal tissue center point coordinates corresponding to each labeled abnormal tissue center point coordinate in each set of original examination data are determined according to the resampled resolution; Based on each resampled examination image and corresponding resampled abnormal tissue segmentation result, globally rigidly register each resampled abnormal tissue center point coordinate to obtain a corresponding registration resampled abnormal tissue center point identifier; For each registration resampled abnormal tissue center point identifier, according to the examination time sequence of each original examination data, the registration resampled abnormal tissue center point identifier is used to center on the resampled abnormal tissue center point coordinates corresponding to each set of original examination data, and a pre-set size of image and segmentation result is respectively cut out in the resampled image and resampled abnormal tissue segmentation result corresponding to each set of original examination data as examination image and corresponding abnormal tissue segmentation result generation training sample and added to the training sample sequence corresponding to the registration resampled abnormal tissue center point identifier.

9. An abnormal tissue growth prediction apparatus characterized by comprising: including: The model obtaining module is configured to obtain an abnormal tissue growth prediction model, the abnormal tissue growth prediction model comprising a time sequence feature extraction network and a generative model, the abnormal tissue growth prediction model being configured to predict image noise and abnormal tissue segmentation results after a growth time based on examination information sequences and the growth time, the examination information sequences being composed of examination images and corresponding abnormal tissue segmentation results obtained by examining abnormal tissues at different times in a sequence from early to late according to examination times, and the generative model being a diffusion model comprising a time sequence convolutional neural network and an attention extraction network; The input obtaining module is configured to obtain a historical sample sequence and a target growth time, the historical sample sequence being arranged in a sequence according to examination times of M historical samples obtained by examining abnormal tissues, M being a natural number, and the historical samples comprising examination images and corresponding abnormal tissue segmentation results; The time sequence feature extraction module is configured to perform the following time sequence feature extraction operation: inputting the M historical samples and examination time intervals between corresponding historical samples and next historical samples in a sequence according to the order of the historical samples in the historical sample sequence into the time sequence feature extraction network, and outputting time sequence features of the historical sample sequence, wherein the examination time interval between the Mth historical sample and the next historical sample is the target growth time; The generative module is configured to input the time sequence features of the historical sample sequence and the Mth historical sample with added noise data into the generative model, and output post-growth image noise and post-growth abnormal tissue segmentation results; The denoising module is configured to obtain post-growth image prediction results based on a preset denoising formula, the Mth historical sample with added noise data and the post-growth image noise.

10. An electronic device, comprising: One or more processors; A storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-8. A computer program is stored thereon, wherein the computer program is executed by one or more processors to implement the method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer program / instructions are executed by the processor to implement the method of any one of claims 1-8.

12. A computer program product, characterised in that, ​

Images