Abnormal tissue growth prediction and model training method and related product

Abstract

The invention provides an abnormal tissue growth prediction and model training method and a related product. According to a specific embodiment of the abnormal tissue growth prediction model training method, multiple medical examination images and corresponding abnormal tissue segmentation results are utilized, a time sequence feature extraction network is adopted to learn time sequence features of different levels, and a first layer of time sequence features is utilized to predict a growth probability value of an abnormal tissue; boundary related information of the abnormal tissue is predicted by using the second layer of time sequence characteristics, and a post-growth image and a post-growth segmentation result of the abnormal tissue are predicted by using the last layer of time sequence characteristics, so that the examination results of different examination times are utilized, and the abnormal tissue growth prediction model is constrained through the post-growth boundary related information; the growth probability prediction model can learn related information of the abnormal tissue boundary, so that the growth event of the abnormal tissue can be accurately predicted even if the size of the abnormal tissue slightly changes.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of medical image processing technology, and specifically to abnormal tissue growth prediction and model training methods and related products. Background Art

[0002] In actual medical practice, multiple follow-up medical imaging examinations are often performed on body parts where abnormal tissue (e.g., nodules) is present during early clinical screening in order to facilitate early screening of the patient's health status.

[0003] In order to automatically predict whether abnormal tissue will grow, most existing methods use deep neural networks to automatically predict the growth pattern of a single abnormal tissue based on follow-up medical images to evaluate the future evolution trend of the abnormal tissue.

[0004] However, existing methods using deep neural networks to predict the growth of single abnormal tissue and the results of pixel-level segmentation have the following problems:

[0005] First, when using training samples to train the deep neural network, since most abnormal tissues grow very slowly, the network tends to have a strong bias in predicting that nodules will not grow. This makes it easy for the network to judge that abnormal tissues with small growth amplitudes have not grown, which leads to missed screening in early screening. In addition, the experiment also shows that the segmentation results predicted by the deep neural network for abnormal tissues with obvious growth are often smaller than the actual situation.

[0006] Secondly, the existing pixel-level prediction of abnormal tissue only uses one previous medical image as a basis. However, since clinical guidelines recommend follow-up examinations for patients with abnormal tissue (e.g., patients with lung nodules), medical examination images at multiple time points can usually be obtained for each patient in clinical practice, and these past medical imaging examinations are not utilized. Summary of the invention

[0007] The embodiments of the present disclosure provide abnormal tissue growth prediction and model training methods, devices, electronic devices, storage media and computer program products.

[0008] In a first aspect, an embodiment of the present disclosure provides a method for training an abnormal tissue growth prediction model, the method comprising:

[0009] Acquire a set of inspection result sequences, wherein the inspection result sequence is formed by arranging at least three groups of inspection results obtained by inspecting a part including the same abnormal tissue at different times in a chronological order, and the inspection results include inspection images and corresponding abnormal tissue segmentation results;

[0010] For the inspection result sequence in the inspection result sequence set, perform the following overall loss function calculation operation to obtain the overall loss function value of the inspection result sequence: starting from the first inspection result in the inspection result sequence to the second to last inspection result, perform the following prediction loss function value calculation operation in the order of inspection time from front to back to obtain the prediction loss function value of the inspection result: input the inspection result and the inspection time interval between the inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back to obtain the first timing feature, the second timing feature and the third timing feature respectively; input the first timing feature, the second timing feature and the third timing feature respectively into A growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model are used to obtain the predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval; based on the difference between the predicted growth probability value and the annotated growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted post-growth boundary information and the annotated post-growth boundary information, the difference between the predicted post-growth image and the inspection image in the next inspection result, and the difference between the predicted post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result in the next inspection result, the prediction loss function value of the inspection result is determined; based on the prediction loss function value of each inspection result in the inspection result sequence, the overall loss function value of the inspection result sequence is determined;

[0011] Based on the overall loss function value of each inspection result sequence, parameters of the temporal feature extraction network, the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model are optimized.

[0012] In some optional embodiments, the first timing feature extraction network updates the first timing feature output this time according to the input inspection results and time intervals and the first timing feature output last time, the second timing feature extraction network updates the second timing feature output this time according to the first timing feature input from the first timing feature extraction network and the second timing feature output last time, and the third timing feature extraction network updates the third timing feature output this time according to the second timing feature input from the second timing feature extraction network and the third timing feature output last time.

[0013] In some optional embodiments, the method of determining the prediction loss function value of the inspection result based on the difference between the predicted growth probability value and the labeled growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted boundary information after growth and the labeled boundary information after growth, the difference between the predicted image after growth and the inspection image in the next inspection result, and the difference between the predicted abnormal tissue segmentation result after growth and the abnormal tissue segmentation result in the next inspection result, comprises:

[0014] Determining a growth probability loss function value of the inspection result based on a difference between a predicted growth probability value and a marked growth probability value of the inspection result after a corresponding inspection time interval;

[0015] Determine a post-growth boundary information loss function value of the inspection result based on a difference between the predicted post-growth boundary information and the annotated post-growth boundary information of the inspection result after a corresponding inspection time interval;

[0016] Determine a post-growth image loss function value of the inspection result based on a difference between a predicted post-growth image of the inspection result after a corresponding inspection time interval and an inspection image in a next inspection result;

[0017] Determine a loss function value of the abnormal tissue segmentation result after growth of the inspection result based on a difference between a predicted abnormal tissue segmentation result after growth of the inspection result sequence and an abnormal tissue segmentation result in a next inspection result after a corresponding inspection time interval;

[0018] Based on the growth probability loss function value, the post-growth boundary information loss function value, the post-growth image loss function value and the post-growth abnormal tissue segmentation result loss function value of the inspection result, the prediction loss function value of the inspection result is determined.

[0019] In some optional embodiments, the post-growth boundary information prediction model includes a post-growth abnormal tissue surface prediction model, the post-growth boundary information includes a post-growth abnormal tissue surface segmentation result, the labeled post-growth boundary information includes a labeled post-growth abnormal tissue surface segmentation result, and the post-growth boundary information loss function value includes a post-growth abnormal tissue surface segmentation result loss function value; and

[0020] The first time series feature, the second time series feature and the third time series feature are respectively input into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval, including:

[0021] Inputting the second time series feature into the post-growth abnormal tissue surface prediction model to obtain a predicted post-growth abnormal tissue surface segmentation result of the inspection result after a corresponding inspection time interval; and

[0022] The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes:

[0023] Based on the difference between the predicted post-growth abnormal tissue surface segmentation result and the marked post-growth abnormal tissue surface segmentation result of the inspection result after the corresponding inspection time interval, the abnormal tissue surface segmentation result loss function value of the inspection result is determined.

[0024] In some optional embodiments, the post-growth boundary information prediction model further includes a post-growth abnormal tissue surface coordinate prediction model, the post-growth boundary information further includes a post-growth abnormal tissue surface coordinate set, the labeled post-growth boundary information further includes a labeled post-growth abnormal tissue surface coordinate set, and the post-growth boundary information loss function value further includes a post-growth abnormal tissue surface coordinate set loss function value; and

[0025] The step of inputting the first time series feature, the second time series feature and the third time series feature into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model respectively to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval also includes:

[0026] Inputting the predicted abnormal tissue surface segmentation result after the corresponding inspection time interval of the inspection result into the abnormal tissue surface coordinate prediction model after the growth to obtain the predicted abnormal tissue surface coordinate set after the corresponding inspection time interval of the inspection result; and

[0027] The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes:

[0028] Based on the difference between the predicted post-growth abnormal tissue surface coordinate set and the marked post-growth abnormal tissue surface coordinate set of the inspection result after the corresponding inspection time interval, the post-growth abnormal tissue surface coordinate set loss function value of the inspection result is determined.

[0029] In some optional implementations, the post-growth boundary information prediction model further includes a post-growth distance map prediction model, the post-growth boundary information further includes a post-growth distance map, and the post-growth boundary information loss function value includes a post-growth distance map loss function value; and

[0030] The first time series feature, the second time series feature and the third time series feature are respectively input into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval, including:

[0031] Inputting the second time series feature into the post-growth distance map prediction model to obtain a predicted post-growth distance map of the inspection result after a corresponding inspection time interval; and

[0032] The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes:

[0033] Based on the difference between the predicted post-growth distance map and the annotated post-growth distance map of the inspection result after the corresponding inspection time interval, a post-growth distance map loss function value of the inspection result is determined.

[0034] In a second aspect, an embodiment of the present disclosure provides a method for predicting abnormal tissue growth, the method comprising:

[0035] Obtaining a sequence of inspection results to be predicted and a growth duration to be predicted, wherein the sequence of inspection results to be predicted is composed of at least two groups of inspection results obtained by inspecting a part including the abnormal tissue to be predicted at different times and arranged in chronological order, and the inspection results include inspection images and corresponding abnormal tissue segmentation results;

[0036] Sequentially inputting the first inspection result to the last inspection result in the inspection result sequence to be predicted and the inspection time interval between the corresponding inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back, wherein the inspection time interval between the last inspection result and the next inspection result is the growth duration to be predicted;

[0037] The first time series feature to be predicted is input into the growth probability prediction model to obtain the predicted growth probability value of the abnormal tissue to be predicted after the predicted growth time, wherein the first time series feature to be predicted is the first time series feature finally output by the first time series feature extraction network, wherein the time series feature extraction network and the growth probability prediction model are pre-trained using the method described in any implementation method of the first aspect.

[0038] In some optional embodiments, the method further comprises:

[0039] The third time series feature to be predicted is input into the post-growth image prediction model to obtain the predicted post-growth image of the abnormal tissue to be predicted after the predicted growth time and the predicted post-growth abnormal tissue segmentation result, wherein the third time series feature to be predicted is the third time series feature finally output by the third time series feature extraction network, and the post-growth image prediction model is pre-trained using the method described in any implementation method of the first aspect.

[0040] In some optional embodiments, the method further comprises:

[0041] The second time series feature to be predicted is input into the post-growth boundary information prediction model to obtain the predicted post-growth boundary information of the abnormal tissue to be predicted after the predicted growth period, wherein the second time series feature to be predicted is the second time series feature finally output by the second time series feature extraction network, and the post-growth boundary information prediction model is pre-trained using the method described in any implementation method of the first aspect.

[0042] In a third aspect, an embodiment of the present disclosure provides a device for training an abnormal tissue growth prediction model, the device comprising:

[0043] A first acquisition module is configured to acquire a set of inspection result sequences, wherein the inspection result sequence is composed of at least three groups of inspection results obtained by inspecting a part including the same abnormal tissue at different times and arranged in order of inspection time, and the inspection results include inspection images and corresponding abnormal tissue segmentation results;

[0044] The loss function calculation module is configured to perform the following overall loss function calculation operation on the inspection result sequence in the inspection result sequence set to obtain the overall loss function value of the inspection result sequence: starting from the first inspection result in the inspection result sequence to the second to last inspection result, in the order of inspection time from front to back, perform the following prediction loss function value calculation operation to obtain the prediction loss function value of the inspection result: input the inspection result and the inspection time interval between the inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back to obtain the first timing feature, the second timing feature and the third timing feature respectively; the first timing feature, the second timing feature and the third timing feature are respectively input into the timing feature extraction network ... The sequence features are respectively input into the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model to obtain the predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval; based on the difference between the predicted growth probability value and the annotated growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted post-growth boundary information and the annotated post-growth boundary information, the difference between the predicted post-growth image and the inspection image in the next inspection result, and the difference between the predicted post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result in the next inspection result, the prediction loss function value of the inspection result is determined; based on the prediction loss function value of each inspection result in the inspection result sequence, the overall loss function value of the inspection result sequence is determined;

[0045] The parameter optimization module is configured to optimize the parameters of the temporal feature extraction network, the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model based on the overall loss function value of each inspection result sequence.

[0046] In some optional embodiments, the first timing feature extraction network updates the first timing feature output this time according to the input inspection results and time intervals and the first timing feature output last time, the second timing feature extraction network updates the second timing feature output this time according to the first timing feature input from the first timing feature extraction network and the second timing feature output last time, and the third timing feature extraction network updates the third timing feature output this time according to the second timing feature input from the second timing feature extraction network and the third timing feature output last time.

[0047] In some optional embodiments, the method of determining the prediction loss function value of the inspection result based on the difference between the predicted growth probability value and the labeled growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted boundary information after growth and the labeled boundary information after growth, the difference between the predicted image after growth and the inspection image in the next inspection result, and the difference between the predicted abnormal tissue segmentation result after growth and the abnormal tissue segmentation result in the next inspection result, comprises:

[0048] Determining a growth probability loss function value of the inspection result based on a difference between a predicted growth probability value and a marked growth probability value of the inspection result after a corresponding inspection time interval;

[0049] Determine a post-growth boundary information loss function value of the inspection result based on a difference between the predicted post-growth boundary information and the annotated post-growth boundary information of the inspection result after a corresponding inspection time interval;

[0050] Determine a post-growth image loss function value of the inspection result based on a difference between a predicted post-growth image of the inspection result after a corresponding inspection time interval and an inspection image in a next inspection result;

[0051] Determine a loss function value of the abnormal tissue segmentation result after growth of the inspection result based on a difference between a predicted abnormal tissue segmentation result after growth of the inspection result sequence and an abnormal tissue segmentation result in a next inspection result after a corresponding inspection time interval;

[0052] Based on the growth probability loss function value, the post-growth boundary information loss function value, the post-growth image loss function value and the post-growth abnormal tissue segmentation result loss function value of the inspection result, the prediction loss function value of the inspection result is determined.

[0053] In some optional embodiments, the post-growth boundary information prediction model includes a post-growth abnormal tissue surface prediction model, the post-growth boundary information includes a post-growth abnormal tissue surface segmentation result, the labeled post-growth boundary information includes a labeled post-growth abnormal tissue surface segmentation result, and the post-growth boundary information loss function value includes a post-growth abnormal tissue surface segmentation result loss function value; and

[0054] The first time series feature, the second time series feature and the third time series feature are respectively input into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval, including:

[0055] Inputting the second time series feature into the post-growth abnormal tissue surface prediction model to obtain a predicted post-growth abnormal tissue surface segmentation result of the inspection result after a corresponding inspection time interval; and

[0056] The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes:

[0057] Based on the difference between the predicted post-growth abnormal tissue surface segmentation result and the marked post-growth abnormal tissue surface segmentation result of the inspection result after the corresponding inspection time interval, the abnormal tissue surface segmentation result loss function value of the inspection result is determined.

[0058] In some optional embodiments, the post-growth boundary information prediction model further includes a post-growth abnormal tissue surface coordinate prediction model, the post-growth boundary information further includes a post-growth abnormal tissue surface coordinate set, the labeled post-growth boundary information further includes a labeled post-growth abnormal tissue surface coordinate set, and the post-growth boundary information loss function value further includes a post-growth abnormal tissue surface coordinate set loss function value; and

[0059] The step of inputting the first time series feature, the second time series feature and the third time series feature into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model respectively to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval also includes:

[0060] Inputting the predicted abnormal tissue surface segmentation result after the corresponding inspection time interval of the inspection result into the abnormal tissue surface coordinate prediction model after the growth to obtain the predicted abnormal tissue surface coordinate set after the corresponding inspection time interval of the inspection result; and

[0061] The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes:

[0062] Based on the difference between the predicted post-growth abnormal tissue surface coordinate set and the marked post-growth abnormal tissue surface coordinate set of the inspection result after the corresponding inspection time interval, the post-growth abnormal tissue surface coordinate set loss function value of the inspection result is determined.

[0063] In some optional implementations, the post-growth boundary information prediction model further includes a post-growth distance map prediction model, the post-growth boundary information further includes a post-growth distance map, and the post-growth boundary information loss function value includes a post-growth distance map loss function value; and

[0064] The first time series feature, the second time series feature and the third time series feature are respectively input into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval, including:

[0065] Inputting the second time series feature into the post-growth distance map prediction model to obtain a predicted post-growth distance map of the inspection result after a corresponding inspection time interval; and

[0066] The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes:

[0067] Based on the difference between the predicted post-growth distance map and the annotated post-growth distance map of the inspection result after the corresponding inspection time interval, a post-growth distance map loss function value of the inspection result is determined.

[0068] In a fourth aspect, an embodiment of the present disclosure provides a device for predicting abnormal tissue growth, the device comprising:

[0069] A second acquisition module is configured to acquire a sequence of inspection results to be predicted and a growth duration to be predicted, wherein the sequence of inspection results to be predicted is composed of at least two groups of inspection results obtained by inspecting the parts including the abnormal tissue to be predicted at different times and arranged in order of inspection time, and the inspection results include inspection images and corresponding abnormal tissue segmentation results;

[0070] The feature extraction module is configured to sequentially input the first inspection result to the last inspection result in the inspection result sequence to be predicted and the inspection time interval between the corresponding inspection result and the next inspection result into a time series feature extraction network formed by connecting the first time series feature extraction network, the second time series feature extraction network and the third time series feature extraction network in series from front to back, wherein the inspection time interval between the last inspection result and the next inspection result is the growth time to be predicted;

[0071] The probability prediction module is configured to input the first time series feature to be predicted into the growth probability prediction model to obtain the predicted growth probability value of the abnormal tissue to be predicted after the predicted growth time, wherein the first time series feature to be predicted is the first time series feature finally output by the first time series feature extraction network, wherein the time series feature extraction network and the growth probability prediction model are pre-trained using the method described in any implementation method of the first aspect.

[0072] In some optional embodiments, the device further comprises:

[0073] The image prediction module is configured to input the third time series feature to be predicted into a post-growth image prediction model to obtain a predicted post-growth image of the abnormal tissue to be predicted after the predicted growth time and a predicted post-growth abnormal tissue segmentation result, wherein the third time series feature to be predicted is the third time series feature finally output by the third time series feature extraction network, and the post-growth image prediction model is pre-trained using the method described in any implementation method of the first aspect.

[0074] In some optional embodiments, the device further comprises:

[0075] The boundary prediction module is configured to input the second time series feature to be predicted into a post-growth boundary information prediction model to obtain the predicted post-growth boundary information of the abnormal tissue to be predicted after the predicted growth period, wherein the second time series feature to be predicted is the second time series feature finally output by the second time series feature extraction network, and the post-growth boundary information prediction model is pre-trained using the method described in any implementation method of the first aspect.

[0076] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect and / or the second aspect.

[0077] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner in the first aspect and / or the second aspect.

[0078] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method described in any implementation manner in the first aspect and / or the second aspect.

[0079] In order to solve the problems that the existing deep neural network method for predicting abnormal tissues may have, that is, abnormal tissues with slower growth rates are easily predicted to not grow, and the abnormal tissue segmentation results are predicted to be too small, the abnormal tissue growth prediction and model training method provided by the embodiments of the present invention obtains a set of inspection result sequences, wherein the inspection result sequence is composed of at least three groups of inspection results obtained by inspecting parts including the same abnormal tissue at different times and arranged in chronological order of inspection time, and the inspection results include inspection images and corresponding abnormal tissue segmentation results; then, for the inspection result sequence in the inspection result sequence set, the following overall loss function calculation operation is performed to obtain the overall loss function value of the inspection result sequence: starting from the first inspection result in the inspection result sequence to the second to last inspection result, in the order of inspection time from front to back, the following prediction loss function value calculation operation is performed to obtain the prediction loss function value of the inspection result: the inspection result and the inspection time interval between the inspection result and the next inspection result are input into the first temporal feature extraction network, the second temporal feature extraction network and the prediction loss function value of the inspection result. A temporal feature extraction network formed by connecting the temporal feature extraction network and the third temporal feature extraction network in series from front to back obtains the first temporal feature, the second temporal feature and the third temporal feature respectively; the first temporal feature, the second temporal feature and the third temporal feature are respectively input into the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model to obtain the predicted growth probability value, the predicted post-growth boundary information and the predicted post-growth image and the predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval; based on the difference between the predicted growth probability value and the annotated growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted post-growth boundary information and the annotated post-growth boundary information, the difference between the predicted post-growth image and the inspection image in the next inspection result, and the difference between the predicted post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result in the next inspection result, the prediction loss function value of the inspection result is determined; based on the prediction loss function value of each inspection result in the inspection result sequence, the overall loss function value of the inspection result sequence is determined. Finally, based on the overall loss function value of each inspection result sequence, the parameters of the temporal feature extraction network, growth probability prediction model, post-growth boundary information prediction model and post-growth image prediction model are optimized.That is, by utilizing multiple medical examination images and corresponding abnormal tissue segmentation results, a temporal feature extraction network is used to learn temporal features at different levels, and the first-level temporal features are used to predict the growth probability value of the abnormal tissue, the second-level temporal features are used to predict the boundary-related information of the abnormal tissue, and the last-level temporal features are used to predict the post-growth image and post-growth segmentation result of the abnormal tissue. This not only utilizes the examination results at different examination times, but also constrains the abnormal tissue growth prediction model through the post-growth boundary-related information, so that the growth probability prediction model can learn the abnormal tissue boundary-related information, and even if the volume of the abnormal tissue changes slightly, the abnormal tissue boundary will change significantly. Therefore, the growth probability prediction model can learn the above-mentioned boundary changes, which is helpful to predict abnormal tissue growth events with small volume changes, while utilizing multiple examination results at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the drawings:

[0081] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0082] Figure 2A is a flow chart of an embodiment of a method for training a model for predicting abnormal tissue growth according to the present disclosure;

[0083] Figure 2B is a decomposed flow chart of one embodiment of step 202 according to the present disclosure;

[0084] Figure 2C is a decomposed flow chart of one embodiment of step 2021 according to the present disclosure;

[0085] Figure 2D is a decomposed flow chart of one embodiment of step 20213 according to the present disclosure;

[0086] Figure 2E is a decomposed flow chart of one embodiment of step 202132 according to the present disclosure;

[0087] Figure 3A is an exemplary structural schematic diagram of an abnormal tissue growth prediction model according to the present disclosure;

[0088] Figure 3B A schematic diagram showing the data flow of two consecutive inputs of data into the abnormal tissue growth prediction model;

[0089] Figure 4is a flow chart of an embodiment of a method for predicting abnormal tissue growth according to the present disclosure;

[0090] Figure 5 A schematic structural diagram of an embodiment of an abnormal tissue growth prediction model training device according to the present disclosure;

[0091] Figure 6 A schematic structural diagram of an embodiment of an abnormal tissue growth prediction device according to the present disclosure;

[0092] Figure 7 A schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0093] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0094] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0095] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the abnormal tissue growth prediction model training and abnormal tissue growth prediction methods, apparatuses, electronic devices, and storage media disclosed herein can be applied.

[0096] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0097] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as abnormal tissue growth prediction model training applications, abnormal tissue growth prediction applications, etc.

[0098] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with information input devices (e.g., keyboard, mouse, touch screen, microphone, camera, etc.) and information output devices (e.g., display screen, speaker, etc.), including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, used to provide abnormal tissue growth prediction model training services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0099] In some cases, the abnormal tissue growth prediction model training method and abnormal tissue growth prediction method provided by the present disclosure may be executed by the terminal devices 101, 102, and 103, and accordingly, the abnormal tissue growth prediction model training method and abnormal tissue growth prediction method apparatus may be set in the terminal devices 101, 102, and 103. In this case, the system architecture 100 may also not include the server 105.

[0100] In some cases, the abnormal tissue growth prediction model training method and abnormal tissue growth prediction method provided by the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining a set of inspection result sequences" can be performed by the terminal devices 101, 102, 103, and the steps of "for the inspection result sequences in the inspection result sequence set, performing the overall loss function calculation operation" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the abnormal tissue growth prediction model training device and the abnormal tissue growth prediction device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0101] In some cases, the abnormal tissue growth prediction model training method and the abnormal tissue growth prediction method provided by the present disclosure can be executed by the server 105, and accordingly the abnormal tissue growth prediction model training device and the abnormal tissue growth prediction device can also be set in the server 105. In this case, the system architecture 100 may not include the terminal devices 101, 102, and 103.

[0102] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0103] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0104] Continue to refer Figure 2A , which shows a process 200 of an embodiment of the abnormal tissue growth prediction model training method according to the present disclosure, the abnormal tissue growth prediction model training method comprises the following steps:

[0105] Step 201: Obtain a set of inspection result sequences.

[0106] Here, the inspection result sequence in the inspection result sequence set is composed of at least three groups of inspection results obtained by inspecting a part including the same abnormal tissue at different times, arranged in chronological order of inspection time, and the inspection results include inspection images and corresponding abnormal tissue segmentation results.

[0107] It should be noted that the abnormal tissues corresponding to different examination result sequences in the examination result sequence set may be at least two different abnormal tissues, and optionally, may be at least two different abnormal tissues of at least two different patients.

[0108] Here, the examination image can be a medical examination image, and optionally can be a three-dimensional medical examination image, for example, it can be three-dimensional image data of CT image, magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET-CT), etc., more specifically, it can be a lung CT image for lung examination.

[0109] The abnormal tissue segmentation result may be a pixel-level abnormal tissue segmentation result. For example, it may be a three-dimensional mask image, in which each voxel represents an area in the inspection site, and the difference in the corresponding value of the voxel may be used to characterize the area as abnormal tissue or non-abnormal tissue. Here, the abnormal tissue may refer to tissue with a specific lesion, such as nodule tissue. More specifically, it may be lung nodule tissue.

[0110] Step 202: For an inspection result sequence in the inspection result sequence set, perform an overall loss function calculation operation to obtain an overall loss function value of the inspection result sequence.

[0111] Here, the overall loss function calculation operation may be performed on one, multiple or all inspection result sequences in the inspection result sequence set obtained in step 201 to obtain the overall loss function value of the inspection result sequence.

[0112] Here, the overall loss function calculation operation may include: Figure 2BThe following steps 2021 and 2022 are shown:

[0113] Step 2021, starting from the first inspection result in the inspection result sequence to the second to last inspection result, in the order of inspection time from front to back, perform the predicted loss function value calculation operation to obtain the predicted loss function value of the inspection result.

[0114] Assume that there are n inspection results in the inspection result sequence, and n can be a positive integer greater than or equal to 3. Then, starting from t=1 until t=n-1, the prediction loss function value calculation operation is performed on the t-th inspection result to obtain the prediction loss function value of the t-th inspection result, and finally obtain the prediction loss function value of each inspection result from the first inspection result to the n-1-th inspection result in the inspection result sequence.

[0115] Specifically, the operation of calculating the predicted loss function value for the t-th inspection result may include the following: Figure 2C The following steps 20211 to 20213 are shown:

[0116] Step 20211, input the inspection result and the inspection time interval between the inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back, and obtain the first timing feature, the second timing feature and the third timing feature respectively.

[0117] That is, the t-th examination result and the examination time interval between the t-th examination result and the t+1-th examination result are input into the time series feature extraction network. The time series feature extraction network is used to extract the examination images and corresponding abnormal tissue segmentation results of the same abnormal tissue at different follow-up times and the time interval between two adjacent follow-ups. The time series feature extraction network is composed of the first time series feature extraction network, the second time series feature extraction network and the third time series feature extraction network connected in series from front to back. It is used to extract time series features at different levels. The features output by the first time series feature extraction network are the first time series features, the features output by the second time series feature extraction network are the second time series features, and the features output by the third time series feature extraction network are the third time series features.

[0118] The first temporal feature extraction network, the second temporal feature extraction network and the third temporal feature extraction network can be various neural networks for extracting temporal features, for example, they can be recurrent convolutional neural networks (R-CNN, Recurrent Convolutional Neural Networks,), long short-term memory (LSTM, Long Short-Term Memory) neural networks, gated recurrent units (GRU Gated Recurrent Unit) and other networks that can simultaneously learn spatial and / or temporal features in image sequences.

[0119] In some optional embodiments, the first time series feature extraction network updates the first time series feature output this time according to the input inspection result and time interval and the first time series feature output last time by the first time series feature extraction network. The second time series feature extraction network updates the second time series feature output this time according to the first time series feature input from the first time series feature extraction network and the second time series feature output last time by the second time series feature extraction network. The third time series feature extraction network updates the third time series feature output this time according to the second time series feature input from the second time series feature extraction network and the third time series feature output last time by the third time series feature extraction network.

[0120] Specifically, for the t-th inspection result, that is, the t-th inspection in the inspection result sequence, the t-th inspection result in the inspection result sequence and the inspection time interval between the t-th inspection and the t+1-th inspection in the inspection result sequence are input together into the first time series feature extraction network. Also synchronously input into the first time series feature extraction network is the first time series feature input into the first time series feature extraction network this time (that is, the t-1-th time if there is one, and not input if there is none) by the last time series feature extraction network. The first time series feature extraction network obtains the first time series feature corresponding to the t-th inspection result through calculation.

[0121] The first time series feature corresponding to the t-th inspection result will be input into the second time series feature extraction network, and also input into the time series feature extraction part in the next time series feature extraction network (if there is a t+1-th time).

[0122] Next, the second time series feature extraction network calculates the second time series feature of this time (i.e., the tth time) based on the first time series feature input from the first time series feature extraction network and the second time series feature input from the second time series feature extraction network last time (if it exists, it is the t-1th time, if it does not exist, it can be omitted) to obtain the second time series feature of this time (i.e., the tth time).

[0123] The second time series feature calculated this time is input to the time series feature extraction part in the second time series feature extraction network for the next time (if there is a t+1 time), and is also input to the time series feature extraction part in the third time series feature extraction network.

[0124] Then, the third time series feature extraction network calculates the third time series feature of this time (i.e., the tth time) based on the second time series feature input from the second time series feature extraction network and the third time series feature input to the third time series feature extraction network last time (i.e., the t-1th time if it exists, and it can be omitted if it does not exist) by the third time series feature extraction network.

[0125] Step 20212, input the first time series feature, the second time series feature and the third time series feature into the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model respectively, to obtain the predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval.

[0126] After obtaining the first time series feature corresponding to the t-th inspection result, the first time series feature will be input into the growth probability prediction model to output the predicted growth probability value of the inspection result after the corresponding inspection time interval.

[0127] Here, the growth probability prediction model is used to characterize the correspondence between the first time series feature output by the first time series feature extraction network and the predicted growth probability value, and the predicted growth probability value is used to characterize the probability of determining that the abnormal tissue has grown or enlarged. The growth probability prediction model can be a variety of models that obtain probability values ​​based on time series features, such as but not limited to fully connected neural networks, multi-layer perceptrons, convolutional neural networks, deep neural networks, logistic regression, naive Bayes classifiers, support vector machines, Bayesian networks, hybrid models, and the like.

[0128] Therefore, by inputting the first time series feature corresponding to the tth examination result into the growth probability prediction model, the probability that the abnormal tissue has grown or enlarged between the tth examination and the t+1th examination can be obtained, and then the degree of further deterioration of the disease between the above two examination times can be determined.

[0129] The applicant has found through research that when predicting abnormal tissue, if only the abnormal tissue segmentation results are predicted, when the volume of the abnormal tissue changes slightly, the label of the abnormal tissue segmentation result changes little before and after growth, and the overlap ratio between the segmentation label after growth and the segmentation label before growth is very high, which is not conducive to the network learning the slight changes in abnormal tissue. However, even when the volume of the abnormal tissue changes slightly, the overlap ratio of the abnormal tissue boundary label after growth and before growth is very small, and the change ratio is relatively high, which is conducive to the network learning the slight changes in the volume of abnormal tissue through the large changes in the abnormal tissue boundary related information. Therefore, the abnormal tissue boundary related information is introduced here to predict the abnormal tissue boundary information.

[0130] Therefore, after obtaining the second time series feature corresponding to the t-th inspection result, the second time series feature will be input into the post-growth boundary information prediction model to output the predicted post-growth boundary information of the inspection result after the corresponding inspection time interval. Here, the post-growth boundary information prediction model is used to characterize the correspondence between the second time series feature output by the second time series feature extraction network and the predicted post-growth boundary information. Therefore, by inputting the second time series feature corresponding to the t-th inspection result into the post-growth boundary information prediction model, it is possible to predict the inspection time corresponding to the next inspection result of the t-th inspection result (i.e., the inspection time of the t+1-th inspection), various information related to the boundary of the abnormal tissue, that is, the predicted post-growth boundary information. Here, the abnormal tissue boundary refers to the boundary between the abnormal tissue and the external non-abnormal tissue.

[0131] After obtaining the third time series feature corresponding to the tth inspection result, the third time series feature will be input into the post-growth image prediction model to output the predicted post-growth image and the predicted post-growth abnormal tissue segmentation result of the inspection time corresponding to the next inspection result of the tth inspection result (i.e., the inspection time of the t+1th inspection).

[0132] Here, the post-growth image prediction model is used to characterize the correspondence between the third temporal feature output by the third temporal feature extraction network and the predicted post-growth image and the predicted post-growth abnormal tissue segmentation result. The post-growth image prediction model can be a variety of models for generating images based on temporal features, such as but not limited to convolutional neural networks (CNNs), variational autoencoders (VAEs), U-Nets, and the like.

[0133] Step 20213, based on the difference between the predicted growth probability value and the marked growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted boundary information after growth and the marked boundary information after growth, the difference between the predicted image after growth and the inspection image in the next inspection result, and the difference between the predicted abnormal tissue segmentation result after growth and the abnormal tissue segmentation result in the next inspection result, determine the prediction loss function value of the inspection result.

[0134] Here, each inspection result may correspond to a labeled growth probability value. Taking the t-th inspection result as an example, the labeled growth probability value of the t-th inspection result is used to characterize the actual probability that the abnormal tissue corresponding to the inspection result sequence grows or grows between the t-th inspection and the t+1-th inspection. Specifically, the t-th inspection result and the t+1-th inspection result may be manually read and labeled, or the abnormal tissue segmentation results in the t-th inspection result and the t+1-th inspection result may be calculated according to the preset abnormal tissue growth event determination rule to determine the labeled growth probability value corresponding to the t-th inspection result. For example, the abnormal tissue segmentation results in the t-th inspection result and the t+1-th inspection result may be first used to determine the abnormal tissue diameter corresponding to each inspection result. If the abnormal tissue diameter corresponding to the t+1-th inspection result is greater than the abnormal tissue diameter corresponding to the penultimate t-th inspection result, and the diameter difference between the two is greater than the preset diameter change threshold (e.g., 1.5 cm), the labeled growth probability value corresponding to the t+1-th inspection result may be set to 1, otherwise it may be set to 0. Here, the preset diameter change threshold may be predetermined based on various medical expertise.

[0135] Each inspection result may also correspond to annotated post-growth boundary information. Taking the t-th inspection result as an example, the annotated post-growth boundary information of the t-th inspection result may be calculated based on the t+1-th inspection result. Since the t-th inspection result is the actual inspection result, the annotated post-growth boundary information calculated based on the t-th inspection result may be used for the actual post-growth boundary situation of the abnormal tissue corresponding to the inspection result sequence at the t+1-th inspection. Specifically, the corresponding boundary information calculation method may be adopted according to the difference in boundary information, and the boundary information may be obtained by calculation based on the abnormal tissue segmentation result in the t-th inspection result.

[0136] In some optional implementations, step 20213 may include: Figure 2D The following steps 202131 to 202135 are shown:

[0137] Step 202131, determining the growth probability loss function value of the inspection result based on the difference between the predicted growth probability value and the marked growth probability value of the inspection result after the corresponding inspection time interval.

[0138] Here, various loss function calculation methods may be used to determine the growth probability loss function value of the inspection result based on the difference between the predicted growth probability value and the marked growth probability value of the inspection result after the corresponding inspection time interval, for example, including but not limited to the absolute value of the difference.

[0139] Step 202132, determining the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval.

[0140] Here, various loss function calculation methods can be used to determine the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the annotated post-growth boundary information of the inspection result after the corresponding inspection time interval. Specifically, the post-growth boundary information loss function value of the inspection result can be determined by using a corresponding loss function calculation method according to the specific form of the post-growth boundary information.

[0141] Optionally, the post-growth boundary information prediction model includes a post-growth abnormal tissue surface prediction model, the post-growth boundary information includes a post-growth abnormal tissue surface segmentation result, the labeled post-growth boundary information includes a labeled post-growth abnormal tissue surface segmentation result, and the post-growth boundary information loss function value includes a post-growth abnormal tissue surface segmentation result loss function value. As an example, it may include but is not limited to a convolutional neural network (CNN), a variational autoencoder (VAE), a U-Net, and the like.

[0142] Here, the abnormal tissue surface segmentation result may be a pixel-level abnormal tissue surface segmentation result, for example, a three-dimensional mask image, each voxel in the three-dimensional mask image represents an area in the inspection site, and the difference in the corresponding value of the voxel can be used to characterize the area as an abnormal tissue surface or a non-abnormal tissue surface.

[0143] In this way, step 20212, inputting the first time series feature, the second time series feature and the third time series feature into the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model respectively, to obtain the predicted growth probability value, the predicted post-growth boundary information, the predicted post-growth image and the predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval, can include: inputting the second time series feature into the post-growth abnormal tissue surface prediction model to obtain the predicted post-growth abnormal tissue surface segmentation result of the inspection result after the corresponding inspection time interval.

[0144] Here, each inspection result may correspond to a segmentation result of the abnormal tissue surface after annotated growth. Taking the t-th inspection result as an example, the segmentation result of the abnormal tissue surface after annotated growth of the t-th inspection result may be obtained by calculating various surface generation algorithms based on the abnormal tissue segmentation result in the t+1-th inspection result in the inspection result sequence. For example, including but not limited to Marching Cubes algorithm, surface reconstruction algorithm based on point cloud, region growing algorithm, active contour model (Snakes model), etc.

[0145] Accordingly, step 202132, based on the difference between the predicted post-growth boundary information and the annotated post-growth boundary information of the inspection result after the corresponding inspection time interval, determines the post-growth boundary information loss function value of the inspection result, which may include: Figure 2E Steps 2021321 shown:

[0146] Step 2021321, based on the difference between the predicted post-growth abnormal tissue surface segmentation result and the marked post-growth abnormal tissue surface segmentation result of the inspection result after the corresponding inspection time interval, determine the loss function value of the post-growth abnormal tissue surface segmentation result of the inspection result.

[0147] Here, various image segmentation result loss function calculation methods can be used to determine the post-growth abnormal tissue surface segmentation result loss function value of the inspection result based on the difference between the predicted post-growth abnormal tissue surface segmentation result and the labeled post-growth abnormal tissue surface segmentation result after the corresponding inspection time interval. As an example, including but not limited to the Dice loss function (Dice Loss) can be used.

[0148] Optionally, the post-growth boundary information prediction model may also include a post-growth abnormal tissue surface coordinate prediction model, the post-growth boundary information may also include a post-growth abnormal tissue surface coordinate set, the labeled post-growth boundary information may also include a labeled post-growth abnormal tissue surface coordinate set, and the post-growth boundary information loss function value may also include a post-growth abnormal tissue surface coordinate set loss function value.

[0149] The post-growth abnormal tissue surface coordinate prediction model may be any model that generates a coordinate set based on an image segmentation result, and may include, but is not limited to, a fully connected neural network (FCN) and the like.

[0150] Here, the post-growth abnormal tissue surface coordinate set is composed of the voxel coordinates of each voxel in the post-growth abnormal tissue surface.

[0151] Thus, in step 20212, after the second time series feature is input into the post-growth abnormal tissue surface prediction model to obtain the predicted post-growth abnormal tissue surface segmentation result of the inspection result after the corresponding inspection time interval, the predicted post-growth abnormal tissue surface segmentation result of the inspection result after the corresponding inspection time interval can also be input into the post-growth abnormal tissue surface coordinate prediction model to obtain the predicted post-growth abnormal tissue surface coordinate set of the inspection result after the corresponding inspection time interval. Accordingly, step 202132, based on the difference between the predicted post-growth boundary information of the inspection result after the corresponding inspection time interval and the annotated post-growth boundary information, determines the post-growth boundary information loss function value of the inspection result, and can also include the following: Figure 2E Steps 2021322 shown:

[0152] Step 2021322, based on the difference between the predicted post-growth abnormal tissue surface coordinate set and the marked post-growth abnormal tissue surface coordinate set of the inspection result after the corresponding inspection time interval, determine the post-growth abnormal tissue surface coordinate set loss function value of the inspection result.

[0153] Here, various loss function calculation methods suitable for calculating the difference between coordinate sets can be used to determine the loss function value of the abnormal tissue surface coordinate set after growth of the inspection result based on the difference between the predicted abnormal tissue surface coordinate set after growth and the marked abnormal tissue surface coordinate set after growth of the inspection result after the corresponding inspection time interval. As an example, it can include but is not limited to mean square error loss, mean absolute error loss, distance-based loss function, probability distribution-based loss function, improved IoU (Intersection over Union) loss function, Hungarian loss function, Dice loss function, etc.

[0154] Optionally, the post-growth boundary information prediction model may further include a post-growth distance map prediction model, the post-growth boundary information may further include a post-growth distance map, and the post-growth boundary information loss function value may include a post-growth distance map loss function value.

[0155] In this way, step 20212, the first time series feature, the second time series feature and the third time series feature are respectively input into the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model to obtain the predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval, which can include: inputting the second time series feature into the post-growth distance map prediction model to obtain the predicted post-growth distance map of the inspection result after the corresponding inspection time interval.

[0156] Here, the distance map after growth can be a pixel-level distance map of the same size as the inspection image in the inspection result, each voxel in the distance map corresponds to a corresponding voxel in the inspection image, and the value of each voxel in the distance map represents the shortest distance between the voxel and the abnormal tissue surface. In other words, the distance map can be considered as an abnormal tissue surface distance map.

[0157] Each inspection result may correspond to a labeled distance map after growth. Taking the t-th inspection result as an example, the labeled distance map after growth corresponding to the t-th inspection result may be based on the abnormal tissue segmentation result in the t+1-th inspection result. First, various surface generation algorithms are used to calculate the abnormal tissue surface segmentation result, and then various distance map generation algorithms are used to generate a distance map based on the abnormal tissue surface segmentation result. For example, including but not limited to Euclidean distance transform, signed distance field (SDF), nearest neighbor search based on point cloud, Chamfer distance transform, etc.

[0158] It should be noted that the post-growth distance map has a more obvious effect on abnormal tissues with unclear boundaries. This is because after the inspection result sequence corresponding to the abnormal tissue with unclear boundaries is input into the abnormal tissue growth prediction model, the abnormal tissue growth prediction model needs to determine the final boundary based on some areas around the unclear boundary. If the post-growth distance map is used for supervision, since each pixel (voxel) point in the post-growth distance map represents the distance between the point and the abnormal tissue boundary, the post-growth image prediction model can learn the mapping relationship between the characteristics of each point in the area around the abnormal tissue boundary and the distance between the abnormal tissue boundary, and can learn the ability to infer the actual abnormal tissue boundary based on the unclear abnormal tissue boundary. If only manual segmentation results or manually extracted boundaries are used for supervision, these labels can only prompt the abnormal tissue growth prediction model whether a certain pixel point is the internal area of ​​the nodule, and cannot prompt how far it is from the boundary. It is equivalent to asking the abnormal tissue growth prediction model to forcibly determine which pixels belong to the abnormal tissue from some relatively similar pixels, which will reduce the learning effect of the abnormal tissue growth prediction model.

[0159] Accordingly, step 202132, based on the difference between the predicted post-growth boundary information and the annotated post-growth boundary information of the inspection result after the corresponding inspection time interval, determines the post-growth boundary information loss function value of the inspection result, which may include: Figure 2E Steps 2021323 shown:

[0160] Step 2021323, based on the difference between the predicted post-growth distance map and the labeled post-growth distance map of the inspection result after the corresponding inspection time interval, determine the post-growth distance map loss function value of the inspection result.

[0161] Here, various distance map loss function calculation methods can be used to determine the post-growth distance map loss function value of the inspection result based on the difference between the predicted post-growth distance map and the labeled post-growth distance map of the inspection result after the corresponding inspection time interval. As an example, methods including but not limited to mean square error loss (MSE), mean absolute error loss (MAE), Euclidean distance loss, Smooth L1 loss, Chamfer distance, signed distance field (SDF) loss, Dice loss function (Dice Loss), etc. can be used.

[0162] After step 202132, the post-growth boundary information loss function value of the inspection result can be obtained, for example, the abnormal tissue surface segmentation result loss function value, the post-growth abnormal tissue surface coordinate set loss function value, and the post-growth distance map loss function value.

[0163] Step 202133, determining the post-growth image loss function value of the inspection result based on the difference between the predicted post-growth image of the inspection result after the corresponding inspection time interval and the inspection image in the next inspection result.

[0164] Here, various loss function calculation methods for determining the difference between three-dimensional images can be used to determine the loss function value of the growth image of the inspection result based on the difference between the predicted growth image of the inspection result after the corresponding inspection time interval and the inspection image in the next inspection result. As an example, a loss function including but not limited to SSIM (Structural Similarity Index) can be used.

[0165] Step 202134, based on the difference between the predicted post-growth abnormal tissue segmentation result of the inspection result sequence after the corresponding inspection time interval and the abnormal tissue segmentation result in the next inspection result, determine the post-growth abnormal tissue segmentation result loss function value of the inspection result.

[0166] Here, various loss function calculation methods for determining the differences between image segmentation results can be used to determine the loss function value of the post-growth abnormal tissue segmentation result of the inspection result based on the difference between the predicted post-growth abnormal tissue segmentation result of the inspection result sequence after the corresponding inspection time interval and the abnormal tissue segmentation result in the next inspection result.

[0167] As an example, a function including but not limited to the Dice Loss function may be used.

[0168] Step 202135, based on the growth probability loss function value of the inspection result, the post-growth boundary information loss function value, the post-growth image loss function value and the post-growth abnormal tissue segmentation result loss function value, determine the prediction loss function value of the inspection result.

[0169] Here, the predicted loss function value of the inspection result can be calculated based on the different loss function calculation methods corresponding to the growth probability loss function value, the post-growth boundary information loss function value, the post-growth image loss function value and the post-growth abnormal tissue segmentation result loss function value of the inspection result, and the above loss function values ​​are combined to calculate the predicted loss function value of the inspection result.

[0170] For example, the growth probability loss function value, the post-growth boundary information loss function value, the post-growth image loss function value, and the post-growth abnormal tissue segmentation result loss function value of the inspection result may be weightedly summed to obtain the predicted loss function value of the inspection result.

[0171] After step 2021, the prediction loss function value of each inspection result in the inspection result sequence can be obtained. Among them, for the t-th inspection result, the prediction loss function value of the t-th inspection result is used to characterize the difference between the prediction information (including the predicted growth probability value, the predicted boundary information after growth, the predicted image after growth, and the predicted abnormal tissue segmentation result after growth) predicted based on various relevant information of the inspection time of the t+1-th inspection result from the first to the t-th inspection results in the inspection result sequence and the various annotation information corresponding to the t+1-th inspection result in the inspection result.

[0172] Step 2022: determine the overall loss function value of the inspection result sequence based on the predicted loss function value of each inspection result in the inspection result sequence.

[0173] Here, various implementations can be used to determine the overall loss function value of the inspection result sequence based on the predicted loss function value of each inspection result in the inspection result sequence. As an example, the average of the predicted loss function values ​​of each inspection result in the inspection result sequence can be used as the overall loss function value of the inspection result sequence.

[0174] After step 202, the overall loss function value of the inspection result sequence can be obtained.

[0175] Step 203 , based on the overall loss function value of each inspection result sequence, the parameters of the temporal feature extraction network, the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model are optimized.

[0176] Here, there is no specific limitation on the parameter optimization method, for example, a stochastic gradient descent method or the like may be used.

[0177] Through steps 201 to 203, parameter optimization of the temporal feature extraction network, the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model can be achieved based on multiple inspection result sequences.

[0178] Reference below Figure 3A and Figure 3B , Figure 3A A specific example of an abnormal tissue growth prediction model is shown. Figure 3B A schematic diagram showing the data flow when two consecutive data are input into the abnormal tissue growth prediction model is shown.

[0179] like Figure 3A As shown, the inspection image and abnormal tissue segmentation result in the tth inspection result (t starts from 1 to the second to last in the inspection result sequence) in the inspection result sequence, as well as the inspection time interval between the tth inspection and the t+1th inspection in the inspection result sequence are input into the first temporal feature extraction network together.

[0180] Then, the first time series feature extraction network calculates the first time series feature corresponding to the t-th inspection result according to the input inspection result and time interval, as well as the time series feature input by the last time series feature extraction network (if there is t-1 time) to the current time series feature extraction network (i.e., the t-th time), and inputs the calculated first time series feature to the second time series feature extraction network, and also inputs it to the time series feature extraction part of the first time series feature extraction network for the next time (if there is t+1 time), and also inputs it to the t-th growth probability prediction model, which outputs the predicted growth probability value of the t+1-th inspection time.

[0181] Next, the second time series feature extraction network calculates the second time series feature corresponding to the t-th inspection result based on the first time series feature corresponding to the t-th inspection result input from the first time series feature extraction network and the second time series feature input from the second time series feature extraction network last time (if there is t-1 time) to the second time series feature extraction network this time (i.e., the t-th time) to obtain the second time series feature corresponding to the t-th inspection result. The calculated second time series feature corresponding to the t-th inspection result is given to the time series feature extraction part of the second time series feature extraction network next time (if there is t+1 time), and is simultaneously input to the t-th third time series feature extraction network, the post-growth abnormal tissue surface prediction model, and the post-growth distance map prediction model.

[0182] Among them, the post-growth distance map prediction model calculates according to the second time series feature corresponding to the t-th inspection result input from the second time series feature extraction network, and outputs the predicted post-growth abnormal tissue surface distance map at the t+1-th inspection time.

[0183] Among them, the post-growth abnormal tissue surface prediction model calculates according to the second time series feature corresponding to the t-th inspection result input from the second time series feature extraction network, and outputs the predicted post-growth abnormal tissue surface segmentation result at the t+1-th inspection time.

[0184] The predicted post-growth abnormal tissue surface segmentation result at the t+1th inspection time is then input into the post-growth abnormal tissue surface coordinate prediction model to obtain the predicted post-growth abnormal tissue surface coordinate set at the t+1th inspection time.

[0185] Among them, the third temporal feature extraction network calculates the third temporal feature corresponding to the t-th inspection result based on the second temporal feature corresponding to the t-th inspection result input from the second temporal feature extraction network and the third temporal feature input to the third temporal feature extraction network this time (i.e., the t-th time) by the third temporal feature extraction network last time (if there is t-1 time), and gives the calculated third temporal feature corresponding to the t-th inspection result to the temporal feature extraction part in the third temporal feature extraction network next time (if there is t+1 time), and inputs it to the grown image prediction model at the same time.

[0186] The post-growth image prediction model is calculated based on the third time series feature corresponding to the t-th inspection result input from the third time series feature extraction network to obtain the predicted post-growth image and predicted post-growth abnormal tissue segmentation result at the t+1-th inspection time.

[0187] After the above operations, the predicted growth probability value of the t+1th inspection time corresponding to the tth inspection result, the predicted abnormal tissue surface distance map after growth, the predicted abnormal tissue surface segmentation result after growth, the predicted abnormal tissue surface coordinate set after growth, the predicted post-growth image and the predicted abnormal tissue segmentation result after growth can be obtained.

[0188] Next, the inspection result of the t+1th time and the inspection time interval between the t+1th inspection and the t+2th inspection can be input into the first time series feature extraction network. In this way, the first time series feature extraction from the first time series feature extraction network can be repeated until the predicted growth probability value of the t+2th inspection time, the predicted abnormal tissue surface distance map after growth, the predicted abnormal tissue surface segmentation result after growth, the predicted abnormal tissue surface coordinate set after growth, the predicted image after growth and the predicted abnormal tissue segmentation result after growth are obtained.

[0189] Continue to repeat the above data input operation until the second to last examination result in the examination result sequence and the examination time interval between the second to last examination and the last examination are input into the first time series feature extraction network to obtain the predicted growth probability value of the last examination time corresponding to the second to last examination result, the predicted abnormal tissue surface distance map after growth, the predicted abnormal tissue surface segmentation result after growth, the predicted abnormal tissue surface coordinate set after growth, the predicted post-growth image and the predicted post-growth abnormal tissue segmentation result.

[0190] That is, after the above-mentioned input from the first inspection result in the inspection result sequence to the second to last inspection result, the predicted growth probability value, predicted abnormal tissue surface distance map after growth, predicted abnormal tissue surface segmentation result after growth, predicted abnormal tissue surface coordinate set after growth, predicted image after growth, and predicted abnormal tissue segmentation result after growth of each inspection result after the corresponding inspection time interval can be obtained. That is, assuming that there are n inspection results in the inspection result sequence, n can be a positive integer greater than or equal to 3. Then, starting from t=1 to t=n-1, the predicted growth probability value, predicted abnormal tissue surface segmentation result after growth, predicted abnormal tissue surface coordinate set after growth, predicted abnormal tissue surface distance map after growth, predicted image after growth, and predicted abnormal tissue segmentation result after growth of the tth inspection result can be obtained.

[0191] Then, by determining the prediction loss function value of each of the above n-1 inspection results according to the difference between the predicted growth probability value and the marked growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted abnormal tissue surface segmentation result after growth and the marked abnormal tissue surface segmentation result after growth, the difference between the predicted abnormal tissue surface coordinate set after growth and the marked abnormal tissue surface coordinate set after growth, the difference between the predicted abnormal tissue surface distance map after growth and the marked abnormal tissue surface distance map after growth, the difference between the predicted image after growth and the inspection image in the next inspection result, and the difference between the predicted abnormal tissue segmentation result after growth and the abnormal tissue segmentation result in the next inspection result.

[0192] Finally, the overall loss function value of the inspection result sequence may be determined based on the predicted loss function value of each inspection result in the inspection result sequence.

[0193] Furthermore, based on the overall loss function value of each inspection result sequence in the inspection result sequence set, parameters of the first time series feature extraction network, the second time series feature extraction network, the third time series feature extraction network, the growth probability prediction model, the post-growth abnormal tissue surface prediction model, the post-growth abnormal tissue surface coordinate prediction model, the post-growth distance map prediction model and the post-growth image prediction model can be optimized.

[0194] The abnormal tissue growth prediction model training method provided by the above-mentioned embodiment of the present disclosure obtains a set of inspection result sequences, wherein the inspection result sequence is composed of at least three groups of inspection results obtained by inspecting a part including the same abnormal tissue at different times and arranged in chronological order of inspection time, and the inspection result includes an inspection image and a corresponding abnormal tissue segmentation result; then, for the inspection result sequence in the inspection result sequence set, the following overall loss function calculation operation is performed to obtain the overall loss function value of the inspection result sequence: starting from the first inspection result in the inspection result sequence to the second to last inspection result, in the order of inspection time from front to back, the following prediction loss function value calculation operation is performed to obtain the prediction loss function value of the inspection result: the inspection result and the inspection time interval between the inspection result and the next inspection result are input into the timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back. The first time series feature extraction network is used to obtain the first time series feature, the second time series feature and the third time series feature respectively; the first time series feature, the second time series feature and the third time series feature are respectively input into the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model to obtain the predicted growth probability value, the predicted post-growth boundary information, the predicted post-growth image and the predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval; based on the difference between the predicted growth probability value and the annotated growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted post-growth boundary information and the annotated post-growth boundary information, the difference between the predicted post-growth image and the inspection image in the next inspection result, and the difference between the predicted post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result in the next inspection result, the prediction loss function value of the inspection result is determined; based on the prediction loss function value of each inspection result in the inspection result sequence, the overall loss function value of the inspection result sequence is determined. Finally, based on the overall loss function value of each inspection result sequence, the parameters of the time series feature extraction network, the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model are optimized.That is, by utilizing multiple medical examination images and corresponding abnormal tissue segmentation results, a temporal feature extraction network is used to learn temporal features at different levels, and the first-level temporal features are used to predict the growth probability value of the abnormal tissue, the second-level temporal features are used to predict the boundary-related information of the abnormal tissue, and the last-level temporal features are used to predict the post-growth image and post-growth segmentation result of the abnormal tissue. This not only utilizes the examination results at different examination times, but also constrains the abnormal tissue growth prediction model through the post-growth boundary-related information, so that the growth probability prediction model can learn the abnormal tissue boundary-related information, and even if the volume of the abnormal tissue changes slightly, the abnormal tissue boundary will change significantly. Therefore, the growth probability prediction model can learn the above-mentioned boundary changes, which is helpful to predict abnormal tissue growth events with small volume changes, while utilizing multiple examination results at the same time.

[0195] Embodiment 1:

[0196] The applicant adopts Figure 3A and Figure 3B The abnormal tissue growth prediction model shown is trained and tested. Specifically:

[0197] Dataset and pre-processing: A follow-up dataset of lung nodules was collected. There were 508 patients in total, and each patient had three or more follow-up CT images and pixel-level segmentation results of all nodules in the images annotated by professional physicians. In this study, each set of three consecutive images was first taken as a group, and then the three CT images in each group were resampled to the same resolution and globally rigidly registered. The corresponding nodules in the image were extracted with the center of the nodule as the center, and the corresponding segmentation results were extracted from the corresponding segmentation results to form a set of follow-up nodule data. Finally, 2603 sets of follow-up nodule data were extracted. The data that were too small (maximum 2D diameter less than 4 mm), too large (maximum 2D diameter greater than 30 mm), or with a follow-up interval that was too short (less than 30 days) or too long (greater than 1500 days) were removed, and finally 1919 groups of follow-up nodule data were obtained (i.e., each group of follow-up data included CT image data of the same lung nodule at three time points arranged in chronological order, the corresponding pixel-level segmentation results, and the acquisition time, and the segmentation result was a binary mask image). They were divided into training, validation, and test sets in a ratio of 7:1:2 for model training, validation, and testing.

[0198] The test set is divided into two parts. The first part of the test set is the test set separated from all lung nodule follow-up data. The second part of the test set is a growth data set extracted from the first part of the test set with the standard that the volume of the last nodule increases by more than 50% compared with the second to last nodule, which is used to measure the model's prediction performance for nodule volume changes.

[0199] For the predicted results of abnormal tissue segmentation after growth, the Dice index is used to measure its similarity with the segmentation results after actual growth (the third nodule in this data set). The test is performed on the first and second test sets, and the results are recorded as dice and dice growth respectively.

[0200] For the prediction results of abnormal tissue images after growth, the union of the segmentation results of the three follow-ups is first taken as the mask of the area of ​​interest, and then the mask is applied to the image prediction results and the real nodule image after growth, and the SSIM values ​​of the two are calculated. The SSIM value is the Structural Similarity Index, which is an indicator used to evaluate image quality. It measures the quality of the image by comparing the similarities of brightness, contrast and structure of two images. The test was performed on the first and second test sets respectively, and the results were recorded as SSIM and SSIM growth respectively.

[0201] The results are shown in Table 1 below:

[0202] Table 1

[0203]

[0204] As can be seen from Table 1, Dice and Dice growth are basically the same, and SSIM and SSIM growth are also basically the same.

[0205] In practice, most nodules do not grow, that is, corresponding to the first part of the test set, most of the lung nodules in the follow-up data of lung nodules do not grow. For the test results of the first part of the test set, both Dice and SSIM performed well, indicating that the abnormal tissue growth prediction model shown in Figure 3 performs well in most cases.

[0206] For the traditional method of abnormal tissue (pulmonary nodule) segmentation based on deep learning model, since the supervised training of the model only involves segmentation learning, and since most of the nodules in the training data do not grow, this will lead to the model's weak ability to predict abnormal tissue growth. The abnormal tissue growth prediction model obtained by the abnormal tissue training method shown in Figure 3, due to the further constraints and learning of the model in three aspects: abnormal tissue surface segmentation results, surface coordinates, and abnormal tissue distance map, the model has the ability to predict abnormal tissue segmentation results and growth at the same time, that is, only used to predict the second part of the test set with more obvious growth, it can still achieve better test results.

[0207] Reference below Figure 4, which shows a process 400 of an embodiment of the abnormal tissue growth prediction method according to the present disclosure. The abnormal tissue growth prediction method comprises the following steps:

[0208] Step 401, obtaining a sequence of inspection results to be predicted and a growth duration to be predicted.

[0209] Here, the sequence of inspection results to be predicted is formed by inspecting the parts including the abnormal tissue to be predicted at different times and arranging them in chronological order. The inspection results may include inspection images and corresponding abnormal tissue segmentation results.

[0210] Here, for the explanation of the examination images and abnormal tissue segmentation results, please refer to Figure 2A The relevant parts of step 201 in the illustrated embodiment will not be described in detail here.

[0211] The growth duration to be predicted is used to indicate how long after the last examination result in the sequence of examination results to be predicted the growth status of the abnormal tissue is expected to be predicted.

[0212] Step 402, sequentially input the first inspection result to the last inspection result in the inspection result sequence to be predicted and the inspection time interval between the corresponding inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back.

[0213] Specifically, assuming that there are n inspection results in the sequence of inspection results to be predicted, here, in order of the inspection time corresponding to the inspection results from front to back, starting from the first inspection result in the sequence of inspection results to be predicted to each inspection result in the last inspection result, taking the tth inspection result as an example, t is a positive integer between 1 and n, the tth inspection result and the inspection time interval between the tth inspection result and the t+1th inspection result (also referred to as the inspection time interval between the tth inspection and the t+1th inspection) are input to the time series feature extraction network. The inspection time interval between the last inspection result and the next inspection result is the growth duration to be predicted. That is, the nth inspection result, the nth inspection and the growth duration to be predicted will be input to the time series feature extraction network for the last time.

[0214] Here, the temporal feature extraction network is adopted Figure 2A The illustrated embodiment and its alternative implementations are pre-trained.

[0215] Step 403, inputting the first time series feature to be predicted into the growth probability prediction model to obtain the predicted growth probability value of the abnormal tissue to be predicted after the growth time to be predicted.

[0216] Here, the first time series feature to be predicted is the first time series feature finally output by the first time series feature extraction network, that is, the first time series feature output by the first time series feature extraction network after the last inspection result in the inspection result sequence to be predicted and the growth duration to be predicted are input into the time series feature extraction network.

[0217] The growth probability prediction model is based on Figure 2A The illustrated embodiment and its alternative implementations are pre-trained.

[0218] It can be understood that, based on the predicted growth probability value obtained in step 403, various further processed information can be output.

[0219] For example, the predicted growth probability value of the abnormal tissue to be predicted after the growth time to be predicted can be directly output.

[0220] For another example, when the predicted growth probability value is greater than a preset growth probability threshold (e.g., 85%), growth prompt information indicating that the abnormal tissue to be predicted will grow after the predicted growth time is output. Conversely, when the predicted growth probability value is not greater than the preset growth probability threshold (e.g., 85%), maintenance prompt information indicating that the abnormal tissue to be predicted will not grow after the predicted growth time is output.

[0221] By outputting the above-mentioned predicted growth probability value, growth prompt information or maintenance prompt information, it can assist doctors in judging the future development trend of the patient's condition so as to give corresponding treatment plans later, such as adjusting the follow-up time and frequency, etc.

[0222] In some optional implementations, the abnormal tissue growth prediction method 400 may further include the following steps 404:

[0223] Step 404 , input the third time series feature to be predicted into the post-growth image prediction model to obtain the predicted post-growth image of the abnormal tissue to be predicted after the predicted growth time and the predicted post-growth abnormal tissue segmentation result.

[0224] Here, the third time series feature to be predicted is the third time series feature finally output by the third time series feature extraction network. The image prediction model after growth can be adopted Figure 2A The illustrated embodiment and its alternative implementations are pre-trained.

[0225] By outputting the predicted growth image of the abnormal tissue to be predicted after the predicted growth time and the segmentation result of the abnormal tissue after the predicted growth, it can help doctors to visually predict the future development of the abnormal tissue, so as to help doctors give corresponding treatment plans later.

[0226] In some optional implementations, the abnormal tissue growth prediction method 400 may further include the following steps 405:

[0227] Step 405 , input the second time series feature to be predicted into the post-growth boundary information prediction model to obtain the predicted post-growth boundary information of the abnormal tissue to be predicted after the growth time to be predicted.

[0228] Here, the second time series feature to be predicted is the second time series feature finally output by the second time series feature extraction network, and the post-growth boundary information prediction model can be adopted Figure 2A The illustrated embodiment and its alternative implementations are pre-trained.

[0229] Here, for detailed information about post-growth boundary information and post-growth boundary information prediction model, please refer to Figure 2A The relevant parts of its optional implementation methods will not be repeated here.

[0230] By outputting the predicted post-growth boundary information of the abnormal tissue to be predicted after the predicted growth time, doctors can be further assisted in predicting the future development of the abnormal tissue, so as to assist doctors in giving corresponding treatment plans later.

[0231] The abnormal tissue growth prediction method provided by the above-mentioned embodiment of the present disclosure can improve the accuracy of abnormal tissue growth prediction by utilizing the sequence of examination results to be predicted from medical examinations conducted at multiple follow-up times and the time period in which the abnormal tissue growth is expected to be predicted.

[0232] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an abnormal tissue growth prediction model training device. Figure 2A Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0233] like Figure 5As shown, the abnormal tissue growth prediction model training device 500 of this embodiment includes: a first acquisition module 501, a loss function calculation module 502 and a parameter optimization module 503. Among them, the first acquisition module 501 is configured to obtain a set of inspection result sequences, wherein the inspection result sequence is composed of at least three groups of inspection results obtained by inspecting a part including the same abnormal tissue at different times and arranged in chronological order of inspection time, and the inspection results include inspection images and corresponding abnormal tissue segmentation results; the loss function calculation module 502 is configured to perform the following overall loss function calculation operation on the inspection result sequence in the inspection result sequence set to obtain the overall loss function value of the inspection result sequence: starting from the first inspection result in the inspection result sequence to the second to last inspection result, in the order of inspection time from front to back, perform the following prediction loss function value calculation operation to obtain the prediction loss function value of the inspection result: input the inspection result and the inspection time interval between the inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back to obtain the first timing feature, the second timing feature and the third timing feature respectively; the first timing feature, the second timing feature and the third timing feature are respectively input into the timing feature extraction network ... The time series feature and the third time series feature are respectively input into the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model to obtain the predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval; based on the difference between the predicted growth probability value and the annotated growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted post-growth boundary information and the annotated post-growth boundary information, the difference between the predicted post-growth image and the inspection image in the next inspection result, and the difference between the predicted post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result in the next inspection result, the prediction loss function value of the inspection result is determined; based on the prediction loss function value of each inspection result in the inspection result sequence, the overall loss function value of the inspection result sequence is determined; the parameter optimization module 503 is configured to optimize the parameters of the time series feature extraction network, the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model based on the overall loss function value of each inspection result sequence.

[0234] In this embodiment, the specific processing of the first acquisition module 501, the loss function calculation module 502 and the parameter optimization module 503 of the abnormal tissue growth prediction model training device 500 and the technical effects thereof can be referred to respectively. Figure 2A The relevant descriptions of step 201, step 202 and step 203 in the corresponding embodiment are not repeated here.

[0235] In some optional embodiments, the first timing feature extraction network can update the first timing feature output this time according to the input inspection results and time intervals and the first timing feature output last time, the second timing feature extraction network can update the second timing feature output this time according to the first timing feature input from the first timing feature extraction network and the second timing feature output last time, and the third timing feature extraction network can update the third timing feature output this time according to the second timing feature input from the second timing feature extraction network and the third timing feature output last time.

[0236] In some optional embodiments, the determining of the prediction loss function value of the inspection result based on the difference between the predicted growth probability value and the annotated growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted boundary information after growth and the annotated boundary information after growth, the difference between the predicted image after growth and the inspection image in the next inspection result, and the difference between the predicted abnormal tissue segmentation result after growth and the abnormal tissue segmentation result in the next inspection result may include:

[0237] Determining a growth probability loss function value of the inspection result based on a difference between a predicted growth probability value and a marked growth probability value of the inspection result after a corresponding inspection time interval;

[0238] Determine a post-growth boundary information loss function value of the inspection result based on a difference between the predicted post-growth boundary information and the annotated post-growth boundary information of the inspection result after a corresponding inspection time interval;

[0239] Determine a post-growth image loss function value of the inspection result based on a difference between a predicted post-growth image of the inspection result after a corresponding inspection time interval and an inspection image in a next inspection result;

[0240] Determine a loss function value of the abnormal tissue segmentation result after growth of the inspection result based on a difference between a predicted abnormal tissue segmentation result after growth of the inspection result sequence and an abnormal tissue segmentation result in a next inspection result after a corresponding inspection time interval;

[0241] Based on the growth probability loss function value, the post-growth boundary information loss function value, the post-growth image loss function value and the post-growth abnormal tissue segmentation result loss function value of the inspection result, the prediction loss function value of the inspection result is determined.

[0242] In some optional embodiments, the post-growth boundary information prediction model may include a post-growth abnormal tissue surface prediction model, the post-growth boundary information may include a post-growth abnormal tissue surface segmentation result, marking the post-growth boundary information may include marking the post-growth abnormal tissue surface segmentation result, and the post-growth boundary information loss function value may include a post-growth abnormal tissue surface segmentation result loss function value; and

[0243] The step of inputting the first time series feature, the second time series feature and the third time series feature into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model respectively to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval may include:

[0244] Inputting the second time series feature into the post-growth abnormal tissue surface prediction model to obtain a predicted post-growth abnormal tissue surface segmentation result of the inspection result after a corresponding inspection time interval; and

[0245] Determining the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval may include:

[0246] Based on the difference between the predicted post-growth abnormal tissue surface segmentation result and the marked post-growth abnormal tissue surface segmentation result of the inspection result after the corresponding inspection time interval, the abnormal tissue surface segmentation result loss function value of the inspection result is determined.

[0247] In some optional embodiments, the post-growth boundary information prediction model may further include a post-growth abnormal tissue surface coordinate prediction model, the post-growth boundary information may further include a post-growth abnormal tissue surface coordinate set, marking the post-growth boundary information may further include marking the post-growth abnormal tissue surface coordinate set, and the post-growth boundary information loss function value may further include a post-growth abnormal tissue surface coordinate set loss function value; and

[0248] The step of inputting the first time series feature, the second time series feature and the third time series feature into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model respectively to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval may also include:

[0249] Inputting the predicted abnormal tissue surface segmentation result after the corresponding inspection time interval of the inspection result into the abnormal tissue surface coordinate prediction model after the growth to obtain the predicted abnormal tissue surface coordinate set after the corresponding inspection time interval of the inspection result; and

[0250] Determining the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval may include:

[0251] Based on the difference between the predicted post-growth abnormal tissue surface coordinate set and the marked post-growth abnormal tissue surface coordinate set of the inspection result after the corresponding inspection time interval, the post-growth abnormal tissue surface coordinate set loss function value of the inspection result is determined.

[0252] In some optional implementations, the post-growth boundary information prediction model may further include a post-growth distance map prediction model, the post-growth boundary information may further include a post-growth distance map, and the post-growth boundary information loss function value may include a post-growth distance map loss function value; and

[0253] The step of inputting the first time series feature, the second time series feature and the third time series feature into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model respectively to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval may include:

[0254] Inputting the second time series feature into the post-growth distance map prediction model to obtain a predicted post-growth distance map of the inspection result after a corresponding inspection time interval; and

[0255] Determining the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval may include:

[0256] Based on the difference between the predicted post-growth distance map and the annotated post-growth distance map of the inspection result after the corresponding inspection time interval, a post-growth distance map loss function value of the inspection result is determined.

[0257] In some optional implementations, the first temporal feature extraction network, the second temporal feature extraction network, and the third temporal feature extraction network may be long short-term memory networks.

[0258] In some optional embodiments, the growth probability prediction model may include a fully connected neural network, and the post-growth image prediction model may include a convolutional neural network.

[0259] In some optional embodiments, the post-growth abnormal tissue surface prediction model may include a convolutional neural network, and the post-growth abnormal tissue surface coordinate prediction model may include a fully connected neural network.

[0260] It should be noted that the implementation details and technical effects of each module in the abnormal tissue growth prediction model training device provided in the embodiments of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0261] Reference below Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an abnormal tissue growth prediction device. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0262] like Figure 6 As shown, the abnormal tissue growth prediction device 600 of this embodiment includes: a second acquisition module 601 , a feature extraction module 602 and a probability prediction module 603 . Among them, the second acquisition module 601 is configured to obtain a sequence of inspection results to be predicted and a growth time to be predicted, wherein the sequence of inspection results to be predicted is composed of at least two groups of inspection results obtained by inspecting the parts including the abnormal tissue to be predicted at different times and arranged in order of inspection time, and the inspection results include inspection images and corresponding abnormal tissue segmentation results; the feature extraction module 602 is configured to sequentially input the first inspection result to the last inspection result in the sequence of inspection results to be predicted and the inspection time interval between the corresponding inspection result and the next inspection result into a time series feature extraction network formed by connecting the first time series feature extraction network, the second time series feature extraction network and the third time series feature extraction network in series from front to back, wherein the inspection time interval between the last inspection result and the next inspection result is the growth time to be predicted; the probability prediction module 603 is configured to input the first time series feature to be predicted into the growth probability prediction model to obtain the predicted growth probability value of the abnormal tissue to be predicted after the growth time to be predicted, and the first time series feature to be predicted is the first time series feature finally output by the first time series feature extraction network, wherein the time series feature extraction network and the growth probability prediction model are adopted as follows Figure 2A The method described in the illustrated embodiment and its optional implementation is pre-trained.

[0263] In this embodiment, the specific processing of the second acquisition module 601, the feature extraction module 602 and the probability prediction module 603 of the abnormal tissue growth prediction device 600 and the technical effects thereof can be referred to respectively. Figure 4The relevant descriptions of step 401, step 402 and step 403 in the corresponding embodiment are not repeated here.

[0264] In some optional embodiments, the abnormal tissue growth prediction device 600 may further include:

[0265] The image prediction module 604 is configured to input the third time series feature to be predicted into the post-growth image prediction model to obtain the predicted post-growth image of the abnormal tissue to be predicted after the predicted growth time and the predicted post-growth abnormal tissue segmentation result, wherein the third time series feature to be predicted is the third time series feature finally output by the third time series feature extraction network, and the post-growth image prediction model is adopted as follows Figure 2A The method described in the illustrated embodiment and its optional implementation is pre-trained.

[0266] In some optional embodiments, the abnormal tissue growth prediction device 600 may further include:

[0267] The boundary prediction module 605 is configured to input the second time series feature to be predicted into the post-growth boundary information prediction model to obtain the predicted post-growth boundary information of the abnormal tissue to be predicted after the predicted growth time, wherein the second time series feature to be predicted is the second time series feature finally output by the second time series feature extraction network, and the post-growth boundary information prediction model is as follows: Figure 2A The method described in the illustrated embodiment and its optional implementation is pre-trained.

[0268] It should be noted that the implementation details and technical effects of each module in the abnormal tissue growth prediction device provided in the embodiments of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0269] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 suitable for implementing the electronic device of the present disclosure. Figure 7 The computer system 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0270] like Figure 7As shown, the computer system 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. Various programs and data required for the operation of the computer system 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0271] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the computer system 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The computer system 700 of the electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0272] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0273] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0274] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0275] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2A The abnormal tissue growth prediction model training method and / or the like shown in the embodiment and its optional implementation manner Figure 4 The illustrated embodiment and its alternative implementations illustrate a method for predicting abnormal tissue growth.

[0276] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Python, Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0277] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0278] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first acquisition module may also be described as a "module for acquiring a set of inspection result sequences".

[0279] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A method for training an abnormal tissue growth prediction model, comprising: Acquire a set of inspection result sequences, wherein the inspection result sequence is formed by arranging at least three groups of inspection results obtained by inspecting a part including the same abnormal tissue at different times in a chronological order, and the inspection results include inspection images and corresponding abnormal tissue segmentation results; For the inspection result sequence in the inspection result sequence set, perform the following overall loss function calculation operation to obtain the overall loss function value of the inspection result sequence: starting from the first inspection result in the inspection result sequence to the second to last inspection result, perform the following prediction loss function value calculation operation in the order of inspection time from front to back to obtain the prediction loss function value of the inspection result: input the inspection result and the inspection time interval between the inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back to obtain the first timing feature, the second timing feature and the third timing feature respectively; input the first timing feature, the second timing feature and the third timing feature respectively into A growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model are used to obtain the predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval; based on the difference between the predicted growth probability value and the annotated growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted post-growth boundary information and the annotated post-growth boundary information, the difference between the predicted post-growth image and the inspection image in the next inspection result, and the difference between the predicted post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result in the next inspection result, the prediction loss function value of the inspection result is determined; based on the prediction loss function value of each inspection result in the inspection result sequence, the overall loss function value of the inspection result sequence is determined; Based on the overall loss function value of each inspection result sequence, parameters of the temporal feature extraction network, the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model are optimized.

2. The method according to claim 1, wherein: The first timing feature extraction network updates the first timing feature output this time according to the input inspection results and time intervals and the first timing feature output last time. The second timing feature extraction network updates the second timing feature output this time according to the first timing feature input from the first timing feature extraction network and the second timing feature output last time. The third timing feature extraction network updates the third timing feature output this time according to the second timing feature input from the second timing feature extraction network and the third timing feature output last time.

3. The method according to claim 1, wherein: The method of determining the prediction loss function value of the inspection result based on the difference between the predicted growth probability value and the labeled growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted boundary information after growth and the labeled boundary information after growth, the difference between the predicted image after growth and the inspection image in the next inspection result, and the difference between the predicted abnormal tissue segmentation result after growth and the abnormal tissue segmentation result in the next inspection result, comprises: Determining a growth probability loss function value of the inspection result based on a difference between a predicted growth probability value and a marked growth probability value of the inspection result after a corresponding inspection time interval; Determine a post-growth boundary information loss function value of the inspection result based on a difference between the predicted post-growth boundary information and the annotated post-growth boundary information of the inspection result after a corresponding inspection time interval; Determine a post-growth image loss function value of the inspection result based on a difference between a predicted post-growth image of the inspection result after a corresponding inspection time interval and an inspection image in a next inspection result; Determine a loss function value of the abnormal tissue segmentation result after growth of the inspection result based on a difference between a predicted abnormal tissue segmentation result after growth of the inspection result sequence and an abnormal tissue segmentation result in a next inspection result after a corresponding inspection time interval; Based on the growth probability loss function value, the post-growth boundary information loss function value, the post-growth image loss function value and the post-growth abnormal tissue segmentation result loss function value of the inspection result, the prediction loss function value of the inspection result is determined.

4. The method according to claim 3, wherein: The post-growth boundary information prediction model includes a post-growth abnormal tissue surface prediction model, the post-growth boundary information includes a post-growth abnormal tissue surface segmentation result, the labeled post-growth boundary information includes a labeled post-growth abnormal tissue surface segmentation result, and the post-growth boundary information loss function value includes a post-growth abnormal tissue surface segmentation result loss function value; and The first time series feature, the second time series feature and the third time series feature are respectively input into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval, including: Inputting the second time series feature into the post-growth abnormal tissue surface prediction model to obtain a predicted post-growth abnormal tissue surface segmentation result of the inspection result after a corresponding inspection time interval; and The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes: Based on the difference between the predicted post-growth abnormal tissue surface segmentation result and the marked post-growth abnormal tissue surface segmentation result of the inspection result after the corresponding inspection time interval, the abnormal tissue surface segmentation result loss function value of the inspection result is determined.

5. The method according to claim 4, wherein: The post-growth boundary information prediction model also includes a post-growth abnormal tissue surface coordinate prediction model, the post-growth boundary information also includes a post-growth abnormal tissue surface coordinate set, the labeled post-growth boundary information also includes a labeled post-growth abnormal tissue surface coordinate set, and the post-growth boundary information loss function value also includes a post-growth abnormal tissue surface coordinate set loss function value; and The step of inputting the first time series feature, the second time series feature and the third time series feature into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model respectively to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval also includes: Inputting the predicted abnormal tissue surface segmentation result after the corresponding inspection time interval of the inspection result into the abnormal tissue surface coordinate prediction model after the growth to obtain the predicted abnormal tissue surface coordinate set after the corresponding inspection time interval of the inspection result; and The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes: Based on the difference between the predicted post-growth abnormal tissue surface coordinate set and the marked post-growth abnormal tissue surface coordinate set of the inspection result after the corresponding inspection time interval, the post-growth abnormal tissue surface coordinate set loss function value of the inspection result is determined.

6. The method according to claim 3, wherein: The post-growth boundary information prediction model further includes a post-growth distance map prediction model, the post-growth boundary information further includes a post-growth distance map, and the post-growth boundary information loss function value includes a post-growth distance map loss function value; and The first time series feature, the second time series feature and the third time series feature are respectively input into a growth probability prediction model, a post-growth boundary information prediction model and a post-growth image prediction model to obtain a predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after a corresponding inspection time interval, including: Inputting the second time series feature into the post-growth distance map prediction model to obtain a predicted post-growth distance map of the inspection result after a corresponding inspection time interval; and The determining of the post-growth boundary information loss function value of the inspection result based on the difference between the predicted post-growth boundary information and the marked post-growth boundary information of the inspection result after the corresponding inspection time interval includes: Based on the difference between the predicted post-growth distance map and the annotated post-growth distance map of the inspection result after the corresponding inspection time interval, a post-growth distance map loss function value of the inspection result is determined.

7. A method for predicting abnormal tissue growth, comprising: Obtaining a sequence of inspection results to be predicted and a growth duration to be predicted, wherein the sequence of inspection results to be predicted is composed of at least two groups of inspection results obtained by inspecting a part including the abnormal tissue to be predicted at different times and arranged in chronological order, and the inspection results include inspection images and corresponding abnormal tissue segmentation results; Sequentially inputting the first inspection result to the last inspection result in the inspection result sequence to be predicted and the inspection time interval between the corresponding inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back, wherein the inspection time interval between the last inspection result and the next inspection result is the growth duration to be predicted; The first time series feature to be predicted is input into the growth probability prediction model to obtain the predicted growth probability value of the abnormal tissue to be predicted after the predicted growth time, wherein the first time series feature to be predicted is the first time series feature finally output by the first time series feature extraction network, wherein the time series feature extraction network and the growth probability prediction model are pre-trained using the method described in any one of claims 1 to 6.

8. The method according to claim 7, wherein: The method further comprises: The third time series feature to be predicted is input into the post-growth image prediction model to obtain the predicted post-growth image of the abnormal tissue to be predicted after the predicted growth time and the predicted post-growth abnormal tissue segmentation result, wherein the third time series feature to be predicted is the third time series feature finally output by the third time series feature extraction network, and the post-growth image prediction model is pre-trained using the method described in any one of claims 1-6.

9. The method according to claim 7, wherein: The method further comprises: The second time series feature to be predicted is input into the post-growth boundary information prediction model to obtain the predicted post-growth boundary information of the abnormal tissue to be predicted after the predicted growth period, wherein the second time series feature to be predicted is the second time series feature finally output by the second time series feature extraction network, and the post-growth boundary information prediction model is pre-trained using the method described in any one of claims 1-6.

10. An abnormal tissue growth prediction model training device, comprising: A first acquisition module is configured to acquire a set of inspection result sequences, wherein the inspection result sequence is composed of at least three groups of inspection results obtained by inspecting a part including the same abnormal tissue at different times and arranged in order of inspection time, and the inspection results include inspection images and corresponding abnormal tissue segmentation results; The loss function calculation module is configured to perform the following overall loss function calculation operation on the inspection result sequence in the inspection result sequence set to obtain the overall loss function value of the inspection result sequence: starting from the first inspection result in the inspection result sequence to the second to last inspection result, in the order of inspection time from front to back, perform the following prediction loss function value calculation operation to obtain the prediction loss function value of the inspection result: input the inspection result and the inspection time interval between the inspection result and the next inspection result into a timing feature extraction network formed by connecting the first timing feature extraction network, the second timing feature extraction network and the third timing feature extraction network in series from front to back to obtain the first timing feature, the second timing feature and the third timing feature respectively; the first timing feature, the second timing feature and the third timing feature are respectively input into the timing feature extraction network ... The sequence features are respectively input into the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model to obtain the predicted growth probability value, predicted post-growth boundary information, predicted post-growth image and predicted post-growth abnormal tissue segmentation result of the inspection result after the corresponding inspection time interval; based on the difference between the predicted growth probability value and the annotated growth probability value of the inspection result after the corresponding inspection time interval, the difference between the predicted post-growth boundary information and the annotated post-growth boundary information, the difference between the predicted post-growth image and the inspection image in the next inspection result, and the difference between the predicted post-growth abnormal tissue segmentation result and the abnormal tissue segmentation result in the next inspection result, the prediction loss function value of the inspection result is determined; based on the prediction loss function value of each inspection result in the inspection result sequence, the overall loss function value of the inspection result sequence is determined; The parameter optimization module is configured to optimize the parameters of the temporal feature extraction network, the growth probability prediction model, the post-growth boundary information prediction model and the post-growth image prediction model based on the overall loss function value of each inspection result sequence.

11. An abnormal tissue growth prediction device, comprising: A second acquisition module is configured to acquire a sequence of inspection results to be predicted and a growth duration to be predicted, wherein the sequence of inspection results to be predicted is composed of at least two groups of inspection results obtained by inspecting the parts including the abnormal tissue to be predicted at different times and arranged in order of inspection time, and the inspection results include inspection images and corresponding abnormal tissue segmentation results; The feature extraction module is configured to sequentially input the first inspection result to the last inspection result in the inspection result sequence to be predicted and the inspection time interval between the corresponding inspection result and the next inspection result into a time series feature extraction network formed by connecting the first time series feature extraction network, the second time series feature extraction network and the third time series feature extraction network in series from front to back, wherein the inspection time interval between the last inspection result and the next inspection result is the growth time to be predicted; A probability prediction module is configured to input the first time series feature to be predicted into a growth probability prediction model to obtain a predicted growth probability value of the abnormal tissue to be predicted after the predicted growth time, wherein the first time series feature to be predicted is the first time series feature finally output by the first time series feature extraction network, wherein the time series feature extraction network and the growth probability prediction model are pre-trained using the method described in any one of claims 1 to 6.

12. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6 and / or the method according to any one of claims 7 to 9.

13. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 6 and / or the method according to any one of claims 7 to 9 is implemented.

14. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 6 and / or the method according to any one of claims 7 to 9 is implemented.

Images