An image processing method, apparatus, device, and storage medium

Abstract

The embodiments of the present application provide an image processing method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology. The method includes: obtaining a to-be-processed image of a target body part at a first moment, where the to-be-processed image includes a first target abnormal region of the target body part. Extracting image features of the to-be-processed image through a target region prediction model, and obtaining a predicted image of the target body part at a second moment based on the image features of the to-be-processed image, where the predicted image includes a second target abnormal region associated with the first target abnormal region, and the target region prediction model is obtained by performing multiple iterative trainings on an initialized region prediction model based on multi-task learning. Obtaining the target region prediction model in the way of multi-task learning avoids the model falling into local optimum and overfitting, improves the prediction ability and generalization ability of the model, and further improves the accuracy of predicting the changes of abnormal regions at different moments.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to an image processing method, apparatus, device, and storage medium. Background Art

[0002] With the development of medical technology and the popularization of medical imaging, currently doctors generally understand the current abnormal state of a patient's body through medical images. In addition to obtaining the current abnormal state, doctors also need to make a prediction about the development of the abnormal state, so as to formulate a targeted plan. One important prediction is how the morphology of the abnormal area on the body changes, which is closely related to the subsequent countermeasures. However, taking medical images can only obtain the current abnormal areas on the patient's body. Doctors need to rely on personal experience to predict the changes in the abnormal areas, which requires high personal experience. When personal experience is insufficient, it will lead to low prediction accuracy and efficiency. Summary of the Invention

[0003] Embodiments of the present application provide an image processing method, apparatus, device, and storage medium, which are used to improve the accuracy and efficiency of predicting the changes in abnormal areas at different times, and reduce the dependence on personal experience.

[0004] On the one hand, embodiments of the present application provide an image processing method, which includes:

[0005] Obtain a to-be-processed image of a target body part at a first moment, where the to-be-processed image includes a first target abnormal area of the target body part;

[0006] Extract the image features of the to-be-processed image, and based on the image features of the to-be-processed image, obtain a predicted image of the target body part at a second moment, where the predicted image includes a second target abnormal area associated with the first target abnormal area.

[0007] On the one hand, embodiments of the present application provide a method for training a region prediction model, which includes:

[0008] Based on multi-task learning, perform multiple iterative trainings on an initialized region prediction model to obtain a target region prediction model. The region prediction model includes a shared network, a main task network, and an auxiliary task network. In each iteration process, perform the following operations:

[0009] Extract the first sample image features of a first sample image through the shared network, where the first sample image includes a pre-marked first sample abnormal area;

[0010] Through an auxiliary task network, based on the first sample image features, obtain a first predicted image including a first predicted abnormal region, and determine a first difference degree between the first predicted abnormal region and the first sample abnormal region;

[0011] Through a main task network, based on the first sample image features, obtain a second predicted image including a second predicted abnormal region, and determine a second difference degree between the second predicted abnormal region and a second sample abnormal region pre-marked in a second sample image, where the first sample image and the second sample image are images of a target body part collected at different times;

[0012] Based on the first difference degree and the second difference degree, respectively adjust the parameters of the shared network, the main task network, and the auxiliary task network.

[0013] On the one hand, an embodiment of the present application provides an image processing device, and the device includes:

[0014] An acquisition module, configured to acquire a to-be-processed image of a target body part at a first moment, where the to-be-processed image includes a first target abnormal region of the target body part;

[0015] A processing module, configured to extract image features of the to-be-processed image, and based on the image features of the to-be-processed image, obtain a predicted image of the target body part at a second moment, where the predicted image includes a second target abnormal region associated with the first target abnormal region.

[0016] Optionally, the processing module is specifically configured to:

[0017] Extract image features of the to-be-processed image through a trained target region prediction model, and based on the image features of the to-be-processed image, obtain a predicted image of the target body part at a second moment, where the target region prediction model is obtained by performing multiple iterative trainings on an initialized region prediction model based on multi-task learning.

[0018] Optionally, the region prediction model includes a shared network, a main task network, and an auxiliary task network;

[0019] The processing module is specifically configured to:

[0020] Extract first sample image features of a first sample image through a shared network, where the first sample image includes a pre-marked first sample abnormal region;

[0021] Through an auxiliary task network, based on the first sample image features, obtain a first predicted image including a first predicted abnormal region, and determine a first difference degree between the first predicted abnormal region and the first sample abnormal region;

[0022] Through the main task network, based on the first sample image features, obtain a second predicted image including a second predicted abnormal region, and determine a second difference degree between the second predicted abnormal region and a second sample abnormal region pre-labeled in the second sample image, where the first sample image and the second sample image are images of the target body part collected at different times;

[0023] Based on the first difference degree and the second difference degree, respectively adjust the parameters of the shared network, the main task network, and the auxiliary task network.

[0024] Optionally, the processing module is specifically configured to:

[0025] Based on the first difference degree, adjust the parameters of the auxiliary task network;

[0026] Based on the second difference degree, adjust the parameters of the main task network;

[0027] According to the first difference degree and the second difference degree, determine a comprehensive difference degree;

[0028] Based on the comprehensive difference degree, adjust the parameters of the shared network.

[0029] Optionally, the processing module is specifically configured to:

[0030] According to the first difference degree and the second difference degree, determine a comprehensive difference degree;

[0031] Based on the comprehensive difference degree, respectively adjust the parameters of the shared network, the main task network, and the auxiliary task network.

[0032] Optionally, it further includes a selection module;

[0033] The selection module is specifically configured to:

[0034] After each iteration process ends, obtain a candidate region prediction model correspondingly;

[0035] After multiple iteration processes end, based on the validation sample set, determine a target region prediction model from multiple candidate region prediction models.

[0036] Optionally, the selection module is specifically configured to:

[0037] Based on the validation sample set, respectively verify the multiple candidate region prediction models, and obtain performance parameter values corresponding to the multiple candidate region prediction models;

[0038] Among the multiple candidate region prediction models, use the candidate region prediction model whose performance parameter value meets the preset condition as the target region prediction model.

[0039] On the one hand, an embodiment of the present application provides a device for training a region prediction model. The device includes:

[0040] A model training module, configured to perform multiple iterative trainings on an initialized region prediction model based on multi-task learning to obtain a target region prediction model. The region prediction model includes a shared network, a main task network, and an auxiliary task network;

[0041] The model training module includes a feature extraction module, a first prediction module, a second prediction module, and a parameter adjustment module;

[0042] The feature extraction module is configured to extract first sample image features of a first sample image through the shared network. The first sample image includes a pre-marked first sample abnormal region;

[0043] The first prediction module is configured to obtain a first prediction image including a first predicted abnormal region based on the first sample image features through the auxiliary task network, and determine a first difference degree between the first predicted abnormal region and the first sample abnormal region;

[0044] The second prediction module obtains a second prediction image including a second predicted abnormal region based on the first sample image features through the main task network, and determines a second difference degree between the second predicted abnormal region and a second sample abnormal region pre-marked in a second sample image. The first sample image and the second sample image are images of a target body part collected at different times;

[0045] The parameter adjustment module is configured to perform parameter adjustment on the shared network, the main task network, and the auxiliary task network respectively based on the first difference degree and the second difference degree.

[0046] Optionally, the parameter adjustment module is specifically configured to:

[0047] Perform parameter adjustment on the auxiliary task network based on the first difference degree;

[0048] Perform parameter adjustment on the main task network based on the second difference degree;

[0049] Determine a comprehensive difference degree according to the first difference degree and the second difference degree;

[0050] Perform parameter adjustment on the shared network based on the comprehensive difference degree.

[0051] Optionally, the parameter adjustment module is specifically configured to:

[0052] Determine a comprehensive difference degree according to the first difference degree and the second difference degree;

[0053] Based on the comprehensive difference degree, perform parameter adjustment on the shared network, the main task network, and the auxiliary task network respectively.

[0054] Optionally, the model training module further includes a screening module;

[0055] The screening module is specifically configured to:

[0056] After each iteration process ends, a candidate region prediction model is obtained correspondingly;

[0057] After multiple iteration processes end, determine a target region prediction model from multiple candidate region prediction models based on a validation sample set.

[0058] Optionally, the screening module is specifically configured to:

[0059] Based on the validation sample set, validate the multiple candidate region prediction models respectively, and obtain the performance parameter values corresponding to the multiple candidate region prediction models respectively;

[0060] Use the candidate region prediction model whose performance parameter value meets a preset condition among the multiple candidate region prediction models as the target region prediction model.

[0061] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the above image processing method or the steps of the above method for training a region prediction model.

[0062] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, it causes the computer device to execute the steps of the above image processing method or the steps of the above method for training a region prediction model.

[0063] In an embodiment of the present application, based on the image features of the image to be processed of the target body part at the first moment, the predicted image of the target body part at the second moment is automatically predicted, and at the same time, the second target abnormal region corresponding to the first target abnormal region in the image to be processed is obtained in the predicted image, without the need for manual judgment of the change of the abnormal region at different moments, thereby greatly reducing the dependence on personal experience and improving the prediction accuracy and prediction efficiency at the same time.

[0064] In the embodiments of the present application, since the learning objectives of the main task and the auxiliary task are different, the local minima of the main task and the auxiliary task are located at different positions. The interaction between the main task and the auxiliary task can help the main task escape from the local minimum, thereby improving the prediction ability of the target area prediction model. At the same time, the main task and the auxiliary task share shallow-layer parameters, so the overfitting degree of the target area prediction model is reduced, and the generalization ability of the model is improved, thereby improving the accuracy of predicting the changes of the abnormal area at different times and providing a more valuable treatment plan reference for doctors. Brief Description of the Drawings

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0066] Figure 1 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0067] Figure 2 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0068] Figure 3 It is a schematic diagram of a brain CT image provided by an embodiment of the present application;

[0069] Figure 4 It is a schematic diagram of a brain CT image provided by an embodiment of the present application;

[0070] Figure 5 It is a schematic diagram of the structure of a region prediction model provided by an embodiment of the present application;

[0071] Figure 6 It is a schematic flowchart of a method for training a region prediction model provided by an embodiment of the present application;

[0072] Figure 7 It is a schematic diagram of a brain CT image provided by an embodiment of the present application;

[0073] Figure 8 It is a schematic diagram of the structure of a region prediction model provided by an embodiment of the present application;

[0074] Figure 9 It is a schematic diagram of a brain CT image provided by an embodiment of the present application;

[0075] Figure 10 It is a schematic diagram of the structure of an image processing device provided by an embodiment of the present application;

[0076] Figure 11 The structural schematic diagram of an apparatus for training a regional prediction model provided by an embodiment of the present application;

[0077] Figure 12 The structural schematic diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0078] In order to make the objectives, technical solutions and beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0079] For the convenience of understanding, the nouns involved in the embodiments of the present invention will be explained below.

[0080] Artificial Intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, a theory, method, technology and application system that can perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0081] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0082] Machine Learning (ML) is an interdisciplinary subject in multiple fields, involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how a computer simulates or realizes human learning behaviors to acquire new knowledge or skills, and reorganizes the existing knowledge structure to continuously improve its own performance. Machine learning is the core of artificial intelligence and the fundamental way to make a computer intelligent. Its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rote learning. For example, in the embodiments of the present application, machine learning technology is used to predict the change of the cerebral hemorrhage area.

[0083] Multi-task learning: Multi-task learning refers to a machine learning method in which multiple related tasks are learned together and the results affect each other.

[0084] Deep convolutional neural network: A machine learning method.

[0085] CT: Computed Tomography, computerized tomography.

[0086] Intracerebral hemorrhage (parenchymal hemorrhage): It refers to bleeding caused by the rupture of blood vessels in the non-traumatic brain parenchyma. It belongs to a type of "stroke" and is a common and serious brain complication in middle-aged and elderly hypertensive patients. Acute intracerebral hemorrhage patients usually undergo a head CT scan to observe the bleeding site, the amount of bleeding, the hematoma morphology, whether it has broken into the ventricle, and whether there is a low-density edema zone and mass effect around the hematoma. In addition to the current state, doctors also need to make a prediction about the development of the condition in order to formulate a targeted treatment plan. One important prediction is how the morphology of the intracerebral hemorrhage area changes, which is related to whether to perform surgery and how to perform the surgery.

[0087] Generalization ability: The adaptability of machine learning / deep learning algorithms to fresh samples.

[0088] The design concept of the embodiments of the present application will be introduced below.

[0089] Acute intracerebral hemorrhage patients usually undergo a head CT scan to observe the bleeding site, the amount of bleeding, the hematoma morphology, whether it has broken into the ventricle, and whether there is a low-density edema zone and mass effect around the hematoma. In addition to the current state, doctors also need to make a prediction about the development of the condition in order to formulate a targeted treatment plan. One important prediction is how the morphology of the intracerebral hemorrhage area changes, which is related to whether to perform surgery and how to perform the surgery. However, taking a head CT scan can only obtain the current intracerebral hemorrhage area of the patient. Doctors need to rely on personal experience to predict the changes in the intracerebral hemorrhage area, which requires a high level of personal experience. When personal experience is insufficient, it will lead to low prediction accuracy and prediction efficiency.

[0090] If the changes in the future intracerebral hemorrhage area are automatically predicted based on the intracerebral hemorrhage area in the currently taken head CT, this will greatly reduce the dependence on personal experience and improve the processing efficiency at the same time. In view of this, the embodiments of the present application provide an image processing method, which specifically includes: obtaining a to-be-processed image of a target body part at a first moment, where the to-be-processed image includes a first target abnormal area of the target body part. Then, extracting the image features of the to-be-processed image, and based on the image features of the to-be-processed image, obtaining a predicted image of the target body part at a second moment, where the predicted image includes a second target abnormal area associated with the first target abnormal area.

[0091] In the embodiments of the present application, based on the image features of the to-be-processed image of the target body part at the first moment, the predicted image of the target body part at the second moment is automatically predicted, and at the same time, the corresponding second target abnormal area of the first target abnormal area in the to-be-processed image is obtained in the predicted image, without the need for manual judgment of the changes of the abnormal area at different moments, thereby greatly reducing the dependence on personal experience and improving the prediction accuracy and prediction efficiency at the same time.

[0092] Optionally, through analysis, it is found that when training a prediction model based on single-task learning, since the prediction model is only limited to the learning of a single target task, it is easy to fall into a local optimum and cannot exert the optimal prediction ability of the model. At the same time, the prediction model is prone to overfitting, resulting in a low generalization ability of the prediction model.

[0093] In view of this, in the embodiments of the present application, through the trained target area prediction model, the image features of the to-be-processed image are extracted, and based on the image features of the to-be-processed image, the predicted image of the target body part at the second moment is obtained, where the target area prediction model is obtained by performing multiple iterative trainings on the initialized area prediction model based on multi-task learning.

[0094] In multi-task learning, the local minima of different tasks are in different positions. Interaction can help the hidden layer escape from the local minimum, enabling the target area prediction model to learn a better prediction ability. At the same time, multiple tasks share shallow-layer parameters, reducing the overfitting degree of the target area prediction model, thereby improving the generalization ability of the model, and further improving the accuracy of predicting the changes of abnormal areas at different moments, providing a more valuable treatment plan reference for doctors.

[0095] Refer to Figure 1 , which is the system architecture diagram of the image processing method provided by the embodiments of the present application. This architecture at least includes a terminal device 101 and a server 102.

[0096] The target application is installed in the terminal device 101, where the target application can be a client application, a web application, a mini-program application, etc. The target application is used to predict the changes of abnormal areas on the target body part at different moments. The terminal device 101 may include one or more processors 1011, a memory 1012, an I / O interface 1013 for interacting with the server 102, a display panel 1014, etc. The terminal device 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0097] The server 102 may be the background server of the target application, providing corresponding services for the target application. The server 102 may include one or more processors 1021, a memory 1022, and an I / O interface 1023 for interacting with the terminal device 101, etc. In addition, the server 102 may also be configured with a database 1024. The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The terminal device 101 and the server 102 may be directly or indirectly connected through wired or wireless communication methods, and this application does not limit this here.

[0098] The image processing method in the embodiments of this application may be executed by the terminal device 101 or the server 102.

[0099] In the first case, the image processing method may be executed by the terminal device 101.

[0100] The terminal device 101 obtains a to-be-processed image of the target body part at the first moment, where the to-be-processed image includes a first target abnormal area of the target body part. Then, the image features of the to-be-processed image are extracted, and based on the image features of the to-be-processed image, a predicted image of the target body part at the second moment is obtained, where the predicted image includes a second target abnormal area associated with the first target abnormal area. The terminal device 101 displays the predicted image of the target body part at the second moment on the display interface.

[0101] In the second case, the image processing method may be executed by the server 102.

[0102] The terminal device 101 obtains a to-be-processed image of the target body part at the first moment, where the to-be-processed image includes a first target abnormal area of the target body part, and then sends the to-be-processed image to the server 102. The server 102 extracts the image features of the to-be-processed image, and based on the image features of the to-be-processed image, a predicted image of the target body part at the second moment is obtained, where the predicted image includes a second target abnormal area associated with the first target abnormal area. The server 102 sends the predicted image to the terminal device 101, and the terminal device 101 displays the predicted image of the target body part at the second moment on the display interface.

[0103] Based on Figure 1 the system architecture diagram shown, the embodiments of this application provide a flow of an image processing method, as Figure 2As shown, the process of this method can be executed by the Figure 1 terminal device 101 or the server 102 shown below, and includes the following steps:

[0104] Step S201, obtain the image to be processed of the target body part at the first moment.

[0105] Specifically, the target body part can be a body part on any target object, where the target object can be a person, an animal, a plant, etc. The target body part can be the brain, heart, liver, lungs, stomach, kidneys, etc. The image to be processed can be various medical images taken. For example, CT images, magnetic resonance images, etc. The image to be processed includes the first target abnormal area of the target body part, where the first target abnormal area can be a bleeding area, a lesion area, a fluid accumulation area, etc.

[0106] Taking the image to be processed as a brain CT image as an example, Figure 3 is the brain CT image taken of the patient at the first moment, and this brain CT image includes a cerebral hemorrhage area 301.

[0107] Step S202, extract the image features of the image to be processed, and based on the image features of the image to be processed, obtain the predicted image of the target body part at the second moment.

[0108] Specifically, the first moment and the second moment are different moments. For example, the interval between the first moment and the second moment is 24 hours, and the first moment is before the second moment. The predicted image includes the second target abnormal area associated with the first target abnormal area, and it can be understood that the second target abnormal area is the first target abnormal area after change.

[0109] Taking the image to be processed as a brain CT image as an example, Figure 3 is the brain CT image taken of the patient at the first moment, and this brain CT image includes the cerebral hemorrhage area 301 at the first moment. Figure 4 is the predicted brain image at the second moment obtained based on the brain CT image taken at the first moment. This brain predicted image includes the cerebral hemorrhage area 401 at the second moment, where the cerebral hemorrhage area 401 at the second moment is the cerebral hemorrhage area after the change of the cerebral hemorrhage area 301 at the first moment over time. By comparing Figure 3 and Figure 4 it can be seen that compared with the cerebral hemorrhage area 301 at the first moment, the predicted cerebral hemorrhage area 401 at the second moment is larger, which indicates that over time, the cerebral hemorrhage area of this patient may become larger.

[0110] In the embodiment of the present application, based on the image features of the image to be processed of the target body part at the first moment, the predicted image of the target body part at the second moment is automatically predicted, and at the same time, the corresponding second target abnormal region of the first target abnormal region in the image to be processed is obtained in the predicted image, without the need for manual judgment of the changes of the abnormal region at different moments, thereby greatly reducing the dependence on personal experience and improving the prediction accuracy and prediction efficiency at the same time.

[0111] Optionally, in the above step S202, the image features of the image to be processed are extracted through the trained target region prediction model, and based on the image features of the image to be processed, the predicted image of the target body part at the second moment is obtained, where the target region prediction model is obtained by performing multiple iterative trainings on the initialized region prediction model based on multi-task learning.

[0112] Specifically, the target region prediction model can be a convolutional neural network model, a fully convolutional network model, a support vector machine model, etc. Multi-task learning refers to a machine learning method in which multiple related tasks are learned together and the results affect each other. There are many forms of multi-task learning, such as joint learning, learning to learn, learning with auxiliary tasks, etc. Taking learning with auxiliary tasks as an example, the auxiliary tasks can be related tasks, adversarial tasks, hint tasks, etc. Optionally, in the embodiment of the present application, the trained target region prediction model can be stored on the blockchain.

[0113] Since in multi-task learning, the local minima of different tasks are in different positions. The interaction can help the hidden layer escape from the local minimum, so that the target region prediction model learns better prediction ability. At the same time, multiple tasks share shallow parameters, reducing the overfitting degree of the target region prediction model and improving the generalization ability of the model, thereby improving the accuracy of predicting the changes of the abnormal region at different moments and providing a more valuable treatment plan reference for doctors.

[0114] It should be noted that when training the target region prediction model, the embodiment of the present application is not limited to the multi-task learning method, and can also be based on the single-task learning method. In this regard, the present application does not make specific limitations.

[0115] Optionally, the structure of the region prediction model is as Figure 5 shown. The region prediction model includes a shared network, a main task network, and an auxiliary task network, where the main task network and the auxiliary task network share the parameters of the above shared network.

[0116] Based on multi-task learning, the initialized region prediction model is iteratively trained multiple times to obtain the target region prediction model. During each iteration, the following operations are performed, as Figure 6 shown:

[0117] Step S601: Extract the first sample image features of the first sample image through the shared network, where the first sample image includes a pre-marked first sample abnormal region.

[0118] Specifically, the region prediction model is initialized by using a random initialization method. First, the first sample images of the target body part are collected, then the first sample abnormal region is segmented from the first sample images, and then the first sample images are preprocessed, where the preprocessing includes removing interference regions, image registration, and image normalization.

[0119] Step S602: Through the auxiliary task network, based on the first sample image features, obtain a first prediction image including a first predicted abnormal region, and determine the first difference degree between the first predicted abnormal region and the first sample abnormal region.

[0120] Specifically, during the training process, the first prediction image output by the auxiliary task network continuously approaches the first sample image, and the first predicted abnormal region in the first prediction image continuously approaches the first sample abnormal region in the first sample image.

[0121] Step S603: Through the main task network, based on the first sample image features, obtain a second prediction image including a second predicted abnormal region, and determine the second difference degree between the second predicted abnormal region and the second sample abnormal region pre-marked in the second sample image.

[0122] Specifically, the second sample images of the target body part are collected first, then the second sample abnormal region is segmented from the second sample images, and then the second sample images are preprocessed, where the preprocessing includes removing interference regions, image registration, and image normalization. The first sample image and the second sample image are images of the target body part collected at different times, and the time interval between the first sample image and the second sample image can be set according to actual needs.

[0123] During the training process, the second prediction image output by the main task network continuously approaches the second sample image, and the second predicted abnormal region in the second prediction image continuously approaches the second sample abnormal region in the second sample image.

[0124] Step S604: Based on the first difference degree and the second difference degree, adjust the parameters of the shared network, the main task network, and the auxiliary task network respectively.

[0125] Specifically, based on the principle that the first difference degree and the second difference degree continuously decrease, the parameters of the shared network, the main task network, and the auxiliary task network are adjusted respectively.

[0126] In the embodiment of the present application, since the learning objectives of the main task and the auxiliary task are different, the local minima of the main task and the auxiliary task are in different positions. The interaction between the main task and the auxiliary task can help the main task escape from the local minimum, thereby improving the prediction ability of the target area prediction model. At the same time, the main task and the auxiliary task share shallow-layer parameters, so the overfitting degree of the target area prediction model is reduced, and the generalization ability of the model is improved, thereby improving the accuracy of predicting the changes of the abnormal area at different times and providing a more valuable treatment plan reference for doctors.

[0127] Optionally, in the above step S604, the embodiment of the present application provides at least two implementation manners for adjusting the parameters of the shared network, the main task network, and the auxiliary task network respectively based on the first difference degree and the second difference degree:

[0128] Implementation manner 1: Adjust the parameters of the auxiliary task network based on the first difference degree. Adjust the parameters of the main task network based on the second difference degree. Determine the comprehensive difference degree according to the first difference degree and the second difference degree, and then adjust the parameters of the shared network based on the comprehensive difference degree.

[0129] Specifically, based on the principle that the first difference degree continuously decreases, the parameters of the auxiliary task network are adjusted so that the first predicted abnormal area in the first predicted image output by the auxiliary task network continuously approaches the first sample abnormal area in the first sample image. Based on the principle that the second difference degree continuously decreases, the parameters of the main task network are adjusted so that the second predicted abnormal area in the second predicted image output by the main task network continuously approaches the second sample abnormal area in the second sample image. The first difference degree and the second difference degree can be summed or weighted and summed to obtain the comprehensive difference degree, and then based on the principle that the comprehensive difference degree continuously decreases, the parameters of the shared network are adjusted.

[0130] In the embodiment of the present application, based on the first difference degree and the second difference degree, the parameters of the auxiliary task network and the main task network are respectively optimized, and the prediction abilities of the auxiliary task network and the main task network are continuously improved. The parameters of the shared network are jointly optimized based on the first difference degree and the second difference degree, so that the learning processes of the main task network and the auxiliary task network influence and complement each other, thereby improving the prediction ability and generalization ability of the target area prediction model obtained by training.

[0131] Implementation manner 2: Determine the comprehensive difference degree according to the first difference degree and the second difference degree. Then, based on the comprehensive difference degree, adjust the parameters of the shared network, the main task network, and the auxiliary task network respectively.

[0132] Specifically, the first difference degree and the second difference degree can be summed or weighted and summed to obtain a comprehensive difference degree, and then based on the principle that the comprehensive difference degree continuously decreases, the parameters of the shared network, the main task network, and the auxiliary task network are adjusted respectively.

[0133] In the embodiments of the present application, the parameters of the shared network, the main task network, and the auxiliary task network are jointly optimized based on the first difference degree and the second difference degree, so that the learning processes of the main task network and the auxiliary task network affect and complement each other, thereby improving the prediction ability and generalization ability of the target region prediction model obtained by training.

[0134] Optionally, after each iteration process ends, a candidate region prediction model is obtained correspondingly. After multiple iteration processes end, the present application provides at least the following two implementation manners for determining the target region prediction model from multiple candidate region prediction models:

[0135] Implementation manner 1: Determine the target region prediction model from multiple candidate region prediction models based on the validation sample set.

[0136] Specifically, the validation sample set is a sample set set aside separately during the model training process and is different from the sample set used for training the model. For example, 100 sample images are obtained in advance, 80 of which are selected for training the region prediction model, and the remaining 20 sample images are used as the validation sample set. Another example is that 100 sample images are obtained in advance, 70 of which are selected for training the region prediction model, 20 of the remaining 30 sample images are selected as the validation sample set, and the remaining 10 sample images are used as the test sample set.

[0137] Optionally, based on the validation sample set, multiple candidate region prediction models are respectively verified to obtain the performance parameter values corresponding to the multiple candidate region prediction models, and then the candidate region prediction models among the multiple candidate region prediction models whose performance parameter values meet the preset conditions are used as the target region prediction models.

[0138] In specific implementation, for a candidate region prediction model, the candidate region prediction model includes a shared network, a main task network, and an auxiliary task network. When verifying the candidate region prediction model based on the validation sample set, one or more of the shared network, the main task network, and the auxiliary task network can be verified to obtain the performance parameter value corresponding to the candidate region prediction model.

[0139] For example, based on the validation sample set, the main task network is verified to obtain the performance parameter value of the main task network, and then the performance parameter value of the main task network is directly used as the performance parameter value corresponding to the candidate region prediction model.

[0140] For another example, the main task network and the auxiliary task network are respectively verified based on a verification sample set to obtain the performance parameter values corresponding to the main task network and the auxiliary task network, and then the performance parameter value corresponding to the candidate region prediction model is determined based on the performance parameter values corresponding to the main task network and the auxiliary task network.

[0141] The performance parameter value can be accuracy, false detection rate, precision, recall, receiver operating characteristic (ROC for short), etc. The performance parameter value meeting the preset condition can be that the performance parameter value is the largest, or the performance parameter value is the smallest, or the performance parameter value is within a preset range, etc.

[0142] In the embodiment of the present application, since the candidate region prediction model obtained in the last iteration after multiple iteration processes is not necessarily the candidate region prediction model with the optimal performance, and by using the verification sample set, multiple candidate region prediction models can be respectively verified to obtain the candidate region prediction model with the optimal performance. Using the candidate region prediction model with the optimal performance as the target region prediction model can effectively improve the accuracy of predicting the changes of the abnormal region at different times, thereby providing a more valuable reference for the doctor's treatment plan.

[0143] Embodiment 2: Determine the target region prediction model from multiple candidate region prediction models based on the first difference degree and the second difference degree.

[0144] Specifically, in the current iteration process, the first sample image feature of the first sample image is extracted through a shared network, and the first sample image includes a pre-marked first sample abnormal region. Through the auxiliary task network, based on the first sample image feature, a first prediction image including a first predicted abnormal region is obtained, and the first difference degree between the first predicted abnormal region and the first sample abnormal region is determined. Through the main task network, based on the first sample image feature, a second prediction image including a second predicted abnormal region is obtained, and the second difference degree between the second predicted abnormal region and the second sample abnormal region pre-marked in the second sample image is determined. The first sample image and the second sample image are images of the target body part collected at different times.

[0145] Judge whether the first difference degree and the second difference degree meet the preset conditions. If so, stop training, and use the candidate region prediction model composed of the shared network, the main task network, and the auxiliary task network in the current iteration process as the target region prediction model. Otherwise, based on the first difference degree and the second difference degree, the parameters of the shared network, the main task network, and the auxiliary task network are respectively adjusted, and the next iteration process is entered based on the shared network, the main task network, and the auxiliary task network with adjusted parameters.

[0146] In the embodiments of the present application, when it is determined whether the first difference degree and the second difference degree meet the preset conditions, it indicates that the candidate region prediction model basically fits the training data, that is, the prediction capabilities of the main task network and the auxiliary task network in the candidate region prediction model meet certain requirements. At this time, the training is stopped, and the candidate region prediction model obtained in the last iteration is used as the target region prediction model, which can ensure the prediction ability of the target region prediction model.

[0147] To better explain the embodiments of the present application, taking the prediction of the change of the cerebral hemorrhage region at different times as an example, an image processing method provided by the embodiments of the present application is introduced. First, the process of training the target region prediction model is introduced, which specifically includes the following steps:

[0148] Sample set preparation: Collect the brain CT images of a patient before and after. The brain CT image collected for the first time is called CT1, and the brain CT image collected for the second time is called CT2. The two brain CT images need to meet the following conditions: both are taken before surgery, the interval time is < 24 hours, the bleeding type is parenchymal hemorrhage, and the image has no obvious artifacts, etc. Further, segment and label the cerebral hemorrhage regions in CT1 and CT2 to form a data pair (CT1, CT1 mask, CT2 mask), where CT1 mask represents CT1 with the cerebral hemorrhage region labeled, CT2 mask represents CT2 with the cerebral hemorrhage region labeled, and CT1 mask and CT2 mask are specifically as Figure 7 shown, where CT1 mask includes the cerebral hemorrhage region 701, and CT2 mask includes the cerebral hemorrhage region 702.

[0149] Data preprocessing: Due to the differences caused by different CT devices and doctor operations, CT1 and CT2 may have different numbers of layers, slice thicknesses, scanning regions, angles, etc. Therefore, it is necessary to align CT1 and CT2 as much as possible and remove the image changes caused by factors other than the development of the lesion itself, which specifically includes three parts:

[0150] First, remove the interference regions. Use the image cropping method to remove the regions other than the brain tissue in the brain CT image, including removing the regions outside the head and deboning.

[0151] Second, image registration. Use the image registration technology to register CT1 and CT2 to the same angle so that the positions of the lesions are aligned.

[0152] Third, image normalization processing. Resample all images to the same slice thickness, and unify all images to a fixed size by the methods of padding zeros at the end and cropping. Normalize the images and map their value ranges to the range from -1 to 1.

[0153] Model Initialization: The regional prediction model is initialized in a random initialization manner.

[0154] Model Training: Based on multi-task learning, the initialized regional prediction model is iteratively trained multiple times to obtain the target regional prediction model. The structure of the regional prediction model is as Figure 8 shown. The regional prediction model includes a shared network, a main task network, and an auxiliary task network. Among them, the shared network includes an input layer (Input), an encoder (Encoder), and an embedding layer (Embedding). The auxiliary task network includes a first decoder (Decoder1), a segmentation prediction module (Segmentation Prediction), and a first output layer. The main task network includes a second decoder (Decoder2), a hematoma prediction module (Hematoma Prediction), and a second output layer.

[0155] During the model training process, CT1 and CT1 mask are input into the initialized regional prediction model. The shared network extracts the image features of CT1 and CT1 mask through the encoder and the embedding layer. The auxiliary task network predicts the intracerebral hemorrhage region in CT1 based on the image features of CT1 and CT1 mask through the first decoder and the segmentation prediction module. The main task network predicts the intracerebral hemorrhage region in CT2 based on the image features of CT1 and CT1 mask through the second decoder and the hematoma prediction module.

[0156] For the auxiliary task network, calculate the first difference degree between the pre-labeled intracerebral hemorrhage region in CT1 and the predicted intracerebral hemorrhage region in CT1. For the main task network, calculate the second difference degree between the pre-labeled intracerebral hemorrhage region in CT2 and the predicted intracerebral hemorrhage region in CT2. Then, adjust the parameters of the shared network, the main task network, and the auxiliary task network respectively through the first difference degree and the second difference degree.

[0157] After each iteration process ends, a candidate regional prediction model is obtained correspondingly. After multiple iteration processes end, the candidate regional prediction model basically fits the training data. Then, based on the validation sample set, the main task networks in multiple candidate regional prediction models are verified respectively to obtain the performance parameter values corresponding to multiple candidate regional prediction models. Then, select the candidate regional prediction model whose performance parameter value meets the preset conditions among multiple candidate regional prediction models as the target regional prediction model.

[0158] Model Testing: Input CT1 into the target regional prediction model, and the main task network in the target regional prediction model outputs the predicted CT2. The taken CT2 and the predicted CT2 are as Figure 9As shown, among them, the pre-labeled cerebral hemorrhage area in the captured CT2 is 901, and the predicted cerebral hemorrhage area in the predicted CT2 is 902. By comparing the pre-labeled cerebral hemorrhage area 901 with the predicted cerebral hemorrhage area 902, it can be seen that the pre-labeled cerebral hemorrhage area 901 is basically the same as the predicted cerebral hemorrhage area 902. Then, it can be concluded that the prediction ability of the target area prediction model to predict the change of the cerebral hemorrhage area is relatively strong.

[0159] After training the target area prediction model, the target area prediction model is applied to predict the change of the cerebral hemorrhage area. Specifically, obtain the brain CT image (CT3) of the target patient at the current moment, where CT3 includes the cerebral hemorrhage area at the current moment. Input CT3 into the target area prediction model, and the main task network in the target area prediction model outputs the brain CT image (CT4) of the target patient 24 hours later, where CT4 includes the cerebral hemorrhage area 24 hours later.

[0160] In the embodiments of the present application, for the problem of cerebral hemorrhage, the interaction between different tasks in multi-task learning is utilized to help the hidden layer escape from the local minimum, so that the target area prediction model learns better prediction ability. At the same time, multiple tasks share shallow-layer parameters, reduce the overfitting degree of the target area prediction model, improve the generalization ability of the model, and also improve the accuracy of predicting the future morphology of the cerebral hemorrhage area based on the CT images collected at the current time, providing more valuable reference for doctors to judge the development of the patient's condition.

[0161] The embodiments of the present application also provide a process of a method for training a region prediction model. The process of this method can be executed by Figure 1 the terminal device 101 or the server 102 shown, specifically including: based on multi-task learning, performing multiple iterative trainings on the initialized region prediction model to obtain the target region prediction model. Among them, the region prediction model includes a shared network, a main task network, and an auxiliary task network. In each iteration process, perform the following operations:

[0162] Extract the first sample image features of the first sample image through a shared network, where the first sample image includes a pre-marked first sample abnormal area. Through an auxiliary task network, based on the first sample image features, obtain a first predicted image including a first predicted abnormal area, and determine a first difference degree between the first predicted abnormal area and the first sample abnormal area. Through a main task network, based on the first sample image features, obtain a second predicted image including a second predicted abnormal area, and determine a second difference degree between the second predicted abnormal area and a second sample abnormal area pre-marked in a second sample image, where the first sample image and the second sample image are images of a target body part collected at different times. Based on the first difference degree and the second difference degree, adjust the parameters of the shared network, the main task network, and the auxiliary task network respectively.

[0163] In the embodiments of the present application, since the learning objectives of the main task and the auxiliary task are different, the local minima of the main task and the auxiliary task are in different positions. The interaction between the main task and the auxiliary task can help the main task escape from the local minimum, thereby improving the prediction ability of the target area prediction model. At the same time, the main task and the auxiliary task share shallow parameters, so the overfitting degree of the target area prediction model is reduced, and the generalization ability of the model is improved, thereby improving the accuracy of predicting the changes of the predicted abnormal area at different times and providing a more valuable treatment plan reference for doctors.

[0164] Optionally, when adjusting the parameters of the shared network, the main task network, and the auxiliary task network respectively based on the first difference degree and the second difference degree, the embodiments of the present application provide at least the following two implementation manners:

[0165] Implementation manner 1: Adjust the parameters of the auxiliary task network based on the first difference degree. Adjust the parameters of the main task network based on the second difference degree. Determine a comprehensive difference degree according to the first difference degree and the second difference degree, and then adjust the parameters of the shared network based on the comprehensive difference degree.

[0166] In the embodiments of the present application, based on the first difference degree and the second difference degree, the parameters of the auxiliary task network and the main task network are optimized respectively, and the prediction ability of the auxiliary task network and the main task network is continuously improved. The parameters of the shared network are jointly optimized based on the first difference degree and the second difference degree, so that the learning processes of the main task network and the auxiliary task network influence and complement each other, thereby improving the prediction ability and generalization ability of the obtained target area prediction model.

[0167] Implementation manner 2: Determine a comprehensive difference degree according to the first difference degree and the second difference degree, and then adjust the parameters of the shared network, the main task network, and the auxiliary task network respectively based on the comprehensive difference degree.

[0168] In the embodiments of the present application, based on the first difference degree and the second difference degree, the parameters of the shared network, the main task network, and the auxiliary task network are jointly optimized, so that the learning processes of the main task network and the auxiliary task network influence and complement each other, thereby improving the prediction ability and generalization ability of the target region prediction model obtained through training.

[0169] Optionally, after each iteration process ends, a candidate region prediction model is obtained correspondingly. After multiple iteration processes end, based on the validation sample set, a target region prediction model is determined from multiple candidate region prediction models.

[0170] Specifically, based on the validation sample set, multiple candidate region prediction models are respectively verified to obtain the performance parameter values corresponding to the multiple candidate region prediction models. Then, among the multiple candidate region prediction models, the candidate region prediction model whose performance parameter value meets the preset condition is used as the target region prediction model.

[0171] In the embodiments of the present application, since after multiple iteration processes end, it is not necessarily the case that the candidate region prediction model obtained in the last iteration is the candidate region prediction model with the optimal performance. By using the validation sample set to verify multiple candidate region prediction models respectively, a candidate region prediction model with the optimal performance can be obtained. Using the candidate region prediction model with the optimal performance as the target region prediction model can effectively improve the accuracy of predicting the changes of abnormal regions at different times, thereby providing a more valuable treatment plan reference for doctors.

[0172] Based on the same technical concept, the embodiments of the present application provide a structural schematic diagram of an image processing device, as Figure 10 shown. The device 1000 includes:

[0173] An acquisition module 1001, configured to acquire a to-be-processed image of a target body part at a first moment, where the to-be-processed image includes a first target abnormal region of the target body part;

[0174] A processing module 1002, configured to extract image features of the to-be-processed image, and based on the image features of the to-be-processed image, obtain a predicted image of the target body part at a second moment, where the predicted image includes a second target abnormal region associated with the first target abnormal region.

[0175] Optionally, the processing module 1002 is specifically configured to:

[0176] Extract the image features of the to-be-processed image through a trained target region prediction model, and based on the image features of the to-be-processed image, obtain a predicted image of the target body part at a second moment, where the target region prediction model is obtained by performing multiple iterative trainings on an initialized region prediction model based on multi-task learning.

[0177] Optionally, the region prediction model includes a shared network, a main task network, and an auxiliary task network;

[0178] The processing module 1002 is specifically configured to:

[0179] Extract first sample image features of a first sample image through the shared network, where the first sample image includes a pre-marked first sample abnormal region;

[0180] Through the auxiliary task network, based on the first sample image features, obtain a first prediction image including a first predicted abnormal region, and determine a first difference degree between the first predicted abnormal region and the first sample abnormal region;

[0181] Through the main task network, based on the first sample image features, obtain a second prediction image including a second predicted abnormal region, and determine a second difference degree between the second predicted abnormal region and a second sample abnormal region pre-marked in a second sample image, where the first sample image and the second sample image are images of a target body part collected at different times;

[0182] Based on the first difference degree and the second difference degree, respectively adjust the parameters of the shared network, the main task network, and the auxiliary task network.

[0183] Optionally, the processing module 1002 is specifically configured to:

[0184] Based on the first difference degree, adjust the parameters of the auxiliary task network;

[0185] Based on the second difference degree, adjust the parameters of the main task network;

[0186] According to the first difference degree and the second difference degree, determine a comprehensive difference degree;

[0187] Based on the comprehensive difference degree, adjust the parameters of the shared network.

[0188] Optionally, the processing module 1002 is specifically configured to:

[0189] According to the first difference degree and the second difference degree, determine a comprehensive difference degree;

[0190] Based on the comprehensive difference degree, respectively adjust the parameters of the shared network, the main task network, and the auxiliary task network.

[0191] Optionally, it further includes a selection module 1002;

[0192] The selection module 1002 is specifically configured to:

[0193] After each iteration process ends, a candidate region prediction model is correspondingly obtained.

[0194] After multiple iteration processes end, based on the validation sample set, a target region prediction model is determined from multiple candidate region prediction models.

[0195] Optionally, the selection module 1002 is specifically configured to:

[0196] Based on the validation sample set, the multiple candidate region prediction models are respectively verified to obtain the performance parameter values corresponding to the multiple candidate region prediction models.

[0197] The candidate region prediction model with the performance parameter value meeting the preset condition among the multiple candidate region prediction models is used as the target region prediction model.

[0198] In the embodiment of the present application, based on the image features of the to-be-processed image of the target body part at the first moment, the predicted image of the target body part at the second moment is automatically predicted, and at the same time, the second target abnormal region corresponding to the first target abnormal region in the to-be-processed image in the predicted image is obtained, without the need for manual judgment of the change of the abnormal region at different moments, thereby greatly reducing the dependence on personal experience, and at the same time improving the prediction accuracy and prediction efficiency. In multi-task learning, the local minima of different tasks are at different positions. Through interaction, it can help the hidden layer escape from the local minimum, so that the target region prediction model learns better prediction ability. At the same time, multiple tasks share shallow-layer parameters, reducing the overfitting degree of the target region prediction model, thereby improving the generalization ability of the model, and further improving the accuracy of predicting the change of the abnormal region at different moments, providing a more valuable treatment plan reference for doctors.

[0199] Based on the same technical concept, the embodiment of the present application provides a structural schematic diagram of a device for training a region prediction model, as Figure 11 shown. The device 1100 includes:

[0200] A model training module 1101, configured to perform multiple iterative trainings on the initialized region prediction model based on multi-task learning to obtain a target region prediction model, where the region prediction model includes a shared network, a main task network, and an auxiliary task network.

[0201] The model training module 1101 includes a feature extraction module 1102, a first prediction module 1103, a second prediction module 1104, and a parameter adjustment module 1105.

[0202] The feature extraction module 1102 is configured to extract first sample image features of a first sample image through a shared network, where the first sample image includes a pre-marked first sample abnormal region;

[0203] The first prediction module 1103 is configured to obtain a first prediction image including a first predicted abnormal region based on the first sample image features through an auxiliary task network, and determine a first difference degree between the first predicted abnormal region and the first sample abnormal region;

[0204] The second prediction module 1104 obtains a second prediction image including a second predicted abnormal region based on the first sample image features through a main task network, and determines a second difference degree between the second predicted abnormal region and a second sample abnormal region pre-marked in a second sample image, where the first sample image and the second sample image are images of a target body part collected at different times;

[0205] The parameter adjustment module 1105 is configured to perform parameter adjustment on the shared network, the main task network, and the auxiliary task network respectively based on the first difference degree and the second difference degree.

[0206] Optionally, the parameter adjustment module 1105 is specifically configured to:

[0207] Perform parameter adjustment on the auxiliary task network based on the first difference degree;

[0208] Perform parameter adjustment on the main task network based on the second difference degree;

[0209] Determine a comprehensive difference degree according to the first difference degree and the second difference degree;

[0210] Perform parameter adjustment on the shared network based on the comprehensive difference degree.

[0211] Optionally, the parameter adjustment module 1105 is specifically configured to:

[0212] Determine a comprehensive difference degree according to the first difference degree and the second difference degree;

[0213] Perform parameter adjustment on the shared network, the main task network, and the auxiliary task network respectively based on the comprehensive difference degree.

[0214] Optionally, the model training module 1101 further includes a screening module 1106;

[0215] The screening module 1106 is specifically configured to:

[0216] After each iteration process ends, a candidate region prediction model is obtained correspondingly;

[0217] After the end of multiple iteration processes, based on the verification sample set, a target region prediction model is determined from multiple candidate region prediction models.

[0218] Optionally, the screening module 1106 is specifically configured to:

[0219] Based on the verification sample set, verify the multiple candidate region prediction models respectively, and obtain the performance parameter values corresponding to the multiple candidate region prediction models respectively;

[0220] Use the candidate region prediction models among the multiple candidate region prediction models whose performance parameter values meet the preset conditions as the target region prediction models.

[0221] In the embodiments of the present application, since the learning objectives of the main task and the auxiliary task are different, the local minima of the main task and the auxiliary task are in different positions. The interaction between the main task and the auxiliary task can help the main task escape from the local minimum, thereby improving the prediction ability of the target region prediction model. At the same time, the main task and the auxiliary task share shallow parameters, so the overfitting degree of the target region prediction model is reduced, and the generalization ability of the model is improved, thereby improving the accuracy of predicting the changes of the abnormal region at different times and providing a more valuable treatment plan reference for doctors.

[0222] Based on the same technical concept, the embodiments of the present application provide a computer device, such as Figure 12 shown, including at least one processor 1201 and a memory 1202 connected to the at least one processor. In the embodiments of the present application, the specific connection medium between the processor 1201 and the memory 1202 is not limited. Figure 12 Taking the example that the processor 1201 and the memory 1202 are connected through a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0223] In the embodiments of the present application, the memory 1202 stores instructions executable by at least one processor 1201. By executing the instructions stored in the memory 1202, the at least one processor 1201 can execute the steps included in the above image processing method or the above method for training a region prediction model.

[0224] Among them, the processor 1201 is the control center of the computer device. It can connect various parts of the computer device through various interfaces and circuits. By running or executing the instructions stored in the memory 1202 and calling the data stored in the memory 1202, image processing or training the area prediction model can be performed. Optionally, the processor 1201 may include one or more processing units. The processor 1201 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 1201. In some embodiments, the processor 1201 and the memory 1202 may be implemented on the same chip. In some embodiments, they may also be separately implemented on independent chips.

[0225] The processor 1201 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0226] The memory 1202, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 1202 can include at least one type of storage medium. For example, it can include flash memory, hard disks, multimedia cards, card-type memories, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memories, magnetic disks, optical disks, and so on. The memory 1202 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1202 in the embodiments of the present application can also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0227] Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, it causes the computer device to execute the steps of the above image processing method or the above method for training a region prediction model.

[0228] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a computer program product, or a combination thereof. Therefore, the present invention can take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0229] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the processes Figure 1means for the functions specified in one or more processes and / or blocks Figure 1 means for the functions specified in one or more blocks

[0230] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions in the process Figure 1 means for the functions specified in one or more processes and / or blocks Figure 1 means for the functions specified in one or more blocks

[0231] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions in the process Figure 1 means for the functions specified in one or more processes and / or blocks Figure 1 means for the functions specified in one or more blocks

[0232] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn of the basic inventive concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention

[0233] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations

Claims

1. An image processing method, characterized in that, comprising: Obtaining a to-be-processed image of a target body part at a first moment, where the to-be-processed image includes a first target abnormal area of the target body part; Extracting image features of the to-be-processed image through a trained target area prediction model, and obtaining a predicted image of the target body part at a second moment based on the image features of the to-be-processed image, where the predicted image includes a second target abnormal area associated with the first target abnormal area; the area prediction model includes a shared network, a main task network, and an auxiliary task network; The target area prediction model is obtained by performing multiple iterative trainings on an initialized area prediction model based on multi-task learning. In each iteration process, the following operations are performed: Extracting first sample image features of a first sample image through the shared network, where the first sample image includes a pre-marked first sample abnormal area; Through the auxiliary task network, obtaining a first predicted image including a first predicted abnormal area based on the first sample image features, and determining a first difference degree between the first predicted abnormal area and the first sample abnormal area; Through the main task network, obtaining a second predicted image including a second predicted abnormal area based on the first sample image features, and determining a second difference degree between the second predicted abnormal area and a second sample abnormal area pre-marked in a second sample image, where the first sample image and the second sample image are images of the target body part collected at different moments; Based on the first difference degree and the second difference degree, respectively adjusting parameters of the shared network, the main task network, and the auxiliary task network.

2. The method according to claim 1, characterized in that, The respectively adjusting parameters of the shared network, the main task network, and the auxiliary task network based on the first difference degree and the second difference degree includes: Based on the first difference degree, adjusting parameters of the auxiliary task network; Based on the second difference degree, adjusting parameters of the main task network; Determining a comprehensive difference degree according to the first difference degree and the second difference degree; Based on the comprehensive difference degree, adjusting parameters of the shared network.

3. The method according to claim 2, characterized in that, The respectively adjusting parameters of the shared network, the main task network, and the auxiliary task network based on the first difference degree and the second difference degree includes: Determining a comprehensive difference degree according to the first difference degree and the second difference degree; Based on the comprehensive difference degree, respectively adjusting parameters of the shared network, the main task network, and the auxiliary task network.

4. The method according to claim 2 or 3, characterized in that, further comprising: After each iteration process ends, a candidate area prediction model is correspondingly obtained; After multiple iteration processes end, determining a target area prediction model from multiple candidate area prediction models based on a validation sample set.

5. The method according to claim 4, characterized in that, Determining a target region prediction model from multiple candidate region prediction models based on a validation sample set includes: Based on the validation sample set, validating the multiple candidate region prediction models respectively to obtain performance parameter values corresponding to the multiple candidate region prediction models; Regarding the candidate region prediction models among the multiple candidate region prediction models whose performance parameter values meet the preset conditions as the target region prediction models.

6. A method for training a region prediction model, characterized in that, it includes: Based on multi-task learning, performing multiple iterative trainings on an initialized region prediction model to obtain a target region prediction model. The region prediction model includes a shared network, a main task network, and an auxiliary task network. In each iteration process, perform the following operations: Extract first sample image features of a first sample image through the shared network. The first sample image includes a pre-marked first sample abnormal region; Through the auxiliary task network, based on the first sample image features, obtain a first prediction image including a first predicted abnormal region, and determine a first difference degree between the first predicted abnormal region and the first sample abnormal region; Through the main task network, based on the first sample image features, obtain a second prediction image including a second predicted abnormal region, and determine a second difference degree between the second predicted abnormal region and a second sample abnormal region pre-marked in a second sample image. The first sample image and the second sample image are images of a target body part collected at different times; Based on the first difference degree and the second difference degree, adjust the parameters of the shared network, the main task network, and the auxiliary task network respectively.

7. The method according to claim 6, characterized in that, The adjusting the parameters of the shared network, the main task network, and the auxiliary task network respectively based on the first difference degree and the second difference degree includes: Based on the first difference degree, adjusting the parameters of the auxiliary task network; Based on the second difference degree, adjusting the parameters of the main task network; Determine a comprehensive difference degree according to the first difference degree and the second difference degree; Based on the comprehensive difference degree, adjust the parameters of the shared network.

8. The method according to claim 6, characterized in that, The adjusting the parameters of the shared network, the main task network, and the auxiliary task network respectively based on the first difference degree and the second difference degree includes: Determine a comprehensive difference degree according to the first difference degree and the second difference degree; Based on the comprehensive difference degree, adjust the parameters of the shared network, the main task network, and the auxiliary task network respectively.

9. The method according to claim 7 or 8, characterized in that, it further includes: After each iteration process ends, a corresponding candidate region prediction model is obtained; After multiple iteration processes end, determine a target region prediction model from multiple candidate region prediction models based on a validation sample set.

10. The method according to claim 9, characterized in that, Determining a target region prediction model from multiple candidate region prediction models based on a validation sample set includes: Based on the validation sample set, validating the multiple candidate region prediction models respectively to obtain the performance parameter values corresponding to the multiple candidate region prediction models respectively; Regarding the candidate region prediction model whose performance parameter value meets the preset condition among the multiple candidate region prediction models as the target region prediction model.

11. An image processing device Characterized in that It includes: An acquisition module for acquiring a to-be-processed image of a target body part at a first moment, where the to-be-processed image includes a first target abnormal region of the target body part; A processing module for extracting image features of the to-be-processed image through a trained target region prediction model, and obtaining a predicted image of the target body part at a second moment based on the image features of the to-be-processed image, where the predicted image includes a second target abnormal region associated with the first target abnormal region; the region prediction model includes a shared network, a main task network, and an auxiliary task network; The target region prediction model is obtained by performing multiple iterative trainings on an initialized region prediction model based on multi-task learning. In each iteration process, the following operations are performed: Extracting first sample image features of a first sample image through the shared network, where the first sample image includes a pre-marked first sample abnormal region; Through the auxiliary task network, obtaining a first predicted image including a first predicted abnormal region based on the first sample image features, and determining a first difference degree between the first predicted abnormal region and the first sample abnormal region; Through the main task network, obtaining a second predicted image including a second predicted abnormal region based on the first sample image features, and determining a second difference degree between the second predicted abnormal region and a second sample abnormal region pre-marked in a second sample image, where the first sample image and the second sample image are images of the target body part acquired at different moments; Based on the first difference degree and the second difference degree, adjusting the parameters of the shared network, the main task network, and the auxiliary task network respectively.

12. A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, Characterized in that When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 5, or the steps of the method according to any one of claims 6 to 10.

13. A computer-readable storage medium Characterized in that It stores a computer program executable by a computer device. When the program runs on the computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 5, or the steps of the method according to any one of claims 6 to 10.

Images