A method for processing lesion images and related devices

Abstract

The present application discloses a method for processing lesion images and related devices, which are applied to the machine learning technology of artificial intelligence. By seeing the image to be predicted and inputting it into the encoder in the target model to obtain an encoded vector; then inputting the encoded vector into the mapper in the target model to obtain a target vector, where the mapper is obtained by training based on lesion images; and further inputting the target vector into the decoder in the target model to obtain a predicted image including the target lesion after changing based on the temporal relationship. Thus, the process of predicting lesion changes based on artificial intelligence is realized. Since the target model is trained by an image-to-image training method and the training data is collected based on the actual change process of the lesion, the target model has the ability to infer lesion images, improving the accuracy of predicting lesion changes.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular, to a method for processing lesion images and related devices. Background Art

[0002] With the rapid development of medical technology, more and more detection items appear in medical scenarios. How to identify and predict lesions in the detection images generated in medical examinations has become a difficult problem.

[0003] Generally, the prediction of lesion areas in medical images requires identification based on doctors' professional knowledge and experience, and has relatively high requirements for doctors' professional skills.

[0004] However, the process of manual prediction is time-consuming and laborious, and affected by subjective factors, the results of lesion prediction are not stable, affecting the accuracy of lesion change prediction. Summary of the Invention

[0005] In view of this, the present application provides a method for processing lesion images, which can effectively improve the accuracy of lesion prediction.

[0006] The first aspect of the present application provides a method for processing lesion images, which can be applied to a system or program with a lesion image processing function in a terminal device, and specifically includes:

[0007] Obtain a to-be-predicted image containing a target lesion and input it into an encoder in a target model for vector representation to obtain a coded vector;

[0008] Input the coded vector into a mapper in the target model for vector conversion in the image dimension to obtain a target vector. The mapper is obtained by training based on lesion images, and the lesion image pairs are composed of training images detected based on a temporal relationship, and the training images contain the target lesion;

[0009] Input the target vector into a decoder in the target model for image prediction to obtain a predicted image containing the target lesion after changing based on the temporal relationship.

[0010] Optionally, in some possible implementation manners of the present application, a plurality of lesion image pairs are collected. The lesion image pairs include a first image and a second image as the training images. The first image is an image obtained by detecting a target lesion at a first time node, and the second image is an image obtained by detecting the target lesion at a second time node, and the second time node is after the first time node;

[0011] Training the target model based on the lesion images, the training process of the target model includes a reconstruction task and a prediction task, the reconstruction task is used to reconstruct the image based on the first image, and the prediction task is used to learn the correspondence between the first image and the second image.

[0012] Optionally, in some possible implementation manners of the present application, the training the target model based on the lesion images includes:

[0013] Performing an identity mapping on the first image in the lesion image pair to obtain a reconstructed decoded image;

[0014] Performing the reconstruction task according to the process of restoring the reconstructed decoded image to the first image;

[0015] If the similarity between the reconstructed decoded image and the first image reaches a threshold, end the reconstruction task;

[0016] Performing the prediction task based on the correspondence between the first image and the second image to train the target model.

[0017] Optionally, in some possible implementation manners of the present application, the performing an identity mapping on the first image in the lesion image pair to obtain a reconstructed decoded image includes:

[0018] Performing an encoding operation on the first image in the lesion image pair to compress the feature dimension of the first image to obtain a first feature vector;

[0019] Performing a decoding operation on the first feature vector to obtain a reconstructed decoded image by performing similarity constraint based on a first constraint function, and the feature dimension of the reconstructed decoded image is restored from the first feature vector to the feature dimension corresponding to the first image;

[0020] Performing a discrimination operation on the reconstructed decoded image and the first image to obtain a discrimination result, the objective of the discrimination operation is that the discrimination result output based on a second constraint function is the first image, and the objectives of the encoding operation and the decoding operation are that the discrimination result output based on a third constraint function is the reconstructed decoded image;

[0021] The performing the reconstruction task according to the process of restoring the reconstructed decoded image to the first image includes:

[0022] Performing the reconstruction task according to the adjustment process of the second constraint function and the third constraint function.

[0023] Optionally, in some possible implementation manners of the present application, the method further includes:

[0024] Obtain the preset preheating steps;

[0025] Based on the preset preheating steps, perform the similarity constraint process of the first constraint function in a loop to preheat the processes of the encoding operation and the decoding operation.

[0026] Optionally, in some possible implementation manners of the present application, the performing the prediction task based on the correspondence between the first image and the second image to train the target model includes:

[0027] Perform vector representation on the first image to obtain a low-dimensional vector;

[0028] Input the low-dimensional vector into a mapper to obtain a prediction vector;

[0029] Based on the prediction vector, perform conversion to the vector corresponding to the second image to adjust the fourth constraint function;

[0030] Perform the prediction task based on the adjustment process of the fourth constraint function to train the target model.

[0031] Optionally, in some possible implementation manners of the present application, the method further includes:

[0032] Crop the images in the lesion image pair;

[0033] Align the images in the cropped lesion image pair based on a preset angle to obtain an aligned image pair;

[0034] Adjust the images in the aligned image pair to the same feature dimension;

[0035] Perform normalization processing on the images in the adjusted aligned image pair to update the lesion image pair.

[0036] Optionally, in some possible implementation manners of the present application, the cropping the images in the lesion image pair includes:

[0037] Determine the body part information corresponding to the target lesion;

[0038] Determine the cropping item based on the body part information;

[0039] Crop the images in the lesion image pair according to the cropping item.

[0040] Optionally, in some possible implementation manners of the present application, the collecting multiple lesion image pairs includes:

[0041] Determine the type information corresponding to the target lesion;

[0042] Determine the lesion change period based on the type information;

[0043] Take the change duration corresponding to the lesion change period as the acquisition interval, and acquire multiple pairs of the lesion images.

[0044] Optionally, in some possible implementation manners of the present application, the method further includes:

[0045] Obtain a reference lesion area;

[0046] Determine a negative sample parameter based on the similarity between the reference lesion area and the lesion area of the first image;

[0047] Determine a first difference part between the reference lesion area and the lesion area of the first image;

[0048] Determine a second difference part between the lesion area of the second image and the lesion area of the first image;

[0049] Determine a positive sample parameter according to the first difference part and the second difference part;

[0050] Determine an evaluation index based on the negative sample parameter and the positive sample parameter, and the evaluation index is used to indicate the prediction accuracy of the target model.

[0051] Optionally, in some possible implementation manners of the present application, the method further includes:

[0052] If the evaluation index indicates that the target model does not meet the preset index, determine the difference information in the pair of lesion images;

[0053] Audit the pair of lesion images based on the difference information to update the pair of lesion images, and train the target model based on the updated pair of lesion images.

[0054] Optionally, in some possible implementation manners of the present application, the method further includes:

[0055] Input the prediction image into the trained target model to obtain a cyclic image;

[0056] If the cyclic image is the same as the prediction image, determine the prediction image as the prediction result;

[0057] If the cyclic image is different from the prediction image, predict the change condition of the target lesion based on the cyclic image.

[0058] Optionally, in some possible implementation manners of the present application, the pair of lesion images is obtained by computer tomography, the target lesion is the cerebral hemorrhage area obtained by the computer tomography, the target model is a deep generative adversarial network, and the predicted image is used to predict the change trend of the cerebral hemorrhage area.

[0059] The second aspect of the present application provides a processing device for lesion images, including:

[0060] An encoding unit, configured to obtain a to-be-predicted image including a target lesion and input it into an encoder in the target model for vector representation to obtain an encoded vector;

[0061] A mapping unit, configured to input the encoded vector into a mapper in the target model for vector conversion of the image dimension to obtain a target vector, where the mapper is obtained by training based on a pair of lesion images, and the pair of lesion images consists of training images detected based on a temporal relationship, and the training images include the target lesion;

[0062] A processing unit, configured to input the target vector into a decoder in the target model for image prediction to obtain a predicted image including the target lesion after change based on the temporal relationship.

[0063] Optionally, in some possible implementation manners of the present application, the processing device for lesion images further includes:

[0064] A training unit, specifically configured to collect multiple pairs of lesion images, where the pair of lesion images includes a first image and a second image as the training images, the first image is an image obtained by detecting a target lesion at a first time node, and the second image is an image obtained by detecting the target lesion at a second time node, and the second time node is after the first time node;

[0065] The training unit is specifically configured to train the target model based on the pair of lesion images. The training process of the target model includes a reconstruction task and a prediction task. The reconstruction task is used to perform image reconstruction based on the first image, and the prediction task is used to learn the correspondence between the first image and the second image.

[0066] Optionally, in some possible implementation manners of the present application, the training unit is specifically configured to perform an identity mapping on the first image in the pair of lesion images to obtain a reconstructed decoded image;

[0067] The training unit is specifically configured to execute the reconstruction task according to the process of restoring the reconstructed decoded image to the first image;

[0068] The training unit is specifically configured to end the reconstruction task if the similarity between the reconstructed decoded image and the first image reaches a threshold;

[0069] The training unit is specifically configured to perform the prediction task based on the correspondence between the first image and the second image to train the target model.

[0070] Optionally, in some possible implementation manners of the present application, the training unit is specifically configured to perform an encoding operation on the first image in the lesion image pair to compress the feature dimension of the first image to obtain a first feature vector;

[0071] The training unit is specifically configured to perform a decoding operation on the first feature vector to obtain a reconstructed decoded image by performing similarity constraint based on a first constraint function, and the feature dimension of the reconstructed decoded image is restored from the first feature vector to the feature dimension corresponding to the first image;

[0072] The training unit is specifically configured to perform a discrimination operation on the reconstructed decoded image and the first image to obtain a discrimination result. The objective of the discrimination operation is that the discrimination result output based on a second constraint function is the first image, and the objectives of the encoding operation and the decoding operation are that the discrimination result output based on a third constraint function is the reconstructed decoded image;

[0073] The training unit is specifically configured to perform the reconstruction task based on the adjustment processes of the second constraint function and the third constraint function.

[0074] Optionally, in some possible implementation manners of the present application, the training unit is specifically configured to obtain a preset warm-up step number;

[0075] The training unit is specifically configured to perform the similarity constraint process of the first constraint function in a loop based on the preset warm-up step number to warm up the processes of the encoding operation and the decoding operation.

[0076] Optionally, in some possible implementation manners of the present application, the training unit is specifically configured to perform vector representation on the first image to obtain a low-dimensional vector;

[0077] The training unit is specifically configured to input the low-dimensional vector into a mapper to obtain a prediction vector;

[0078] The training unit is specifically configured to perform conversion based on the prediction vector to a vector corresponding to the second image to adjust a fourth constraint function;

[0079] The training unit is specifically configured to execute the prediction task based on the adjustment process of the fourth constraint function to train the target model.

[0080] Optionally, in some possible implementation manners of the present application, the training unit is specifically configured to crop the images in the pair of lesion images;

[0081] The training unit is specifically configured to align the images in the cropped pair of lesion images based on a preset angle to obtain an aligned pair of images;

[0082] The training unit is specifically configured to adjust the images in the aligned pair of images to the same feature dimension;

[0083] The training unit is specifically configured to perform normalization processing on the images in the adjusted aligned pair of images to update the pair of lesion images.

[0084] Optionally, in some possible implementation manners of the present application, the training unit is specifically configured to determine the body part information corresponding to the target lesion;

[0085] The training unit is specifically configured to determine a cropping item based on the body part information;

[0086] The training unit is specifically configured to crop the images in the pair of lesion images according to the cropping item.

[0087] Optionally, in some possible implementation manners of the present application, the training unit is specifically configured to determine the type information corresponding to the target lesion;

[0088] The training unit is specifically configured to determine a lesion change cycle based on the type information;

[0089] The training unit is specifically configured to use the change duration corresponding to the lesion change cycle as a collection interval to collect multiple pairs of lesion images.

[0090] Optionally, in some possible implementation manners of the present application, the training unit is specifically configured to obtain a reference lesion area;

[0091] The training unit is specifically configured to determine a negative sample parameter based on the similarity between the reference lesion area and the lesion area of the first image;

[0092] The training unit is specifically configured to determine a first difference part between the reference lesion area and the lesion area of the first image;

[0093] The training unit is specifically configured to determine a second difference part between the lesion area of the second image and the lesion area of the first image;

[0094] The training unit is specifically configured to determine positive sample parameters according to the first difference part and the second difference part;

[0095] The training unit is specifically configured to determine an evaluation index based on the negative sample parameters and the positive sample parameters, and the evaluation index is used to indicate the prediction accuracy of the target model.

[0096] Optionally, in some possible implementation manners of this application, the training unit is specifically configured to determine the difference information in the pair of lesion images if the evaluation index indicates that the target model does not meet the preset index;

[0097] The training unit is specifically configured to review the pair of lesion images based on the difference information, so as to update the pair of lesion images, and train the target model based on the updated pair of lesion images.

[0098] Optionally, in some possible implementation manners of this application, the processing unit is specifically configured to input the predicted image into the trained target model to obtain a cyclic image;

[0099] The processing unit is specifically configured to determine the predicted image as the prediction result if the cyclic image is the same as the predicted image;

[0100] The processing unit is specifically configured to predict the change situation of the target lesion based on the cyclic image if the cyclic image is different from the predicted image.

[0101] A third aspect of this application provides a computer device, including: a memory, a processor, and a bus system; the memory is used to store program codes; the processor is configured to execute the processing method of the lesion image according to the instructions in the program codes in the first aspect or any item of the first aspect.

[0102] A fourth aspect of this application provides a computer-readable storage medium, in which instructions are stored, and when the instructions are run on a computer, the computer is caused to execute the processing method of the lesion image according to the first aspect or any item of the first aspect.

[0103] According to one aspect of this application, there is provided a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the processing method of the lesion image provided in the first aspect or various optional implementation manners of the first aspect.

[0104] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

[0105] By obtaining the to-be-predicted image containing the target lesion and inputting it into the encoder in the target model for vector representation to obtain an encoded vector; then inputting the encoded vector into the mapper in the target model to obtain a target vector, where the mapper is trained based on lesion images, and the lesion image pairs are composed of training images detected based on the temporal relationship, and the training images contain the target lesion; further inputting the target vector into the decoder in the target model for image prediction to obtain a predicted image containing the target lesion after changing based on the temporal relationship. Thus, the process of predicting lesion changes based on artificial intelligence is realized. Since the target model is trained in an image-to-image training manner and the training data is collected based on the actual change process of the lesion, the target model has the ability to infer lesion images, improving the accuracy of predicting lesion changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0107] Figure 1 It is a network architecture diagram for the operation of the processing system of lesion images;

[0108] Figure 2 It is a process architecture diagram for the processing of a kind of lesion images provided by the embodiments of the present application;

[0109] Figure 3 It is a flowchart of a method for processing a kind of lesion images provided by the embodiments of the present application;

[0110] Figure 4 It is a scenario schematic diagram of a method for processing a kind of lesion images provided by the embodiments of the present application;

[0111] Figure 5 It is a scenario schematic diagram of another method for processing a kind of lesion images provided by the embodiments of the present application;

[0112] Figure 6 It is a scenario schematic diagram of another method for processing a kind of lesion images provided by the embodiments of the present application;

[0113] Figure 7 It is a scenario schematic diagram of another method for processing a kind of lesion images provided by the embodiments of the present application;

[0114] Figure 8Flowchart of another method for processing lesion images provided by an embodiment of the present application;

[0115] Figure 9 Scenario schematic diagram of another method for processing lesion images provided by an embodiment of the present application;

[0116] Figure 10 Structural schematic diagram of a device for processing lesion images provided by an embodiment of the present application;

[0117] Figure 11 Structural schematic diagram of a terminal device provided by an embodiment of the present application;

[0118] Figure 12 Structural schematic diagram of a server provided by an embodiment of the present application. Detailed implementation manners

[0119] An embodiment of the present application provides a method for processing lesion images and related devices, which can be applied to a system or program with a lesion image processing function in a terminal device. By obtaining a to-be-predicted image containing a target lesion and inputting it into an encoder in a target model for vector representation to obtain a coded vector; then inputting the coded vector into a mapper in the target model to obtain a target vector, where the mapper is trained based on lesion images, and the lesion image pairs are composed of training images detected based on a temporal relationship, and the training images contain the target lesion; furthermore, inputting the target vector into a decoder in the target model for image prediction to obtain a predicted image containing the target lesion after change based on the temporal relationship. Thus, the process of predicting lesion changes based on artificial intelligence is realized. Since the target model is trained in an image-to-image training manner and the training data is collected based on the actual change process of the lesion, the target model has the ability to infer lesion images, improving the accuracy of lesion change prediction.

[0120] Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily limit to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0121] First, some terms that may appear in the embodiments of the present application are explained.

[0122] Computed Tomography (CT): It uses precisely collimated X-ray beams, gamma rays, ultrasonic waves, etc., together with highly sensitive detectors, to perform successive cross-sectional scans around a certain part of the human body. It features fast scanning time and clear images, and can be used for the examination of various diseases.

[0123] Intracerebral hemorrhage: Refers to the bleeding caused by the rupture of blood vessels in the non-traumatic cerebral parenchyma.

[0124] Dice: A medical image segmentation evaluation index, usually used to calculate the similarity between two samples. The value ranges from 0 to 1. The best segmentation result has a value of 1, and the worst has a value of 0.

[0125] Dice of Change: An index for predicting the accuracy of the hematoma change area between two consecutive CT scans.

[0126] It should be understood that the method for processing lesion images provided in this application can be applied to a system or program with a lesion image processing function in a terminal device, such as an auxiliary medical application. Specifically, the lesion image processing system can run in a network architecture as Figure 1 shown. As Figure 1 shown, it is the network architecture diagram for the operation of the lesion image processing system. As can be seen from the figure, the lesion image processing system can provide the processing process of lesion images from multiple information sources, that is, obtain lesion images and sequential images at different time points through the detection operation on the terminal side (medical detection instrument), and then send them to the server for analysis, so that the server can organize and train after collecting a certain number of lesion images to predict the development of the lesion; it can be understood that Figure 1 shows various terminal devices. The terminal device can be a computer device. In actual scenarios, more or fewer types of terminal devices may participate in the process of processing lesion images. The specific quantity and types depend on the actual scenario and are not limited here. Additionally, Figure 1 shows one server, but in actual scenarios, multiple servers may also participate. The specific number of servers depends on the actual scenario.

[0127] In this embodiment, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a medical detection device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods. The terminal and the server can be connected to form a blockchain network, which is not limited in this application.

[0128] It can be understood that the above-mentioned lesion image processing system can run on a personal mobile terminal, for example, as an application such as an auxiliary medical application, or can run on a server, or can also be run on a third-party device to provide lesion image processing to obtain the processing result of the lesion image of the information source; The specific lesion image processing system can run in the above-mentioned device in the form of a program, or can run as a system component in the above-mentioned device, or can also be a kind of cloud service program. The specific operation mode depends on the actual scenario and is not limited here.

[0129] With the rapid development of medical technology, more and more detection items appear in medical scenarios. How to identify and predict lesions in the detection images generated in medical detections has become a difficult problem.

[0130] Generally, the prediction of lesion areas in medical images requires identification based on doctors' professional knowledge and experience, and has relatively high requirements for doctors' professional skills.

[0131] However, the process of manual prediction is time-consuming and laborious, and is affected by subjective factors. The results of lesion prediction are not stable, affecting the accuracy of lesion change prediction.

[0132] To solve the above problems, this application proposes a method for processing lesion images using machine learning. Machine Learning (ML) is a multi-disciplinary cross-discipline that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specializes in studying how computers simulate or implement human learning behaviors to acquire new knowledge or skills, and reorganize the existing knowledge structure to continuously improve its own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0133] Specifically, this method is applied to Figure 2 the process framework for processing the lesion images shown in Figure 2 As shown in Figure 2 , it is a flowchart of a process architecture for processing lesion images provided by an embodiment of the present application. The user controls a medical device to perform an inspection operation, so that the server receives the lesion images and sorts them according to time, thereby obtaining a pair of lesion images. Therefore, the model can be trained based on the pair of lesion images, and the trained model can be used for predicting lesion changes. The present application can be applied to an auxiliary analysis device for CT images.

[0134] It can be understood that the method provided by the present application can be a program writing to serve as a processing logic in a hardware system, or can also be a processing device for lesion images, and the above processing logic is implemented in an integrated or external connection manner. As an implementation method, the processing device for lesion images obtains a to-be-predicted image containing a target lesion and inputs it into an encoder in the target model for vector representation to obtain an encoded vector; then the encoded vector is input into a mapper in the target model to obtain a target vector. The mapper is trained based on a pair of lesion images, and the pair of lesion images consists of training images detected based on a temporal relationship. The training images contain the target lesion; furthermore, the target vector is input into a decoder in the target model for image prediction to obtain a predicted image containing the target lesion after change based on the temporal relationship. Thus, the process of predicting lesion changes based on artificial intelligence is realized. Since the target model is trained by an image-to-image training method and the training data is collected based on the actual change process of the lesion, the target model has the ability to infer lesion images, improving the accuracy of predicting lesion changes.

[0135] The solution provided by the embodiment of the present application relates to the machine learning technology of artificial intelligence, and is specifically described through the following embodiments:

[0136] Combined with the above process architecture, the method for processing lesion images in the present application will be introduced below. Please refer to Figure 3 , Figure 3 As shown in Figure 3 , it is a flowchart of a method for processing lesion images provided by an embodiment of the present application. This prediction method can be executed by a terminal, or by a server, or jointly by a terminal and a server. Here, taking the execution by the terminal as an example, the embodiment of the present application at least includes the following steps:

[0137] 301. Obtain a to-be-predicted image containing a target lesion and input it into an encoder in the target model for vector representation to obtain an encoded vector.

[0138] In this embodiment, the target lesion can be images of lesions in different parts, including but not limited to disease lesion types such as cerebral hematoma, tumor, tuberculosis, etc. The present application takes the change prediction of cerebral hematoma as an example for illustration.

[0139] Specifically, the target model is a generative adversarial network, which specifically includes an encoder, a mapper, a decoder, and a discriminator. Among them, the encoder can be composed of multiple convolutional layers and a fully connected layer, and its function is to vectorize the image to be predicted for subsequent mapping processing.

[0140] 302. Input the encoded vector into the mapper in the target model to perform vector conversion of the image dimension to obtain a target vector.

[0141] In this embodiment, through the vector conversion of the image dimension, a target vector for indicating the image after the development of the target lesion is obtained. This is because the target model of this application is based on the discrimination and training of the correspondence relationship of the images, that is, the mapper is obtained by training based on the lesion images, and the lesion image pairs are composed of training images detected based on the temporal relationship. The training images contain the target lesion. The temporal relationship can be a fixed time period (such as two or more images within adjacent 24 hours), or a dynamic time period (such as two or more lesion development images at different treatment stages), or a time period set based on the lesion development (such as two or more images before and after the lesion change, that is, the interval duration is not fixed).

[0142] It can be understood that the training process of the mapper is the overall training process of the target model. During the training process of the mapper, parameter adjustment of the encoder, decoder, and discriminator can be involved.

[0143] The training process of the model will be described below.

[0144] First, collect multiple lesion image pairs. The lesion image pair includes a first image and a second image. The first image is an image obtained by detecting the target lesion at the first time node, and the second image is an image obtained by detecting the target lesion at the second time node. The second time node is after the first time node.

[0145] In a possible scenario, as Figure 4 shown, Figure 4 This is a schematic diagram of the scenario of another method for processing lesion images provided by the embodiment of the present application; a pair of lesion image pairs are shown in the figure. That is, the first image contains the lesion area A1 at the first time node, and the second image contains the lesion area A2 at the second time node. That is, the lesion area A2 is changed from the lesion area A1, thus showing the change of the lesion.

[0146] This application constructs image pairs for training by collecting CT images of a large number of patients at different time points, enabling the model to ultimately predict the possible future state based on a single CT image. In this application, the CT obtained at the previous time node is defined as CT1, and the image obtained at the subsequent time node is defined as CT2.

[0147] Optionally, since the change cycles of different lesions are different, when collecting lesion image pairs, the cycle duration can be considered. That is, first determine the type information corresponding to the target lesion; then determine the lesion change cycle based on the type information; and then use the change duration corresponding to the lesion change cycle as the acquisition interval to collect multiple lesion image pairs. This ensures the accuracy of the collection of lesion image pairs, that is, the development of the collected lesion image pairs is relatively determined.

[0148] In a possible scenario, the lesion image pair is obtained by computed tomography, and the target lesion is the cerebral hemorrhage area obtained by computed tomography. The prediction image is used to predict the change trend of the cerebral hemorrhage area. The following embodiments will be described by taking this scenario as an example, and the specific detection equipment and lesion types are not limited.

[0149] Then, the target model is trained based on the lesion image pair. Among them, the training process of the target model includes a reconstruction task and a prediction task. The reconstruction task is used to reconstruct the image based on the first image, and the prediction task is used to learn the corresponding relationship between the first image and the second image; specifically, the target model can be a deep generative adversarial network, which includes 4 modules, namely an encoder, a decoder, a discriminator, and a mapper. Specifically, first perform an identity mapping (decoding by the decoder after encoding by the encoder) on the first image in the lesion image pair to obtain a reconstructed decoded image; then execute the reconstruction task according to the process of the discriminator reconstructing and decoding the image to recover to the first image; if the similarity between the reconstructed decoded image and the first image reaches the threshold, end the reconstruction task; and then the mapper executes the prediction task based on the corresponding relationship between the first image and the second image to train the target model.

[0150] Specifically, the process of predicting the subsequent changes based on a single CT image is decomposed into two subtasks: the first subtask is to reconstruct the image. When the model has the ability to completely recover a CT from the network, the model will perform the second subtask, that is, change prediction. For the two subtasks, the training of the model is divided into two stages, namely identity mapping and target mapping. Next, the training processes of the two stages will be introduced separately.

[0151] For the process of identity mapping, refer to Figure 5 the scene architecture shown Figure 5Schematic diagram of the scenario of another method for processing lesion images provided by the embodiments of the present application; that is, first, the encoder performs an encoding operation on the first image in the pair of lesion images to compress the feature dimension of the first image to obtain a first feature vector; then, the decoder performs a decoding operation on the first feature vector to obtain a reconstructed decoded image based on a similarity constraint using a first constraint function. The feature dimension of the reconstructed decoded image is restored from the first feature vector to the feature dimension corresponding to the first image. Specifically, the first constraint function can be:

[0152]

[0153] where E is the output of the encoder, D is the output of the decoder, and this function is used to minimize the difference between the first image and the reconstructed decoded image.

[0154] Further, the discriminator performs a discrimination operation on the reconstructed decoded image and the first image to obtain a discrimination result. The objective of the discriminator's discrimination operation is that the discrimination result output based on the second constraint function is the first image, and the objectives of the encoder's encoding operation and the decoder's decoding operation are that the discrimination result output based on the third constraint function is the reconstructed decoded image; where the second constraint function can be:

[0155]

[0156] where E is the output of the encoder, D is the output of the decoder, and Dis is the output of the discriminator. This function is used to instruct the discriminator to distinguish and generate the real image (the first image) as much as possible.

[0157] The third constraint function can be:

[0158]

[0159] where E is the output of the encoder, D is the output of the decoder, and Dis is the output of the discriminator. This function is used to instruct the encoder and the decoder to deceive the discriminator as much as possible, that is, the discriminator generates the reconstructed decoded image.

[0160] Thus, the reconstruction task can be performed based on the adjustment processes of the second constraint function and the third constraint function.

[0161] Optionally, for the above identity mapping process, the encoder and decoder can also be preheated, that is, attempt to perform the process of restoring the reconstructed decoded image to the first image. That is, first, obtain a preset number of preheating steps; then, based on the preset number of preheating steps, perform the similarity constraint process of the first constraint function in a loop to preheat the encoding operation and the decoding operation processes.

[0162] In a possible scenario, the above identity mapping first warms up the encoder and decoder modules. This warm-up compresses a preprocessed CT1 from the encoder to a low-dimensional representation (for example, compressing a 128*128*128 CT image to a 1024-dimensional vector), and then the decoder restores the corresponding low-dimensional representation to the original dimension and constrains it with the original image, and its constraint function is the first constraint function. Specifically, the encoder in the model consists of multiple convolutional layers and a fully connected layer, and the decoder consists of a fully connected layer and multiple deconvolutional layers.

[0163] Further, after warming up for a certain number of steps, the discriminator is added to the training. The discriminator consists of multiple convolutional layers and a fully connected layer. The image decoded by the decoder and the real image are simultaneously input into the discriminator for discrimination. The discriminator needs to distinguish and generate real images as much as possible, while the encoder and decoder need to deceive the discriminator as much as possible.

[0164] Specifically, the constraint function of the discriminator is the second constraint function, and the constraint functions of the decoder and generator are the third constraint function. During the training process, the encoder, generator, and discriminator will be alternately trained according to their respective optimization constraint functions until the model converges.

[0165] It can be understood that through the continuous alternating training of two networks with opposite objectives. When finally converging, if the discriminator can no longer determine the source of a sample, then it is equivalent to the generation network being able to generate samples that conform to the real data distribution, thus ensuring the authenticity of the output image of the target model, that is, similar to the image output by the actual medical detection instrument.

[0166] On the other hand, for the process of the target mapping, refer to Figure 6 the scenario architecture shown in Figure 6 which is a schematic diagram of the scenario of another method for processing lesion images provided by the embodiments of the present application; that is, first, the first image is vectorized based on the encoder to obtain a low-dimensional vector; then the low-dimensional vector is input into the mapper to obtain a prediction vector; then, based on the prediction vector, a conversion is made to the vector corresponding to the second image to adjust the fourth constraint function; and finally, a prediction task is performed based on the adjustment process of the fourth constraint function to train the target model. Specifically, the fourth constraint function can be:

[0167] Among them, E is the output of the encoder, D is the output of the decoder, M is the output of the mapper, and L is the similarity constraint.

[0168] In a possible scenario, the modules participating in the training during the target mapping process are the mapper, encoder, and decoder, where the parameters of the encoder and decoder are inherited from the previous stage. During the training process, after CT1 obtains the low-dimensional representation embedding (feature vector 1) through the encoder, it is input into the mapper to obtain the predicted embedding. The predicted embedding is decoded by the decoder to obtain the predicted CT image CTpred, and CTpred is made close to the vector representation (feature vector 2) of CT2. The mapper consists of multiple fully connected layers, and its function is to perform the conversion from CT1 to CT2 in the embedding dimension. During the entire training process, the parameters of the modules other than the mapper are fixed.

[0169] It can be understood that the similarity constraint L in the above two training stages is the mean square error constraint. Since the optimization objective of the first stage is to reconstruct the image, a large amount of unlabeled data can be used in this training stage to ensure the reconstruction quality of the data.

[0170] Optionally, the first image and the second image participating in the training can be preprocessed, that is, first crop the images in the lesion image pair; then align the images in the cropped lesion image pair based on a preset angle to obtain an aligned image pair; and adjust the images in the aligned image pair to the same feature dimension; furthermore, perform normalization processing on the images in the adjusted aligned image pair to update the lesion image pair. Thus, it is ensured that the lesion positions of the first image and the second image are aligned, facilitating subsequent vector processing.

[0171] Optionally, since different lesions correspond to different body parts, a targeted cropping process can be performed, that is, first determine the body part information corresponding to the target lesion; then determine the cropping items based on the body part information; and then crop the images in the lesion image pair according to the cropping items.

[0172] In a possible scenario, the main objective of the above preprocessing process is to align CT1 and CT2 as much as possible and remove all differences except those caused by the development of the lesion itself. First, the present application will use an image cropping method to remove the areas in the CT except for the brain tissue, including head cropping and bone removal. Then, an image registration technique will be used to register CT1 and CT2 to the same angle so that the positions of the lesions are aligned. Further, in order to align the different scan slice thicknesses, all images will be resampled to the same slice thickness, and all data will be fixed to the same dimension by means of padding and cropping. Finally, the images will be normalized and their value ranges will be mapped to the range from -1 to 1, thus ensuring the alignment relationship between CT1 and CT2 and improving the accuracy of the data in the model training process.

[0173] Optionally, after training the target model, a model effect evaluation process can also be carried out. This is because the prediction target of the present application is to accurately depict the hematoma area on CT2. For samples with little change (negative samples), it is hoped to remain as original as possible, and for samples with more changes, the changed parts can be depicted as accurately as possible. Therefore, the reference lesion area can be obtained first; then the negative sample parameters can be determined based on the similarity between the reference lesion area and the lesion area of the first image; and the first difference part between the reference lesion area and the lesion area of the first image can be determined; further, the second difference part between the lesion area of the second image and the lesion area of the first image can be determined; then the positive sample parameters can be determined according to the first difference part and the second difference part; and then the evaluation index can be determined based on the negative sample parameters and the positive sample parameters. The evaluation index is used to indicate the prediction accuracy of the target model. For example, first, a hematoma segmentation method is used to segment the hematoma area of the generated image to obtain M gen (reference lesion area), and then the dice between M gen and the hematoma area of CT1 is calculated for negative samples (samples that do not meet the clinical judgment of enlargement conditions) and defined as D neg ; for positive samples, the present application will calculate the dice between the different part between M gen and CT1 and the different part between CT2 and CT1, and define it as D pos . Then, the average between the two is calculated through a weighted formula as the index for evaluating the model. The weighted formula can be:

[0174] Avg = 2 * (D neg * D pos ) / (D neg + D pos )

[0175] where Avg is the model index, and D neg is the similarity between M gen and the hematoma area of CT1, and Dpos For M gen The similarity between the different part between M and CT1 and the different part between CT2 and CT1.

[0176] Optionally, if the evaluation index indicates that the target model does not meet the preset index, the lesion image pairs participating in the training can be audited, that is, the difference information in the lesion image pairs is determined; then the lesion image pairs are audited based on the difference information (for example, manually auditing the corresponding situations of the lesion images with smaller differences), so as to update the lesion image pairs, and the target model is trained based on the updated lesion image pairs.

[0177] 303. Input the target vector into the decoder in the target model for image prediction to obtain a predicted image including the target lesion after changing based on the temporal relationship.

[0178] In this embodiment, the parameter configuration of the decoder refers to the above process of model training. After image restoration, the result of the lesion development prediction process is obtained. The prediction process is applicable to the prediction process of the same type of lesions. For example, in Figure 7 the scenario shown, Figure 7 is a schematic diagram of the scenario of another method for processing lesion images provided by the embodiment of the present application; the figure shows a comparison diagram of obtaining a predicted image from a to-be-predicted image including a target lesion based on the trained target model. It can be seen that the image is clear and consistent with the real image.

[0179] It can be understood that the to-be-predicted images in the present application can be those that have been simply analyzed, that is, with recognition annotations; they can also be those without recognition annotations. Even without recognition annotations (such as hematoma area annotations), the data itself can be used for training. The supervision information relied on by the model comes from the objective image state, rather than human subjective annotations. The influence brought by the annotations is smaller, and the applicable range is wide.

[0180] Combined with the above embodiments, the present application presents the change trend of the hematoma from the perspective of image prediction. Specifically, a to-be-predicted image including a target lesion is obtained and input into the encoder in the target model for vector representation to obtain a coded vector; then the coded vector is input into the mapper in the target model to obtain a target vector. The mapper is obtained by training based on lesion image pairs, and the lesion image pairs are composed of training images detected based on the temporal relationship. The training images include the target lesion; furthermore, the target vector is input into the decoder in the target model for image prediction to obtain a predicted image including the target lesion after changing based on the temporal relationship. Thus, the process of predicting the change of the lesion based on artificial intelligence is realized. Since the target model is trained by an image-to-image training method, and the training data is collected based on the actual change process of the lesion, the target model has the ability to speculate on the lesion image, improving the accuracy of predicting the change of the lesion.

[0181] The above embodiments introduce the process of a single prediction. In a possible scenario, the present application can also perform a process of cyclic recognition. The following describes this scenario. Please refer to Figure 8 , Figure 8 which is a flowchart of another method for processing lesion images provided by an embodiment of the present application. The embodiment of the present application at least includes the following steps:

[0182] 801. Obtain a target detection image in response to a detection operation.

[0183] In this embodiment, the detection operation can be an operation performed by a detection person on a medical detection device, that is, after initiating the detection to obtain a preliminary target detection image, the subsequent prediction process is automatically triggered.

[0184] 802. Input the target detection image into a target model to obtain a first prediction image.

[0185] In this embodiment, the process of predicting the target detection image is similar to step 303 of the Figure 3 illustrated embodiment, and will not be elaborated here.

[0186] 803. Judge the case information of the target detection image.

[0187] In this embodiment, after obtaining the target detection image, medical staff can obtain preliminary case information, such as: hematoma diffusion, hematoma reduction, etc.

[0188] 804. Input the first prediction image into the target model to obtain a second prediction image.

[0189] In this embodiment, since it is uncertain whether the first prediction image is the final form of the lesion, that is, whether the lesion may still spread, shrink or remain unchanged, etc., the process of inputting into the target model for a second time and predicting can be performed, so as to obtain a second prediction image.

[0190] 805. Perform case inference based on the second prediction image, and display the first prediction image and the second prediction image on the basis of the target detection image.

[0191] In this embodiment, if the cyclic image (the second prediction image) is the same as the prediction image (the first prediction image), it indicates that the first prediction image may be the current stable state of the lesion, and the prediction image can be determined as the prediction result; if the cyclic image (the second prediction image) is different from the prediction image (the first prediction image), then predict the change situation of the target lesion based on the cyclic image, or determine the second prediction image as the prediction result, that is, the lesion is still in a changing state, and the first prediction image and the second prediction image can be displayed on the basis of the target detection image.

[0192] In a possible scenario, such asFigure 9 as shown Figure 9 is a schematic diagram of a scenario of another method for processing lesion images provided by an embodiment of the present application; the figure shows the process of displaying a first prediction image and a second prediction image based on a target detection image, that is, the target detection image B1, the first prediction image B2, and the second prediction image B3 are a process of gradually expanding outwards, thereby dynamically displaying the changes of the lesion.

[0193] Based on the method of the generative adversarial network, the above embodiment can obtain the change trend of the target lesion through the collected CT images, and generate corresponding CT images, providing a richer reference basis for the prognosis analysis of patients.

[0194] To better implement the above solution of the embodiment of the present application, the following also provides a related device for implementing the above solution. Please refer to Figure 10 , Figure 10 is a schematic structural diagram of a device for processing lesion images provided by an embodiment of the present application. The processing device 1000 includes:

[0195] An encoding unit 1001, configured to obtain a to-be-predicted image including a target lesion and input it into an encoder in a target model for vector representation to obtain an encoded vector;

[0196] A mapping unit 1002, configured to input the encoded vector into a mapper in the target model for vector conversion in the image dimension to obtain a target vector, where the mapper is obtained by training based on lesion images, and the lesion image pairs are composed of training images detected based on a time sequence relationship, and the training images include the target lesion;

[0197] A processing unit 1003, configured to input the target vector into a decoder in the target model for image prediction to obtain a prediction image including the target lesion after changing based on the time sequence relationship.

[0198] Optionally, in some possible implementation manners of the present application, the device for processing lesion images further includes:

[0199] A training unit 1004, specifically configured to collect a plurality of lesion image pairs, where the lesion image pairs include a first image and a second image as the training images, the first image is an image obtained by detecting a target lesion at a first time node, and the second image is an image obtained by detecting the target lesion at a second time node, and the second time node is after the first time node;

[0200] The training unit 1004 is specifically configured to train a target model based on the lesion images. The training process of the target model includes a reconstruction task and a prediction task. The reconstruction task is used to reconstruct an image based on the first image, and the prediction task is used to learn the correspondence between the first image and the second image.

[0201] Optionally, in some possible implementation manners of the present application, the training unit 1004 is specifically configured to perform an identity mapping on the first image in the lesion image pair to obtain a reconstructed decoded image;

[0202] The training unit 1004 is specifically configured to perform the reconstruction task according to the process of restoring the reconstructed decoded image to the first image;

[0203] The training unit 1004 is specifically configured to end the reconstruction task if the similarity between the reconstructed decoded image and the first image reaches a threshold;

[0204] The training unit 1004 is specifically configured to perform the prediction task based on the correspondence between the first image and the second image to train the target model.

[0205] Optionally, in some possible implementation manners of the present application, the training unit 1004 is specifically configured to perform an encoding operation on the first image in the lesion image pair to compress the feature dimension of the first image to obtain a first feature vector;

[0206] The training unit 1004 is specifically configured to perform a decoding operation on the first feature vector to obtain a reconstructed decoded image by performing similarity constraint based on a first constraint function, and the feature dimension of the reconstructed decoded image is restored from the first feature vector to the feature dimension corresponding to the first image;

[0207] The training unit 1004 is specifically configured to perform a discrimination operation on the reconstructed decoded image and the first image to obtain a discrimination result. The objective of the discrimination operation is that the discrimination result output based on a second constraint function is the first image, and the objectives of the encoding operation and the decoding operation are that the discrimination result output based on a third constraint function is the reconstructed decoded image;

[0208] The training unit 1004 is specifically configured to perform the reconstruction task based on the adjustment process of the second constraint function and the third constraint function.

[0209] Optionally, in some possible implementation manners of the present application, the training unit 1004 is specifically configured to obtain a preset warm-up step count;

[0210] The training unit 1004 is specifically configured to perform the similarity constraint process of the first constraint function in a loop based on the preset warm-up steps to warm up the processes of the encoding operation and the decoding operation.

[0211] Optionally, in some possible implementation manners of this application, the training unit 1004 is specifically configured to perform vector representation on the first image to obtain a low-dimensional vector;

[0212] The training unit 1004 is specifically configured to input the low-dimensional vector into a mapper to obtain a prediction vector;

[0213] The training unit 1004 is specifically configured to perform conversion based on the prediction vector to a vector corresponding to the second image to adjust a fourth constraint function;

[0214] The training unit 1004 is specifically configured to perform the prediction task based on the adjustment process of the fourth constraint function to train the target model.

[0215] Optionally, in some possible implementation manners of this application, the training unit 1004 is specifically configured to crop the images in the lesion image pair;

[0216] The training unit 1004 is specifically configured to align the images in the cropped lesion image pair based on a preset angle to obtain an aligned image pair;

[0217] The training unit 1004 is specifically configured to adjust the images in the aligned image pair to the same feature dimension;

[0218] The training unit 1004 is specifically configured to perform normalization processing on the images in the adjusted aligned image pair to update the lesion image pair.

[0219] Optionally, in some possible implementation manners of this application, the training unit 1004 is specifically configured to determine the body part information corresponding to the target lesion;

[0220] The training unit 1004 is specifically configured to determine a cropping item based on the body part information;

[0221] The training unit 1004 is specifically configured to crop the images in the lesion image pair according to the cropping item.

[0222] Optionally, in some possible implementation manners of this application, the training unit 1004 is specifically configured to determine the type information corresponding to the target lesion;

[0223] The training unit 1004 is specifically configured to determine the lesion change period based on the type information;

[0224] The training unit 1004 is specifically configured to collect multiple pairs of the lesion images at an acquisition interval based on the change duration corresponding to the lesion change cycle.

[0225] Optionally, in some possible implementation manners of the present application, the training unit 1004 is specifically configured to obtain a reference lesion area;

[0226] The training unit 1004 is specifically configured to determine negative sample parameters based on the similarity between the reference lesion area and the lesion area of the first image;

[0227] The training unit 1004 is specifically configured to determine a first difference part between the reference lesion area and the lesion area of the first image;

[0228] The training unit 1004 is specifically configured to determine a second difference part between the lesion area of the second image and the lesion area of the first image;

[0229] The training unit 1004 is specifically configured to determine positive sample parameters according to the first difference part and the second difference part;

[0230] The training unit 1004 is specifically configured to determine an evaluation index based on the negative sample parameters and the positive sample parameters, and the evaluation index is used to indicate the prediction accuracy of the target model.

[0231] Optionally, in some possible implementation manners of the present application, the training unit 1004 is specifically configured to determine the difference information in the pair of lesion images if the evaluation index indicates that the target model does not meet the preset index;

[0232] The training unit 1004 is specifically configured to review the pair of lesion images based on the difference information to update the pair of lesion images, and train the target model based on the updated pair of lesion images.

[0233] Optionally, in some possible implementation manners of the present application, the processing unit 1003 is specifically configured to input the prediction image into the trained target model to obtain a cyclic image;

[0234] The processing unit 1003 is specifically configured to determine the prediction image as the prediction result if the cyclic image is the same as the prediction image;

[0235] The processing unit 1003 is specifically configured to predict the change condition of the target lesion based on the cyclic image if the cyclic image is different from the prediction image.

[0236] By obtaining a to-be-predicted image containing a target lesion and inputting it into an encoder in a target model for vector representation to obtain an encoded vector; then inputting the encoded vector into a mapper in the target model to obtain a target vector, where the mapper is obtained by training based on lesion images, and the lesion image pairs are composed of training images detected based on a temporal relationship, and the training images contain the target lesion; furthermore, inputting the target vector into a decoder in the target model for image prediction to obtain a predicted image containing the target lesion after change based on the temporal relationship. Thus, the process of predicting lesion changes based on artificial intelligence is realized. Since the target model is trained in an image-to-image training manner and the training data is collected based on the actual change process of the lesion, the target model has the ability to infer lesion images, improving the accuracy of predicting lesion changes.

[0237] The embodiment of the present application also provides a terminal device. As Figure 11 shown, it is a schematic structural diagram of another terminal device provided by the embodiment of the present application. For the sake of convenience of description, only the parts related to the embodiment of the present application are shown. For the specific technical details not disclosed, please refer to the method part of the embodiment of the present application. This terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), an in-vehicle computer, etc. Taking the terminal as a mobile phone as an example:

[0238] Figure 11 Shown is a block diagram of a part of the structure of the mobile phone related to the terminal provided by the embodiment of the present application. Referring to Figure 11 , the mobile phone includes: a radio frequency (RF) circuit 1110, a memory 1120, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a wireless fidelity (WiFi) module 1170, a processor 1180, and a power supply 1190 and other components. Those skilled in the art can understand that Figure 11 the structure of the mobile phone shown in

[0239] does not limit the mobile phone, and it may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements. Figure 11 The following specifically introduces each component of the mobile phone in combination with

[0240] The RF circuit 1110 can be used for receiving and transmitting information or signals during a call. Specifically, after receiving the downlink information from the base station, it is sent to the processor 1180 for processing; in addition, the uplink data designed is sent to the base station. Generally, the RF circuit 1110 includes, but is not limited to, antennas, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 1110 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0241] The memory 1120 can be used to store software programs and modules. The processor 1180 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1120. The memory 1120 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory 1120 can include high-speed random access memory and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0242] The input unit 1130 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the mobile phone. Specifically, the input unit 1130 can include a touch panel 1131 and other input devices 1132. The touch panel 1131, also known as a touch screen, can collect touch operations of the user on or near it (such as operations of the user using any suitable object or accessory such as a finger, a stylus, etc. on or near the touch panel 1131, and air-touch operations within a certain range on the touch panel 1131), and drive corresponding connection devices according to a pre-set program. Optionally, the touch panel 1131 can include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the user, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1180, and can receive commands sent by the processor 1180 and execute them. In addition, various types such as resistive, capacitive, infrared, and surface acoustic wave can be used to implement the touch panel 1131. In addition to the touch panel 1131, the input unit 1130 can also include other input devices 1132. Specifically, the other input devices 1132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power-on keys, etc.), a trackball, a mouse, a joystick, etc.

[0243] The display unit 1140 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1140 can include a display panel 1141. Optionally, the display panel 1141 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1131 can cover the display panel 1141. When the touch panel 1131 detects a touch operation on or near it, it transmits it to the processor 1180 to determine the type of touch event. Subsequently, the processor 1180 provides a corresponding visual output on the display panel 1141 according to the type of touch event. Although in Figure 11 the touch panel 1131 and the display panel 1141 are implemented as two independent components to realize the input and input functions of the mobile phone, in some embodiments, the touch panel 1131 and the display panel 1141 can be integrated to realize the input and output functions of the mobile phone.

[0244] The mobile phone may further include at least one sensor 1150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. Among them, the ambient light sensor can adjust the brightness of the display panel 1141 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1141 and / or the backlight when the mobile phone is moved to the ear. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, and can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be elaborated here.

[0245] The audio circuit 1160, the speaker 1161, and the microphone 1162 can provide an audio interface between the user and the mobile phone. The audio circuit 1160 can transmit the electrical signal converted from the received audio data to the speaker 1161, and the speaker 1161 converts it into a sound signal for output; on the other hand, the microphone 1162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1160 and then converted into audio data. After the audio data is output to the processor 1180 for processing, it is sent through the RF circuit 1110 to, for example, another mobile phone, or the audio data is output to the memory 1120 for further processing.

[0246] WiFi belongs to short - range wireless transmission technology. The mobile phone can help users send and receive emails, browse the web, and access streaming media through the WiFi module 1170, which provides users with wireless broadband Internet access. Although Figure 11 the WiFi module 1170 is shown, it can be understood that it does not belong to the essential composition of the mobile phone and can be omitted entirely within the scope of not changing the essence of the invention according to needs.

[0247] The processor 1180 is the control center of the mobile phone, connecting various parts of the entire mobile phone using various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 1120, and by calling the data stored in the memory 1120, it executes various functions of the mobile phone and processes data. Optionally, the processor 1180 may include one or more processing units; optionally, the processor 1180 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above - mentioned modem processor may not be integrated into the processor 1180 either.

[0248] The mobile phone further includes a power supply 1190 (such as a battery) for powering each component. Optionally, the power supply can be logically connected to the processor 1180 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system.

[0249] Although not shown, the mobile phone may further include a camera, a Bluetooth module, etc., which will not be elaborated here.

[0250] In the embodiment of the present application, the processor 1180 included in the terminal further has the function of executing each step of the above-mentioned page processing method.

[0251] The embodiment of the present application also provides a server. Please refer to Figure 12 , Figure 12 FIG. is a schematic structural diagram of a server provided by the embodiment of the present application. The server 1200 may vary greatly due to different configurations or performances, and may include one or more central processing units (CPUs) 1222 (for example, one or more processors) and a memory 1232, and one or more storage media 1230 for storing application programs 1242 or data 1244 (for example, one or more mass storage devices). Among them, the memory 1232 and the storage media 1230 may be transient storage or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Further, the central processor 1222 may be configured to communicate with the storage media 1230 and execute a series of instruction operations in the storage media 1230 on the server 1200.

[0252] The server 1200 may further include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1258, and / or one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0253] The steps performed by the management device in the above embodiment may be based on the Figure 12 shown server structure.

[0254] The embodiment of the present application also provides a computer-readable storage medium, in which processing instructions for lesion images are stored. When it runs on a computer, it enables the computer to execute the steps performed by the processing device for lesion images in the method described in the foregoing Figures 3 to 9 shown embodiment.

[0255] In an embodiment of the present application, there is also provided a computer program product including a processing instruction for a lesion image. When it runs on a computer, it causes the computer to execute the steps performed by the processing device for the lesion image in the method described in the foregoing Figures 3 to 9 embodiment.

[0256] In an embodiment of the present application, there is also provided a processing system for a lesion image. The processing system for the lesion image may include Figure 10 the processing device for the lesion image in the described embodiment, or Figure 11 the terminal device in the described embodiment, or Figure 12 the described server.

[0257] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0258] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0259] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0261] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a lesion image processing device, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0262] As described above, the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of various embodiments of this application.

Claims

1. A method for processing lesion images, characterized in that, Including: Obtain a to-be-predicted image including a target lesion and input it into an encoder in a target model for vector representation to obtain an encoded vector; Input the encoded vector into a mapper in the target model for vector conversion of the image dimension to obtain a target vector. The mapper is obtained by training based on lesion image pairs. The lesion image pairs include a first image and a second image collected based on a temporal relationship. The first image is an image obtained by detecting the target lesion at a first time node, and the second image is an image obtained by detecting the target lesion at a second time node, and the second time node is after the first time node; Input the target vector into a decoder in the target model for image prediction to obtain a predicted image including the target lesion after change based on the temporal relationship; The method further includes: obtaining a reference lesion region; Determine a negative sample parameter based on the similarity between the reference lesion region and the lesion region of the first image; Determine a first difference part between the reference lesion region and the lesion region of the first image; Determine a second difference part between the lesion region of the second image and the lesion region of the first image; Determine a positive sample parameter according to the first difference part and the second difference part; Determine an evaluation index based on the negative sample parameter and the positive sample parameter, and the evaluation index is used to indicate the prediction accuracy of the target model.

2. The method according to claim 1, characterized in that The method further includes: Collect a plurality of lesion image pairs; Train the target model based on the lesion image pairs. The training process of the target model includes a reconstruction task and a prediction task. The reconstruction task is used to perform image reconstruction based on the first image, and the prediction task is used to learn the correspondence between the first image and the second image.

3. The method according to claim 2, wherein The training of the target model based on the lesion image pairs includes: Perform an identity mapping on the first image in the lesion image pair to obtain a reconstructed decoded image; Execute the reconstruction task according to the process of restoring the reconstructed decoded image to the first image; If the similarity between the reconstructed decoded image and the first image reaches a threshold, end the reconstruction task; Execute the prediction task based on the correspondence between the first image and the second image to train the target model.

4. The method according to claim 3, wherein The performing an identity mapping on the first image in the lesion image pair to obtain a reconstructed decoded image includes: Perform an encoding operation on the first image in the lesion image pair to compress the feature dimension of the first image to obtain a first feature vector; Perform a decoding operation on the first feature vector to obtain a reconstructed decoded image based on a similarity constraint of a first constraint function. The feature dimension of the reconstructed decoded image is restored from the first feature vector to the feature dimension corresponding to the first image; Perform a discrimination operation on the reconstructed decoded image and the first image to obtain a discrimination result. The objective of the discrimination operation is that the discrimination result output based on the second constraint function is the first image, and the objectives of the encoding operation and the decoding operation are that the discrimination result output based on the third constraint function is the reconstructed decoded image; The process of performing the reconstruction task by restoring from the reconstructed decoded image to the first image includes: Performing the reconstruction task based on the adjustment process of the second constraint function and the third constraint function.

5. The method according to claim 4, characterized in that, The method further includes: Obtaining a preset warm-up step count; Based on the preset warm-up step count, cyclically perform the similarity constraint process of the first constraint function to warm up the processes of the encoding operation and the decoding operation.

6. The method according to claim 3, characterized in that, The process of performing the prediction task based on the correspondence between the first image and the second image to train the target model includes: Performing vector representation on the first image to obtain a low-dimensional vector; Inputting the low-dimensional vector into a mapper to obtain a prediction vector; Based on the prediction vector, performing a conversion to the vector corresponding to the second image to adjust the fourth constraint function; Performing the prediction task based on the adjustment process of the fourth constraint function to train the target model.

7. The method according to claim 3, wherein The method further includes: Cropping the images in the lesion image pair; Aligning the images in the cropped lesion image pair based on a preset angle to obtain an aligned image pair; Adjusting the images in the aligned image pair to the same feature dimension; Performing normalization processing on the images in the adjusted aligned image pair to update the lesion image pair.

8. The method according to claim 2, wherein The process of collecting multiple lesion image pairs includes: Determining the type information corresponding to the target lesion; Based on the type information, determining the lesion change period; Using the change duration corresponding to the lesion change period as the collection interval to collect multiple lesion image pairs.

9. The method according to claim 1, characterized in that The method further includes: If the evaluation index indicates that the target model does not meet the preset index, determining the difference information in the lesion image pair; Based on the difference information, auditing the lesion image pair to update the lesion image pair, and training the target model based on the updated lesion image pair.

10. The method according to any one of claims 1-8, characterized in that, The method further includes: Inputting the predicted image into the target model to obtain a cyclic image; If the cyclic image is the same as the predicted image, determining the predicted image as the prediction result; If the cyclic image is different from the predicted image, predicting the change situation of the target lesion based on the cyclic image.

11. The method according to claim 1, wherein The lesion image pair is obtained by computed tomography. The target lesion is the cerebral hemorrhage area obtained by computed tomography, the target model is a deep generative adversarial network, and the predicted image is used to predict the change trend of the cerebral hemorrhage area.

12. A processing device for lesion images, characterized in that, Includes: An encoding unit for obtaining a to-be-predicted image containing a target lesion and inputting it into an encoder in a target model for vector representation to obtain an encoded vector; A mapping unit for inputting the encoded vector into a mapper in the target model for vector conversion of the image dimension to obtain a target vector, where the mapper is obtained by training based on lesion images, and the lesion image pair includes a first image and a second image collected based on a temporal relationship. The first image is an image obtained by detecting the target lesion at a first time node, and the second image is an image obtained by detecting the target lesion at a second time node, and the second time node is after the first time node; A processing unit for inputting the target vector into a decoder in the target model for image prediction to obtain a predicted image including the target lesion after change based on the temporal relationship; A training unit for obtaining a reference lesion area; Determining a negative sample parameter based on the similarity between the reference lesion area and the lesion area of the first image; Determining a first difference part between the reference lesion area and the lesion area of the first image; Determining a second difference part between the lesion area of the second image and the lesion area of the first image; Determining a positive sample parameter according to the first difference part and the second difference part; determining an evaluation index based on the negative sample parameter and the positive sample parameter, and the evaluation index is used to indicate the prediction accuracy of the target model.

13. The device according to claim 12, characterized in that, The training unit is further configured to: Collect a plurality of lesion image pairs; Training the target model based on the lesion image pair. The training process of the target model includes a reconstruction task and a prediction task. The reconstruction task is used for image reconstruction based on the first image, and the prediction task is used for learning the correspondence between the first image and the second image.

14. The device according to claim 13, characterized in that, Specifically, the training unit is configured to: Perform an identity mapping on the first image in the lesion image pair to obtain a reconstructed decoded image; Performing the reconstruction task according to the process of restoring the reconstructed decoded image to the first image; If the similarity between the reconstructed decoded image and the first image reaches a threshold, end the reconstruction task; Performing the prediction task based on the correspondence between the first image and the second image to train the target model.

15. The device according to claim 14, wherein Specifically, the training unit is configured to: Perform an encoding operation on the first image in the lesion image pair to compress the feature dimension of the first image to obtain a first feature vector; Performing a decoding operation on the first feature vector to obtain a reconstructed decoded image by similarity constraint based on a first constraint function, and the feature dimension of the reconstructed decoded image is restored from the first feature vector to the feature dimension corresponding to the first image; Performing a discrimination operation on the reconstructed decoded image and the first image to obtain a discrimination result. The target of the discrimination operation is that the discrimination result output based on a second constraint function is the first image, and the targets of the encoding operation and the decoding operation are that the discrimination result output based on a third constraint function is the reconstructed decoded image; Performing the reconstruction task based on the adjustment process of the second constraint function and the third constraint function.

16. The device according to claim 15, characterized in that, Specifically, the training unit is configured to: Obtain a preset warm-up step number; Based on the preset number of warm-up steps, the similarity constraint process of the first constraint function is cyclically performed to warm up the processes of the encoding operation and the decoding operation.

17. The device according to claim 14, wherein The training unit is specifically configured to: Perform vector representation on the first image to obtain a low-dimensional vector; Input the low-dimensional vector into a mapper to obtain a prediction vector; Based on the prediction vector, perform conversion towards the vector corresponding to the second image to adjust the fourth constraint function; Execute the prediction task based on the adjustment process of the fourth constraint function to train the target model.

18. The device according to claim 14, characterized in that, The training unit is specifically configured to: Crop the images in the pair of lesion images; Align the images in the pair of cropped lesion images based on a preset angle to obtain an aligned pair of images; Adjust the images in the aligned pair of images to the same feature dimension; Perform normalization processing on the images in the aligned pair of images after adjustment to update the pair of lesion images.

19. The device according to claim 13, characterized in that, The training unit is specifically configured to: Determine the type information corresponding to the target lesion; Determine the lesion change cycle based on the type information; Using the change duration corresponding to the lesion change cycle as the acquisition interval, acquire multiple pairs of lesion images.

20. The device according to claim 12, characterized in that, The training unit is specifically configured to: If the evaluation metric indicates that the target model does not meet the preset metric, determine the difference information in the pair of lesion images; Audit the pair of lesion images based on the difference information to update the pair of lesion images, and train the target model based on the updated pair of lesion images.

21. The device according to any one of claims 12-19, characterized in that, The processing unit is specifically configured to: Input the prediction image into the target model to obtain a cyclic image; If the cyclic image is the same as the prediction image, determine the prediction image as the prediction result; If the cyclic image is different from the prediction image, predict the change situation of the target lesion based on the cyclic image.

22. The device according to claim 12, characterized in that, The pair of lesion images is obtained by computed tomography, the target lesion is the cerebral hemorrhage area obtained by computed tomography, the target model is a deep generative adversarial network, and the prediction image is used to predict the change trend of the cerebral hemorrhage area.

23. A computer device, characterized in that, The computer device includes a processor and a memory: The memory is used to store program codes; the processor is used to execute the processing method of the lesion images according to the instructions in the program codes as described in any one of claims 1 to 11.

24. A computer-readable storage medium, in which instructions are stored, and when it runs on a computer, it causes the computer to execute the processing method of the lesion images as described in any one of claims 1 to 11 above.

25. A computer program product, characterized in that, The computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to execute the processing method of the lesion images as described in any one of claims 1 to 11 above.

Images