An image processing method, device, storage medium and terminal

Abstract

The application discloses an image processing method and device, a storage medium and a terminal, and relates to the technical field of image processing. The method comprises the following steps: acquiring a lung CT scan image to be processed; inputting the image into a pre-trained image processing model; and outputting a plurality of lung nodule parameter values corresponding to the lung CT scan image. The image processing model is trained and generated based on a final target vector, the final target vector is generated by feature fusion based on existing clinical information and a plurality of prediction vectors, and the plurality of prediction vector features are generated by feature fusion based on the feature vectors of each image in an existing historical lung CT scan image sequence and the shooting interval length between adjacent images. The application performs feature fusion on the plurality of prediction vectors generated according to the existing CT images shot continuously for many years in terms of time sequence and clinical information, so that the final feature value is more abundant, the accuracy of the trained model is higher, and the accuracy of identifying the lung nodule parameters in the image is improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to an image processing method, apparatus, storage medium, and terminal. Background Technology

[0002] Global cancer statistics show that in 2018, there were 18.1 million new cancer cases and 9.6 million cancer deaths. Lung cancer accounted for the largest proportion of both new cases and deaths, at 11.6% and 18.4% respectively, meaning approximately one in five lung cancer patients would die. The survival rate of lung cancer is closely related to its clinical stage at diagnosis. Because early symptoms of lung cancer are not obvious, it is often diagnosed at an advanced stage, missing the opportunity for surgical treatment and resulting in a poor prognosis.

[0003] Among existing lung cancer screening methods, low-dose computed tomography (CT) has long been considered a potential early screening tool, with evidence suggesting a 20% reduction in lung cancer mortality for high-risk populations. However, due to equipment and personnel costs, as well as the complexity of the task, accurately outputting specific parameters of lung nodules in CT images remains challenging. Lung nodules exhibit a wide range of shapes and features, making it difficult to identify and describe their specific characteristics, thus reducing the accuracy of identifying lung nodule parameters in images. Summary of the Invention

[0004] This application provides an image processing method, apparatus, storage medium, and terminal. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0006] Acquire lung CT scan images to be processed;

[0007] The lung CT scan images are input into a pre-trained image processing model; wherein, the image processing model is trained and generated based on the final target vector, the final target vector is generated by feature fusion based on existing clinical information and multiple prediction vectors, and the features of the multiple prediction vectors are generated by feature fusion based on the feature vectors of each image in the existing historical lung CT scan image sequence and the shooting interval between adjacent images;

[0008] Output the parameter values ​​of multiple lung nodules corresponding to the lung CT scan image.

[0009] Optionally, the pre-trained image processing model is generated according to the following steps, comprising:

[0010] constructing an image processing model; wherein the image processing model comprises a lung nodule detection network, a feature extractor, a normalization layer, and a full connection layer;

[0011] obtaining an existing historical lung CT scan image sequence of multiple continuous time periods;

[0012] generating a plurality of combined feature vector sequences according to the historical lung CT scan image sequence, the lung nodule detection network, the feature extractor, and the normalization layer;

[0013] calculating a first lookup key and a first information content of each feature vector in each combined feature vector sequence according to the full connection layer;

[0014] obtaining a shooting interval duration of the historical lung CT scan image corresponding to each feature vector, and generating a plurality of prediction feature vectors after feature fusion according to the first lookup key, the first information content, and the shooting interval duration of each feature vector in each combined feature vector sequence;

[0015] extracting existing clinical information, and generating a final target vector after feature fusion according to the clinical information and the plurality of prediction feature vectors;

[0016] calculating a cross-entropy loss value based on the final target vector, and generating a pre-trained image processing model when the cross-entropy loss value reaches a minimum.

[0017] Optionally, the plurality of combined feature vector sequences are generated according to the historical lung CT scan image sequence, the lung nodule detection network, the feature extractor, and the normalization layer, comprising:

[0018] determining the image with the smallest interval from the current time in the historical lung CT scan image sequence as a target lung CT scan image;

[0019] inputting the target lung CT scan image into the lung nodule detection network to output a plurality of lung nodule data corresponding to the target lung CT scan image;

[0020] cutting a plurality of CT image patch sequences of a preset size from each historical lung CT scan image according to the plurality of lung nodule data;

[0021] inputting each CT image patch sequence into the feature extractor to output a plurality of first feature vectors of each CT image patch sequence;

[0022] extracting a positive center feature of each first feature vector in the plurality of first feature vectors and inputting the positive center feature into the normalization layer to generate a plurality of second feature vectors of each CT image patch sequence;

[0023] The same lung nodule at different time sequences is selected from the plurality of second feature vectors of each CT image slice sequence to combine a plurality of feature vectors to obtain a plurality of combined feature vector sequences.

[0024] Optionally, a plurality of prediction feature vectors are generated by performing feature fusion according to the first search key, the first information content and the shooting interval duration of each feature vector in each combined feature vector sequence, including:

[0025] Each feature vector in each combined feature vector sequence is fused with the corresponding shooting interval duration to generate a first fusion feature of each feature vector.

[0026] The attention weight value of each feature vector is calculated according to the first fusion feature of each feature vector.

[0027] The attention weight value of each feature vector is multiplied by the corresponding first information content and summed to generate a plurality of prediction feature vectors.

[0028] Optionally, a final target vector is generated by performing feature fusion according to the clinical information and the plurality of prediction feature vectors, including:

[0029] A key feature vector is constructed according to the existing clinical information.

[0030] The second search key and the second information content of each prediction feature vector in the plurality of prediction feature vectors are calculated.

[0031] The attention weight of each prediction feature vector is calculated based on the key feature vector and the second search key of each prediction feature vector.

[0032] The attention weight of each prediction feature vector is multiplied by the corresponding second information content and summed to generate a final target vector.

[0033] Optionally, when the cross-entropy loss value reaches the minimum, a pre-trained image processing model is generated, including:

[0034] When the cross-entropy loss value does not reach the minimum, the network parameters of the image processing model are updated, and the step of obtaining the existing historical lung CT scan image sequence of a plurality of continuous time periods is continued.

[0035] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0036] An image acquisition module is configured to acquire a lung CT scan image to be processed.

[0037] An image input module is configured to input the lung CT scan image into a pre-trained image processing model, wherein the image processing model is generated based on a final target vector, the final target vector is generated based on feature fusion of existing clinical information and a plurality of prediction vectors, and the plurality of prediction vectors are generated based on feature fusion of feature vectors of each image in an existing historical lung CT scan image sequence and a shooting interval length between adjacent images.

[0038] An image output module is configured to output a plurality of lung nodule parameter values corresponding to the lung CT scan image.

[0039] Optionally, the device further comprises:

[0040] A model construction module is configured to construct an image processing model, wherein the image processing model comprises a lung nodule detection network, a feature extractor, a normalization layer, and a full connection layer.

[0041] An image sequence acquisition module is configured to acquire an existing historical lung CT scan image sequence of a plurality of continuous time periods.

[0042] A combined feature vector sequence generation module is configured to generate a plurality of combined feature vector sequences based on the historical lung CT scan image sequence, the lung nodule detection network, the feature extractor, and the normalization layer.

[0043] A parameter calculation module is configured to calculate a first lookup key and a first information content of each feature vector in each combined feature vector sequence based on the full connection layer.

[0044] A feature fusion module is configured to acquire a shooting interval length of a historical lung CT scan image corresponding to each feature vector, and generate a plurality of prediction feature vectors based on feature fusion of the first lookup key, the first information content, and the shooting interval length of each feature vector in each combined feature vector sequence.

[0045] A final target vector generation module is configured to extract the existing clinical information, and generate a final target vector based on feature fusion of the clinical information and the plurality of prediction feature vectors.

[0046] A model generation module is configured to calculate a cross-entropy loss value based on the final target vector, and generate a pre-trained image processing model when the cross-entropy loss value reaches a minimum value.

[0047] In a third aspect, the embodiments of the present application provide a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and performing the method steps described above.

[0048] Fourthly, embodiments of this application provide a terminal that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed by the above-described method steps.

[0049] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0050] In this embodiment, the image processing device first acquires a lung CT scan image to be processed, then inputs the image into a pre-trained image processing model, and finally outputs multiple lung nodule parameter values ​​corresponding to the lung CT scan image. The image processing model is trained and generated based on a final target vector, which is generated by feature fusion of existing clinical information and multiple prediction vectors. The features of the multiple prediction vectors are generated by feature fusion of the feature vectors of each image in an existing historical lung CT scan image sequence and the shooting interval between adjacent images. Because this application first fuses the multiple prediction vectors generated from existing CT images from multiple consecutive periods with the shooting interval between adjacent images, and then fuses the fused features with the vectors corresponding to existing clinical information, the final feature values ​​are richer, resulting in higher accuracy of the trained model, thereby improving the accuracy of identifying lung nodule parameters in the image.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0054] Figure 2 This is a flowchart illustrating an image processing model training method provided in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram illustrating the image processing model training process provided in an embodiment of this application;

[0056] Figure 4 This is a schematic diagram of lung node changes at different times provided in an embodiment of this application;

[0057] Figure 5 This is a schematic diagram illustrating the process of fusing the same nodule features at different time points, as provided in an embodiment of this application.

[0058] Figure 6is a process schematic diagram of fusing existing clinical information and uniform nodule features provided by an embodiment of the present application;

[0059] Figure 7 is a structural schematic diagram of an image processing device provided by an embodiment of the present application;

[0060] Figure 8 is a structural schematic diagram of another image processing device provided by an embodiment of the present application;

[0061] Figure 9 is a structural schematic diagram of a combined feature vector sequence generation module provided by an embodiment of the present application;

[0062] Figure 10 is a structural schematic diagram of a feature fusion module provided by an embodiment of the present application;

[0063] Figure 11 is a structural schematic diagram of a final target vector generation module provided by an embodiment of the present application;

[0064] Figure 12 is a structural schematic diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following description and drawings are illustrative of the specific embodiments of the present application and are not intended to be limiting thereof.

[0066] It should be apparent that the described embodiments are only some - but not all - of the embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative effort, fall within the scope of protection of the present application.

[0067] The following description refers to the accompanying drawings. Wherever possible, the same reference numbers in different drawings refer to the same or similar elements. The following description of the exemplary embodiments is not meant to limit or restrict the scope of the application, as set forth in the appended claims. Rather, the following description is intended to enable those skilled in the art to make and use the application, and to provide several examples of the apparatus and methods consistent with the application.

[0068] In the description of the present application, it should be understood that the terms "first", "second" and the like are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The above terms can be understood in the specific meaning in the present application according to the specific circumstances for those skilled in the art. In addition, in the description of the present application, "a plurality of" means two or more, unless otherwise specified. The association relationship of the associated objects is described, which means that there can be three kinds of relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the front and rear associated objects.

[0069] The present application provides an image processing method, device, storage medium and terminal to solve the problems in the above related technical problems. In the technical scheme provided by the present application, the present application first performs feature fusion on the plurality of prediction vectors generated according to the existing CT images of the continuous plurality of periods and the shooting interval time length between adjacent images, and then performs feature fusion on the plurality of fused features and the vector corresponding to the existing clinical information, so that the final feature value is more rich, the accuracy of the trained model is higher, and the accuracy of identifying the lung nodule parameters in the image is improved. The following exemplary embodiments are described in detail.

[0070] The following will be described in detail with reference to the accompanying drawings Figure 1 -Appendix Figure 6 The image processing method provided by the embodiment of the present application is described in detail. The method can be realized by relying on a computer program and can run on an image processing device based on the von Neumann system. The computer program can be integrated in an application or run as an independent tool application.

[0071] Please refer to Figure 1 A flowchart of an image processing method provided by the embodiment of the present application is provided. As shown in Figure 1 The method of the embodiment of the present application can include the following steps:

[0072] S101, acquiring a lung CT scan image to be processed;

[0073] Wherein, CT, i.e. computed tomography, is a scanning technology that uses precisely collimated X-ray beams, gamma rays, ultrasonic waves, etc. together with a highly sensitive detector to make one after another cross-sectional scans around a certain part of the human body, has the characteristics of fast scanning time and clear image, and can be used for the examination of various diseases. According to the different types of rays used, it can be divided into X-ray CT (X-CT) and gamma-ray CT (γ-CT) and the like. The lung CT scan image is an image obtained by continuously scanning the human lung using computed tomography.

[0074] Typically, the lung CT scan images to be processed can be obtained from a local image library or CT scan images received in real time.

[0075] In one possible implementation, when image processing is required, a local image library containing a large number of pre-saved CT scan images is first opened, and then the lung CT scan image to be processed is retrieved from the large number of CT scan images.

[0076] In another possible implementation, when image processing is required, the user terminal first establishes a communication connection with the CT scanner. After receiving the scanning command, the CT scanner performs multiple scans on the image to be processed and then sends the generated lung CT scan image to the user terminal, which can then obtain the lung CT scan image to be processed.

[0077] S102, Input the lung CT scan image into a pre-trained image processing model;

[0078] The image processing model is trained and generated based on the final target vector. The final target vector is generated by feature fusion based on existing clinical information and multiple prediction vectors. The features of the multiple prediction vectors are generated by feature fusion based on the feature vectors of each image in the existing historical lung CT scan image sequence and the shooting interval between adjacent images.

[0079] Typically, a pre-trained image processing model is a mathematical model that predicts the risk of developing lung cancer. The image processing model consists of a lung nodule detection network, a feature extractor, a normalization layer, and a fully connected layer.

[0080] Specifically, the lung nodule detection network is a neural network that detects lung nodules in lung CT scan images. This network is pre-trained, and as long as the lung CT scan image to be processed is input into the lung nodule detection network, the network can output the lung nodules present in the CT image.

[0081] In one possible implementation, after the user terminal acquires the lung CT scan image based on step S101, it first accesses a pre-trained image processing model, then inputs the lung CT scan image into the pre-trained image processing model for processing, and finally outputs multiple lung nodule parameter values ​​corresponding to the lung CT scan image after the model finishes processing.

[0082] S103, output the parameter values ​​of multiple lung nodules corresponding to the lung CT scan image.

[0083] In one possible implementation, after obtaining multiple lung nodule parameter values, these values ​​can serve as reference data for doctors to diagnose the disease, allowing them to assess the current situation based on this data.

[0084] In the embodiment of the present application, the image processing device first acquires the lung CT scan image to be processed, then inputs the image into the pre-trained image processing model, and finally outputs a plurality of lung nodule parameter values corresponding to the lung CT scan image. The image processing model is generated based on the final target vector training. The final target vector is generated based on the existing clinical information and a plurality of prediction vectors. The plurality of prediction vector features are generated based on the feature vectors of each image in the existing historical lung CT scan image sequence and the shooting interval time length between adjacent images. Since the plurality of prediction vectors generated according to the existing continuous multiple period CT images are first fused with the shooting interval time length between adjacent images, and then the fused plurality of features are fused with the vectors corresponding to the existing clinical information, the final feature value is more rich, resulting in higher accuracy of the trained model, thereby improving the accuracy of identifying lung nodule parameters in the image.

[0085] Please refer to Figure 2 A flowchart of an image processing model training method is provided for the embodiment of the present application. As shown in Figure 2 The method of the embodiment of the present application can include the following steps:

[0086] S201, constructing an image processing model;

[0087] The image processing model includes a lung nodule detection network, a feature extractor, a normalization layer, and a full connection layer.

[0088] In the embodiment of the present application, an initial image processing model needs to be constructed when training the image processing model. First, the existing lung nodule detection network, feature extractor, normalization layer, and full connection layer are obtained according to the model construction parameters. Then, the image processing model is constructed according to the lung nodule detection network, feature extractor, normalization layer, and full connection layer.

[0089] S202, acquiring a historical lung CT scan image sequence of existing continuous multiple periods;

[0090] The historical lung CT scan image sequence has a plurality of lung CT scan images, and the plurality of lung CT scan images are of existing continuous multiple periods.

[0091] For example, the data set of the lung CT scan image in the historical lung CT scan image sequence of existing continuous multiple periods is: Where n represents the number of CT pictures contained in the data set, represents the CT picture taken in the Tth year of the ith patient, and similarly represents the CT picture taken in the (T-1)th year of the ith patient, and y iwhether the patient has cancer within a year.

[0092] S203, generating a plurality of combined feature vector sequences according to the historical lung CT scan image sequence, the lung nodule detection network, the feature extractor and the normalization layer;

[0093] In the embodiment of the present application, after obtaining the historical lung CT scan image sequence, first, the image with the smallest interval from the current time in the historical lung CT scan image sequence is determined as the target lung CT scan image, then the target lung CT scan image is input into the lung nodule detection network, and a plurality of lung nodule data corresponding to the target lung CT scan image is output, then a plurality of CT image patch sequences of a preset size are cut from each historical lung CT scan image according to the plurality of lung nodule data, then each CT image patch sequence is input into the feature extractor, and a plurality of first feature vectors of each CT image patch sequence are output, and then a plurality of second feature vectors of each CT image patch sequence are generated after extracting the positive center feature of each first feature vector in the plurality of first feature vectors and inputting the positive center feature into the normalization layer, finally, a plurality of feature vectors of the same lung nodule at different time sequences are selected from the plurality of second feature vectors of each CT image patch sequence for combination, and a plurality of combined feature vector sequences are obtained.

[0094] In a possible implementation, for example Figure 3 As shown in the figure, first, a historical lung CT scan image closest to the current time is determined from the historical lung CT scan image sequence For example Then is input into the lung nodule detection network D to output corresponding plurality of lung nodule data Wherein m represents the number of nodes contained in the sample, (x j ,y j ,z j ,p j ) represents the center point coordinates of each node and the probability of the node, according to the probability size of each lung nodule, the n lung nodules with the highest probability are selected from the m lung nodules, and then the 96x96x96 cube is cut on the historical lung CT scan image sequence and with the n lung nodule center point coordinates (x j ,y j ,z j ) as the center, so that the predicted patches are and The predicted segments are then input into the feature extractor E, which outputs multiple first feature vectors for each CT image segment sequence, ultimately producing three sets of feature vectors of size 5×64×24×24×24. Since the nodules are all located at the exact center of each segment and are generally small in volume, it is necessary to extract the exact center of each feature vector to obtain a 5×64×12×12×12 feature vector, which is then processed through a maxpooling layer. The final feature vectors for CT scan images at different time points are as follows: and Each of them is 5×64. Finally, multiple feature vectors of the same lung nodule at different time sequences are selected from the feature vectors of CT scan images at different time periods and combined to obtain multiple combined feature vector sequences.

[0095] For example Figure 4 As shown, the trend of nodule changes in CT images of the same patient taken at adjacent times. Among them, (1)-(2) shows that the nodules increase significantly with time, while the nodule size in (3)-(4) does not change significantly in images taken at different times.

[0096] S204, calculate the first lookup key and first information content of each feature vector in each combined feature vector sequence based on the fully connected layer;

[0097] In one possible implementation, for example Figure 5 As shown, after combining multiple feature vectors of the same lung nodule at different time points to obtain multiple combined feature vector sequences, the first lookup key and first information content of each feature vector in each combined feature vector sequence are calculated using a fully connected layer. The calculation formula is as follows:

[0098] q i =W q f i ,v i =W v f i ,i={T-2,T-1,T};

[0099] Among them W q and W v All are fully connected layers, f i Given a 1×64 feature vector, after calculation using Formula 1, we obtain a 1×6 feature vector q and a 1×64 feature vector v. Here, q represents the lookup key for each feature, and v represents the information content of each feature.

[0100] S205, obtain the shooting interval of the historical lung CT scan image corresponding to each feature vector, and generate multiple predicted feature vectors by performing feature fusion based on the first lookup key, first information content and shooting interval of each feature vector in each combined feature vector sequence.

[0101] In the embodiments of the present application, when generating the plurality of prediction feature vectors, first, the feature vector in each combined feature vector sequence and the corresponding shooting interval length are fused to generate a first fused feature of each feature vector, then the attention weight value of each feature vector is calculated according to the first fused feature of each feature vector, and finally the attention weight value of each feature vector and the corresponding first information content are multiplied and summed to generate the plurality of prediction feature vectors.

[0102] For example Figure 5 As shown, after obtaining the first search key and the first information content of each feature vector, since the CT image is the performance of the nodule at different times, the number of days between different times is also added to the vector q, so q becomes a 1x7 vector. The addition of the time interval is conducive to the network learning more rich and useful information. Then the attention value of each nodule feature vector is calculated, and since the most recently photographed nodule feature is the most useful feature information, the search key at time T is used to calculate the attention weight of each time search key with the search keys at T-2 and T-1. The calculation formula is:

[0103] s i =<q i ,q m >,i={T-2,T-1,T};

[0104] Where <.,.> is the inner product operation of the vector. Let q m =q T .

[0105] Finally, the attention weight a i of each time search key and the corresponding first information content v i are multiplied and added to generate a plurality of vectors {h1, h2, h3, h4, h5} with a size of 1x64. The calculation formula is:

[0106]

[0107] S206, extracting existing clinical information, generating a final target vector after feature fusion between the clinical information and the plurality of prediction feature vectors;

[0108] In the embodiments of the present application, first, a key feature vector is constructed according to the existing clinical information, then the second search key and the second information content of each prediction feature vector in the plurality of prediction feature vectors are calculated, and then the attention weight of each prediction feature vector is calculated based on the key feature vector and the second search key of each prediction feature vector, and finally the attention weight of each prediction feature vector and the corresponding second information content are multiplied and summed to generate the final target vector.

[0109] In a possible implementation, for example Figure 6 As shown, first, 7 clinical information of relatively high importance are selected: {age, gender, whether smoking, location of the largest nodule in the CT image, diameter of the largest nodule in the CT image, edge of the largest nodule in the CT image, and total number of nodules in the CT image}. These are constructed into a 1x7 vector, and then a fully connected layer is used to finally obtain a 1x6 vector, that is, a key feature vector q m Then, a plurality of 1x64 size vectors {h1, h2, h3, h4, h5} are transformed to obtain a 1x6 size q and a 1x64 size v. Wherein q represents the lookup key of each feature, and v represents the information content of each feature. Then, the attention value of each nodule feature vector is calculated through the key feature vector q m and the 1x6 size q. The calculation formula of the attention value is: s i <q i ,q m >, i={1, 2,..., 5}, wherein <.,.> is a vector inner product operation. Here, q m is the vector obtained through the clinical information in the first step, so the attention weight of each nodule is calculated using the clinical attributes. Finally, each attention value is multiplied by and added to the corresponding information vector v to obtain the final target vector b. The calculation formula of the final target vector b is:

[0110]

[0111] S207, calculate the cross-entropy loss value based on the final target vector, and when the cross-entropy loss value reaches the minimum, generate the pre-trained image processing model.

[0112] In the embodiments of the present application, after obtaining the final target vector b, first, the cancer probability of the lung cancer patient is calculated according to the final target vector, then the cross-entropy loss value is calculated according to the cancer probability of the lung cancer patient, and finally when the cross-entropy loss value reaches the minimum, the pre-trained image processing model is generated, or when the cross-entropy loss value does not reach the minimum, the network parameters of the image processing model are updated, and the step of obtaining the existing historical lung CT scan image sequence of a plurality of continuous time periods is continued to continue the model training process. The calculation formula of the cross-entropy loss value is: Wherein, y i is the label of whether the patient has cancer within a year, and p i is the cancer probability of the lung cancer patient.

[0113] Wherein, the probability of finally having cancer is calculated as: p=W b b, wherein W bNetwork parameters of the image processing model.

[0114] Further, in order to enhance the explainability of the model, seven clinical information which have greater influence on lung cancer are adopted and normalized. When the self-attention mechanism is used to fuse the nodule features, the normalized clinical information and each nodule are compared to calculate the weight. In this way, the proportion of the nodules similar to the clinical attributes is increased, which can not only improve the prediction accuracy, but also greatly enhance the medical explainability of the model.

[0115] In the embodiment of the present application, the image processing device first acquires the lung CT scan image to be processed, then inputs the image into the pre-trained image processing model, and finally outputs a plurality of lung nodule parameter values corresponding to the lung CT scan image. The image processing model is trained based on the final target vector, which is generated by feature fusion based on existing clinical information and a plurality of prediction vectors. The features of the plurality of prediction vectors are generated by feature fusion based on the feature vectors of each image in the existing historical lung CT scan image sequence and the shooting interval length between adjacent images. Since the present application first performs feature fusion on the plurality of prediction vectors generated according to the existing CT images of a plurality of consecutive time periods and the shooting interval length between adjacent images, and then performs feature fusion on the fused plurality of features and the vector corresponding to the existing clinical information, the final feature value is more rich, which leads to higher accuracy of the trained model, and thus improves the accuracy of identifying lung nodule parameters in the image.

[0116] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0117] Please refer to Figure 7 which shows the structure diagram of the image processing device provided by an exemplary embodiment of the present application. The image processing device can be realized by software, hardware or a combination of both to become all or part of the terminal. The device 1 includes an image acquisition module 10, an image input module 20, and a cancer risk determination module 30.

[0118] The image acquisition module 10 is configured to acquire a lung CT scan image to be processed.

[0119] The image input module 20 is configured to input the lung CT scan image into a pre-trained image processing model. The image processing model is trained based on a final target vector, which is generated by feature fusion based on existing clinical information and a plurality of prediction vectors. The features of the plurality of prediction vectors are generated by feature fusion based on the feature vectors of each image in the existing historical lung CT scan image sequence and the shooting interval length between adjacent images.

[0120] An image output module 30 is configured to output a plurality of lung nodule parameter values corresponding to the lung CT scan image.

[0121] Optionally, as shown in Figure 8 The device 1 further comprises:

[0122] A model construction module 40 is configured to construct an image processing model, wherein the image processing model comprises a lung nodule detection network, a feature extractor, a normalization layer, and a full connection layer.

[0123] An image sequence acquisition module 50 is configured to acquire a historical lung CT scan image sequence of a plurality of continuous time periods.

[0124] A combined feature vector sequence generation module 60 is configured to generate a plurality of combined feature vector sequences according to the historical lung CT scan image sequence, the lung nodule detection network, the feature extractor, and the normalization layer.

[0125] A parameter calculation module 70 is configured to calculate a first lookup key and a first information content of each feature vector in each combined feature vector sequence according to the full connection layer.

[0126] A feature fusion module 80 is configured to acquire a shooting interval duration of a historical lung CT scan image corresponding to each feature vector, and generate a plurality of predicted feature vectors after feature fusion according to the first lookup key, the first information content, and the shooting interval duration of each feature vector in each combined feature vector sequence.

[0127] A final target vector generation module 90 is configured to extract the existing clinical information, and generate a final target vector after feature fusion according to the clinical information and the plurality of predicted feature vectors.

[0128] A model generation module 100 is configured to calculate a cross-entropy loss value based on the final target vector, and generate a pre-trained image processing model when the cross-entropy loss value reaches a minimum value.

[0129] Optionally, as shown in Figure 9 The combined feature vector sequence generation module 60 comprises:

[0130] A target lung CT scan image determination unit 601 is configured to determine an image with the smallest interval from the current time in the historical lung CT scan image sequence as a target lung CT scan image.

[0131] A plurality of lung nodule data output units 602 are configured to input the target lung CT scan image into the lung nodule detection network, and output a plurality of lung nodule data corresponding to the target lung CT scan image.

[0132] The CT image patch cutting unit 603 is configured to cut a plurality of CT image patch sequences of a preset size from each of the historical lung CT scan images according to the plurality of lung nodule data;

[0133] The first feature vector output unit 604 is configured to input each of the CT image patch sequences into the feature extractor and output a plurality of first feature vectors of each of the CT image patch sequences;

[0134] The second feature vector output unit 605 is configured to extract a positive center feature of each of the first feature vectors and generate a plurality of second feature vectors of each of the CT image patch sequences after inputting the plurality of second feature vectors into the normalization layer;

[0135] The combined feature vector sequence generation unit 606 is configured to select a plurality of feature vectors of the same lung nodule at different time sequences from the plurality of second feature vectors of each of the CT image patch sequences and combine the plurality of feature vectors to obtain a plurality of combined feature vector sequences.

[0136] Optionally, as shown in Figure 10 The feature fusion module 80 includes:

[0137] The first fusion feature generation unit 801 is configured to fuse each feature vector in each of the combined feature vector sequences with the shooting interval length corresponding to the feature vector to generate a first fusion feature of each feature vector;

[0138] The attention weight value calculation unit 802 is configured to calculate an attention weight value of each feature vector according to the first fusion feature of the feature vector;

[0139] The predicted feature vector generation unit 803 is configured to multiply the attention weight value of each feature vector by the first information content corresponding to the feature vector and sum the multiplication results to generate a plurality of predicted feature vectors.

[0140] Optionally, as shown in Figure 11 The final target vector generation module 90 includes:

[0141] The key feature vector construction unit 901 is configured to construct a key feature vector according to the existing clinical information;

[0142] The parameter calculation unit 902 is configured to calculate a second lookup key and second information content of each predicted feature vector in the plurality of predicted feature vectors;

[0143] The predicted feature vector attention weight calculation unit 903 is configured to calculate an attention weight of each predicted feature vector based on the key feature vector and the second lookup key of the predicted feature vector;

[0144] The final target vector generation unit 904 is configured to multiply the attention weight of each prediction feature vector by the second information content corresponding to the prediction feature vector and sum the results to generate a final target vector.

[0145] It should be noted that the image processing apparatus provided in the above embodiments is only used to illustrate the division of the above functional modules when the image processing method is performed, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and the image processing method provided in the above embodiments belong to the same concept, and the implementation process is embodied in the method embodiments, which will not be described here.

[0146] The serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0147] In the embodiments of the present application, the image processing apparatus first acquires a lung CT scan image to be processed, then inputs the image into a pre-trained image processing model, and finally outputs a plurality of lung nodule parameter values corresponding to the lung CT scan image. The image processing model is trained based on a final target vector, which is generated by feature fusion based on existing clinical information and a plurality of prediction vectors. The features of the plurality of prediction vectors are generated by feature fusion based on the feature vectors of each image in the existing historical lung CT scan image sequence and the shooting interval duration between adjacent images. Since the present application first performs feature fusion on the plurality of prediction vectors generated according to the existing CT images of a plurality of consecutive time periods and the shooting interval duration between adjacent images, and then performs feature fusion on the plurality of features after fusion and the vectors corresponding to the existing clinical information, the final feature value is more rich, resulting in a higher accuracy of the trained model, and thus the accuracy of identifying the lung nodule parameters in the image is improved.

[0148] The present application also provides a computer readable medium having program instructions stored thereon, which, when executed by a processor, implement the image processing method provided by each of the method embodiments.

[0149] The present application also provides a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the image processing method of each of the method embodiments.

[0150] Please refer to Figure 12 The present application provides a structural schematic diagram of a terminal. As shown in Figure 12 The terminal 1000 can include at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0151] The communication bus 1002 is configured to realize the connection communication between the components.

[0152] The user interface 1003 can include a display screen, a camera, and optionally a standard wired interface and a wireless interface.

[0153] The network interface 1004 can optionally include a standard wired interface and a wireless interface (e.g., a WI-FI interface).

[0154] The processor 1001 can include one or more processing cores. The processor 1001 is connected to various parts of the electronic device 1000 through various interfaces and lines, and performs various functions of the electronic device 1000 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 1001 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU is mainly used to process the operating system, the user interface, and the application program; the GPU is used to render and draw the content to be displayed on the display screen; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 1001, but can be realized by a separate chip.

[0155] The memory 1005 can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1005 can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area can store data involved in the above-mentioned various method embodiments, etc. The memory 1005 can also be at least one storage device located away from the aforementioned processor 1001. As shown in Figure 12 The memory 1005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and an image processing application program.

[0156] In the terminal 1000 shown in Figure 12 In the terminal 1000 shown in

[0157] Obtain a lung CT scan image to be processed;

[0158] Input the lung CT scan image into a pre-trained image processing model; wherein the image processing model is trained and generated based on a final target vector, the final target vector is generated by feature fusion based on existing clinical information and a plurality of prediction vectors, and the plurality of prediction vector features are generated by feature fusion based on the feature vectors of each image in the existing historical lung CT scan image sequence and the shooting interval time between adjacent images;

[0159] Output a plurality of lung nodule parameter values corresponding to the lung CT scan image.

[0160] In one embodiment, the processor 1001 specifically performs the following operations when generating the pre-trained image processing model:

[0161] Construct an image processing model; wherein the image processing model includes a lung nodule detection network, a feature extractor, a normalization layer, and a full connection layer;

[0162] Obtain a historical lung CT scan image sequence of a plurality of continuous time periods;

[0163] generate a plurality of combined feature vector sequences according to the historical lung CT scan image sequence, the lung nodule detection network, the feature extractor, and the normalization layer;

[0164] calculate a first lookup key and a first information content of each feature vector in each combined feature vector sequence according to the full connection layer;

[0165] obtain a shooting interval duration of the historical lung CT scan image corresponding to each feature vector, and generate a plurality of predicted feature vectors after feature fusion according to the first lookup key, the first information content, and the shooting interval duration of each feature vector in each combined feature vector sequence;

[0166] extract existing clinical information, and generate a final target vector after feature fusion according to the clinical information and the plurality of predicted feature vectors;

[0167] calculate a cross-entropy loss value based on the final target vector, and generate a pre-trained image processing model when the cross-entropy loss value reaches a minimum.

[0168] In one embodiment, the processor 1001, when generating a plurality of combined feature vector sequences according to the historical lung CT scan image sequence, the lung nodule detection network, the feature extractor, and the normalization layer, specifically performs the following operations:

[0169] determines the image with the smallest interval from the current time in the historical lung CT scan image sequence as a target lung CT scan image;

[0170] inputs the target lung CT scan image into the lung nodule detection network, and outputs a plurality of lung nodule data corresponding to the target lung CT scan image;

[0171] cuts a plurality of CT image patch sequences of a preset size from each historical lung CT scan image according to the plurality of lung nodule data;

[0172] inputs each CT image patch sequence into the feature extractor, and outputs a plurality of first feature vectors of each CT image patch sequence;

[0173] extracts a positive center feature of each first feature vector in the plurality of first feature vectors, inputs the positive center feature into the normalization layer, and generates a plurality of second feature vectors of each CT image patch sequence;

[0174] selects a plurality of feature vectors of the same lung nodule at different time sequences from the plurality of second feature vectors of each CT image patch sequence for combination, and obtains a plurality of combined feature vector sequences.

[0175] In an embodiment, when the processor 1001 generates the plurality of predicted feature vectors by performing feature fusion according to the first lookup key, the first information content, and the shooting interval duration of each feature vector in each combined feature vector sequence, the processor 1001 specifically performs the following operations:

[0176] Fusing each feature vector in each combined feature vector sequence and the shooting interval duration corresponding thereto to generate a first fusion feature of each feature vector;

[0177] Calculating an attention weight value of each feature vector according to the first fusion feature of the feature vector;

[0178] Multiplying and summing the attention weight value of each feature vector and the first information content corresponding thereto to generate the plurality of predicted feature vectors.

[0179] In an embodiment, when the processor 1001 generates the final target vector by performing feature fusion according to the clinical information and the plurality of predicted feature vectors, the processor 1001 specifically performs the following operations:

[0180] Constructing a key feature vector according to the existing clinical information;

[0181] Calculating a second lookup key and a second information content of each predicted feature vector in the plurality of predicted feature vectors;

[0182] Calculating an attention weight of each predicted feature vector based on the key feature vector and the second lookup key of the predicted feature vector;

[0183] Multiplying and summing the attention weight of each predicted feature vector and the second information content corresponding thereto to generate the final target vector.

[0184] In an embodiment, when the processor 1001 generates the pre-trained image processing model when the cross-entropy loss value reaches the minimum, the processor 1001 specifically performs the following operations:

[0185] When the cross-entropy loss value does not reach the minimum, updating the network parameters of the image processing model, and continuing to perform the step of obtaining the historical lung CT scan image sequence of the existing continuous multiple epochs.

[0186] In the embodiment of the present application, the image processing apparatus first acquires a lung CT scan image to be processed, then inputs the image into a pre-trained image processing model, and finally outputs a plurality of lung nodule parameter values corresponding to the lung CT scan image. The image processing model is generated based on a final target vector. The final target vector is generated by feature fusion based on existing clinical information and a plurality of prediction vectors. The plurality of prediction vector features are generated by feature fusion based on the feature vectors of each image in the existing historical lung CT scan image sequence and the shooting interval length between adjacent images. Since the present application first performs feature fusion on the plurality of prediction vectors generated according to the existing CT images of a plurality of consecutive time periods and the shooting interval length between adjacent images, and then performs feature fusion on the plurality of fused features and the vector corresponding to the existing clinical information, the final feature value is more rich, which leads to higher accuracy of the trained model, and further improves the accuracy of identifying lung nodule parameters in the image.

[0187] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The image processing program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disc, an optical disc, a read-only memory, a random access memory, etc.

[0188] The above only describes the preferred embodiments of the present application, and of course cannot limit the scope of the rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire lung CT scan images to be processed; The lung CT scan images are input into a pre-trained image processing model; Output the parameter values ​​of multiple lung nodules corresponding to the lung CT scan image; Generate a pre-trained image processing model by following these steps: An image processing model is constructed; wherein the image processing model includes a lung nodule detection network, a feature extractor, a normalization layer, and a fully connected layer; Obtain existing historical lung CT scan image sequences from multiple consecutive periods; The image with the smallest interval from the current time in the historical lung CT scan image sequence is determined as the target lung CT scan image; The target lung CT scan image is input into the lung nodule detection network, and multiple lung nodule data corresponding to the target lung CT scan image are output. Based on the multiple lung nodule data, a sequence of multiple CT image blocks of a preset size is cut out from each of the historical lung CT scan images; Each CT image block sequence is input into the feature extractor, which outputs multiple first feature vectors for each CT image block sequence. After extracting the central feature of each of the plurality of first feature vectors and inputting it into the normalization layer, a plurality of second feature vectors are generated for each CT image block sequence. Multiple feature vectors of the same lung nodule at different time sequences are selected from multiple second feature vectors of each CT image block sequence and combined to obtain multiple combined feature vector sequences; Calculate the first lookup key and first information content of each feature vector in each of the combined feature vector sequences based on the fully connected layer; The acquisition of the shooting interval of the historical lung CT scan image corresponding to each feature vector is obtained, and multiple predicted feature vectors are generated by feature fusion based on the first lookup key, first information content and shooting interval of each feature vector in each combined feature vector sequence. Existing clinical information is extracted, and the final target vector is generated by fusing the clinical information with the multiple predicted feature vectors. The cross-entropy loss value is calculated based on the final target vector, and a pre-trained image processing model is generated when the cross-entropy loss value reaches its minimum.

2. The method according to claim 1, characterized in that, The step of generating multiple predicted feature vectors by fusing features based on the first lookup key, first information content, and shooting interval duration of each feature vector in each of the combined feature vector sequences includes: Each feature vector in each of the combined feature vector sequences and its corresponding shooting interval duration are fused to generate a first fused feature for each feature vector; The attention weight value of each feature vector is calculated based on the first fusion feature of each feature vector; The attention weight value of each feature vector is multiplied by the first information content corresponding to it and then summed to generate multiple predicted feature vectors.

3. The method according to claim 2, characterized in that, The step of generating the final target vector by fusing the clinical information with the multiple predicted feature vectors includes: Construct key feature vectors based on the existing clinical information; Calculate the second lookup key and second information content for each of the plurality of predicted feature vectors; The attention weight of each predicted feature vector is calculated based on the key feature vector and the second lookup key of each predicted feature vector; The attention weight of each predicted feature vector and its corresponding second information content are multiplied by scalar and summed to generate the final target vector.

4. The method according to claim 1, characterized in that, When the cross-entropy loss value reaches its minimum, a pre-trained image processing model is generated, including: When the cross-entropy loss value has not reached its minimum, the network parameters of the image processing model are updated, and the step of obtaining the existing historical lung CT scan image sequence from multiple consecutive periods continues.

5. An image processing apparatus implemented using the method according to any one of claims 1-4, characterized in that, The device includes: Image acquisition module, used to acquire lung CT scan images to be processed; An image input module is used to input the lung CT scan image into a pre-trained image processing model; The image output module is used to output multiple lung nodule parameter values ​​corresponding to the lung CT scan image; The model building module is used to build an image processing model; wherein, the image processing model includes a lung nodule detection network, a feature extractor, a normalization layer, and a fully connected layer; The image sequence acquisition module is used to acquire existing historical lung CT scan image sequences from multiple consecutive periods. The combined feature vector sequence generation module is used to generate multiple combined feature vector sequences based on the historical lung CT scan image sequence, lung nodule detection network, feature extractor, and normalization layer. The parameter calculation module is used to calculate the first lookup key and the first information content of each feature vector in each of the combined feature vector sequences based on the fully connected layer; The feature fusion module is used to obtain the shooting interval of the historical lung CT scan images corresponding to each feature vector, and generate multiple predicted feature vectors by performing feature fusion based on the first lookup key, first information content and shooting interval of each feature vector in each combined feature vector sequence. The final target vector generation module is used to extract existing clinical information and generate the final target vector by performing feature fusion with the multiple predicted feature vectors based on the clinical information. The model generation module is used to calculate the cross-entropy loss value based on the final target vector, and generate a pre-trained image processing model when the cross-entropy loss value reaches its minimum.

6. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method as described in any one of claims 1-4.

7. A terminal, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed as described in any one of claims 1-4.

Images