Image processing method and system

Abstract

The embodiments of this specification provide an image processing method and system, wherein the method comprises: acquiring a first image of a first modality; acquiring a second image of a second modality; inputting the first image into a prediction model to obtain a third image of the second modality; and comparing the second image with the third image to determine whether the second image contains artifacts.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method and system. Background Art

[0002] Artifacts may exist in medical images due to abnormal brightness of local images caused by metal implants and / or drug injection residues in the patient's body, image overlap caused by breathing, heartbeat or other limb movements, image truncation caused by system hardware failure, and partial image blanks caused by overestimation of scatter correction coefficients. Since image artifacts have different sources and manifestations, it is difficult to represent them through mathematical modeling or measure them with quantitative indicators.

[0003] At present, the image quality control of medical images is usually done through human observation, that is, doctors judge whether the current image has artifacts based on their own reading experience. For images that are considered to have artifacts, if it affects the diagnosis, the image needs to be reprocessed or the cause of the artifact needs to be eliminated before arranging the patient for a second scan.

[0004] Deep learning technology has shown far superior results to traditional technologies in the field of images. How to use deep learning technology to identify and eliminate image artifacts has gradually become an important research topic. Summary of the invention

[0005] One of the embodiments of the present specification provides an image processing method. The image processing method includes: acquiring a first image of a first modality; acquiring a second image of a second modality; inputting the first image into a prediction model to obtain a third image of the second modality; and comparing the second image and the third image to determine whether the second image contains an artifact.

[0006] One of the embodiments of the present specification provides an image processing system. The image processing system includes: a first acquisition module, used to acquire a first image of a first modality; a second acquisition module, used to acquire a second image of a second modality; an artifact prediction module, used to input the first image into a prediction model to obtain a third image of the second modality; and an artifact judgment module, used to compare the second image and the third image to determine whether the second image contains an artifact.

[0007] One of the embodiments of the present specification provides an image processing device, including a processor, wherein the processor is used to execute the above-mentioned image processing method.

[0008] One of the embodiments of the present specification provides a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the above-mentioned image processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0010] Figure 1 is a schematic diagram of an exemplary application scenario of an image processing system according to some embodiments of this specification;

[0011] Figure 2 is an exemplary module diagram of an image processing system according to some embodiments of this specification;

[0012] Figure 3 is an exemplary flow chart of an image processing method according to some embodiments of this specification;

[0013] Figure 4 is an exemplary flow chart of locating an artifact area according to some embodiments of this specification;

[0014] Figure 5 is a schematic diagram of a positioning artifact area according to some embodiments of this specification;

[0015] Figure 6 is another schematic diagram of a positioning artifact area according to some embodiments of this specification;

[0016] Figure 7 is an exemplary flow chart of artifact correction according to some embodiments of the present specification;

[0017] Figure 8 is a schematic diagram of artifact correction according to some embodiments of this specification;

[0018] Fig. 9 is a schematic diagram of model training according to some embodiments of this specification;

[0019] Fig.10 is a schematic diagram of an exemplary computing device according to some embodiments of the present specification. DETAILED DESCRIPTION

[0020] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0021] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0022] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0023] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0024] At present, with the continuous development of medical imaging technology, the method of using multiple technology fusion detection is widely used in various scenarios, such as research on tumors, nervous system, cardiovascular system and related disease diagnosis. Multiple technology fusion can include PET-CT, PET-MR, etc.

[0025] In some embodiments of the present specification, an automated solution for medical image quality control combining a neural network and a non-machine learning algorithm is proposed: a generative network and an image of the first modality (e.g., CT or MR) in multimodal imaging are used to predict an image of the second modality (e.g., PET), and a similarity comparison is performed with the image of the second modality actually acquired and reconstructed to determine whether the actual second modality image has artifacts; for areas in the image where artifacts are determined to exist, the area is marked and artifact correction is performed using the generative network. Thus, secondary scanning is avoided and the image quality analysis process is simplified.

[0026] It should be noted that the above examples are only for illustrative purposes and are not intended to limit the application scenarios of the technical solutions disclosed in this specification. The technical solutions disclosed in this specification are described in detail below through the description of the accompanying drawings.

[0027] Figure 1 It is a schematic diagram of an exemplary application scenario of the image processing system shown in some embodiments of this specification.

[0028] In some embodiments, the image processing system 100 may acquire a first image of a first modality; may acquire a second image of a second modality; may input the first image into a prediction model to obtain a third image of the second modality; and may compare the second image and the third image to determine whether the second image contains an artifact. For detailed descriptions of each image, modality, and artifact, please refer to Figure 3 Related description.

[0029] like Figure 1 As shown, the image processing system 100 may include an imaging device 110 , a network 120 , a terminal 130 , a processing device 140 , and a storage device 150 .

[0030] The imaging device 110 can be used to scan the target object to obtain scanning data and form an image. The imaging device 110 can be a medical imaging device, for example, a CT (Computed Tomography) imaging device, a PET (Positron Emission Computed Tomography) imaging device, an MR (Magnetic Resonance) imaging device, a PET-CT imaging device, a PET-MR imaging device, etc. In some embodiments, the imaging device 110 can include a gantry 111, a detector 112, a scanning area 113, and a scanning bed 114. The target object can be placed on the scanning bed 114 to be scanned. The gantry 111 can support the detector 112. In some embodiments, the detector 112 can include one or more detector units. The detector unit can be and / or include a single-row detector and / or a multi-row detector. The detector unit can include a scintillation detector (for example, a cesium iodide detector) and other detectors. In some embodiments, the gantry 111 can rotate, for example, in a CT imaging device, the gantry 111 can rotate clockwise or counterclockwise around the gantry rotation axis. In some embodiments, the imaging device 110 may further include a radiation scanning source, which may rotate together with the gantry 111. The radiation scanning source may emit a radiation beam (e.g., X-ray) to a target object, which is detected by the detector 112 after being attenuated by the target object, thereby generating an image signal. In some embodiments, the CT image may be used to correct the PET image.

[0031] The processing device 140 can process data and / or information obtained from the imaging device 110, the terminal 130 and / or the storage device 150. For example, the processing device 140 can process the image data detected and generated by the detector 112 to obtain an image. In some embodiments, the processing device 140 can be a single server or a server group. The server group can be centralized or distributed. In some embodiments, the processing device 140 can be local or remote. For example, the processing device 140 can access information and / or data from the imaging device 110, the terminal 130 and / or the storage device 150 through the network 120. For another example, the processing device 140 can directly connect to the imaging device 110, the terminal 130 and / or the storage device 150 to access information and / or data. In some embodiments, the processing device 140 can be implemented on a cloud platform. For example, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, a cross-cloud, a multi-cloud, etc., or a combination of several of them.

[0032] The terminal 130 may include a mobile device 131, a tablet computer 132, a laptop computer 133, etc. or any combination thereof. In some embodiments, the terminal 130 may interact with other components in the system 100 through a network. For example, the terminal 130 may send one or more control instructions to the imaging device 110 to control the imaging device 110 to scan the target object according to the instructions. In some embodiments, the terminal 130 may be part of the processing device 140. In some embodiments, the terminal 130 may be integrated with the processing device 140 as an operating console of the imaging device 110. For example, a user / operator of the system 100 (e.g., a doctor) may control the operation of the imaging device 110 through the operating console, such as scanning the target object, etc.

[0033] The storage device 150 may store data (e.g., scanned data of a target object, a first image, a second image, etc.), instructions, and / or any other information. In some embodiments, the storage device 150 may store data obtained from the imaging device 110, the terminal 130, and / or the processing device 140. For example, the storage device 150 may store the first image and the second image of the target object obtained from the imaging device 110, etc. In some embodiments, the storage device 150 may store data and / or instructions that the processing device 140 may execute or use to perform the exemplary method described in this specification. In some embodiments, the storage device 150 may include a large-capacity memory, a removable memory, a volatile read-write memory, a read-only memory (ROM), etc., one or a combination of several thereof. In some embodiments, the storage device 150 may be implemented by the cloud platform described in this specification. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, a cross-cloud, a multi-cloud, etc., one or a combination of several thereof.

[0034] In some embodiments, the storage device 150 may be connected to the network 120 to implement communication with one or more components (e.g., the processing device 140, the terminal 130, etc.) in the system 100. One or more components in the system 100 may read data or instructions in the storage device 150 through the network 120. In some embodiments, the storage device 150 may be part of the processing device 140, or may be independent and directly or indirectly connected to the processing device.

[0035] The network 120 may include any suitable network that can facilitate information and / or data exchange of the image processing system 100. In some embodiments, one or more components of the image processing system 100 (e.g., the imaging device 110, the terminal 130, the processing device 140, the storage device 150, etc.) may exchange information and / or data with one or more components of the image processing system 100 via the network 120. For example, the processing device 140 may obtain scan data from the imaging device 110 via the network 120. The network 120 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN), a wide area network (WAN), etc.), a wired network (e.g., an Ethernet), a wireless network (e.g., an 802.11 network, a wireless Wi-Fi network, etc.), a cellular network (e.g., a long term evolution (LTE) network), a frame relay network, a virtual private network (VPN), a satellite network, a telephone network, a router, a hub, a server computer, etc., or a combination thereof. For example, the network 120 may include a wired network, an optical fiber network, a telecommunication network, a local area network, a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), network, In some embodiments, the network 120 may include one or more network access points, such as a wireless network, a near field communication (NFC) network, etc. For example, the network 120 may include a wired and / or wireless network access point, such as a base station and / or an Internet exchange point, through which one or more components of the system 100 may connect to the network 120 to exchange data and / or information.

[0036] Figure 2 is an exemplary module diagram of an image processing system according to some embodiments of this specification. Figure 2 As shown, the image processing system 200 may include a first acquisition module 210 , a second acquisition module 220 , an artifact prediction module 230 , an artifact determination module 240 and an artifact correction module 250 .

[0037] The first acquisition module 210 may be used to acquire a first image of a first modality. In some embodiments, the first modality is CT or MR.

[0038] The second acquisition module 220 can be used to acquire a second image of a second modality. In some embodiments, the second modality is PET.

[0039] The artifact prediction module 230 can be used to input the first image into the prediction model to obtain a third image of the second modality. In some embodiments, the prediction model is a machine learning model. In some embodiments, the artifact prediction module 230 can be used to perform artifact correction on the first image.

[0040] The artifact determination module 240 may be used to compare the second image and the third image to determine whether the second image contains an artifact.

[0041] The artifact correction module 250 may perform artifact correction. In some embodiments, in response to the second image containing an artifact, the artifact correction module 250 may determine an artifact region of the second image. For example, the artifact correction module 250 may be configured to run a sliding window on the second image to obtain at least one candidate image block of the second image; and for each of the candidate image blocks, determine whether it is a target image block.

[0042] In some embodiments, the artifact correction module 250 may be configured to determine the similarity between the candidate image block and the corresponding image block in the third image; and based on the similarity and a preset condition, select the candidate image block that meets the preset condition as the target image block.

[0043] In some embodiments, the second image includes at least one target image block. The artifact correction module 250 can be used to generate an artifact-corrected second image based on the second image and the at least one target image block. In some embodiments, the artifact correction module 250 can generate a missing image based on the second image and the at least one target image block. The artifact correction module 250 can input the missing image into an artifact correction model to generate an artifact-corrected second image. In some embodiments, the artifact correction model can be a machine learning model. In some embodiments, the artifact correction module 250 can be used to simultaneously input the first image and the missing image into the artifact correction model to obtain an artifact-corrected second image.

[0044] For a detailed description of each module of the above system, please refer to the flow chart section of this manual, for example, Figures 3 to 9 The relevant description of is not repeated here. It should be understood that Figure 2The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. For example, in some embodiments, the first acquisition module 210 and the second acquisition module 210 can be implemented based on any combination of software, imaging devices, and computing devices. Those skilled in the art can understand that the above methods and systems can be implemented using computer executable instructions and / or included in processor control codes, such as carrier media such as disks, CDs or DVD-ROMs, programmable memories such as read-only memories (firmware), or data carriers such as optical or electronic signal carriers. Such codes are provided on the system and its modules of this specification. Not only can the hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc. be implemented, they can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above hardware circuits and software (for example, firmware).

[0045] It should be noted that the above description of the image processing system and its modules is only for convenience of description and does not limit the present specification to the scope of the embodiments. It is understandable that, after understanding the principle of the system, those skilled in the art may arbitrarily combine the modules or form a subsystem to connect with other modules without deviating from the principle. In some embodiments, Figure 2 The first image acquisition module, the second image acquisition module, the image processing module and the image correction module disclosed in the specification can be different modules in a system, or one module can realize the functions of two or more modules. For example, each module can share a storage module, or each module can have its own storage module. Such variations are all within the protection scope of this specification.

[0046] Figure 3 is an exemplary flow chart of an image processing method according to some embodiments of the present specification. In some embodiments, process 300 may be executed by a processing device, for example, processing device 140. For example, process 300 may be stored in a storage device (such as a built-in storage unit of the processing device or an external storage device) in the form of a program or instruction, and when the program or instruction is executed, process 300 may be implemented. Process 300 may include the following operations.

[0047] Step 310, acquiring a first image of a first modality. In some embodiments, step 310 may be performed by the first acquisition module 210. The first modality may be CT or MR.

[0048] The first image refers to an image obtained by scanning with an imaging device of a first modality, for example, a CT image or an MR image.

[0049] In some embodiments, the processing device may acquire a first image. The first image may be obtained by scanning a target object using a first modality imaging device (e.g., a CT imaging device, an MR imaging device). In some embodiments, the processing device may acquire the first image acquired in real time or image data corresponding to the first image. In some embodiments, the processing device may acquire the first image by reading from a storage device or a database and calling a data interface.

[0050] The target object may include a patient or other medical experimental subject (e.g., a mouse or other animal for testing), etc. The target object may also be a part of a patient or other medical experimental subject, including an organ and / or tissue, such as a patient's heart or lungs, etc. In some embodiments, the target object may also include a non-biological subject, such as a phantom, an artificial object, etc.

[0051] Step 320 , acquiring a second image of a second modality. In some embodiments, step 320 may be performed by the second acquisition module 220 .

[0052] The second modality may be different from the first modality. For example, the first modality may be CT or MR, and the second modality may be PET, etc.

[0053] The second image refers to an image obtained by scanning with an imaging device of a second modality, for example, a PET image.

[0054] In some embodiments, the processing device may acquire a second image. The second image may be obtained by scanning the target object using a second modality imaging device (e.g., a PET imaging device). In some embodiments, the processing device may acquire the second image acquired in real time or image data corresponding to the second image. In some embodiments, the processing device may acquire the second image by reading from a storage device or a database and calling a data interface.

[0055] See step 310 for a description of the target object.

[0056] Step 330 , input the first image into a prediction model to obtain a third image of the second modality. In some embodiments, step 330 may be performed by the artifact prediction module 230 .

[0057] The third image refers to a predicted image of the second modality corresponding to the first image. The difference between the third image and the second image is that the third image is obtained by processing the first image by the prediction model, and the second image is obtained by scanning by an imaging device of the second modality.

[0058] In some embodiments, since the multimodal images generated by the multimodal imaging device are interrelated and have complementary advantages, the artifact prediction module 230 can use the trained prediction model and the relationship between the multimodal images to obtain a predicted image of another modality corresponding to an image of one modality, for example, to obtain a second modality image corresponding to a first modality image.

[0059] In some embodiments, the processing device may input the first image into a prediction model, and the prediction model may output a third image of the second modality corresponding to the first image.

[0060] In some embodiments, the prediction model may include a generative network and a discriminative network, wherein the generative network may be a deep neural network with image generation capabilities such as GAN, PixelRNNs, DRAW, and any combination thereof, and the discriminative network may be a neural network model, etc.

[0061] In some embodiments, the prediction model can be trained based on multiple training samples and labels. For more information on training the prediction model, see Fig. 9 The related descriptions will not be repeated here.

[0062] In some embodiments, the processing device may perform artifact correction on the first image before inputting the first image into the prediction model. For example, the artifacts in the first image may be corrected by thin-layer scanning, software correction, etc. Due to the association between multimodal images, the artifacts in the first image may affect the predicted third image. Therefore, performing artifact correction on the first image in advance can make the third image more accurate.

[0063] In some embodiments of the present specification, by predicting functional images (PET images) that take a long time to acquire and have a large amount of radiation using anatomical images (CT images, MR images) that are easy to acquire and have a small amount of radiation, it is possible to improve efficiency and reduce the radiation exposure to patients.

[0064] Step 340 , comparing the second image and the third image to determine whether the second image contains an artifact. In some embodiments, step 340 may be performed by the artifact determination module 240 .

[0065] In some embodiments, the artifact determination module 240 may compare the second image and the third image, and determine whether the second image contains an artifact based on the comparison result.

[0066] As previously described, the third image is a predicted image of the second modality associated with the first image after artifact correction. The second image is a second modality image acquired by an imaging device of the second modality. In some embodiments, if the second image has a high degree of similarity with the third image, the second image is determined to contain no artifacts. In some embodiments, if the second image has a low degree of similarity with the third image, the second image is determined to contain artifacts.

[0067] In some embodiments, the artifact determination module 240 may calculate the similarity between the second image and the third image; if the similarity is lower than a preset threshold, the second image is determined to contain an artifact. In some embodiments, the similarity between images x and y may be calculated using the following formula:

[0068]

[0069] Among them, μ x μ y are the pixel means of image x and y respectively, are the variances of x and y, σ xy is the covariance of x and y, c1 and c2 are constant terms. The range of SSIM value is [-1,1]. The larger the value, the higher the similarity of the two images.

[0070] In some embodiments, the similarity between images x and y can be calculated using the following formula:

[0071]

[0072] Where m and n are the dimensions of images x and y respectively. The smaller the MSE value, the higher the similarity between the two images.

[0073] In some embodiments, the artifact determination module 240 may compare the second image and the third image in other ways, such as perceptual hashing algorithm, peak signal-to-noise ratio method, histogram method, etc. In some embodiments, the artifact determination module 240 may compare the second image and the third image using a trained machine learning model.

[0074] In some embodiments, if the similarity between the second image and the third image is lower than a preset threshold, for example, 0.8, the artifact determination module 240 may determine that the second image contains an artifact. The processing device may further locate the artifact area, for example, by using image segmentation or other methods to locate the artifact area. Figure 4 The sliding window method shown is used to locate the artifact area.

[0075] In some embodiments, if the similarity between the second image and the third image is not less than a preset threshold, the artifact determination module 240 may determine that the second image does not contain an artifact. In some embodiments, the processing device directly outputs the second image as the detection result of the second modality.

[0076] In some embodiments of the present specification, a network is generated and an image of one modality in multimodal imaging is used to predict an image of another modality, and a similarity comparison is performed with the image actually acquired and reconstructed to determine whether the actual image has artifacts, thereby avoiding the labor of manual film reading and reducing the reliance on doctor's experience. In some embodiments of the present specification, the determination of image similarity by a non-machine learning algorithm and the determination of artifacts by a neural network are combined, and the process is more controllable and the results are more accurate compared to the method that completely relies on a neural network to determine the presence of artifacts.

[0077] It should be noted that the above description of process 300 is only for example and illustration, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 300 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0078] Figure 4 is an exemplary flow chart of locating an artifact area according to some embodiments of the present specification.

[0079] Step 410 , running a sliding window on the second image to obtain at least one candidate image block of the second image. In some embodiments, step 410 may be performed by the artifact correction module 250 .

[0080] A candidate image block refers to a part of the second image, or is referred to as a sub-image block. In some embodiments, the second image may include multiple candidate image blocks. These candidate image blocks may be of the same size and in different positions. In this specification, if there is no other candidate image block between two candidate image blocks in one direction, the two image blocks may be referred to as adjacent. In some embodiments, at least two adjacent candidate image blocks among these candidate image blocks may at least partially overlap or not overlap at all. In some embodiments, at least two adjacent candidate image blocks may be connected at at least one boundary. In some embodiments, the union of all candidate image blocks constitutes the second image.

[0081] In some embodiments, the artifact correction module 250 may run a sliding window on the second image to obtain at least one candidate image block of the second image. For example, for a second image of 256x256 pixels, the window size may be set to 64x64 pixels, and each horizontal or vertical sliding distance is 32 pixels, so 7x7=49 candidate image blocks may be obtained.

[0082] In some embodiments, the artifact correction module 250 may run a multi-level sliding window on the second image to obtain at least one candidate image block of the second image. For a description of the multi-level sliding window, see Figure 6 .

[0083] Step 420 : For each candidate image block in the at least one candidate image block, perform steps 422 - 424 . In some embodiments, steps 422 - 424 may be performed by the artifact correction module 250 .

[0084] Step 422: Determine the similarity between the candidate image block and the corresponding image block in the third image.

[0085] The corresponding image block in the third image refers to an image block in the third image that has the same coordinates or relative position as the candidate image block.

[0086] In some embodiments, the similarity between the candidate image block and the corresponding image block in the third image can be calculated by the aforementioned formula (1) or (2). In some embodiments, the artifact correction module 250 can use a trained machine learning model (referred to as a similarity model) to calculate the similarity between the candidate image block and the corresponding image block in the third image. In some embodiments, the similarity model can be a part of the artifact correction model. In some embodiments, the similarity model and the artifact correction model can be two separate models.

[0087] Step 424: Based on the similarity and the preset condition, the candidate image block that meets the preset condition is used as the target image block.

[0088] The preset condition refers to a pre-set condition for similarity judgment. In some embodiments, the preset condition can be set by an experienced doctor or technician. In some embodiments, the preset condition can be set based on statistical information.

[0089] In some embodiments, the artifact correction module 250 may use a candidate image block that meets a preset condition as a target image block based on the similarity and a preset condition (eg, the similarity is lower than 0.8), and mark the candidate image block for further processing.

[0090] This manual Figure 5 and Figure 6 A specific example of using a (multi-level) sliding window to locate the artifact area in the second image is given.

[0091] In some embodiments of the present specification, a sliding window can be used to achieve fine-grained artifact positioning. In some embodiments of the present specification, a multi-level sliding window is used to first screen a large area and then specifically locate within the large area, thereby reducing the amount of calculation.

[0092] Figure 5 It is a schematic diagram of positioning artifact areas according to some embodiments of the present specification.

[0093] like Figure 5 As shown, the artifact correction module 250 may determine the artifact region of the second image using a sliding window.

[0094] For the second image 510 of 256x256 pixels, the window size can be set to 64x64 pixels, and each horizontal or vertical sliding distance is 32 pixels (for example, sliding from 520 to 530), so 7x7=49 areas can be obtained.

[0095] The similarities between the 49 regions in the second image and the region corresponding to the third image 540 are calculated respectively, for example, the similarity 570 between 520 and 550 is calculated, the similarity 580 between 530 and 560 is calculated, etc. If the similarity is less than the threshold, it is considered that the region has artifacts, the region is taken as a target image block, and the region is marked for further processing. If the similarity is not less than the threshold, it is considered that the region does not have artifacts and no subsequent processing is performed.

[0096] Figure 6 is another schematic diagram of positioning an artifact area according to some embodiments of the present specification.

[0097] like Figure 6 As shown, the artifact correction module 250 may determine the artifact region of the second image using a multi-level sliding window.

[0098] For the second image 610 of 256x256 pixels, the primary window size can be set to 128x128 pixels, and each horizontal or vertical sliding distance is 64 pixels (for example, sliding from 620 to 630), then 3x3=9 primary areas can be obtained.

[0099] The similarities between the nine primary regions in the second image and the primary region corresponding to the third image 640 are calculated respectively, for example, the similarity 670 between the primary region 620 and the primary region 650 is calculated, the similarity 680 between the primary region 630 and the primary region 660 is calculated, etc. If the similarity (for example, the similarity 680) is less than the first threshold, it is considered that the primary region (for example, the primary region 630) has an artifact. If the similarity (for example, the similarity 670) is not less than the first threshold, it is considered that the primary region (for example, the primary region 620) does not have an artifact, and no subsequent processing is performed.

[0100] For a primary region with artifacts, for example, primary region 630, the primary region can be further divided into secondary sliding windows. For example, the secondary window size can be set to 64x64 pixels, and each horizontal or vertical sliding distance is 32 pixels (for example, sliding from 631 to 633), then 3x3=9 secondary regions can be obtained in the primary region 630.

[0101] The similarities between the nine secondary regions in region 630 and the corresponding secondary regions in region 660 are calculated respectively. If the similarity is less than the second threshold, it is considered that the secondary region has artifacts, and the secondary region is taken as a target image block and marked for further processing.

[0102] Figure 7 is an exemplary flowchart of artifact correction according to some embodiments of the present specification.

[0103] Step 710 : Generate a missing image based on the second image and the at least one target image block. In some embodiments, step 710 may be performed by the artifact correction module 250 .

[0104] The missing image refers to the second image where the information of the artifact area is missing.

[0105] like Figure 8 As shown, the artifact correction module 250 sets the grayscale values ​​of the pixels in the target image blocks 811 - 819 in the second image to 255, that is, sets the target image blocks to all-white image blocks, thereby obtaining a missing image 810 .

[0106] In some embodiments, other methods may be used to generate missing images, for example, setting the grayscale values ​​of all target image blocks to 0 (completely black), drawing the boundary of the union of all target image blocks, etc.

[0107] Step 720 , input the missing image into an artifact correction model. In some embodiments, step 720 may be performed by the artifact correction module 250 .

[0108] In some embodiments, the artifact correction module 250 may input the missing image into an artifact correction model.

[0109] In some embodiments, the artifact correction model may include a generative network and a discriminative network. In some embodiments, the generative network may be a deep neural network with image generation capabilities such as GAN, PixelRNNs, DRAW, and any combination thereof, and the discriminative network may be a neural network model, a deep learning model, etc. The input of the artifact correction model may be a missing image, and the output may be a complete image. The corresponding missing portion (all target image blocks) in the complete image can be fully repaired by the artifact correction model, and the corresponding non-missing portion in the complete image may not be changed. As a result, the artifact correction process is only performed on the area in the second image that is determined to have artifacts, rather than on the entire second image, thereby reducing the images or image data that need to be processed and improving the efficiency of artifact correction.

[0110] In some embodiments, the artifact correction model can be trained based on multiple training samples and labels. For more information about training the artifact correction model, see Fig. 9 The related descriptions will not be repeated here.

[0111] In some embodiments, artifacts may appear in the cross section, sagittal plane or coronal plane of the image. For example, the identification and elimination of artifacts can be performed separately on different sections of the image. An image section (i.e., missing image) input to the artifact correction model may contain information of a normal image. A normal image refers to an image (e.g., a second image) obtained by removing the artifact area. The information of the normal image can be used as the basis for the artifact correction model to generate a complete image. The lower the proportion of the missing part in the missing image to the entire missing image, the higher the accuracy of the missing area of ​​the image repaired by the deep neural network. If the image artifact has a large coverage in a certain direction, such as a continuous layer of the cross section is judged as an artifact area, the sliding window should be placed on the sagittal or coronal plane of the image. The slices of the sagittal or coronal plane of the image are input to the artifact correction model. At this time, the training set of the artifact correction model needs to correspond to the section of the image, that is, if a certain section of the image is used during training, only the same section of the image can be used as input when used.

[0112] Step 730 , input the first image into an artifact correction model. In some embodiments, step 730 may be performed by the artifact correction module 250 .

[0113] Optionally, in some embodiments, the artifact correction module 250 may input the first image into the artifact correction model, that is, the first image and the missing image are simultaneously used as inputs of the artifact correction model to complete the artifact correction process.

[0114] Compared with the method that completely relies on neural networks for artifact correction, in some embodiments of the present specification, the information of the normal image, i.e., the information of the incomplete second image (with the artifact area removed) and the information of the normal anatomical image (the first image, e.g., CT image, MR image) is utilized to obtain a more accurate repaired image.

[0115] Step 740 , obtaining the second image after artifact correction. In some embodiments, step 740 may be performed by the artifact correction module 250 .

[0116] In some embodiments, the artifact correction module 250 may use the image output by the artifact correction model (hereinafter referred to as the output image) as the second image after artifact correction.

[0117] In some embodiments, the artifact correction module 250 may smooth the edge of the artifact region in the output image, and use the smoothed output image as the second image after artifact correction.

[0118] It should be noted that the above description of each process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0119] Figure 8 is a schematic diagram of artifact correction according to some embodiments of the present specification.

[0120] like Figure 8 As shown, the artifact correction module 250 sets the target image blocks 811-819 in the second image as all-white image blocks to obtain a missing image 810, and inputs the missing image 810 and the first image 820 corresponding to the second image into the artifact correction model 830 to obtain an artifact-corrected second image 840. In the image 840, the missing part has been repaired.

[0121] Fig. 9 It is a schematic diagram of the model training method shown in some embodiments of this specification.

[0122] like Fig. 9 As shown, in some embodiments, the initial model 910 can be trained based on a large number of labeled training samples to update the parameters of the initial model to obtain a trained model 920. The initial model 910 can include an initial prediction model and an initial artifact correction model, and the trained model 920 can include a prediction model and an artifact correction model.

[0123] In some embodiments, the prediction model can be trained based on a large number of first training samples with identifiers to update the parameters of the initial prediction model to obtain the prediction model. In some embodiments, the first training samples with identifiers can be input into the initial prediction model, and the parameters of the initial prediction model can be updated through training iterations.

[0124] The first training sample refers to image data used to train the prediction model. The image data may include historical image data.

[0125] In some embodiments, each first training sample may include a sample first image as an input of the training model and a sample second image as a label, wherein the sample first image corresponds to the first modality and the sample second image corresponds to the second modality.

[0126] In some embodiments, the first modality is CT or MR, and the second modality is PET. Accordingly, the first sample image may be a sample CT image or a sample MR image, and the second sample image may be a sample PET image.

[0127] In some embodiments, the PET image corresponding to the label can be acquired by an imaging device corresponding to the second modality. The label can be added manually or automatically, or in other ways, which is not limited in this embodiment.

[0128] In some embodiments, the processing device may obtain a plurality of first training samples, including their corresponding labels, by reading from a database or a storage device or calling a data interface.

[0129] In some embodiments, the processing device may input a sample first image of a first training sample into a prediction model, and obtain a prediction result from the output of the prediction model.

[0130] In some embodiments, the processing device may construct a loss function based on the prediction result and the label of the first training sample (sample second image). The loss function may reflect the difference between the prediction result and the label. The processing device may adjust the parameters of the prediction model based on the loss function to reduce the difference between the prediction result and the label. For example, by continuously adjusting the parameters of the prediction model, the loss function value is reduced or minimized.

[0131] In some embodiments, the prediction model can also be obtained according to other training methods, for example, a corresponding initial learning rate (for example, 0.1) and a learning rate decay strategy are set for the generation network, and the prediction model is obtained through joint training based on the first training sample with the identifier. This application is not limited here.

[0132] In some embodiments, the artifact correction model can be trained based on a large number of second training samples with labels to update the parameters of the initial artifact correction model to obtain the artifact correction model. In some embodiments, the second training samples with labels can be input into the initial artifact correction model, and the parameters of the initial artifact correction model can be updated through training iterations.

[0133] The second training sample may refer to image data used to train the artifact correction model. The image data may include historical image data.

[0134] In some embodiments, each second training sample may include an incomplete sample second image as an input of the training model and a complete sample second image as a label, wherein both the incomplete sample second image and the complete sample second image correspond to the second modality.

[0135] In some embodiments, an incomplete sample second image can be obtained by setting a mask on the complete sample second image. After the mask is set, the grayscale values ​​in the mask area of ​​the sample second image are all set to 0 (or all set to 255), which is visually equivalent to covering the mask area of ​​the sample second image with a completely black (or completely white) opaque image block. In some embodiments, the shape and size of the mask may be related to the candidate image block. For example, the shape and size of the mask may be the same as the size of the candidate image block, the mask may be an image block stretched horizontally by 1.5 times or 2 times, the mask may be an image block stretched vertically by 1.5 times or 2 times, or any combination thereof. The mask may be any combination of one or more candidate image blocks. For example, a mask is composed of at least one adjacent or non-adjacent candidate image block. For a description of the candidate image blocks, see Figure 4 .

[0136] In some embodiments, the position of the mask in the complete sample second image can be randomly set. In some embodiments, the position of the mask can be set according to a preset rule.

[0137] In some embodiments, multiple second training samples can be constructed by setting different masks on the same complete sample second image. For example, the second training sample 1 is the complete sample second image A and the mask 1 constitute the incomplete sample second image 1, and the second training sample 2 is the complete sample second image A and the mask 2 constitute the incomplete sample second image 2, and the labels of sample 1 and sample 2 are both the complete sample second image A.

[0138] In some embodiments, the second modality is PET. Accordingly, the incomplete sample second image may be an incomplete sample PET image, and the complete sample second image may be a complete sample PET image.

[0139] In some embodiments, the PET image corresponding to the label can be acquired by an imaging device corresponding to the second modality.

[0140] In some embodiments, the processing device may obtain a plurality of second training samples, including their corresponding labels, by reading from a database or a storage device or calling a data interface.

[0141] In some embodiments, the processing device may input the sample second image of the second training sample into the artifact correction model, and the artifact correction model outputs a filling result.

[0142] In some embodiments, the processing device may construct a loss function based on the filling result and the label of the second training sample (the complete sample second image). The loss function may reflect the difference between the filling result and the label. The processing device may adjust the parameters of the artifact correction model based on the loss function to reduce the difference between the filling result and the label. For example, by continuously adjusting the parameters of the artifact correction model, the loss function value is reduced or minimized.

[0143] In some embodiments, the artifact correction model can also be obtained according to other training methods, for example, a corresponding initial learning rate (for example, 0.1) and a learning rate decay strategy are set for the generation network, and the artifact correction model is obtained through joint training based on the second training sample with the identifier. This application is not limited here.

[0144] It should be noted that the above description of process 900 is only for illustration and description, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 900 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification. In some embodiments, Fig. 9 The model generation process described in is similar to image processing (e.g., Figure 3-8 Generating a missing image, generating an artifact-corrected image) as described in can be performed on different processing devices. For example, Fig. 9 The model generation process described in can be performed on a processing device of a manufacturer of an imaging device, while part or all of the image processing can be performed on a processing device of a user of the imaging device (such as a hospital).

[0145] Fig.10 is a schematic diagram of an exemplary computing device according to some embodiments of the present specification.

[0146] In some embodiments, one or more elements of the image processing system 100 may be implemented on the computing device 1000. For example, the processing engine 1012 may be implemented on the computing device 1000 and configured to implement the functions and / or methods disclosed herein.

[0147] The computing device 1000 may include any components used to implement the system described in this application. For example, the processing device 140 may be implemented on the computing device 1000 through its hardware, software program, firmware, or a combination thereof. For convenience, only one computer is drawn in the figure, but the computing functions related to the image processing system 100 described in this specification can be implemented in a distributed manner by a group of similar platforms to disperse the processing load of the system.

[0148] The computing device 1000 may include a communication port 1050 connected to the network 120 for implementing data communication. The computing device 1000 may include a processor (e.g., CPU) 1020, which may execute program instructions in the form of one or more processors. The exemplary computing device 1000 may also include an internal bus 1010, and different forms of program memory and data memory. For example, a hard disk 1070, a read-only memory (ROM) 1030, or a random access memory (RAM) 1040 may be used to store various types of data files processed and / or transmitted by a computer. The exemplary computing device 1000 may include program instructions executed by the processor 1020 stored in a read-only memory 1030, a random access memory 1040, and / or other types of non-temporary storage media. The methods and / or processes of this specification may be implemented in the form of program instructions. The computing device 1000 may also include an input / output component 1060 for supporting input / output between a computer and other components. The computing device 1000 may also receive programs and data in this application via network communication.

[0149] For ease of understanding, Fig.10 Only one processor is drawn as an example. However, it should be noted that the computing device 1000 in this specification may include multiple processors, so the operations and / or methods implemented by one processor described in this application may also be implemented jointly or independently by multiple processors. For example, if in this application, the processor of the computing device 1000 performs steps A and B, it should be understood that steps A and B may also be performed jointly or independently by two different processors of the computing device 1000 (for example, the first processor performs step A and the second processor performs step B; or the first processor and the second processor perform steps A and B jointly).

[0150] The beneficial effects that may be brought about by the embodiments of this specification include, but are not limited to: it is possible to automatically determine whether there are artifacts in the image, determine the artifact area, and correct the artifacts without increasing the process time.

[0151] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other beneficial effects that may be obtained.

[0152] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0153] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0154] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0155] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0156] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0157] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0158] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. An image processing method, the method comprising: acquiring a first image of a first modality; acquiring a second image of a second modality; Inputting the first image into a prediction model to obtain a third image of the second modality; as well as comparing the second image and the third image to determine whether the second image contains an artifact; In response to the second image containing an artifact, determining an artifact area of ​​the second image specifically includes: Running a sliding window on the second image to obtain at least one candidate image block of the second image, wherein at least two adjacent candidate image blocks among the candidate image blocks at least partially overlap; and For each candidate image block in the at least one candidate image block, Determining a similarity between the candidate image block and a corresponding image block in the third image; and Based on the similarity and the preset condition, taking the candidate image block that meets the preset condition as the target image block; An artifact region of the second image is determined based on the target image block.

2. The method according to claim 1, wherein: The first modality is CT or MR; The second modality is PET; and The step of inputting the first image into a prediction model to obtain a third image of the second modality includes: Performing artifact correction on the first image.

3. The method according to claim 1, wherein: The prediction model is a machine learning model.

4. The method according to claim 1, wherein: The sliding window is a multi-level sliding window.

5. The method according to claim 4, wherein determining the artifact area of ​​the second image further comprises: Determine a plurality of primary regions of the second image and a plurality of primary regions of the third image respectively according to the primary window; Determine the similarity between each primary region in the second image and the corresponding primary region in the third image; and determine whether there is an artifact in the primary region of the second image according to the similarity of each primary region; In response to the presence of artifacts in the primary region of the second image: Determine, according to the secondary window, a plurality of secondary regions of the primary region of the second image and a plurality of corresponding secondary regions in the third image; determining a similarity between each secondary region in the second image and a corresponding secondary region in the third image; It is determined whether an artifact exists in the secondary region according to the similarity of each secondary region, wherein the primary window is different from the secondary window.

6. The method according to claim 1, wherein: The second image includes at least one target image block, and the method further includes: generating a missing image based on the second image and the at least one target image block; The missing image is input into an artifact correction model to generate an artifact-corrected second image.

7. The method according to claim 6, wherein: Generating the artifact-corrected second image further includes inputting the first image into the artifact correction model.

8. The method according to claim 6, wherein: The artifact correction model is a machine learning model.

9. An image processing system, characterized in that: The system comprises: A first acquisition module, used to acquire a first image of a first modality; A second acquisition module, used to acquire a second image of a second modality; an artifact prediction module, configured to input the first image into a prediction model to obtain a third image of the second modality; and an artifact determination module, used for comparing the second image with the third image to determine whether the second image contains an artifact; In response to the second image containing an artifact, determining an artifact area of ​​the second image specifically includes: Running a sliding window on the second image to obtain at least one candidate image block of the second image, wherein at least two adjacent candidate image blocks among the candidate image blocks at least partially overlap; and For each candidate image block in the at least one candidate image block, Determining a similarity between the candidate image block and a corresponding image block in the third image; and Based on the similarity and the preset condition, taking the candidate image block that meets the preset condition as the target image block; An artifact region of the second image is determined based on the target image block.

10. An image processing device, characterized in that: The apparatus comprises at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is configured to execute at least part of the computer instructions to implement the method according to any one of claims 1 to 8.

11. A computer storable medium, characterized in that: The storage medium stores computer instructions. When at least part of the computer instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images