A method, apparatus, device and storage medium for lesion-oriented synthesis

Abstract

This application discloses a method, apparatus, device, and storage medium for lesion-oriented synthesis. The technical solution provided in this application combines original lesions into new lesions, selects normal tissue in a CT sequence, fuses the normal tissue with the new lesion to obtain fused lesion tissue, and then replaces the normal tissue in the CT sequence with the fused lesion tissue to obtain a synthesized lesion sequence. The fused lesion tissue in the synthesized lesion sequence is synthesized from different original lesions, effectively increasing the diversity of lesion images. Through the directional synthesis of different original lesion groups, data amplification at the lesion level is effectively achieved. Furthermore, by fusing normal tissue with the new lesion, the obtained fused lesion tissue is more realistic. Using the synthesized lesion sequence for training a deep learning algorithm model effectively improves the accuracy and generalization ability of the deep learning algorithm model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for lesion-oriented synthesis. Background Technology

[0002] With the increasing resolution of CT scans, radiologists are faced with a large number of images, which can easily lead to missed diagnoses and misdiagnoses due to work fatigue, making it difficult to meet the requirements for high-quality image interpretation. Computer-aided detection (CADe) systems can automatically detect and perform quantitative analysis, assisting radiologists in interpreting images, reducing their workload, and improving the accuracy and efficiency of image interpretation.

[0003] Deep learning-based lung nodule detection algorithms boast high accuracy, but their training requires a large amount of high-quality labeled data. A single thin-slice chest CT sequence contains 200-500 images, necessitating layer-by-layer annotation. Furthermore, due to variations in image and lesion distribution (image noise, slice thickness, lesion type, size, etc.), a model trained on one dataset often performs poorly on another, requiring retraining or transfer learning on new datasets. This results in weak generalization ability of deep learning algorithms. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for lesion-oriented synthesis to improve the diversity of lesion images.

[0005] In a first aspect, embodiments of this application provide a lesion-directed synthesis method, comprising:

[0006] Based on the original lesions with labeled lesions, a group of original lesions is determined, and new lesions are formed according to the group of original lesions;

[0007] Obtain normal tissue from CT sequences, and fuse the new lesion with the normal tissue to obtain fused lesion tissue;

[0008] The normal tissue in the CT sequence is replaced with the fused lesion tissue to obtain a synthetic lesion sequence.

[0009] In a second aspect, embodiments of this application provide a lesion-oriented synthesis device, including a lesion synthesis module, a tissue fusion module, and a lesion replacement module, wherein:

[0010] The lesion synthesis module is used to determine the original lesion group based on the original lesions with labeled lesions, and to synthesize new lesions based on the original lesion group;

[0011] The tissue fusion module is used to acquire normal tissue from CT sequences and fuse the new lesion with the normal tissue to obtain fused lesion tissue.

[0012] The lesion replacement module is used to replace the normal tissue in the CT sequence with the fused lesion tissue to obtain a synthetic lesion sequence.

[0013] In a third aspect, embodiments of this application provide a lesion-directed synthesis device, including: a memory and one or more processors;

[0014] The memory is used to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the lesion-directed synthesis method as described in the first aspect.

[0016] In a fourth aspect, embodiments of this application provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the lesion-directed synthesis method as described in the first aspect.

[0017] This application embodiment combines original lesions into new lesions, selects normal tissue in the CT sequence, fuses the normal tissue with the new lesions to obtain fused lesion tissue, and then replaces the normal tissue in the CT sequence with the fused lesion tissue to obtain a synthetic lesion sequence. The fused lesion tissue in the synthetic lesion sequence is synthesized from different original lesions, effectively increasing the diversity of lesion images. Through the directional synthesis of different original lesion groups, data amplification at the lesion level is effectively achieved. Furthermore, by fusing normal tissue with new lesions, the obtained fused lesion tissue is more realistic. Using the synthetic lesion sequence for training deep learning algorithm models effectively improves the accuracy of deep learning algorithm models and enhances their generalization ability. Attached Figure Description

[0018] Figure 1 This is a flowchart of a lesion-directed synthesis method provided in an embodiment of this application;

[0019] Figure 2 This is a flowchart of another lesion-directed synthesis method provided in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram illustrating the synthetic effect of a novel lesion provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram illustrating the effect of a synthetic lesion sequence provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an intermediate layer image of a Gaussian mask under different scaling factors provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram illustrating the effect of fusing lesion tissue according to an embodiment of this application;

[0024] Figure 7 This is a schematic diagram comparing the effects of a synthetic lesion sequence and a primary lesion sequence provided in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram of a fused lesion labeled in a synthetic lesion sequence provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of the structure of a lesion-directed synthesis device provided in an embodiment of this application;

[0027] Figure 10 This is a schematic diagram of the structure of a lesion-directed synthesis device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0029] Figure 1 A flowchart of a lesion-directed synthesis method provided in an embodiment of this application is given. The lesion-directed synthesis method provided in this embodiment of the application can be executed by a lesion-directed synthesis device, which can be implemented by hardware and / or software and integrated into a lesion-directed synthesis equipment.

[0030] The following description uses a lesion-directed synthesis device to perform a lesion-directed synthesis method as an example. (Reference) Figure 1 The method for targeted synthesis of this lesion includes:

[0031] S101: Determine the original lesion group based on the original lesions that have been labeled, and form a new lesion group according to the original lesion group.

[0032] The original lesion group provided in this embodiment includes two or more original lesions with labeled lesions. The labeling type of the original lesions can be the size, texture type, lesion type, and degree of danger of the original lesions. For example, for a CT sequence, lesions are labeled for each layer of the image in the CT sequence, such as labeling the size, texture type, lesion type, and degree of danger of the lesions appearing in each layer of the image. The original lesions in the CT sequence and their corresponding labels are determined by the labeling of the lesions in each layer.

[0033] For example, two primary lesions are selected from the sample dataset and grouped together as a single primary lesion group. The sample dataset contains multiple primary lesions with labeled lesions. Optionally, when selecting primary lesions to construct primary lesion groups, two primary lesions with similar labeling are selected, for example, two primary lesions within the same size range, texture type, lesion type, or risk level are selected to form a primary lesion group. Multiple primary lesion groups are then constructed by sequentially selecting primary lesions from the sample dataset.

[0034] Furthermore, after identifying the original lesion group, the original lesion groups within the original lesion group are synthesized to obtain a new synthesized lesion. The new lesion is a fusion of local features of multiple original lesions in the original lesion group. Compared with traditional data amplification targeting a single sample (such as data amplification achieved by rotation, cropping, scaling, etc.), it has stronger representativeness and can increase the number of lesion samples.

[0035] S102: Obtain normal tissue from the CT sequence and fuse the new lesion with the normal tissue to obtain fused lesion tissue.

[0036] For example, a CT sequence is selected, and normal tissue in the CT sequence is obtained. The newly obtained lesion is then fused with the normal tissue to obtain a fused lesion tissue. Optionally, the edges of the new lesion are fused with the normal tissue so that the edges of the new lesion are closer to the normal tissue in the CT sequence.

[0037] Understandably, taking pulmonary nodules as an example, most pulmonary nodules are round or irregular lesions within the lungs. Therefore, when simulating real lesions, the synthesized lesion needs to be integrated into the lung tissue, and the closer it is to the real lesion, the better. If the synthesized lesion directly replaces the normal lung tissue, that is, directly replaces the normal tissue in the CT sequence, the difference between the new lesion and the normal tissue will be significant, resulting in a strong sense of abruptness. This embodiment utilizes the fusion of normal tissue and the new lesion, making the edge of the new lesion closer to the normal tissue in the CT sequence, and the transition between the new lesion and the normal tissue is smoother and more natural.

[0038] S103: Replace the normal tissue in the CT sequence with the fused lesion tissue to obtain a synthetic lesion sequence.

[0039] For example, after fusing the new lesion with normal tissue to obtain a fused lesion tissue, the fused lesion tissue replaces the normal tissue in the CT sequence, thereby synthesizing the fused lesion tissue into the CT sequence to obtain a synthetic lesion sequence. This synthetic lesion sequence can be used in the training of deep learning algorithm models. Furthermore, this application obtains different new lesions by using original lesion groups composed of different original lesions, effectively achieving lesion-level data augmentation and increasing the diversity of lesion images.

[0040] The above describes a process where original lesions are combined to form new lesions, and normal tissue is selected from the CT sequence. The normal tissue is then fused with the new lesions to obtain a fused lesion tissue. This fused lesion tissue is then used to replace the normal tissue in the CT sequence to obtain a synthetic lesion sequence. The fused lesion tissue in the synthetic lesion sequence is synthesized from different original lesions, effectively increasing the diversity of lesion images. Through the directional synthesis of different original lesion groups, data amplification at the lesion level is effectively achieved. Furthermore, by fusing normal tissue with the new lesions, the obtained fused lesion tissue is more realistic. Using the synthetic lesion sequence for training deep learning algorithm models effectively improves the accuracy of deep learning algorithm models and enhances their generalization ability.

[0041] Based on the above embodiments, Figure 2 A flowchart of another lesion-directed synthesis method provided in an embodiment of this application is given, which is a concretization of the above-described lesion-directed synthesis method. (Reference) Figure 2 The method for targeted synthesis of this lesion includes:

[0042] S201: Perform distribution statistics on the original lesions that have been marked, and select the original lesion group based on the distribution statistics results.

[0043] In this embodiment, the original lesions are provided by a sample dataset, and the original lesions are pre-annotated. Based on the annotation information, the size, texture type, lesion type, and risk level of the corresponding original lesions can be determined. The sample dataset can be obtained based on publicly available datasets, such as publicly available lung nodule detection datasets like LNDb19 and LUNA16, or it can be a self-made sample dataset obtained by annotating multiple CT sequences or corresponding image images. It can also be a combination of publicly available datasets and self-made sample datasets. This application does not limit the scope of the dataset.

[0044] After performing distribution statistics on the original lesions in the sample dataset, original lesions are selected within the same distribution range based on the distribution statistics results to construct an original lesion group. In one possible embodiment, selecting the original lesion group from the original lesions based on the distribution statistics results can be done within a set labeled distribution range. Based on this, this embodiment performs distribution statistics on the original lesions of labeled lesions and selects an original lesion group from the original lesions based on the distribution statistics results, including steps S2011-S2012:

[0045] S2011: Based on the original lesion labels corresponding to the labeled original lesions, perform distribution statistics on the original lesions.

[0046] S2012: Based on the distribution statistics, randomly select a group of original lesions from the original lesions according to the set labeled distribution range.

[0047] Specifically, based on the annotation information corresponding to each original lesion in the sample dataset, distribution statistics are performed on each original lesion to obtain distribution statistics results, thereby determining the distribution range corresponding to each original lesion. Further, according to the distribution range of each original lesion in the distribution statistics results, original lesions are selected to form original lesion groups. It can be understood that an original lesion group contains multiple (two or more) original lesions, and the original lesions within the same original lesion all correspond to the same distribution range.

[0048] In one possible embodiment, when constructing the original lesion group, the original lesions can be selected within the same distribution range to construct the original lesion group in a targeted manner. For example, if the original lesions are selected within a specified size range to construct the original lesion group, then all the original lesions in the constructed original lesion group are within the specified size range.

[0049] Taking the public dataset LNDb19 as an example, each original lesion in the public dataset LNDb19 has been labeled, such as the size (diameter) and texture type of the original lesion. After performing distribution statistics on the original lesions based on the size and texture type of each original lesion in the public dataset LNDb19, the proportion of original lesions in each distribution range reflected by the distribution statistics results in the public dataset LNDb19 is shown in the following table:

[0050] Diameter (mm) 0-5 5-10 10-20 20-40 Proportion 0.602 0.329 0.055 0.014 Texture type Ground glass nodules Partial solid nodules solid nodules Proportion 0.049 0.076 0.875

[0051] As shown in the table above, the nodular lesions in the publicly available dataset LNDb19 are mainly micronodules (primary lesions with a diameter of less than 5 mm) and solid nodules, with a highly uneven distribution. Multiple primary lesion groups are obtained by randomly selecting primary lesions within a specified distribution range. For example, within the diameter range of 10-20 mm, two primary lesions are randomly selected sequentially to construct a primary lesion group until all primary lesions within that distribution range are extracted or the remaining primary lesions are insufficient to construct a primary lesion group. This yields primary lesion groups oriented within the diameter range of 10-20 mm. Alternatively, within the range of solid nodules, two primary lesions are randomly selected sequentially to construct a primary lesion group until all primary lesions within that distribution range are extracted or the remaining primary lesions are insufficient to construct a primary lesion group. This yields primary lesion groups oriented within the range of solid nodules.

[0052] It is understood that the distribution statistics of the original lesions in this scheme are based on the size and texture type of the original lesions, and the required samples are extracted according to this distribution. It is understood that the distribution basis can be replaced with other distribution types, such as lung-Rads grading, etc., and this application does not limit it.

[0053] S202: Normalize the size of the original lesion group and form new lesions based on the original lesion group.

[0054] As can be seen from the above, the original lesion group is composed of different original lesions. Before synthesizing new lesions, the original lesion group needs to be normalized in size to ensure the synthesis effect of new lesions.

[0055] In one embodiment, the size normalization of the original lesion group includes steps S2021-S2022:

[0056] S2021: Based on the original lesion size corresponding to the original lesion in the original lesion group, a normalized size is randomly determined among the original lesion sizes.

[0057] Specifically, for each original lesion group, the original lesion size of each original lesion in the original lesion group is determined (which can be determined based on the annotation information of the original lesions), and a normalized size is randomly determined among these original lesion sizes.

[0058] Taking a primary lesion group containing two primary lesions as an example, assume that the primary lesion sizes of the two primary lesions are S1 and S2, and randomly determine a size between S1 and S2 as the normalized size S. Set the normalized size S to follow a uniform distribution between S1 and S2, that is, S ~ U(S1, S2). At this time, the probability of the normalized size S taking any value between S1 and S2 is the same.

[0059] S2022: Normalize the size of the original lesion according to the normalized size.

[0060] Specifically, after determining the normalized size corresponding to the original lesion group, the size of each original lesion in the original lesion group is normalized according to this normalized size, so as to convert the size of each original lesion into the corresponding normalized size. For example, the size of the original lesions is converted into a normalized size through stretching, scaling, cropping, etc. At this time, the size of all original lesions in the original lesion group is converted into a normalized size, that is, the size of all original lesions in the same original lesion group is consistent.

[0061] After normalizing the size of each original lesion in the original lesion group, the original lesions are combined into new lesions. In one possible embodiment, the combination of the original lesions into new lesions can be based on a beta distribution.

[0062] Specifically, combining the original lesions into new lesions according to the beta distribution includes steps S2023-S2024:

[0063] S2023: Determine the synthesis parameters according to the beta distribution, and determine the synthesis ratio of each original lesion in the original lesion group based on the synthesis parameters.

[0064] S2024: Based on the synthesis ratio and the normalized size, combine the original lesions into new lesions.

[0065] Taking two original lesions as an example to form a group of original lesions, the formula for synthesizing new lesions provided in this embodiment is as follows:

[0066] patch_new=λ*patch1+(1-λ)*patch2

[0067] In this example, `patch_new` represents the new lesion, `patch1` and `patch2` represent the two original lesions, and `λ` is the synthesis parameter. The domain of `λ` is [0, 1], and the synthesis parameter `λ` follows a beta distribution, i.e., `λ ~ beta(α, β)`. The beta distribution is a probability distribution with a domain of 0 to 1, i.e., `λ ∈ [0, 1]`. `α` and `β` are two shape parameters of the beta distribution, used to control the shape of the beta distribution, and `α` and `β` ∈ (0, ∞). It should be noted that when `α < 1` and `β < 1`, the shape of the beta distribution is U-shaped. In this embodiment, `λ` is used as the synthesis ratio of the original lesion `patch1`, and `(1-λ)` is used as the synthesis ratio of the original lesion `patch2`.

[0068] In this example, the shape parameter of the beta distribution is set to a U-shape, for example, α = β = 0.2 (which can be set according to the actual situation, and this application does not limit it). At this time, the distribution shape of the beta distribution is a symmetrical U-shape, the center of the beta distribution is 0.5, and the probability of the distribution on both sides is higher than that of the center.

[0069] Specifically, based on the set shape parameters, the value of the synthesis parameter λ is determined according to the beta distribution, and the synthesis ratio of one original lesion patch 1 is λ, while the synthesis ratio of the other original lesion patch 2 is (1-λ). Further, based on the synthesis ratio λ of original lesion patch 1, the synthesis ratio (1-λ) of original lesion patch 2, and the normalized size S, the two original lesions are combined into a new lesion, and the lesion size of the new lesion is the normalized size S.

[0070] Figure 3 A schematic diagram illustrating the synthetic effect of a novel lesion provided in an embodiment of this application is given, such as... Figure 3 As shown, exemplarily, Figure 3 In this model, patch1 and patch2 represent two original lesions from the original lesion group, while patch_new represents a new lesion synthesized from patch1 and patch2. The randomly generated normalized size S is 15.1 mm, and the synthesis parameter λ is 0.65. Therefore, the synthesis ratios of patch1 and patch2 are 0.65 and 0.35, respectively. Thus, patch_new = 0.65 * patch1 + 0.35 * patch2, and the lesion size of patch_new is 15.1 mm. The synthesized new lesion patch_new is generated from the original lesion pair. Each pixel of the new lesion patch_new is obtained by fusing the corresponding pixels of the two original lesions according to the synthesis ratio. This represents a fusion of local features from both lesions, which, compared to traditional data augmentation targeting a single sample, has stronger representativeness and can provide a richer sample of lesions.

[0071] It is understood that the synthesis parameters provided in this application can be applied to the fusion of original lesions in all original lesion groups, or the synthesis parameters can be re-determined when fusing original lesions in each original lesion group. This application does not limit this.

[0072] S203: Generate a Gaussian mask based on the lesion size and three-dimensional Gaussian function of the new lesion.

[0073] Understandably, after obtaining a new lesion, it is necessary to integrate the new lesion into the CT sequence. Since the new lesion obtained by integrating the original lesion group is significantly different from the CT sequence being integrated, directly replacing the normal tissue in the CT sequence with the new lesion would create a significant abruptness. Figure 4 This is a schematic diagram illustrating the effect of a synthetic lesion sequence provided in an embodiment of this application, such as... Figure 4 As shown, when the new lesion and the CT sequence are directly fused together, a clear fusion boundary can be observed between the new lesion and the surrounding normal tissue at the fusion site.

[0074] To achieve a smooth transition between the new lesion and the surrounding normal tissue, this solution introduces a Gaussian function. Since the lesion is a three-dimensional tissue structure, the Gaussian function introduced in this application is specifically a three-dimensional Gaussian function. Specifically, as mentioned above, the lesion size is a normalized size. This embodiment generates a Gaussian mask based on the lesion size and the three-dimensional Gaussian function.

[0075] The formula for calculating the three-dimensional Gaussian function provided in this solution is as follows:

[0076]

[0077] Where f is a three-dimensional Gaussian function, assuming that the three dimensions (X, Y, Z) are independent and have their own distinct variances. x, y, and z represent the distances to the center point, and k (k≥1) is the scaling factor.

[0078] In one possible embodiment, the variances of the three-dimensional Gaussian function provided by this solution are different in each of the three dimensions. Furthermore, the standard deviations of the three-dimensional Gaussian function in each of the three dimensions are randomly generated based on the lesion size of the new lesion. Specifically, the formulas for calculating the standard deviations of the three-dimensional Gaussian function in each dimension are as follows:

[0079]

[0080] Where, σ i Let D be the standard deviation in the i-direction. iLet t1 be the diameter in the direction of lesion i, and t2 be the minimum and maximum values ​​selected. This scheme is described using t1 = 0.34 and t2 = 0.66 as an example, meaning that the radius of the generated Gaussian mask is between 1.51 and 2.94 times the standard deviation (1 / t, the reciprocal relationship).

[0081] After specifying the scaling factor, the variance or standard deviation in the three directions, and the magnitude in the three directions, the corresponding Gaussian mask can be obtained. That is, the Gaussian mask is an instance of the three-dimensional Gaussian function.

[0082] Figure 5 This application provides a schematic diagram of the intermediate layer image of a Gaussian mask under different scaling factors. Only the intermediate layer image of the Gaussian mask is shown in the diagram, where k is set to 1 (…). Figure 5 When k is set to 2, the Gaussian mask has the same variance in all dimensions and high resolution. Figure 5 When (right), the variances of each dimension of the three-dimensional Gaussian function are different, and the resolution is low. Combined with... Figure 5 From the edge to the center of the Gaussian mask, the three-dimensional Gaussian function value gradually increases. When k is set to 1, the maximum value at the center point is 1, while the minimum value at the edge depends on the variance of each dimension. When the standard deviation σ is less than 1 / 3 of the radius, the Gaussian function value at the edge is close to 0. It is understandable that the purpose of fusing new lesions and normal tissue is to make the transition of the new lesion edge in the CT sequence smoother and more natural. Therefore, only the fusion of the new lesion edge and normal tissue is needed, without the need for a gradual transition fusion of the entire new lesion. Therefore, in this embodiment, k is set to 2 (this can be set according to actual conditions, and this application does not limit it). Furthermore, when the three-dimensional Gaussian function value is greater than 1, the three-dimensional Gaussian function value is set to 1. At this time, k can be used as a control fusion ratio coefficient for fusing new lesions and normal tissue; the larger the value, the closer the fused new lesion area is to the edge.

[0083] S204: Randomly acquire normal tissue from a CT sequence, and use the Gaussian mask to fuse the new lesion and the normal tissue to obtain fused lesion tissue.

[0084] A CT sequence is randomly selected, and normal tissue is randomly obtained from the CT sequence. The size of the normal tissue is consistent with the size of the new lesion to be fused, that is, the size of the normal tissue is the normalized size.

[0085] This scheme uses an anisotropic three-dimensional Gaussian function (different variances in each dimension) to fuse new lesions and normal tissues. At the same time, a fusion ratio coefficient k is designed to freely and flexibly control the fusion ratio, making the generated fused lesion tissue more natural, realistic and rich.

[0086] Furthermore, after identifying the normal tissue in the CT sequence, the new lesion and normal tissue are fused using the Gaussian mask obtained above to obtain the fused lesion tissue. Accordingly, the fusion of the new lesion and normal tissue specifically includes steps S2041-S2042:

[0087] S2041: Determine the fusion ratio between the new lesion and the normal tissue based on the Gaussian mask.

[0088] Specifically, this embodiment describes the fusion of two original lesions into a new lesion as an example. In this embodiment, the fusion ratio of the new lesion and the normal tissue is determined according to the Gaussian mask. For example, when the Gaussian mask is determined to be gmask, the fusion ratio of the new lesion is determined to be gmask, and the fusion ratio of the normal tissue is determined to be (1-gmask).

[0089] S2042: The new lesion and the normal tissue are fused according to the fusion ratio to obtain fused lesion tissue.

[0090] The new lesion and normal tissue are fused according to their respective fusion ratios to obtain fused lesion tissue. It is understood that the lesion size of the fused lesion tissue is consistent with the normalized size, and the value of each point in the fused lesion tissue is consistent with the value obtained by fusing the values ​​of the new lesion and normal tissue at the corresponding points according to the fusion ratio.

[0091] Specifically, the fusion formula for the fused lesion tissue provided in this solution is as follows:

[0092] patch_m=gmask*patch_new+(1-gmask)*patch0

[0093] Where patch_m represents the fused lesion tissue, gmask is the Gaussian mask, patch_new represents the new lesion, and patch_0 represents the normal tissue. Correspondingly, the fusion ratio of the new lesion is gmask, and the fusion ratio of the normal tissue is (1-gmask).

[0094] Figure 6 This is a schematic diagram illustrating the effect of fusing lesion tissue in an embodiment of this application. Figure 6 In this diagram, patch0 represents normal tissue, patch_new represents new lesions, and patch_m represents fused lesion tissue. For example... Figure 6 As shown, normal tissue patch0 is extracted from a randomly acquired CT sequence. The fusion ratio of the new lesion patch_new is determined as gmask based on the aforementioned Gaussian mask, and the fusion ratio of normal tissue patch0 is (1-gmask). After fusing the new lesion and normal tissue according to the fusion formula, the result is as follows: Figure 6 The patch_m shown is a fused lesion tissue.

[0095] S205: Replace the normal tissue in the CT sequence with the fused lesion tissue to obtain a synthetic lesion sequence.

[0096] As can be seen from the above, the size of the fused lesion tissue is consistent with the size of the extracted normal tissue. The fused lesion tissue can directly replace the normal tissue extracted from the CT sequence to obtain a synthetic lesion sequence, thus achieving the fusion of the fused lesion tissue and the CT sequence.

[0097] Figure 7 This is a schematic diagram comparing the effects of a synthetic lesion sequence and a primary lesion sequence provided in an embodiment of this application. Figure 7 Figures a through d show intermediate layers of the synthesized lesion sequence, while figures e and f show intermediate layers of the primary lesion sequence. The areas indicated by the arrows represent the fused lesion tissue and the primary lesion tissue. Figure 7 It can be seen that the fusion effect of the fused lesion tissue in the synthetic lesion sequence is very close to the display effect of the original lesion tissue in the original lesion sequence.

[0098] S206: Based on the lesion size corresponding to the fused lesion tissue and the lesion location of the fused lesion tissue in the CT sequence, generate the fused lesion label corresponding to the fused lesion tissue.

[0099] Specifically, after fusing the fused lesion tissue into the CT sequence to obtain the synthetic lesion sequence, the fused lesion tissue is labeled with the corresponding fused lesion label based on the lesion size of the fused lesion tissue (e.g., consistent with the normalized size determined above) and the lesion location of the fused lesion tissue in the CT sequence (consistent with the tissue location of the replaced normal tissue), and the fused lesion label is simultaneously labeled in the synthetic lesion sequence.

[0100] In one possible embodiment, the lesion size of the fused lesion tissue needs to be re-determined before labeling the fused lesion tissue. Accordingly, after replacing the normal tissue in the CT sequence with the fused lesion tissue to obtain the synthetic lesion sequence, steps S2061-S2062 are further included:

[0101] S2061: Determine the lesion size corresponding to the fused lesion tissue based on the standard deviation of the Gaussian mask.

[0102] S2062: Based on the lesion size corresponding to the fused lesion tissue and the lesion location of the fused lesion tissue in the CT sequence, generate the fused lesion label corresponding to the fused lesion tissue.

[0103] In this CT sequence, the location of the fused lesion tissue coincides with the center of the replaced normal tissue, denoted as (x0, y0, z0). However, the size of the fused lesion tissue is not equal to the size of the normal tissue. Understandably, when fusing the new lesion with normal tissue, the edge of the new lesion is blended into the background, resulting in a smaller lesion size compared to the extracted normal tissue. Therefore, it is necessary to re-estimate the lesion size of the fused lesion tissue.

[0104] Since a three-dimensional Gaussian function is used when fusing new lesions and normal tissue, this embodiment estimates the lesion size (radius) based on the standard deviation σ of the Gaussian mask, that is, the radius of the fused lesion tissue in the i-direction is r. i =γσ i γ can be set according to the actual situation, and this application does not limit it. For example, γ is related to the selection of t1 and t2 in step S203.

[0105] Figure 8 This is a schematic diagram illustrating the annotation of fused lesions in a synthetic lesion sequence according to an embodiment of this application. The annotations are displayed in the synthetic lesion sequence based on the size and location of the fused lesion tissue. In this embodiment, γ = 1.5 is used as an example, and the generated fused lesion annotations are as follows: Figure 8 As shown, the generated fused lesion annotations can represent the generated lesions well. The fused lesion annotations are denoted as (x0, y0, z0, 3σ). x ,3σ y ,3σ z The coordinates (σ) represent the center point and diameter of the fused lesion annotation in the synthesized lesion sequence. By re-estimating the radius of the fused lesion tissue using the standard deviation σ of the Gaussian mask as a benchmark, more accurate fused lesion annotations (location, size, etc.) are generated, effectively increasing the amount of annotation data and assisting detection algorithms in improving accuracy.

[0106] As described above, by combining original lesions into new lesions and selecting normal tissue from the CT sequence, the normal tissue is fused with the new lesions to obtain fused lesion tissue. This fused lesion tissue then replaces the normal tissue in the CT sequence to obtain a synthetic lesion sequence. The fused lesion tissue in the synthetic lesion sequence is synthesized from different original lesions, effectively increasing the diversity of lesion images. Through targeted synthesis of different original lesion groups, data augmentation at the lesion level is effectively achieved. Furthermore, by fusing normal tissue with new lesions, the resulting fused lesion tissue is more realistic. Using the synthetic lesion sequence for training deep learning algorithm models effectively improves the accuracy and generalization ability of the deep learning algorithm models. Moreover, by normalizing the size of the original lesion groups, the fusion effect of the original lesions is improved. The resulting new lesion is a fusion of local features from multiple original lesions, which has stronger representativeness and can increase the number of lesion samples compared to traditional data augmentation targeting a single sample. This approach utilizes a 3D Gaussian function to fuse the edges of new lesions with normal tissue, resulting in fused lesion tissue. This leads to a smoother transition of the fused lesion tissue in the synthesized lesion sequence. Furthermore, by re-estimating the lesion size of the fused lesion tissue and combining it with the lesion location in the CT sequence, fused lesion annotations are obtained, automatically generating fused lesion annotations in the synthesized lesion sequence and improving annotation efficiency. Simultaneously, this scheme uses the original lesion group and beta distribution to synthesize new lesions, which has stronger representational capabilities and provides richer lesion samples compared to enhancing a single lesion. In addition, this scheme can perform targeted synthetic amplification of lesions according to the distribution requirements, effectively alleviating the problem of uneven lesion distribution and annotation difficulties, thereby improving the generalization ability of deep learning algorithms for lesion detection.

[0107] Figure 9 A schematic diagram of a lesion-directed synthesis device provided in an embodiment of this application is given. (Reference) Figure 9 The lesion-directed synthesis device includes a lesion synthesis module 31, a tissue fusion module 32, and a lesion replacement module 33.

[0108] The lesion synthesis module 31 is used to determine the original lesion group based on the original lesions with labeled lesions, and to synthesize a new lesion based on the original lesion group; the tissue fusion module 32 is used to obtain normal tissue from the CT sequence and fuse the new lesion and the normal tissue to obtain fused lesion tissue; the lesion replacement module 33 is used to replace the normal tissue in the CT sequence with the fused lesion tissue to obtain the synthesized lesion sequence.

[0109] The above describes a process where original lesions are combined to form new lesions, and normal tissue is selected from the CT sequence. The normal tissue is then fused with the new lesions to obtain a fused lesion tissue. This fused lesion tissue is then used to replace the normal tissue in the CT sequence to obtain a synthetic lesion sequence. The fused lesion tissue in the synthetic lesion sequence is synthesized from different original lesions, effectively increasing the diversity of lesion images. Through the directional synthesis of different original lesion groups, data amplification at the lesion level is effectively achieved. Furthermore, by fusing normal tissue with the new lesions, the obtained fused lesion tissue is more realistic. Using the synthetic lesion sequence for training deep learning algorithm models effectively improves the accuracy of deep learning algorithm models and enhances their generalization ability.

[0110] In one possible embodiment, the lesion synthesis module 31 is specifically used for:

[0111] The distribution of the original lesions that have been marked is statistically analyzed, and the original lesion group is selected from the original lesions based on the distribution statistical results;

[0112] The original lesion group is normalized in size, and new lesions are formed based on the original lesion group.

[0113] In one possible embodiment, when the lesion synthesis module 31 performs distribution statistics on the original lesions of the labeled lesions and selects the original lesion group from the original lesions based on the distribution statistics results, it specifically includes:

[0114] Based on the original lesion labels corresponding to the labeled original lesions, the distribution statistics of the original lesions are performed.

[0115] Based on the distribution statistics, a group of original lesions is randomly selected from the original lesions according to the set labeled distribution range.

[0116] In one possible embodiment, the lesion synthesis module 31, when performing size normalization on the original lesion group, specifically includes:

[0117] Based on the original lesion size corresponding to the original lesion in the original lesion group, a normalized size is randomly determined among the original lesion sizes;

[0118] The original lesion was normalized in size according to the normalized size.

[0119] In one possible embodiment, the lesion synthesis module 31, when combining the original lesions to form a new lesion, specifically includes:

[0120] The original lesions are combined into new lesions according to the beta distribution.

[0121] In one possible embodiment, when the lesion synthesis module 31 combines the original lesions into new lesions according to the beta distribution, it specifically includes:

[0122] The synthesis parameters are determined according to the beta distribution, and the synthesis ratio of each original lesion in the original lesion group is determined based on the synthesis parameters.

[0123] Based on the synthesis ratio and the normalized size, the original lesions are combined into new lesions.

[0124] In one possible embodiment, the formula for synthesizing the new lesion is:

[0125] patch_new=λ*patch1+(1-λ)*patch2

[0126] Where patch_new is the new lesion, patch1 and patch2 are the two original lesions, λ is the synthesis parameter, the domain of λ is [0, 1], and λ follows a beta distribution.

[0127] In one possible embodiment, the shape parameter of the beta distribution is set to a U-shape.

[0128] In one possible embodiment, the tissue fusion module 32 is specifically used for:

[0129] Based on the lesion size and three-dimensional Gaussian function of the new lesion, a Gaussian mask is generated;

[0130] Normal tissue from a CT sequence is randomly obtained, and the new lesion and the normal tissue are fused using the Gaussian mask to obtain fused lesion tissue.

[0131] In one possible embodiment, the formula for calculating the three-dimensional Gaussian function is:

[0132]

[0133] Where f is a three-dimensional Gaussian function, , , and z represent the variances of the three-dimensional Gaussian function in the three dimensions, respectively, where x, y, and z represent the distances to the center point, and k is the scaling factor.

[0134] In one possible embodiment, the variances of the three-dimensional Gaussian function are different in each of the three dimensions.

[0135] In one possible embodiment, the standard deviations of the three-dimensional Gaussian function in the three dimensions are randomly generated based on the lesion size of the new lesion.

[0136] In one possible embodiment, the formula for calculating the standard deviation of the three-dimensional Gaussian function in each dimension is:

[0137]

[0138] Among them, D i Let t1 be the diameter in the direction of lesion i, and t2 be the minimum and maximum values ​​selected.

[0139] In one possible embodiment, when the tissue fusion module 32 uses the Gaussian mask to fuse the new lesion and the normal tissue to obtain fused lesion tissue, it specifically includes:

[0140] The fusion ratio between the new lesion and the normal tissue is determined based on the Gaussian mask.

[0141] The new lesion and the normal tissue are fused according to the fusion ratio to obtain fused lesion tissue.

[0142] In one possible embodiment, the fusion formula for the fused lesion tissue is:

[0143] patch_m=gmask*patch_new+(1-gmask)*patch0

[0144] Where patch_m represents the fused lesion tissue, gmask is the Gaussian mask, patch_new represents the new lesion, and patch_0 represents the normal tissue.

[0145] In one possible embodiment, the apparatus further includes a first annotation module, the first annotation module being used for:

[0146] Based on the lesion size corresponding to the fused lesion tissue and the lesion location of the fused lesion tissue in the CT sequence, a fused lesion label corresponding to the fused lesion tissue is generated.

[0147] In one possible embodiment, the apparatus further includes a second annotation module, the second annotation module being used for:

[0148] The lesion size corresponding to the fused lesion tissue is determined based on the standard deviation of the Gaussian mask;

[0149] Based on the lesion size corresponding to the fused lesion tissue and the lesion location of the fused lesion tissue in the CT sequence, a fused lesion label corresponding to the fused lesion tissue is generated.

[0150] This application also provides a lesion-oriented synthesis device, which can integrate the lesion-oriented synthesis apparatus provided in this application. Figure 10This is a schematic diagram of a lesion-directed synthesis device provided in an embodiment of this application. (Reference) Figure 10 The lesion-directed synthesis device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the lesion-directed synthesis method provided in the above embodiments. The input device 43, output device 44, memory 42, and processors 41 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0151] The memory 42, as a computing device readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the lesion-directed synthesis method described in any embodiment of this application (e.g., the lesion synthesis module 31, tissue fusion module 32, and lesion replacement module 33 in the lesion-directed synthesis device). The memory 42 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 42 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include memory remotely located relative to the processor 41, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] Input device 43 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 44 may include display devices such as a display screen.

[0153] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the above-mentioned lesion-oriented synthesis method.

[0154] The lesion-oriented synthesis apparatus, equipment, and computer provided above can be used to execute the lesion-oriented synthesis method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0155] This application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the lesion-targeted synthesis method provided in the above embodiments.

[0156] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a first computer system in which a program is executed, or may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0157] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the lesion-oriented synthesis method described above, but can also perform related operations in the lesion-oriented synthesis method provided in any embodiment of this application.

[0158] The lesion-directed synthesis apparatus, device, and storage medium provided in the above embodiments can execute the lesion-directed synthesis method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the lesion-directed synthesis method provided in any embodiment of this application.

[0159] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A method for directional synthesis of lesions, characterized in that, include: Based on the original lesion labels corresponding to the labeled lesions, the distribution statistics of the original lesions are performed. Based on the distribution statistics, a group of original lesions is randomly selected from the original lesions according to the set labeled distribution range; the size of the original lesion group is normalized, and a new lesion is formed according to the original lesion group, wherein the original lesions in the same original lesion group are in the same distribution range; Based on the lesion size and three-dimensional Gaussian function of the new lesion, a Gaussian mask is generated; normal tissue from a CT sequence is randomly obtained, and the new lesion and the normal tissue are fused using the Gaussian mask to obtain fused lesion tissue, wherein the variance of the three-dimensional Gaussian function is different in each of the three dimensions; The normal tissue in the CT sequence is replaced with the fused lesion tissue to obtain a synthetic lesion sequence.

2. The lesion-directed synthesis method according to claim 1, characterized in that, The size normalization of the original lesion group includes: Based on the original lesion size corresponding to the original lesion in the original lesion group, a normalized size is randomly determined among the original lesion sizes; The original lesion was normalized in size according to the normalized size.

3. The lesion-directed synthesis method according to claim 2, characterized in that, The process of combining the original lesions to form new lesions includes: The synthesis parameters are determined according to the beta distribution, and the synthesis ratio of each original lesion in the original lesion group is determined based on the synthesis parameters. Based on the synthesis ratio and the normalized size, the original lesions are combined into new lesions.

4. The lesion-directed synthesis method according to claim 1, characterized in that, The process of fusing the new lesion and the normal tissue using the Gaussian mask to obtain the fused lesion tissue includes: The fusion ratio between the new lesion and the normal tissue is determined based on the Gaussian mask. The new lesion and the normal tissue are fused according to the fusion ratio to obtain fused lesion tissue.

5. The lesion-directed synthesis method according to claim 1, characterized in that, After replacing the normal tissue in the CT sequence with the fused lesion tissue to obtain the synthetic lesion sequence, the method further includes: Based on the lesion size corresponding to the fused lesion tissue and the lesion location of the fused lesion tissue in the CT sequence, a fused lesion label corresponding to the fused lesion tissue is generated.

6. The lesion-directed synthesis method according to claim 1, characterized in that, After replacing the normal tissue in the CT sequence with the fused lesion tissue to obtain the synthetic lesion sequence, the method further includes: The lesion size corresponding to the fused lesion tissue is determined based on the standard deviation of the Gaussian mask; Based on the lesion size corresponding to the fused lesion tissue and the lesion location of the fused lesion tissue in the CT sequence, a fused lesion label corresponding to the fused lesion tissue is generated.

7. A lesion-directed synthesis device, characterized in that, It includes a lesion synthesis module, a tissue fusion module, and a lesion replacement module, among which: The lesion synthesis module is used to perform distribution statistics on the original lesions based on the original lesion labels corresponding to the labeled original lesions; based on the distribution statistics results, randomly select a group of original lesions from the original lesions according to the set label distribution range; normalize the size of the original lesion group, and synthesize a new lesion based on the original lesion group, wherein the original lesions in the same original lesion group are within the same distribution range; The tissue fusion module is used to generate a Gaussian mask based on the size of the new lesion and a three-dimensional Gaussian function; randomly acquire normal tissue from a CT sequence, and use the Gaussian mask to fuse the new lesion and the normal tissue to obtain fused lesion tissue, wherein the variance of the three-dimensional Gaussian function is different in each of the three dimensions. The lesion replacement module is used to replace the normal tissue in the CT sequence with the fused lesion tissue to obtain a synthetic lesion sequence.

8. A lesion-directed synthesis device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the lesion-directed synthesis method as described in any one of claims 1-6.

Images