Target typing method, apparatus, device, and storage medium

Abstract

The application discloses a target typing method, device, equipment and storage medium. The target typing method comprises the following steps: performing feature extraction on a plurality of initial images containing a target to be typed, respectively, to obtain initial features of the target in each initial image; fusing the initial features based on a preset correlation relationship between the initial images to obtain target features; and obtaining a typing result of the target by using the target features. The above scheme can improve the accuracy of the typing result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a target classification method, apparatus, device and storage medium. Background Technology

[0002] In recent years, with the rapid development of computer science and technology and medical imaging engineering, many advanced medical imaging devices have emerged worldwide, providing medical images for clinical diagnosis. These images can reflect relevant information about human structure, organs, and diseased tissues. However, in current technology, lesion types are usually identified by doctors observing the characteristics of lesions in medical images. The identification results are closely related to the doctor's clinical experience, especially when the lesion type is difficult to identify, making identification quite challenging. Summary of the Invention

[0003] This application provides at least one target classification method, apparatus, device, and storage medium.

[0004] This application provides a target classification method, comprising: extracting features from several initial images containing the target to be classified, obtaining initial features of each initial image with respect to the target; fusing the initial features based on a preset correlation between the initial images to obtain target features; and using the target features to obtain the target classification result.

[0005] Therefore, by extracting features from the initial image containing the target to be classified, target classification can be achieved without relying on the doctor's clinical experience, thus reducing the difficulty of identification and improving the accuracy of target classification. Furthermore, by fusing the initial features of multiple initial images to obtain target features, and then obtaining the target classification result based on these features, more target features can be fused compared to using the initial features of a single initial image, resulting in a more accurate target classification result. Further, this application also fuses the initial features by combining the preset correlations between the initial images to obtain target features, making the determined target classification result even more accurate.

[0006] The target is the lesion, and the initial images include local images acquired from the target and global images acquired from the organs to which the target belongs. The initial features include a first initial feature corresponding to the local image and a second initial feature corresponding to the global image. Based on the preset correlation between the initial images, the initial features are fused to obtain the target features, including: based on the preset correlation, the first initial feature and the second initial feature are fused to obtain the target features.

[0007] Therefore, extracting features from both global and local images and fusing these features can increase the feature differences between different categories, reduce the occurrence of the same image features being classified into different categories, and improve the accuracy of target classification.

[0008] Among them, several initial images include initial images acquired from the target using different acquisition methods. Based on the preset correlation between each initial feature, the initial features are fused to obtain the target features, including: based on the preset correlation, the initial features corresponding to the initial images acquired by different acquisition methods are fused to obtain the target features.

[0009] Therefore, by fusing the initial features corresponding to the initial images acquired using different acquisition methods, it is possible to obtain accurate classification results even when there are different image features for the same type of target, by referring to the features of multiple sequences or time series.

[0010] The preset association relationship is an adjacency matrix. Based on the preset association relationship, the initial features corresponding to the initial images acquired by different acquisition methods are fused to obtain the target features. This includes: fusing the initial features to obtain fused features; multiplying the fused features with the adjacency matrix; and obtaining the target features based on the result of the multiplication.

[0011] Therefore, by fusing the adjacency matrix with the fusion features, the connections between the initial images can be uncovered, making the determined target features more accurate.

[0012] The target classification method is executed by a classification network, which includes a first multilayer perceptron and a second multilayer perceptron. The network fuses the initial features to obtain fused features, including: processing the initial features using the first multilayer perceptron to obtain the advanced features corresponding to each initial feature; concatenating the advanced features to obtain fused features; and obtaining the target features based on the result of the multiplication, including: processing the fused features using the second multilayer perceptron to obtain the target features.

[0013] Therefore, performing the target typing method through a typing network eliminates the need to rely on the user's clinical experience, making the target typing process more convenient. Furthermore, feature enhancement can be achieved by processing features using a multilayer perceptron.

[0014] The target classification method is executed by a classification network. The method further includes: extracting features from several sample images containing the target to be classified, obtaining initial features of each sample image about the target; each sample image is acquired using a different acquisition method; fusing the initial features of each sample based on the iterative association relationship corresponding to the current training iteration to obtain sample target features; in the case of the first training iteration, the iterative association relationship corresponding to the current training iteration is the initial association relationship; in the case of subsequent training iterations, the iterative association relationship corresponding to the current training iteration is the iterative association relationship adjusted in the previous training iteration; using the sample target features, obtaining sample classification results about the target; adjusting the network parameters of the classification network based on the difference between the sample classification results and the true classification results, the network parameters including the iterative association relationship; wherein, if the difference meets the error condition, the iterative association relationship adjusted in the last training iteration is used as the preset association relationship.

[0015] Therefore, by training the correlation between sample images acquired by each acquisition method during the training of the fractal network, and taking the iterative correlation adjusted after the last iteration of training as the preset correlation when the difference meets the error condition, the preset correlation obtained by training can better reflect the connection between each initial image.

[0016] The target is the lesion. Using sample target features, sample typing results are obtained, including: using sample target features to obtain the first benign / malignant typing result and the target subtype typing result; based on the difference between the sample typing result and the true typing result, the network parameters of the typing network are adjusted, including: obtaining the first difference between the first benign / malignant typing result and the true benign / malignant typing result, and obtaining the second difference between the subtype typing result and the true subtype typing result; combining the first and second differences, the network parameters of the typing network are adjusted.

[0017] Therefore, by utilizing the target features of the samples to obtain the first benign / malignant classification result and the target subtype classification result, and by adjusting the network parameters of the classification network based on the first difference between the first benign / malignant classification result and the true benign / malignant classification result, and the second difference between the subtype classification result and the true subtype classification result, it is possible to reduce misclassification between major categories, stabilize the training process, accelerate the convergence speed of the classification network, and improve the accuracy of the algorithm.

[0018] The method further includes: obtaining the second benign / malignant classification result of the target based on the subtype classification result; obtaining the third difference between the second benign / malignant classification result and the true benign / malignant classification result; and adjusting the network parameters of the classification network by combining the first difference and the second difference, including: adjusting the network parameters of the classification network by combining the first difference, the second difference and the third difference.

[0019] Therefore, by obtaining the second benign / malignant classification result of the target based on the subtype classification result and obtaining the third difference between the second benign / malignant classification result and the true benign / malignant classification result, and then combining the first difference, the second difference and the third difference, the network parameters of the classification network are adjusted so that the network parameters of the classification network are adjusted by utilizing the mutual constraints between upstream and downstream tasks.

[0020] Before extracting features from several initial images containing the target to be classified and obtaining the initial features of each initial image with respect to the target, the method further includes preprocessing the several initial images, which may include adjusting contrast, adjusting window width and window level, adjusting image size, and normalization.

[0021] Therefore, by preprocessing several initial images before feature extraction, the format of each initial image can be unified, thereby improving the accuracy of subsequent classification results.

[0022] This application provides a target classification device, comprising: a feature extraction module, used to extract features from several initial images containing the target to be classified, to obtain initial features of each initial image with respect to the target; a feature enhancement module, used to fuse the initial features based on a preset correlation between the initial images, to obtain target features; and a classification module, used to obtain the classification result of the target using the target features.

[0023] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described target classification method.

[0024] This application provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the aforementioned target fractal method.

[0025] The aforementioned scheme, by extracting features from the initial image containing the target to be classified, enables target classification without relying on the doctor's clinical experience. This not only reduces the difficulty of identification but also improves the accuracy of target classification. Furthermore, by fusing the initial features of multiple initial images to obtain target features, and then using these target features to obtain the target classification result, more features can be fused compared to using the initial features of a single initial image, resulting in a more accurate target classification. Moreover, this application further fuses the initial features by combining preset correlations between the initial images to obtain target features, making the determined target classification result even more accurate.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0028] Figure 1 This is a flowchart illustrating an embodiment of the target classification method of this application;

[0029] Figure 2 This is a schematic diagram of the structure of the fractal network shown in one embodiment of the target fractal method of this application;

[0030] Figure 3 This is a schematic diagram of the structure of an embodiment of the target parting device of this application;

[0031] Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0032] Figure 5 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0033] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0034] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0035] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0036] This application provides several target typing methods and apparatuses. These target typing methods can be applied to medical testing. For example, in lung cancer diagnosis, the initial image can be an internal biological image captured by medical equipment, and the target can be any lesion within the organism requiring typing, such as a tumor. The executing entity of the target typing method can be a target typing apparatus, such as a terminal device, server, or other processing device. The terminal device can be a device for medical image analysis, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, etc. In some possible implementations, the target typing method can be implemented by a processor calling computer-readable instructions stored in memory.

[0037] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the target classification method of this application.

[0038] Specifically, the target subtyping method may include the following steps:

[0039] Step S11: Extract features from several initial images containing the target to be classified, and obtain the initial features of each initial image with respect to the target.

[0040] The initial image can be either two-dimensional or three-dimensional. The type of initial image can also vary; for example, it can be a medical image, a regular camera image, a security image, etc. In some applications, the initial image is a medical image, and the target to be classified can be a lesion. In other applications, the initial image is a security image, and the target to be classified can be an animal body or other objects requiring classification.

[0041] The initial image can be acquired either by a camera component carried by the execution device performing the target fractal method, or by other devices transmitting it to the execution device via various communication methods. Other devices refer to those that do not share the same processor as the execution device.

[0042] The method for extracting features from several initial images can be any neural network with feature extraction capabilities, or an algorithm with feature extraction capabilities.

[0043] Step S12: Based on the preset correlation between each initial image, fuse each initial feature to obtain the target feature.

[0044] The preset correlation between initial images can be used to represent the degree of correlation between the initial features extracted from each initial image. For example, the higher the correlation between two initial features, the higher the mutual referentiality between the two initial features, meaning the fusion weight between them is relatively larger. That is, in some disclosed embodiments, the preset correlation between initial images affects the fusion weight between initial features.

[0045] Step S13: Utilize the target features to obtain the target classification results.

[0046] In some disclosed embodiments, the method of obtaining the target classification result using target features can be by using a network model with classification capabilities. For example, the target features are input into the network model, and the network model outputs the classification result for the target.

[0047] In some application scenarios, the target is the lesion, and the classification result of the target can be the specific disease type to which the target belongs and / or the benign or malignant classification result.

[0048] The aforementioned scheme, by extracting features from the initial image containing the target to be classified, enables target classification without relying on the doctor's clinical experience. This not only reduces the difficulty of identification but also improves the accuracy of target classification. Furthermore, by fusing the initial features of multiple initial images to obtain target features, and then using these target features to obtain the target classification result, more features can be fused compared to using the initial features of a single initial image, resulting in a more accurate target classification. Moreover, this application further fuses the initial features by combining preset correlations between the initial images to obtain target features, making the determined target classification result even more accurate.

[0049] In some disclosed embodiments, the target is a lesion, and the initial images include local images acquired from the target and global images acquired from the organ to which the target belongs. The initial features of each initial image with respect to the target include a first initial feature corresponding to the local image and a second initial feature corresponding to the global image. Based on this, step S12 above may include the following steps:

[0050] Based on a preset association relationship, the first initial feature and the second initial feature are fused to obtain the target feature. In this embodiment, the preset association relationship can be the fusion weight between a local image and a global image, that is, the preset association relationship can be the fusion weight between the first initial feature and the second initial feature. This fusion weight can be customized by the user or a default value can be selected. For example, the fusion weight of the first initial feature is 0.5 and the fusion weight of the second initial feature is 0.5.

[0051] In some disclosed embodiments, several initial images include initial images acquired from the target using different acquisition methods. For example, the target is a lesion, the organ to which the target belongs is the liver, spleen, or kidney, etc., the initial image is a medical image, and the different acquisition methods can be CT and MR, or plain scan and enhanced scan. For example, different acquisition methods can include in-phase and out-of-phase imaging (IP and OP), fat suppression imaging (T2), diffusion-weighted imaging (DWI), etc. Each initial image can be acquired from the target at the same time using its corresponding acquisition method, or it can be acquired from the target at different times using its corresponding acquisition method. For example, this disclosure embodiment uses the example of initial images acquired from the target at different times. In some disclosed embodiments, the target is a lesion, and the initial image acquired by each acquisition method includes a local image acquired from the target and a global image acquired from the organ to which the target belongs. For example, the organ where the lesion is located can be the liver. The local image acquired from the target can be extracted from the global image acquired from the organ to which the target belongs, or it can be acquired from the target alone. The above step S11 can include the following steps:

[0052] For each acquisition method, feature extraction is performed on the local and global images acquired using that method, resulting in a first feature for the local image and a second feature for the global image. Combining the first and second features corresponding to that acquisition method yields the initial feature corresponding to the initial image acquired by that method. In other words, for each acquisition method, there will be a local image of the target and a global image of the organ to which the target belongs, with each acquisition method corresponding to an initial feature. Because the global image, in addition to containing the target, may also contain changes in the overall morphology and texture of the organ, the distribution and direction of blood vessels within the organ, and, for the liver, the state of bile duct dilation, etc., it can provide important reference information for target classification. By extracting features from the local and global images, features of the organ background and lesions can be extracted, achieving the fusion of global and local features, increasing the feature differences between different disease categories, reducing the occurrence of identical images with different diseases, and thus achieving accurate classification of the target (lesion).

[0053] For each acquisition method, by extracting features from global and local images and fusing global and local features, the feature differences between different categories can be increased, the occurrence of the same image features being classified into different categories can be reduced, and the accuracy of target classification can be improved.

[0054] In some disclosed embodiments, before performing step S11 above, the target fracturing method may further include the following steps:

[0055] Several initial images are preprocessed. This preprocessing includes both local and global image preprocessing. Preprocessing may include adjusting contrast, adjusting window width and level, adjusting image size, and normalization (one or more of these). For example, image intensity is normalized to [0,1]. It is worth noting that the pixel distribution range varies considerably across different initial images. Taking MR images as an example, to highlight the organ to which the target belongs and the region where the target is located—that is, to highlight the organ of interest and the lesion region—different images may use the gray value corresponding to 99.9% of the cumulative gray-level distribution function or other thresholds as preprocessing clamp values.

[0056] For local images, image resizing can be achieved by cropping the region containing the target. To avoid feature loss and maintain a certain receptive field, different cropping methods are used for targets of different sizes. For example, small targets are cropped by expanding their area by approximately 100% in multiple directions, while large targets are cropped by expanding their area by 50% in multiple directions, and finally, the images are resized to the same size. For global images, images containing the organs to which the target belongs are extracted using a segmentation model and resized. The size of the local image input into the classification network is the same as the size of the global image.

[0057] By preprocessing several initial images before feature extraction, the format of each initial image can be standardized, thereby improving the accuracy of subsequent classification results.

[0058] Step S12 above may include the following steps:

[0059] Based on preset correlation relationships, initial features corresponding to initial images acquired through different acquisition methods are fused to obtain target features. For example, fusing initial features corresponding to initial images obtained from plain scans, conventional contrast-enhanced scans, in-phase and out-of-phase imaging, fat-suppressed imaging, and diffusion-weighted imaging can yield target features. In some application scenarios, preset correlation relationships can be used to represent the adjacency relationships of each initial feature or the weights for weighted fusion. Specifically, the fusion method can be to stitch together initial features based on preset adjacency relationships or to perform weighted fusion of initial features based on preset weights. Simply stitching together initial features without considering the relationships between them may result in inaccurate classification results. Considering that the initial features are not isolated from each other, taking CT liver lesion diagnosis as an example, after contrast-enhanced scanning, the influence characteristics of the lesions gradually change over time. Therefore, by combining the preset correlations between the initial images, we can extract the contextual information between the initial images acquired at adjacent times, and we can also explore the connections between the initial images acquired at non-adjacent times, thereby more accurately classifying the lesions.

[0060] By fusing the initial features corresponding to the initial images acquired through different acquisition methods, it is possible to obtain accurate classification results even when there are different image features for the same category of targets, by referring to the features of multiple sequences or time series.

[0061] In some publicly available embodiments, the preset association relationship is an adjacency matrix. In some application scenarios, the relationship between images acquired by different acquisition methods is encoded into a weighted directed graph, and then converted into an adjacency matrix.

[0062] The above-mentioned method of fusing initial features corresponding to initial images acquired by different acquisition methods based on preset association relationships to obtain target features can specifically include the following steps:

[0063] The initial features are fused to obtain fused features. Then, the fused features are multiplied by the adjacency matrix. Finally, the target features are obtained based on the result of the multiplication. The adjacency matrix can be a normalized matrix. By fusing the adjacency matrix with the fused features, the connections between the initial images can be uncovered, making the determined target features more accurate. Specifically, the target classification method is executed by a classification network. The classification network includes a first multilayer perceptron and a second multilayer perceptron. The specific method for fusing the initial features to obtain fused features can be as follows: the first multilayer perceptron processes each initial feature to obtain the corresponding advanced features. Then, the advanced features are concatenated to obtain the fused features. The multiplication of the fused features with the adjacency matrix can specifically be a matrix multiplication operation, meaning the fused features are also matrices. The specific method for obtaining the target features based on the result of the multiplication can be as follows: the second multilayer perceptron processes the fused features to obtain the target features. Executing the target classification method through a classification network eliminates the need for user clinical experience, making the target classification process more convenient. In addition, feature enhancement can be achieved by using a multilayer perceptron to process features.

[0064] For a better understanding of the fractal networks described in the embodiments of this disclosure, please also refer to... Figure 2 , Figure 2 This is a schematic diagram illustrating the structure of a fractal network in an embodiment of the target fractal method of this application. Figure 2 As shown, the initial images include plain scan images and enhanced scan images. The plain scan images include T2 images, DWI images, ADC images, IP images, and OP images. The enhanced scan images include preArtery images, Lap images, PV images, and Delay images. The initial images corresponding to each acquisition method include an image of the lesion area (local image) and an image of the liver background (global image).

[0065] Initial features for each image are obtained through the feature extraction network in the fractal network. The feature extraction network can be ResNet, VGG, EfficientNet, or their variants. Specifically, local and global images acquired under the same acquisition method are simultaneously input into the feature extraction network to obtain the initial features corresponding to the initial images acquired under that acquisition method. That is, a local and global image under T2 mode corresponds to one initial feature, a local and global image under DWI mode corresponds to one initial feature, a local and global image under ADC mode corresponds to one initial feature, a local and global image under IP mode corresponds to one initial feature, a local and global image under OP mode corresponds to one initial feature, a local and global image under PreArtery mode corresponds to one initial feature, a local and global image under Lap mode corresponds to one initial feature, a local and global image under PV mode corresponds to one initial feature, and a local and global image under Delay mode corresponds to one initial feature. Step S12 above can be executed by the graph convolutional network in the fractal network. The graph convolutional network can have several groups, constructed in a cascaded manner. The graph convolutional network includes a first multilayer perceptron and a second multilayer perceptron. Then, each initial feature is input into the first multilayer perceptron to obtain the advanced features corresponding to each initial feature. For example, each initial feature is input into a first multilayer perceptron to obtain the advanced features corresponding to that initial feature.

[0066] The above process multiplies the fused features with the adjacency matrix, and based on the result of the multiplication, obtains the target features. The values ​​between the two acquisition methods in the adjacency matrix are used as the fusion weights for the advanced features corresponding to these two acquisition methods. The adjacency matrix can be considered as the network parameters of the fractal network, which can be obtained through fractal network training. At the beginning of training, the adjacency matrix is ​​initialized with 1s for the diagonal elements and 0s for the other positions. The acquisition methods on the horizontal and vertical axes of the adjacency matrix are the same, meaning that the two acquisition methods corresponding to each position on the diagonal are identical. "At the beginning of training" refers to the first iteration, where the initial associations are obtained through initialization. Then, the number, direction, weight, and other parameters of the edges in the directed graph are adjusted through network self-learning. That is, the number, position, and value of non-zero positions in the adjacency matrix are adjusted through network training.

[0067] Then, the target features are input into the pooling layer and the fully connected layer for processing, and the processing results are then input into the corresponding normalization layer. Figure 2 In the normalization layer used to obtain the first benign / malignant classification result, the activation function used can be the sigmoid function, while the activation function used in the normalization layer used to obtain the subtype classification result can be the softmax function.

[0068] In some publicly available embodiments, the target fractal method further includes a training step for the fractal network:

[0069] Feature extraction was performed on several sample images containing the target to be classified, resulting in initial sample features for each image related to the target. Each sample image was acquired using a different acquisition method, as described above, and will not be repeated here.

[0070] Based on the iterative association relationship corresponding to this training iteration, the initial features of each sample are fused to obtain the target features. In the first training iteration, the iterative association relationship is the initial association relationship. This iterative association relationship can be an adjacency matrix, and the initial adjacency matrix is ​​a matrix with 1s on the diagonal and 0s in the remaining positions—the initial adjacency matrix described above. In subsequent training iterations, the iterative association relationship is the adjusted iterative association relationship from the previous training iteration. For details on how to fuse the initial features of each sample based on the iterative association relationship to obtain the target features, please refer to the steps described above for fusing initial features based on a preset association relationship to obtain the target features; these will not be repeated here.

[0071] Then, using the sample target features, a sample classification result for the target is obtained. Optionally, using the sample target features, a first benign / malignant classification result and a subtype classification result for the target are obtained respectively. The subtype classification result can be the specific disease type to which the target belongs. In some disclosed embodiments, the target classification method may further include the following step: obtaining a second benign / malignant classification result for the target based on the subtype classification result. The first and second benign / malignant classification results include whether the target is benign or malignant. Specifically, each specific disease type is generally either benign or malignant; therefore, after obtaining the specific disease type to which the target belongs, the second benign / malignant classification result for the target can be obtained based on whether the specific disease type is benign or malignant.

[0072] Finally, the network parameters of the typing network are adjusted based on the difference between the sample typing results and the true typing results. The network parameters of the typing network include iterative correlation relationships. Optionally, if the difference meets the error condition, the iterative correlation relationship adjusted after the last iteration of training is used as the preset correlation relationship. The method of adjusting the network parameters of the typing network based on the difference between the sample typing results and the true typing results can be: adjusting the network parameters of the typing network based on the first difference between the first benign / malignant typing result and the true benign / malignant typing result; or adjusting the network parameters of the typing network based on the second difference between the subtype typing result and the true subtype typing result; or adjusting the network parameters of the typing network by combining the first difference and the second difference. In some disclosed embodiments, a third difference between the second benign / malignant typing result and the true benign / malignant typing result can also be obtained, and the network parameters of the typing network are adjusted based on the third difference. In some disclosed embodiments, the network parameters of the typing network are adjusted by combining the first difference and the third difference, or the second difference and the third difference, or the first difference, the second difference, and the third difference. Exemplarily, this disclosed embodiment selects to combine the first difference, the second difference, and the third difference to adjust the network parameters of the typing network.

[0073] One way to adjust the network parameters of the typing network based on the difference between the sample typing results and the true typing results is to determine the target loss based on the difference between the sample typing results and the true typing results. The difference must satisfy the error condition that the target loss is less than or equal to a preset loss. Then, the network parameters of the typing network are adjusted based on the target loss. The target loss can be calculated using binary cross-entropy loss (BCE loss), cross-entropy loss, focal loss, or metric learning loss. For example, when adjusting the network parameters of the typing network by combining the first difference, second difference, and third difference, the first loss is determined based on the first difference (e.g., ...). Figure 2 loss1), determining the second loss based on the second difference (e.g.) Figure 2 loss2), and determine the third loss based on the third difference (e.g. Figure 2 The three losses (loss3) are weighted and fused to obtain the target loss, and then the network parameters of the fractal network are adjusted based on the target loss. Optionally, the weight of the loss corresponding to the second difference in the weighted fusion is greater than the weight of the losses corresponding to the first and third differences in the weighted fusion.

[0074] By training the correlation between sample images acquired by different acquisition methods during the training of the fractal network, and taking the iterative correlation adjusted after the last iteration of training as the preset correlation when the difference meets the error condition, the preset correlation obtained by training can better reflect the connection between the initial images.

[0075] The second benign / malignant classification result of the target is obtained based on the subtype classification result, and the third difference between the second benign / malignant classification result and the true benign / malignant classification result is obtained. Then, the network parameters of the classification network are adjusted by combining the first difference, the second difference and the third difference, so that the network parameters of the classification network can be adjusted by utilizing the mutual constraints between upstream and downstream tasks.

[0076] In some disclosed embodiments, step S13 above may include the following steps: obtaining a benign or malignant classification result of the target using target features; or obtaining a subtype classification result of the target using target features; or, after obtaining a subtype classification result of the target using target features, further obtaining a benign or malignant classification result of the target based on the subtype classification result.

[0077] In some application scenarios, based on multi-sequence CT / MR images, the target classification method provided in this disclosure can be used to determine the benign or malignant nature of lesions and classify their subtypes, assisting radiologists in making rapid and accurate lesion diagnoses, saving image reading time and improving diagnostic accuracy. Furthermore, if doctors are confident in the overall result by combining their own judgment with the auxiliary diagnostic results obtained by this method, they can, to some extent, appropriately omit complex pathological sampling.

[0078] This disclosure constructs a multi-task deep framework, utilizing strong priors between subtype classification and benign / malignant classification, which effectively reduces misclassification between major categories, stabilizes the training process, accelerates model convergence, and improves algorithm accuracy. Three-dimensional imaging technology plays an increasingly important role in medical imaging diagnosis. For example, multi-phase CT or multi-sequence MR imaging, through comprehensive analysis of the imaging features of lesions across sequences, can more comprehensively and accurately clarify the nature of lesions. For instance, a single image is often inaccurate in actual diagnosis. For example, taking the liver as an example, there are many types of focal lesions, and the problem of the same disease appearing differently or the same image showing different diseases is common. If identification is based on a single image, differential diagnosis becomes very difficult.

[0079] The aforementioned scheme, by extracting features from the initial image containing the target to be classified, enables target classification without relying on the doctor's clinical experience. This not only reduces the difficulty of identification but also improves the accuracy of target classification. Furthermore, by fusing the initial features of multiple initial images to obtain target features, and then using these target features to obtain the target classification result, more features can be fused compared to using the initial features of a single initial image, resulting in a more accurate target classification. Moreover, this application further fuses the initial features by combining preset correlations between the initial images to obtain target features, making the determined target classification result even more accurate.

[0080] In addition, the embodiments of this disclosure extract multi-level image features of global and local perspectives, which can learn more comprehensive features, further increase the feature gap between different disease categories, and achieve accurate classification of lesions.

[0081] In addition, compared with single-task learning, the embodiments of this disclosure utilize the strong prior constraints between upstream and downstream tasks to achieve cross-task resource and parameter sharing, stabilize the training process, accelerate the convergence of fractal networks, and improve algorithm accuracy.

[0082] In some application scenarios, the target classification method provided in this disclosure can be applied to products such as computer-aided image reading and diagnosis systems, remote medical diagnosis, and cloud platform-assisted intelligent diagnosis.

[0083] Please see Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the target classification device of this application. The target classification device 30 includes a feature extraction module 31, a feature enhancement module 32, and a classification module 33. The feature extraction module is used to extract features from several initial images containing the target to be classified, to obtain initial features of each initial image with respect to the target; the feature enhancement module is used to fuse the initial features based on a preset correlation between the initial images to obtain target features; the classification module is used to obtain the classification result of the target using the target features.

[0084] The above-described scheme, by extracting features from the initial image containing the target to be classified, enables target classification without relying on the doctor's clinical experience. This not only reduces the difficulty of identification but also improves the accuracy of target classification. Furthermore, by fusing the initial features of multiple initial images to obtain target features, and then using these features to derive the target classification result, it can integrate more target features compared to using initial features from a single initial image, resulting in a more accurate target classification.

[0085] In some disclosed embodiments, the target is a lesion, and the initial images include local images acquired from the target and global images acquired from the organs to which the target belongs. The initial features include a first initial feature corresponding to the local image and a second initial feature corresponding to the global image. The feature enhancement module 32 fuses the initial features based on the preset association relationship between the initial images to obtain the target features, including: fusing the first initial feature and the second initial feature based on the preset association relationship to obtain the target features.

[0086] The above scheme extracts features from both global and local images and fuses these features, which increases the feature differences between different categories, reduces the occurrence of the same image features being classified into different categories, and improves the accuracy of target classification.

[0087] In some disclosed embodiments, the initial images include initial images acquired from the target using different acquisition methods. The feature enhancement module 32 fuses the initial features based on the preset correlation between the initial features to obtain the target features, including: fusing the initial features corresponding to the initial images acquired by the different acquisition methods based on the preset correlation to obtain the target features.

[0088] The above scheme fuses the initial features corresponding to the initial images acquired by different acquisition methods, so that it can refer to the features of multiple sequences or time series, and obtain accurate classification results even when there are different image features of the same type of target.

[0089] In some disclosed embodiments, the preset association relationship is an adjacency matrix. Based on the preset association relationship, the feature enhancement module 32 fuses the initial features corresponding to the initial images acquired by different acquisition methods to obtain target features, including: fusing each initial feature to obtain fused features; multiplying the fused features with the adjacency matrix; and obtaining the target features based on the result of the multiplication.

[0090] The above scheme, by fusing the adjacency matrix with the fusion features, can uncover the connections between the initial images, making the determined target features more accurate.

[0091] In some disclosed embodiments, the target classification method is executed by a classification network, which includes a first multilayer perceptron and a second multilayer perceptron. The feature enhancement module 32 fuses the initial features to obtain fused features, including: processing the initial features using the first multilayer perceptron to obtain the advanced features corresponding to each initial feature; concatenating the advanced features to obtain fused features; and obtaining the target features based on the result of multiplication, including: processing the fused features using the second multilayer perceptron to obtain the target features.

[0092] The above-described scheme executes the target typing method through a typing network, eliminating the need for reliance on the user's clinical experience and making the target typing process more convenient. Furthermore, feature enhancement can be achieved by using a multilayer perceptron to process the features.

[0093] In some disclosed embodiments, the target classification method is executed by a classification network, and the target classification device 30 further includes a training module (not shown). The training module is used to: extract features from several sample images containing the target to be classified, respectively, to obtain initial sample features of each sample image about the target, wherein each sample image is acquired using a different acquisition method; fuse the initial sample features based on the iterative association relationship corresponding to the current iteration training to obtain sample target features; in the case of the first iteration training, the iterative association relationship corresponding to the current iteration training is the initial association relationship, and in the case of non-first iteration training, the iterative association relationship corresponding to the current iteration training is the iterative association relationship adjusted in the previous iteration training; obtain sample classification results about the target using the sample target features; adjust the network parameters of the classification network based on the difference between the sample classification results and the true classification results, wherein the network parameters include the iterative association relationship; wherein, when the difference meets the error condition, the iterative association relationship adjusted in the last iteration training is used as the preset association relationship.

[0094] The above scheme trains the correlation between sample images acquired by different acquisition methods during the training of the fractal network. When the difference meets the error condition, the iterative correlation adjusted by the last iteration of training is used as the preset correlation, so that the preset correlation obtained by training can better reflect the connection between the initial images.

[0095] In some disclosed embodiments, the target is a lesion. The training module uses sample target features to obtain sample typing results about the target, including: using sample target features to obtain a first benign / malignant typing result and a subtype typing result of the target, respectively; adjusting the network parameters of the typing network based on the difference between the sample typing result and the true typing result, including: obtaining a first difference between the first benign / malignant typing result and the true benign / malignant typing result, and obtaining a second difference between the subtype typing result and the true subtype typing result; and adjusting the network parameters of the typing network by combining the first difference and the second difference.

[0096] The above scheme obtains the first benign / malignant classification result and the subtype classification result of the target by utilizing the target features of the sample. Based on the first difference between the first benign / malignant classification result and the true benign / malignant classification result, and the second difference between the subtype classification result and the true subtype classification result, the network parameters of the classification network are adjusted. This can reduce misclassification between major categories, stabilize the training process, accelerate the convergence speed of the classification network, and improve the accuracy of the algorithm.

[0097] In some publicly available embodiments, the training module is further configured to: obtain a second benign / malignant classification result of the target based on the subtype classification result; obtain a third difference between the second benign / malignant classification result and the true benign / malignant classification result; and adjust the network parameters of the classification network by combining the first difference and the second difference, including: adjusting the network parameters of the classification network by combining the first difference, the second difference and the third difference.

[0098] The above scheme obtains the second benign / malignant classification result of the target based on the subtype classification result and obtains the third difference between the second benign / malignant classification result and the true benign / malignant classification result. Then, it combines the first difference, the second difference and the third difference to adjust the network parameters of the classification network, so that the network parameters of the classification network can be adjusted by utilizing the mutual constraints between upstream and downstream tasks.

[0099] In some disclosed embodiments, the target classification device 30 includes a preprocessing module (not shown) that, before performing feature extraction on several initial images containing the target to be classified to obtain initial features of each initial image with respect to the target, the preprocessing module is used to: preprocess the several initial images, the preprocessing including one or more of adjusting contrast, adjusting window width and window level, adjusting image size, and normalization processing.

[0100] The above scheme preprocesses several initial images before feature extraction, thereby unifying the format of each initial image and improving the accuracy of subsequent classification results.

[0101] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 40 includes a memory 41 and a processor 42. The processor 42 is used to execute program instructions stored in the memory 41 to implement the steps in the above-described target classification method embodiment. In a specific implementation scenario, the electronic device 40 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 40 may also include mobile devices such as laptops and tablets, which are not limited here.

[0102] Specifically, processor 42 controls itself and memory 41 to implement the steps in the above-described target fractal method embodiment. Processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.

[0103] The aforementioned scheme, by extracting features from the initial image containing the target to be classified, enables target classification without relying on the doctor's clinical experience. This not only reduces the difficulty of identification but also improves the accuracy of target classification. Furthermore, by fusing the initial features of multiple initial images to obtain target features, and then using these target features to obtain the target classification result, more features can be fused compared to using the initial features of a single initial image, resulting in a more accurate target classification. Moreover, this application further fuses the initial features by combining preset correlations between the initial images to obtain target features, making the determined target classification result even more accurate.

[0104] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 50 stores program instructions 501 that can be executed by a processor. The program instructions 501 are used to implement the steps in the above-described target fractal method embodiment.

[0105] The aforementioned scheme, by extracting features from the initial image containing the target to be classified, enables target classification without relying on the doctor's clinical experience. This not only reduces the difficulty of identification but also improves the accuracy of target classification. Furthermore, by fusing the initial features of multiple initial images to obtain target features, and then using these target features to obtain the target classification result, more features can be fused compared to using the initial features of a single initial image, resulting in a more accurate target classification. Moreover, this application further fuses the initial features by combining preset correlations between the initial images to obtain target features, making the determined target classification result even more accurate.

[0106] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0107] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0109] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0110] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A target classification method, characterized in that, include: Feature extraction is performed on several initial images containing the target to be classified to obtain initial features of each initial image with respect to the target, wherein the target is a lesion, and the several initial images include local images of the target acquired using different acquisition methods and global images of the organ to which the target belongs; Based on the preset association relationship between each initial image, the initial features are fused to obtain the target features, where the preset association relationship is an adjacency matrix; Using the target features, the classification result of the target is obtained; The step of extracting features from several initial images containing the target to be classified, to obtain initial features of each initial image with respect to the target, includes: For each acquisition method, feature extraction is performed on the local image and the global image under the acquisition method to obtain the first feature of the local image and the second feature of the global image. The first feature and the second feature are then combined to obtain the initial feature corresponding to the acquisition method. The target classification method is executed by a classification network, which includes a first multilayer perceptron and a second multilayer perceptron; the step of fusing the initial features based on the preset correlation between the initial images to obtain the target features includes: The first multilayer perceptron is used to process each of the initial features to obtain the advanced features corresponding to each of the initial features; The advanced features described above are concatenated to obtain the fused features; Multiply the fusion feature by the adjacency matrix; The fused features are processed using the second multilayer perceptron to obtain the target features.

2. The method according to claim 1, characterized in that, The method further includes: Feature extraction is performed on several sample images containing the target to be classified to obtain the initial sample features of each sample image with respect to the target. Each sample image is acquired using a different acquisition method. Based on the iterative correlation relationship corresponding to this iteration training, the initial features of each sample are fused to obtain the target features of the sample; in the case of the first iteration training, the iterative correlation relationship corresponding to this iteration training is the initial correlation relationship, and in the case of non-first iteration training, the iterative correlation relationship corresponding to this iteration training is the iterative correlation relationship adjusted in the previous iteration training; Using the target features of the samples, sample classification results for the target are obtained; Based on the difference between the sample typing results and the true typing results, the network parameters of the typing network are adjusted, and the network parameters include the iterative correlation relationship; Wherein, if the difference satisfies the error condition, the iterative correlation after the last iteration of training is used as the preset correlation.

3. The method according to claim 2, characterized in that, The step of obtaining sample classification results for the target using the sample target features includes: Using the target features of the sample, the first benign / malignant classification result and the subtype classification result of the target are obtained respectively; The step of adjusting the network parameters of the typing network based on the difference between the sample typing results and the true typing results includes: Obtain the first difference between the first benign / malignant classification result and the true benign / malignant classification result, and obtain the second difference between the subtype classification result and the true subtype classification result; The network parameters of the fractal network are adjusted by combining the first difference and the second difference.

4. The method according to claim 3, characterized in that, The method further includes: Based on the subtype classification results, a second benign / malignant classification result for the target is obtained; Obtain the third difference between the second benign / malignant classification result and the true benign / malignant classification result; The step of adjusting the network parameters of the fractal network by combining the first difference and the second difference includes: The network parameters of the fractal network are adjusted by combining the first difference, the second difference, and the third difference.

5. The method according to any one of claims 1-4, characterized in that, Before performing feature extraction on several initial images containing the target to be classified, and obtaining initial features of each initial image with respect to the target, the method further includes: The initial images are preprocessed, and the preprocessing includes one or more of the following: adjusting contrast, adjusting window width and window level, adjusting image size, and normalization.

6. A target classification device, characterized in that, include: The feature extraction module is used to extract features from several initial images containing the target to be classified, and to obtain the initial features of each initial image with respect to the target. The target is a lesion. The several initial images include local images of the target acquired using different acquisition methods and global images of the organs to which the target belongs. The feature enhancement module is used to fuse the initial features based on the preset association relationship between the initial images to obtain the target features, wherein the preset association relationship is an adjacency matrix; The classification module is used to obtain the classification result of the target by utilizing the target features; The feature extraction module performs feature extraction on several initial images containing the target to be classified, and obtains the initial features of each initial image with respect to the target in the following ways: for each acquisition method, the local image and the global image under the acquisition method are extracted to obtain the first feature of the local image and the second feature of the global image, and the first feature and the second feature are combined to obtain the initial features corresponding to the acquisition method; The target classification method is executed by a classification network, which includes a first multilayer perceptron and a second multilayer perceptron. The feature enhancement module fuses the initial features based on a preset correlation between the initial images to obtain the target features. The fusion method includes: processing the initial features using the first multilayer perceptron to obtain the advanced features corresponding to each initial feature; concatenating the advanced features to obtain the fused features; multiplying the fused features with the adjacency matrix; and processing the fused features using the second multilayer perceptron to obtain the target features.

7. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images