Feature processing method, device and electronic equipment based on medical images

Abstract

The present application discloses a feature processing method, device and electronic device based on medical images, which relates to the field of medical technology. The method includes: obtaining the probability that each pixel in the medical image belongs to the lesion area; determining L connected domains of the medical image according to the probability that each pixel belongs to the lesion area, wherein L is an integer greater than or equal to 1, the probability that a lesion exists in each connected domain is greater than a preset probability, and each connected domain includes at least one block image; performing feature fusion on the image features of all block images in each connected domain to obtain the connected domain features of each connected domain; performing feature fusion on the connected domain features of the L connected domains to obtain the pathological image features of the medical image. The present application solves the technical problem in the prior art that when constructing a prediction model, a partial area of ​​the medical image is randomly selected for analysis, resulting in inaccurate feature extraction and a small amount of effective information.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and more specifically, to a method, device and electronic equipment for feature processing based on medical images. Background Art

[0002] Pathology slides are considered the gold standard for cancer diagnosis and prognosis. Pathology laboratories scan and digitize pathology slides, and then histopathologists make diagnoses of biopsy specimens based on them. However, considering the large size of pathology slides and the increasing number of cases in which pathology techniques are used for cancer detection, manual analysis of pathology slides is a time-consuming task, and manual analysis can only make diagnoses based on pathology slides and cannot predict prognostic information. Therefore, it is of great clinical significance to automatically process digital pathology slides, segment lesion areas, and predict therapeutic effects. This can not only help histopathologists make accurate and efficient preliminary diagnoses, but also provide important basis for the selection and decision-making of treatment options.

[0003] However, in the prior art, when constructing a prediction model, a method of randomly selecting a partial area of ​​a medical image for analysis is usually used to train the prediction model. For example, at least one block image in the medical image is randomly selected for feature processing, and then the model is trained based on the image features combined with deep learning or machine learning. However, the partial image area selected in this random manner may not be an ideal image area suitable for modeling, which leads to technical problems such as inaccurate subsequent image feature extraction and low amount of effective information.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a feature processing method, device and electronic device based on medical images, so as to at least solve the technical problem in the prior art that, when building a prediction model, partial areas of the medical image are randomly selected for analysis, resulting in inaccurate feature extraction and a small amount of effective information.

[0006] According to one aspect of the present application, a feature processing method based on medical images is provided, including: obtaining the probability that each pixel in the medical image belongs to a lesion area; determining L connected domains of the medical image according to the probability that each pixel belongs to the lesion area, wherein L is an integer greater than or equal to 1, the probability that a lesion exists in each connected domain is greater than a preset probability, and each connected domain includes at least one block image; performing feature fusion on image features of all block images in each connected domain to obtain connected domain features of each connected domain; performing feature fusion on the connected domain features of the L connected domains to obtain pathological image features of the medical image.

[0007] Optionally, in the feature processing method based on medical images proposed in the present application, L connected domains of the medical image are determined according to the probability that each pixel belongs to the lesion area, including: determining that each pixel is a first pixel or a second pixel according to the probability that each pixel belongs to the lesion area, wherein the probability that the first pixel belongs to the lesion area is greater than a preset threshold, and the probability that the second pixel belongs to the lesion area is less than or equal to the preset threshold; taking the area where the first pixel is located as the target area; and performing aggregation operations on adjacent target areas to obtain L connected domains.

[0008] Optionally, in the feature processing method based on medical images proposed in the present application, before performing feature fusion on the image features of all block images in each connected domain to obtain the connected domain features of each connected domain, the block images in each connected domain are converted into target block images of target resolution, wherein the target resolution is higher than the resolution of the block images; the probabilities of all pixels in the target block images belonging to the lesion area are summed to obtain a first value corresponding to the target block image; the ratio of the first value corresponding to the target block image to the number of pixels in the target block image is calculated to obtain a lesion probability mean of the target block image; the target block image with the largest lesion probability mean in each connected domain is used as the main block image of the connected domain; and the target block images other than the main block image in each connected domain are used as the secondary block images of the connected domain.

[0009] Optionally, in the process of performing feature fusion on the image features of all the block images in each connected domain to obtain the connected domain features of each connected domain, the feature processing method based on medical images further includes: extracting the feature vector of each target block image in the I-th connected domain, wherein the feature vector of each target block image represents the image feature of the block image corresponding to the target block image; performing a first-class linear transformation on the feature vector of each target block image to obtain the information vector of the target block image; determining the weight of each secondary block image and the weight of the primary block image according to the similarity between each secondary block image and the primary block image; performing weighted sum calculation on the information vector of each secondary block image and the information vector of the primary block image according to the weight of each secondary block image and the weight of the primary block image, and using the calculation result as the connected domain features of the I-th connected domain.

[0010] Optionally, in the feature processing method based on medical images proposed in the present application, before determining the weight of each secondary block image and the weight of the primary block image according to the similarity between each secondary block image and the primary block image, a second type linear transformation is performed on the feature vector of each secondary block image to obtain a query vector of the secondary block image; a second type linear transformation is performed on the feature vector of the primary block image to obtain a query vector of the primary block image; and a vector similarity between the query vector of each secondary block image and the query vector of the primary block image is calculated, and the vector similarity is used as the similarity between each secondary block image and the primary block image.

[0011] Optionally, in the feature processing method based on medical images proposed in the present application, before performing feature fusion on the connected domain features of L connected domains to obtain the pathological image features of the medical image, the probabilities of all pixels in each connected domain belonging to the lesion area are summed to obtain the second value of each connected domain; the ratio of the second value of each connected domain to the number of pixels in the connected domain is calculated to obtain the mean lesion probability of the connected domain; the connected domain with the largest mean lesion probability among the L connected domains is taken as the main connected domain; and each connected domain among the L connected domains except the main connected domain is taken as a secondary connected domain.

[0012] Optionally, in the feature processing method based on medical images proposed in the present application, feature fusion is performed on the connected domain features of L connected domains to obtain pathological image features of the medical image, including: performing a first-class linear transformation on the connected domain features of each connected domain to obtain an information vector of each connected domain; determining the weight of each secondary connected domain and the weight of the main connected domain based on the similarity between each secondary connected domain and the main connected domain; and performing weighted sum calculation on the information vector of each secondary connected domain and the information vector of the main connected domain based on the weight of each secondary connected domain and the weight of the main connected domain to obtain the pathological image feature.

[0013] Optionally, in the feature processing method based on medical images proposed in the present application, before determining the weight of each secondary connected domain and the weight of the main connected domain according to the similarity between each secondary connected domain and the main connected domain, a second kind of linear transformation is performed on the connected domain features of each secondary connected domain to obtain a query vector of each secondary connected domain; a second kind of linear transformation is performed on the connected domain features of the main connected domain to obtain a query vector of the main connected domain; and the vector similarity between the query vector of each secondary connected domain and the query vector of the main connected domain is calculated, and the vector similarity is used as the similarity between each secondary connected domain and the main connected domain.

[0014] Optionally, in the feature processing method based on medical images proposed in the present application, after the connected domain features of L connected domains are fused to obtain the pathological image features of the medical image, the pathological image features are converted into a target feature vector through a target model, and the prediction result is determined based on the target feature vector, wherein the target model is a neural network model pre-trained using a reference image as a training sample.

[0015] According to another aspect of the present application, a feature processing device based on medical images is also provided, wherein the feature processing device based on medical images includes: an acquisition unit, used to acquire the probability that each pixel in the medical image belongs to the lesion area; a determination unit, used to determine L connected domains of the medical image according to the probability that each pixel belongs to the lesion area, wherein L is an integer greater than or equal to 1, the probability that a lesion exists in each connected domain is greater than a preset probability, and each connected domain includes at least one block image; a first feature processing unit, used to perform feature fusion on image features of all block images in each connected domain to obtain a connected domain feature of each connected domain; and a second feature processing unit, used to perform feature fusion on the connected domain features of the L connected domains to obtain a pathological image feature of the medical image.

[0016] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute any one of the above-mentioned medical image-based feature processing methods.

[0017] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes any one of the above-mentioned medical image-based feature processing methods.

[0018] In the present application, the probability that each pixel in the medical image belongs to the lesion area is first obtained, and then L connected domains of the medical image are determined according to the probability that each pixel belongs to the lesion area, where L is an integer greater than or equal to 1, the probability that a lesion exists in each connected domain is greater than a preset probability, and each connected domain includes at least one block image. Subsequently, the image features of all the block images in each connected domain are feature fused to obtain the connected domain features of each connected domain, and finally, the connected domain features of the L connected domains are feature fused to obtain the pathological image features of the medical image.

[0019] As can be seen from the above content, the present application first obtains the probability that each pixel may belong to the lesion area, and then determines at least one connected domain of the medical image based on the probability that the pixel belongs to the lesion area, wherein each connected domain includes at least one block image, and subsequently the present application mainly performs feature processing on the block images in the connected domain. Compared with the prior art that uses a random method to determine the block images that need to be processed, the present application is more targeted when selecting the block images that need to be processed, and mainly selects based on the probability that the pixel belongs to the lesion area as a reference, so that the selected block images can include more information related to the lesion area, that is, the block images in the connected domain determined according to the present application include more valuable lesion information, so the image features extracted from these block images are also more accurate, and there will not be too many redundant features, which solves the technical problem of inaccurate feature extraction and small amount of effective information caused by all random or partially random selection of block images in medical images for feature processing in the prior art.

[0020] In addition, by determining at least one connected domain, the present application can avoid processing a large number of invalid image areas, thereby reducing the size of the input image in the subsequent modeling process and reducing the demand for computing resources during modeling. Secondly, the present application also defines two feature fusion processing processes, one is the feature fusion of the block image within the connected domain, and the other is the feature fusion between connected domains, so that not only can the features between different lesion areas be cleverly fused, but also the features of the entire pathological section can be fused to improve the model accuracy when the pathological image features are used for subsequent modeling.

[0021] It can be seen that through the technical solution of the present application, block images and connected domains that are more strongly correlated with the lesion area are selected for feature processing in a targeted manner, thereby solving the technical problems of inaccurate feature extraction and small amount of effective information caused by randomly selecting partial areas of medical images for analysis when building a prediction model in the prior art, thereby achieving the technical effect of improving the prediction accuracy of the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a flowchart of an optional feature processing method based on medical images according to an embodiment of the present application;

[0024] Figure 2 is an optional flow chart for determining a connected domain of a medical image according to an embodiment of the present application;

[0025] Figure 3 is an optional flow chart for determining the weight of a target block image according to an embodiment of the present application;

[0026] Figure 4 is a flowchart of an optional feature processing and model training according to an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of an optional feature processing device based on medical images according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0031] According to an embodiment of the present application, an embodiment of a feature processing method based on medical images is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] It should also be noted that a feature processing system can be used as the executor of the feature processing method based on medical images in the embodiment of the present application, wherein the feature processing system can be a software system or an immersive system combining software and hardware. In addition, those skilled in the art can understand that in addition to the feature processing system, the executor of the feature processing method based on medical images in the embodiment of the present application can also be other devices or equipment.

[0033] Figure 1 is a flowchart of an optional feature processing method based on medical images according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0034] Step S101, obtaining the probability that each pixel in the medical image belongs to the lesion area.

[0035] Optionally, the feature processing system may calculate the probability that each pixel in the medical image belongs to the lesion area by mathematical statistics, or may use a pre-trained neural network model to determine the probability that each pixel in the medical image belongs to the lesion area.

[0036] Optionally, the medical images include but are not limited to various types of images in the medical field such as MR, CT, and PET.

[0037] Step S102, determining L connected domains of the medical image according to the probability that each pixel belongs to the lesion area.

[0038] In step S102, L is an integer greater than or equal to 1, the probability that a lesion exists in each connected domain is greater than a preset probability, and each connected domain includes at least one block image.

[0039] Optionally, when determining the probability that each pixel in the medical image belongs to the lesion area, the feature processing system can also set a classification threshold, such as 0.5. If the probability that a certain pixel belongs to the lesion area is greater than the set classification threshold, the pixel is determined to belong to the lesion area. If the probability that a certain pixel belongs to the lesion area is less than or equal to the set classification threshold, the pixel is determined not to belong to the lesion area. In this way, pixels with a higher probability of belonging to the lesion area are actually directly classified in the lesion area, and pixels with a lower probability of belonging to the lesion area are directly classified in the non-lesion area, thereby facilitating the subsequent connected domain division.

[0040] Optionally, after pixels with a higher probability of belonging to the lesion area are directly classified into the lesion area and pixels with a lower probability of belonging to the lesion area are directly classified into the non-lesion area in the above manner, the feature processing system can aggregate adjacent lesion areas, and perform appropriate corrosion and expansion and other processing, and then fill unnecessary holes in the aggregated lesion areas, and finally obtain a mask (outline) of the lesion area based on the entire medical image. At this time, the mask based on the entire medical image should have multiple connected domains (i.e., various connected areas where lesions may exist).

[0041] It should be noted that since the pixels within the connected domain are more likely to belong to the lesion area, the above-mentioned connected domain is the area with a higher probability of the existence of lesions in the medical image. It can be seen that through the above-mentioned process of the present application, the purpose of determining the connected area in the medical image that is strongly correlated with the lesion area is achieved.

[0042] Step S103: performing feature fusion on the image features of all block images in each connected domain to obtain the connected domain features of each connected domain.

[0043] Optionally, the feature processing system may adopt an attention mechanism to perform feature fusion (eg, weighted sum calculation) on the image features of all block images in each connected domain, thereby obtaining the connected domain features of each connected domain.

[0044] Step S104, performing feature fusion on the connected domain features of the L connected domains to obtain pathological image features of the medical image.

[0045] Optionally, after obtaining the connected domain features of each connected domain, the feature processing system may use an attention mechanism to perform feature fusion on the connected domain features of the L connected domains, thereby using the fused features as pathological image features of the medical image.

[0046] Optionally, after obtaining the pathological image features, the medical diagnosis information and efficacy information of the patient can be predicted based on the pathological image features, a neural network model can be trained based on the pathological image features, and the pathological image features can be used to reconstruct and amplify features, etc. The present application does not specifically limit the specific use of the pathological image features.

[0047] Based on the contents of the above steps S101 to S104, it can be known that the present application first obtains the probability that each pixel may belong to the lesion area, and then determines at least one connected domain of the medical image according to the probability that the pixel belongs to the lesion area, wherein each connected domain includes at least one block image, and subsequently the present application mainly performs feature processing on the block images in the connected domain. Compared with the prior art that uses a random method to determine the block images that need to be processed, the present application is more targeted when selecting the block images that need to be processed, and mainly selects them based on the probability that the pixel belongs to the lesion area as a reference, so that the selected block images can include more information related to the lesion area, that is, the block images in the connected domain determined according to the present application include more valuable lesion information, so the image features extracted from these block images are also more accurate, and there will not be too many redundant features, which solves the technical problem of inaccurate feature extraction and small amount of effective information caused by completely or partially randomly selecting block images in medical images for feature processing in the prior art.

[0048] In addition, by determining at least one connected domain, the present application can avoid processing a large number of invalid image areas, thereby reducing the size of the input image in the subsequent modeling process and reducing the demand for computing resources during modeling. Secondly, the present application also defines two feature fusion processing processes, one is the feature fusion of the block image within the connected domain, and the other is the feature fusion between connected domains, so that not only can the features between different lesion areas be cleverly fused, but also the features of the entire pathological section can be fused to improve the model accuracy when the pathological image features are used for subsequent modeling.

[0049] It can be seen that through the technical solution of the present application, block images and connected domains that are more strongly correlated with the lesion area are selected for feature processing in a targeted manner, thereby solving the technical problems of inaccurate feature extraction and small amount of effective information caused by randomly selecting partial areas of medical images for analysis when building a prediction model in the prior art, thereby achieving the technical effect of improving the prediction accuracy of the prediction model.

[0050] In an optional embodiment, Figure 2 is an optional flow chart for determining a connected domain of a medical image according to an embodiment of the present application, such as Figure 2 As shown, the following steps are included:

[0051] Step S201 , determining whether each pixel is a first pixel or a second pixel according to the probability that each pixel belongs to a lesion area.

[0052] In step S201, the probability that the first pixel belongs to the lesion area is greater than a preset threshold, and the probability that the second pixel belongs to the lesion area is less than or equal to the preset threshold.

[0053] Optionally, the preset threshold value may also be referred to as a classification threshold value, wherein the preset threshold value may be custom set, for example, 0.5, 0.6, 0.75, etc. The feature processing system classifies each pixel in the medical image by comparing the probability of each pixel belonging to the lesion area with the preset threshold value. If the probability of a certain pixel belonging to the lesion area is greater than the preset threshold value, the feature processing system classifies the pixel as a first pixel; if the probability of a certain pixel belonging to the lesion area is less than or equal to the preset threshold value, the feature processing system classifies the pixel as a second pixel.

[0054] Step S202: taking the area where the first pixel is located as the target area.

[0055] Optionally, the feature processing system may use the region where each first pixel in the medical image is located as the target region. Since there may be multiple first pixels, there may also be multiple target regions, and any two different target regions may be discrete or connected. The target region may be understood as the lesion region determined by the feature processing system.

[0056] Step S203: performing aggregation operation on adjacent target regions to obtain L connected regions.

[0057] Optionally, the feature processing system can aggregate adjacent target areas and perform appropriate corrosion and expansion processing. In addition, the feature processing system can also fill unnecessary holes in the aggregated target areas, and finally obtain multiple connected domains based on the entire pathological image (i.e., various connected areas where lesions may exist).

[0058] In an optional embodiment, before performing feature fusion on the image features of all block images in each connected domain to obtain the connected domain features of each connected domain, the feature processing system may convert the block images in each connected domain into target block images of target resolution, wherein the target resolution is higher than the resolution of the block images. Then, the feature processing system sums the probabilities of all pixels in the target block image belonging to the lesion area to obtain a first value corresponding to the target block image. Subsequently, the feature processing system calculates the ratio of the first value corresponding to the target block image to the number of pixels in the target block image to obtain the mean lesion probability of the target block image. Finally, the feature processing system uses the target block image with the largest mean lesion probability in each connected domain as the main block image of the connected domain; and uses the block images other than the main block image in each connected domain as the secondary block images of the connected domain.

[0059] Optionally, each connected domain includes at least one block image. In order to improve the quality of image features, before starting to extract image features, the resolution of the block images in the connected domain may be first enhanced, that is, the block images in the connected domain are converted into target block images of target resolution, wherein the target resolution is higher than the original resolution of the block images.

[0060] Optionally, the feature processing system may further calculate a mean value of the lesion probability of each target block image, wherein the calculation formula for the mean value of the lesion probability of each target block image is as shown in the following formula (1).

[0061]

[0062] Alternatively, in formula (1), represents the mean probability of the lesion in the target block image, represents the probability that the i-th pixel in a target block image belongs to the lesion area, and G represents the total number of pixels in the target block image. It can be seen that in Formula 1 That is, it is the first value obtained after summing up the probabilities that all pixels in the target block image belong to the lesion area.

[0063] Optionally, after obtaining the lesion probability mean of each target block image in each connected domain, the feature processing system may use the target block image with the largest lesion probability mean in each connected domain as the main block image of the connected domain, and use the target block images other than the main block image in each connected domain as the secondary block images of the connected domain.

[0064] Optionally, the feature processing system may not convert the block images in the connected domain into target block images of the target resolution, but directly calculate the mean value of the lesion probability of each block image in each connected domain, and use the block image with the largest mean value of the lesion probability in the connected domain as the main block image of the connected domain; and use the block images in the connected domain other than the main block image as the secondary block images of the connected domain.

[0065] In an optional embodiment, the feature processing system can extract the feature vector of each target block image in the I-th connected domain, wherein the feature vector of each target block image represents the image feature of the block image corresponding to the target block image. The feature processing system can also perform a first-class linear transformation on the feature vector of each target block image to obtain the information vector of the target block image; and determine the weight of each secondary block image and the weight of the primary block image according to the similarity between each secondary block image and the primary block image. Finally, the feature processing system performs a weighted sum calculation on the information vector of each secondary block image and the information vector of the primary block image according to the weight of each secondary block image and the weight of the primary block image, and uses the calculation result as the connected domain feature of the I-th connected domain.

[0066] Optionally, assuming that the I-th connected domain includes at least one target block image, Represents any target block image in the I-th connected domain. The feature processing system can process the target block image Perform feature extraction to obtain the target block image The eigenvector of Among them, the feature processing system can use a pre-trained feature extractor to Extract feature vectors.

[0067] Optionally, the feature processing system can also process the feature vector Perform the first type of linear transformation to obtain the information vector of the target block image. For example, the feature processing system can perform the first type of linear transformation The feature vector Convert to query vector . Among them, the query vector The calculation formula is as follows (2).

[0068]

[0069] Optionally, the feature processing system may also calculate the similarity between each secondary block image in each connected domain and the primary block image in the connected domain, and determine the weight of each secondary block image and the weight of the primary block image based on the calculated similarity, wherein the weight is also referred to as the attention weight. For example, assuming that the weight of the primary block image is 1, and the similarity between the secondary block image b1 and the primary block image is 80%, the weight of the secondary block image b1 may be set to 0.8; and the similarity between the secondary block image b2 and the primary block image is 60%, the weight of the secondary block image b2 may be set to 0.6.

[0070] also, Figure 3 is an optional flow chart for determining the weight of a target block image according to an embodiment of the present application, such as Figure 3 As shown, the following steps are included:

[0071] Step S301 : performing a second type of linear transformation on the feature vector of each secondary block image to obtain a query vector of the secondary block image.

[0072] Step S302: Perform a second type of linear transformation on the feature vector of the main block image to obtain a query vector of the main block image.

[0073] Optionally, the feature processing system may perform a second type linear transformation on the feature vector of the target block image (secondary block image or primary block image) by the following formula (3).

[0074]

[0075] In formula (3), Characterizes the second kind of linear transformation, Represents the target block image The eigenvector of , Represents the target block image The query vector of .

[0076] Step S303: Calculate the vector similarity between the query vector of each secondary block image and the query vector of the primary block image, and use the vector similarity as the similarity between each secondary block image and the primary block image.

[0077] Alternatively, assuming is a query vector of a secondary patch image, is the query vector of the main patch image, then the feature processing system can calculate and The vector similarity (such as cosine similarity) between the secondary block image and the primary block image is taken as the similarity (which can be recorded as ). It should be noted that A query vector that can refer to any secondary patch image.

[0078] In an optional embodiment, the feature processing system may also calculate the similarity between each secondary block image and the primary block image by using the following formula (4).

[0079]

[0080] In formula (4), Indicates secondary block image With the main block image The attention weight, Indicates secondary block image The query vector With the main block image The query vector The vector similarity between them, wherein the calculation method of vector similarity includes but is not limited to the dot product calculation method of two feature vectors. The above formula (4) It means that the similarity between all N-1 secondary block images in the connected domain and the primary block image is calculated and summed up (assuming that the connected domain includes N target block images).

[0081] In an optional embodiment, before performing feature fusion on the connected domain features of the L connected domains to obtain the pathological image features of the medical image, the feature processing system may sum the probabilities of all pixels in each connected domain belonging to the lesion area to obtain the second value of each connected domain, and then calculate the ratio of the second value of each connected domain to the number of pixels in the connected domain to obtain the mean lesion probability of the connected domain. Finally, the feature processing system takes the connected domain with the largest mean lesion probability among the L connected domains as the main connected domain; and takes each connected domain among the L connected domains except the main connected domain as the secondary connected domain.

[0082] Optionally, the feature processing system may calculate the mean value of the lesion probability of each connected domain, wherein the calculation formula for the mean value of the lesion probability of each connected domain is as shown in the following formula (5).

[0083]

[0084] Optionally, in formula (5), S represents the mean probability of lesions in a connected domain, It represents the probability that the i-th pixel of the connected domain belongs to the lesion area, and Q represents the total number of pixels in the connected domain.

[0085] It can be seen that the That is, the second value obtained after summing up the probabilities that all pixels in a connected domain belong to the lesion area.

[0086] Optionally, after obtaining the mean lesion probability of each connected domain, the feature processing system may take the connected domain with the largest mean lesion probability in the medical image as the main connected domain, and take the connected domains in the medical image other than the main connected domain as the secondary connected domain.

[0087] In an optional embodiment, in the process of performing feature fusion on the connected domain features of L connected domains to obtain the pathological image features of the medical image, the feature processing system can perform a first-class linear transformation on the connected domain features of each connected domain to obtain the information vector of each connected domain, and then determine the weight of each secondary connected domain and the weight of the main connected domain according to the similarity between each secondary connected domain and the main connected domain. Finally, the feature processing system performs a weighted sum calculation on the information vector of each secondary connected domain and the information vector of the main connected domain according to the weight of each secondary connected domain and the weight of the main connected domain to obtain the pathological image features.

[0088] Optionally, assume that the connected domain feature of any connected domain among the L connected domains is represented as a connected domain feature vector Indicates that the feature processing system can process the connected domain feature vector Perform the first kind of linear transformation to obtain the information vector of the connected domain. For example, the feature processing system can be transformed by the first kind of linear transformation Connected component feature vector Convert to query vector . Among them, the query vector The calculation formula is as follows (6).

[0089]

[0090] Optionally, the feature processing system may also calculate the similarity between each secondary connected domain and the primary connected domain, and then determine the weight of each secondary connected domain and the weight of the primary connected domain based on the calculated similarity, wherein the weight is also referred to as the attention weight. For example, assuming that the weight of the primary connected domain is 1, and the similarity between the secondary connected domain A1 and the primary connected domain is 80%, the weight of the secondary connected domain A1 may be set to 0.8; the similarity between the secondary connected domain A2 and the primary connected domain is 60%, and the weight of the secondary connected domain A2 may be set to 0.6.

[0091] In an optional embodiment, before determining the weight of each secondary connected domain and the weight of the primary connected domain according to the similarity between each secondary connected domain and the primary connected domain, the feature processing system may also perform a second type of linear transformation on the connected domain features of each secondary connected domain to obtain a query vector of each secondary connected domain, and simultaneously perform a second type of linear transformation on the connected domain features of the primary connected domain to obtain a query vector of the primary connected domain. Finally, the feature processing system calculates the vector similarity between the query vector of each secondary connected domain and the query vector of the primary connected domain, and uses the vector similarity as the similarity between each secondary connected domain and the primary connected domain.

[0092] Optionally, the feature processing system may perform a second type linear transformation on the connected domain feature vector of any connected domain (primary connected domain or secondary connected domain) by using the following formula (7).

[0093]

[0094] In formula (7), Characterizes the second kind of linear transformation, represents the feature vector of any connected domain, Represents the query vector of the connected component.

[0095] Alternatively, assuming is a query vector of a secondary connected domain, is the query vector of the main connected domain, then the feature processing system can calculate and The vector similarity (such as cosine similarity) between the secondary connected domain and the main connected domain is taken as the similarity (which can be recorded as ). It should be noted that It can represent the query vector of any secondary connected domain.

[0096] In an optional embodiment, the feature processing system may also calculate the similarity between each secondary connected domain and the primary connected domain by using the following formula (8).

[0097]

[0098] In formula (8), Represents secondary connected domain With the main block image The attention weight, Represents secondary connected domain The query vector With the main block image The query vector The vector similarity between them, wherein the calculation method of vector similarity includes but is not limited to the dot product calculation method of two feature vectors. The above formula (8) It means that the similarity between all L-1 secondary connected domains in the L connected domains and the main connected domain is calculated and summed.

[0099] In an optional embodiment, after performing feature fusion on the connected domain features of L connected domains to obtain the pathological image features of the medical image, the feature processing system can also convert the pathological image features into a target feature vector through a target model, and determine the prediction result based on the target feature vector, wherein the target model is a neural network model pre-trained using a reference image as a training sample.

[0100] Optionally, the prediction result includes, but is not limited to, medical diagnosis information and efficacy information of the target object, for example, a negative or positive diagnosis result of the target object, prognosis information of the target object after receiving radiotherapy, survival estimation information of the target object (such as survival time or survival rate), radiotherapy dose information or radiotherapy plan information of the target object. The target object may be a patient corresponding to the medical image.

[0101] In addition, the present application does not specifically limit the structure of the target model. For example, the target model can be a model of an attention mechanism, a deep learning model, or a machine learning model.

[0102] In an optional embodiment, Figure 4 is a flowchart of an optional feature processing and model training according to an embodiment of the present application, such as Figure 4As shown, the following steps are included:

[0103] First, a medical image is obtained, wherein the medical image includes L connected domains, the probability of the existence of a lesion in each connected domain is greater than a preset probability, and each connected domain includes at least one block image. Secondly, a feature extractor is used to extract a feature vector of each block image, and then the feature vector of each block image is subjected to a first type linear transformation and a second type linear transformation respectively, so as to obtain a query vector and an information vector of the block image.

[0104] Subsequently, the query vector of the main block image and the query vector of each secondary block image in each connected domain are used to calculate the similarity between the main block image and each secondary block image in each connected domain, and the attention weight of each block image in each connected domain is determined according to the similarity. Finally, the connected domain features are calculated according to the attention weights and information vectors of each block image in each connected domain to obtain the connected domain feature vector of each connected domain.

[0105] Then, the connected domain feature vector of each connected domain is subjected to a first type of linear transformation and a second type of linear transformation, respectively, so as to obtain a query vector and an information vector of each connected domain. Further, the feature processing system can calculate the attention weight of each connected domain according to the query vector of each connected domain (including the primary connected domain and the secondary connected domain), and then calculate the pathological image features of the medical image based on the attention weight and the information vector of each connected domain.

[0106] Finally, the feature processing system can input the calculated pathological images into the target model, and then the target model outputs the prediction results or prediction endpoints. The prediction endpoints can be diagnostic results (negative, positive), prognostic information (disease remission, disease non-remission), survival analysis information (whether survival for different lengths of time), etc.

[0107] It should be noted that the training process of the target model also involves the linear transformation of the feature vector of the block image and the linear transformation of the feature vector of the connected domain. If computing resources permit, the feature processor can also be updated and trained during the training process of the target model.

[0108] As can be seen from the above content, the present application first obtains the probability that each pixel may belong to the lesion area, and then determines at least one connected domain of the medical image based on the probability that the pixel belongs to the lesion area, wherein each connected domain includes at least one block image, and subsequently the present application mainly performs feature processing on the block images in the connected domain. Compared with the prior art that uses a random method to determine the block images that need to be processed, the present application is more targeted when selecting the block images that need to be processed, and mainly selects based on the probability that the pixel belongs to the lesion area as a reference, so that the selected block images can include more information related to the lesion area, that is, the block images in the connected domain determined according to the present application include more valuable lesion information, so the image features extracted from these block images are also more accurate, and there will not be too many redundant features, which solves the technical problem of inaccurate feature extraction and small amount of effective information caused by all random or partially random selection of block images in medical images for feature processing in the prior art.

[0109] In addition, by determining at least one connected domain, the present application can avoid processing a large number of invalid image areas, thereby reducing the size of the input image in the subsequent modeling process and reducing the demand for computing resources during modeling. Secondly, the present application also defines two feature fusion processing processes, one is the feature fusion of the block image within the connected domain, and the other is the feature fusion between connected domains, so that not only can the features between different lesion areas be cleverly fused, but also the features of the entire pathological section can be fused to improve the model accuracy when the pathological image features are used for subsequent modeling.

[0110] It can be seen that through the technical solution of the present application, block images and connected domains that are more strongly correlated with the lesion area are selected for feature processing in a targeted manner, thereby solving the technical problems of inaccurate feature extraction and small amount of effective information caused by randomly selecting partial areas of medical images for analysis when building a prediction model in the prior art, thereby achieving the technical effect of improving the prediction accuracy of the prediction model.

[0111] According to another aspect of the embodiment of the present application, a feature processing device based on medical images is also provided, wherein: Figure 5 is a schematic diagram of an optional feature processing device based on medical images according to an embodiment of the present application, such as Figure 5 As shown, it includes the following units:

[0112] The acquisition unit 501 is used to obtain the probability that each pixel in the medical image belongs to the lesion area; the determination unit 502 is used to determine L connected domains of the medical image according to the probability that each pixel belongs to the lesion area, wherein L is an integer greater than or equal to 1, the probability that a lesion exists in each connected domain is greater than a preset probability, and each connected domain includes at least one block image; the first feature processing unit 503 is used to perform feature fusion on the image features of all the block images in each connected domain to obtain the connected domain features of each connected domain; the second feature processing unit 504 is used to perform feature fusion on the connected domain features of the L connected domains to obtain the pathological image features of the medical image.

[0113] Optionally, the determination unit 502 includes: a first determination subunit, a first processing subunit, and a second processing subunit. The first determination subunit is used to determine whether each pixel is a first pixel or a second pixel according to the probability that each pixel belongs to the lesion area, wherein the probability that the first pixel belongs to the lesion area is greater than a preset threshold, and the probability that the second pixel belongs to the lesion area is less than or equal to the preset threshold; the first processing subunit is used to take the area where the first pixel is located as the target area; and the second processing subunit is used to perform aggregation operations on adjacent target areas to obtain L connected domains.

[0114] Optionally, the feature processing device based on medical images further includes: a conversion unit, a first calculation unit, a second calculation unit, a first processing unit and a second processing unit. The conversion unit is used to convert the block image in each connected domain into a target block image of a target resolution, wherein the target resolution is higher than the resolution of the block image; the first calculation unit is used to sum the probabilities of all pixels in the target block image belonging to the lesion area to obtain a first value corresponding to the target block image; the second calculation unit is used to calculate the ratio of the first value corresponding to the target block image to the number of pixels in the target block image to obtain a mean lesion probability of the target block image; the first processing unit is used to use the target block image with the largest mean lesion probability in each connected domain as the main block image of the connected domain; the second processing unit is used to use the target block images other than the main block image in each connected domain as the secondary block images of the connected domain.

[0115] Optionally, the feature processing device based on medical images further includes: a feature extraction unit, a third processing unit, a first determination unit, and a third calculation unit. The feature extraction unit is used to extract a feature vector of each target block image in the first connected domain, wherein the feature vector of each target block image represents the image feature of the block image corresponding to the target block image; the third processing unit is used to perform a first type of linear transformation on the feature vector of each target block image to obtain an information vector of the target block image; the first determination unit is used to determine the weight of each secondary block image and the weight of the primary block image according to the similarity between each secondary block image and the primary block image; the third calculation unit is used to perform a weighted sum calculation on the information vector of each secondary block image and the information vector of the primary block image according to the weight of each secondary block image and the weight of the primary block image, and use the calculation result as the connected domain feature of the first connected domain.

[0116] Optionally, the feature processing device based on medical images further includes: a fourth processing unit, a fifth processing unit and a sixth processing unit, wherein the fourth processing unit is used to perform a second type linear transformation on the feature vector of each secondary block image to obtain a query vector of the secondary block image; the fifth processing unit is used to perform a second type linear transformation on the feature vector of the primary block image to obtain a query vector of the primary block image; and the sixth processing unit is used to calculate the vector similarity between the query vector of each secondary block image and the query vector of the primary block image, and use the vector similarity as the similarity between each secondary block image and the primary block image.

[0117] Optionally, the feature processing device based on medical images further includes: a fourth calculation unit, a fifth calculation unit, a seventh processing unit and an eighth processing unit. The fourth calculation unit is used to sum the probabilities of all pixels in each connected domain belonging to the lesion area to obtain the second value of each connected domain; the fifth calculation unit is used to calculate the ratio of the second value of each connected domain to the number of pixels in the connected domain to obtain the mean lesion probability of the connected domain; the seventh processing unit is used to take the connected domain with the largest mean lesion probability among the L connected domains as the main connected domain; the eighth processing unit is used to take each connected domain among the L connected domains except the main connected domain as a secondary connected domain.

[0118] Optionally, the second feature processing unit includes: a third processing subunit, a fourth processing subunit and a fifth processing subunit. The third processing subunit is used to perform a first-class linear transformation on the connected domain feature of each connected domain to obtain the information vector of each connected domain; the fourth processing subunit is used to determine the weight of each secondary connected domain and the weight of the main connected domain according to the similarity between each secondary connected domain and the main connected domain; the fifth processing subunit is used to perform a weighted sum calculation on the information vector of each secondary connected domain and the information vector of the main connected domain according to the weight of each secondary connected domain and the weight of the main connected domain to obtain the pathological image feature.

[0119] Optionally, the feature processing device based on medical images further includes: a ninth processing unit, a tenth processing unit, and a similarity determination unit. The ninth processing unit is used to perform a second-class linear transformation on the connected domain features of each secondary connected domain to obtain a query vector of each secondary connected domain; the tenth processing unit is used to perform a second-class linear transformation on the connected domain features of the primary connected domain to obtain a query vector of the primary connected domain; the similarity determination unit is used to calculate the vector similarity between the query vector of each secondary connected domain and the query vector of the primary connected domain, and use the vector similarity as the similarity between each secondary connected domain and the primary connected domain.

[0120] Optionally, the feature processing device based on medical images also includes: a prediction result determination unit, which is used to convert the pathological image features into a target feature vector through a target model, and determine the prediction result based on the target feature vector, wherein the target model is a neural network model pre-trained using a reference image as a training sample.

[0121] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute any one of the above-mentioned medical image-based feature processing methods.

[0122] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes any one of the above-mentioned medical image-based feature processing methods.

[0123] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0124] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0127] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0129] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A feature processing method based on medical images, characterized in that: include: Get the probability that each pixel in the medical image belongs to the lesion area; Determine L connected domains of the medical image according to the probability that each pixel belongs to the lesion area, wherein L is an integer greater than or equal to 1, the probability that a lesion exists in each connected domain is greater than a preset probability, and each connected domain includes at least one block image; Performing feature fusion on image features of all block images in each connected domain to obtain connected domain features of each connected domain; The connected domain features of the L connected domains are subjected to feature fusion to obtain the pathological image features of the medical image, including: summing up the probabilities that all pixels in each connected domain belong to the lesion area to obtain the second value of each connected domain; calculating the ratio of the second value of each connected domain to the number of pixels in the connected domain to obtain the mean lesion probability of the connected domain; taking the connected domain with the largest mean lesion probability among the L connected domains as the main connected domain; taking each connected domain except the main connected domain among the L connected domains as the secondary connected domain; performing a first kind of linear transformation on the connected domain features of each connected domain to obtain the information vector of each connected domain; determining the weight of each secondary connected domain and the weight of the main connected domain according to the similarity between each secondary connected domain and the main connected domain; performing a weighted sum calculation on the information vector of each secondary connected domain and the information vector of the main connected domain according to the weight of each secondary connected domain and the weight of the main connected domain to obtain the pathological image features.

2. The feature processing method based on medical images according to claim 1, characterized in that: Determining L connected domains of the medical image according to the probability that each pixel belongs to the lesion area includes: Determining whether each pixel is a first pixel or a second pixel according to the probability that each pixel belongs to the lesion area, wherein the probability that the first pixel belongs to the lesion area is greater than a preset threshold, and the probability that the second pixel belongs to the lesion area is less than or equal to the preset threshold; taking the area where the first pixel is located as the target area; An aggregation operation is performed on the adjacent target regions to obtain the L connected regions.

3. The feature processing method based on medical images according to claim 1, characterized in that: Before performing feature fusion on the image features of all block images in each connected domain to obtain the connected domain features of each connected domain, the feature processing method based on medical images further includes: Converting the block image in each connected domain into a target block image with a target resolution, wherein the target resolution is higher than the resolution of the block image; Summing up the probabilities that all pixels in the target block image belong to the lesion area to obtain a first value corresponding to the target block image; Calculating a ratio of a first value corresponding to the target block image to the number of pixels in the target block image to obtain a lesion probability mean value of the target block image; Taking the target block image with the largest mean probability of lesions in each connected domain as the main block image of the connected domain; The target block images other than the main block images in each connected domain are used as the secondary block images of the connected domain.

4. The feature processing method based on medical images according to claim 3 is characterized in that: In the process of fusing the image features of all the block images in each connected domain to obtain the connected domain features of each connected domain, the feature processing method based on medical images further includes: Extracting a feature vector of each target block image in the I-th connected domain, wherein the feature vector of each target block image represents an image feature of a block image corresponding to the target block image; Performing a first type linear transformation on the feature vector of each target block image to obtain an information vector of the target block image; Determining the weight of each secondary block image and the weight of the primary block image according to the similarity between each secondary block image and the primary block image; According to the weight of each of the secondary block images and the weight of the primary block image, a weighted sum calculation is performed on the information vector of each of the secondary block images and the information vector of the primary block image, and the calculation result is used as the connected domain feature of the I-th connected domain.

5. The feature processing method based on medical images according to claim 4, characterized in that: Before determining the weight of each secondary block image and the weight of the primary block image according to the similarity between each secondary block image and the primary block image, the feature processing method based on medical images further includes: Performing a second type of linear transformation on the feature vector of each secondary block image to obtain a query vector of the secondary block image; Performing a second type of linear transformation on the feature vector of the main block image to obtain a query vector of the main block image; The vector similarity between the query vector of each of the secondary block images and the query vector of the primary block image is calculated, and the vector similarity is used as the similarity between each of the secondary block images and the primary block image.

6. The feature processing method based on medical images according to claim 1, characterized in that: Before determining the weight of each secondary connected domain and the weight of the primary connected domain according to the similarity between each secondary connected domain and the primary connected domain, the feature processing method based on medical images further includes: Performing a second type of linear transformation on the connected domain features of each secondary connected domain to obtain a query vector of each secondary connected domain; Performing a second type of linear transformation on the connected domain features of the main connected domain to obtain a query vector of the main connected domain; The vector similarity between the query vector of each secondary connected domain and the query vector of the primary connected domain is calculated, and the vector similarity is used as the similarity between each secondary connected domain and the primary connected domain.

7. The feature processing method based on medical images according to claim 1, characterized in that: After the connected domain features of the L connected domains are fused to obtain the pathological image features of the medical image, the feature processing method based on medical images further includes: The pathological image features are converted into target feature vectors through a target model, and a prediction result is determined based on the target feature vectors, wherein the target model is a neural network model pre-trained using reference images as training samples.

8. A feature processing device based on medical images, characterized in that: include: An acquisition unit, used for acquiring the probability that each pixel in the medical image belongs to the lesion area; A determination unit, configured to determine L connected domains of the medical image according to the probability that each pixel belongs to the lesion area, wherein L is an integer greater than or equal to 1, the probability that a lesion exists in each connected domain is greater than a preset probability, and each connected domain includes at least one block image; A first feature processing unit is used to perform feature fusion on image features of all block images in each connected domain to obtain a connected domain feature of each connected domain; A second feature processing unit is used to perform feature fusion on the connected domain features of the L connected domains to obtain the pathological image features of the medical image; Among them, the second feature processing unit includes: a fourth calculation unit, which is used to sum the probabilities that all pixels in each connected domain belong to the lesion area to obtain the second value of each connected domain; a fifth calculation unit, which is used to calculate the ratio of the second value of each connected domain to the number of pixels in the connected domain to obtain the mean lesion probability of the connected domain; a seventh processing unit, which is used to take the connected domain with the largest mean lesion probability among the L connected domains as the main connected domain; an eighth processing unit, which is used to take each connected domain except the main connected domain among the L connected domains as a secondary connected domain; a third processing subunit, which is used to perform a first-class linear transformation on the connected domain features of each connected domain to obtain the information vector of each connected domain; a fourth processing subunit, which is used to determine the weight of each secondary connected domain and the weight of the main connected domain according to the similarity between each secondary connected domain and the main connected domain; and a fifth processing subunit, which is used to perform weighted sum calculation on the information vector of each secondary connected domain and the information vector of the main connected domain according to the weight of each secondary connected domain and the weight of the main connected domain to obtain the pathological image feature.

9. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the feature processing method based on medical images as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the feature processing method based on medical images as described in any one of claims 1 to 7.

Images