A follow-up case data processing method, device and equipment and a storage medium

Abstract

The application provides a processing method and device for follow-up case data, equipment and a storage medium. The processing method comprises: determining a first detection result of a target lesion in a first space according to position information and size information of the target lesion in a first medical image; performing transformation processing on the first detection result according to a registration transformation matrix between the first medical image and a second medical image to obtain an initial lesion matching result of the first detection result in a second space; and inputting the first medical image, the second medical image, the first detection result and the initial lesion matching result into a feature extraction model to output a lesion feature extraction result of the second medical image. In this way, the application enables the model to effectively combine the anatomical structure information of the lesion on the basis of the medical image information, thereby improving the feature extraction accuracy of the model for the same lesion in different medical images and the precision of matching and positioning of the same lesion in follow-up case data.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method, apparatus, device, and storage medium for processing follow-up case data. Background Technology

[0002] A lesion refers to a diseased part of the body, most commonly seen in acute lung infections such as pneumonia. For example, if a part of the lung is destroyed by tuberculosis bacteria, the destroyed part is called a pulmonary tuberculosis lesion. In the medical field, lesions are often treated using follow-up tracking. Doctors can monitor the changes in lesions over time by continuously collecting medical imaging data from the same patient at different times (i.e., patient follow-up case data). However, when analyzing patient follow-up case data, because the spatial information of medical images differs at different times, it is often necessary to match and locate the same target lesion in different medical images.

[0003] Currently, the matching and localization of target lesions in follow-up cases is mainly achieved by comparing the similarity between lesions in different medical images. When the similarity between a first lesion in a first medical image and a second lesion in a second medical image is higher than a preset threshold, it is determined that the first and second lesions belong to the same lesion, thus achieving the matching and localization of the same target lesion in the follow-up case data. However, with this method, when patients undergo treatments such as surgery, the characteristics of the lesions in subsequent follow-up case data can change drastically, leading to significant differences in the same lesion across different medical images. This significantly reduces the accuracy of lesion matching results based on similarity comparison. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, apparatus, device and storage medium for processing follow-up case data, so that the model can effectively combine the anatomical structure information of the lesion on the basis of medical image information, improve the accuracy of feature extraction of the same lesion in different medical images, and help improve the accuracy of matching and locating the same lesion in follow-up case data.

[0005] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0006] In a first aspect, embodiments of this application provide a method for processing follow-up case data, wherein the follow-up case data includes at least a first medical image and a second medical image; the first medical image and the second medical image are medical images collected from the same object at different times; the processing method includes:

[0007] Based on the location and size information of the target lesion in the first medical image, a first detection result of the target lesion in a first space is determined; wherein, the first space represents the coordinate space in which the first medical image is located;

[0008] Based on the registration transformation matrix between the first medical image and the second medical image, the first detection result is transformed to obtain the initial lesion matching result of the first detection result in the second space; wherein, the second space represents the coordinate space where the second medical image is located;

[0009] The first medical image, the second medical image, the first detection result, and the initial lesion matching result are input into the feature extraction model, and the lesion feature extraction result of the second medical image is output; wherein, the lesion feature extraction result is used at least for the lesion segmentation task of the second medical image.

[0010] Secondly, embodiments of this application provide a processing apparatus for follow-up case data, wherein the follow-up case data includes at least a first medical image and a second medical image; the first medical image and the second medical image are medical images collected from the same object at different times; the processing apparatus includes:

[0011] The determination module is used to determine a first detection result of the target lesion in a first space based on the location and size information of the target lesion in the first medical image; wherein, the first space represents the coordinate space where the first medical image is located;

[0012] The registration module is used to transform the first detection result according to the registration transformation matrix between the first medical image and the second medical image to obtain the initial lesion matching result of the first detection result in the second space; wherein, the second space represents the coordinate space where the second medical image is located.

[0013] The processing module is used to input the first medical image, the second medical image, the first detection result, and the initial lesion matching result into the feature extraction model, and output the lesion feature extraction result of the second medical image; wherein, the lesion feature extraction result is used at least for the lesion segmentation task of the second medical image.

[0014] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for processing follow-up case data.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method for processing follow-up case data.

[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0017] This application provides a method, apparatus, device, and storage medium for processing follow-up case data. Based on the location and size information of the target lesion in a first medical image, a first detection result of the target lesion in a first space is determined. The first detection result is then transformed according to the registration transformation matrix between the first and second medical images to obtain an initial lesion matching result in a second space. The first medical image, the second medical image, the first detection result, and the initial lesion matching result are input into a feature extraction model, which outputs the lesion feature extraction result of the second medical image. Thus, this application enables the model to effectively combine anatomical structural information of the lesion with medical image information, improving the accuracy of feature extraction for the same lesion in different medical images and the precision of matching and locating the same lesion in follow-up case data. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for processing follow-up case data provided in an embodiment of this application is shown.

[0020] Figure 2 This illustration shows a schematic diagram of the model structure of a feature extraction model provided in an embodiment of this application;

[0021] Figure 3 This paper illustrates a flowchart of a method for obtaining a first enhanced feature corresponding to a first input data within a first feature extraction window, as provided in an embodiment of this application.

[0022] Figure 4 The illustration shows a flowchart of a method for obtaining a second enhanced feature corresponding to a second input data within a first feature extraction window, as provided in an embodiment of this application.

[0023] Figure 5The illustration shows a flowchart of a method for obtaining a third enhanced feature corresponding to third input data within a first feature extraction window, as provided in an embodiment of this application.

[0024] Figure 6 The illustration shows a flowchart of a method for obtaining a fourth enhanced feature corresponding to a fourth input data within a first feature extraction window, as provided in an embodiment of this application.

[0025] Figure 7 A flowchart illustrating the first method for extracting lesion features from a second medical image, as provided in an embodiment of this application, is shown.

[0026] Figure 8 A flowchart illustrating the second method for extracting lesion features from a second medical image, as provided in an embodiment of this application, is shown.

[0027] Figure 9 This illustration shows a schematic diagram of a follow-up case data processing device provided in an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the structure of a computer device 1000 provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0030] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0031] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0032] Currently, the matching and localization of target lesions in follow-up cases is mainly achieved by comparing the similarity between lesions in different medical images. When the similarity between a first lesion in a first medical image and a second lesion in a second medical image is higher than a preset threshold, it is determined that the first and second lesions belong to the same lesion, thus achieving the matching and localization of the same target lesion in the follow-up case data. However, with this method, when patients undergo treatments such as surgery, the characteristics of the lesions in subsequent follow-up case data can change drastically, leading to significant differences in the same lesion across different medical images. This significantly reduces the accuracy of lesion matching results based on similarity comparison.

[0033] Based on this, embodiments of this application provide a method, apparatus, device, and storage medium for processing follow-up case data. The method involves determining a first detection result of the target lesion in a first space based on its location and size information in a first medical image; transforming the first detection result according to the registration transformation matrix between the first and second medical images to obtain an initial lesion matching result in a second space; and inputting the first medical image, the second medical image, the first detection result, and the initial lesion matching result into a feature extraction model to output the lesion feature extraction result of the second medical image. In this way, this application enables the model to effectively combine anatomical structural information of the lesion with medical image information, improving the accuracy of feature extraction for the same lesion in different medical images and the precision of matching and locating the same lesion in follow-up case data.

[0034] It should be noted that the method for processing follow-up case data provided in this application embodiment is applicable to a follow-up case data processing device, which can be integrated into a computer device.

[0035] Specifically, the aforementioned computer equipment can be terminal devices, such as mobile phones, tablets, laptops, desktop computers, etc.; the aforementioned computer equipment can also be servers, which can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but are not limited to these.

[0036] To facilitate understanding of the embodiments of this application, a detailed description is provided below of a method, apparatus, device, and storage medium for processing follow-up case data provided in the embodiments of this application.

[0037] Reference Figure 1 As shown, Figure 1 This illustration shows a flowchart of a method for processing follow-up case data according to an embodiment of this application. The processing method includes steps S101-S103; specifically:

[0038] S101, based on the location and size information of the target lesion in the first medical image, determine the first detection result of the target lesion in the first space.

[0039] Here, the follow-up case data includes at least a first medical image and a second medical image; wherein the first medical image and the second medical image are medical images collected from the same object at different times. That is, the follow-up case data is used to characterize multiple medical images continuously collected from the target lesion of the same patient at different times. The specific number of medical images included in the follow-up case data is not limited in this embodiment.

[0040] It should be noted that different medical images in the follow-up case data may belong to the same type of medical images. For example, medical images in the follow-up case data may be CT (Computed Tomography) images collected at different times for the same patient's target lesion. This application embodiment does not limit the specific image type to which the medical images in the follow-up case data belong.

[0041] Here, two different medical images can be arbitrarily selected from the follow-up case data as the first medical image and the second medical image mentioned above. The location and size information of the target lesion in each medical image (i.e., the shape, size, etc. of the target lesion in the medical image) are known information. However, since there are differences in the spatial information of different medical images, in this embodiment of the application, it is necessary to register the image information of the same lesion (i.e., the target lesion mentioned above) in different spaces (i.e., different medical images) to the same reference space, so as to obtain the anatomical structure information of the target lesion (i.e., the shape, size, and other structural information of the target lesion itself).

[0042] Specifically, the first space represents the coordinate space where the first medical image is located. When executing step S101, a first spatial matrix that can represent the position and size information of the target lesion in the first space can be generated based on the position and size information of the target lesion in the first medical image as the first detection result.

[0043] As an example, since medical images are usually in the form of three-dimensional image data, as an optional embodiment, the first spatial matrix (i.e. the first detection result) can be a 3D (three-dimensional) Gaussian matrix, wherein the center of the sphere of the 3D Gaussian matrix is ​​the center point of the target lesion in the first medical image, and the radius of the 3D Gaussian matrix is ​​determined according to the size of the target lesion in the first medical image.

[0044] It should be noted that, in addition to the 3D Gaussian matrix mentioned above, the first detection result can also be other types of spatial matrices, as long as the first detection result can characterize the position and size information of the target lesion in the first space. This application embodiment does not limit the specific form of the first detection result.

[0045] S102, based on the registration transformation matrix between the first medical image and the second medical image, the first detection result is transformed to obtain the initial lesion matching result of the first detection result in the second space.

[0046] Here, the second space represents the coordinate space where the second medical image is located.

[0047] Specifically, before executing step S102, using the second medical image as the reference image and the first medical image as the floating image, a graphic transformation matrix that can transform the first medical image (i.e., the floating image) to the same coordinate space as the second medical image (i.e., the reference image) can be determined through rigid registration. The determined graphic transformation matrix is ​​the registration transformation matrix mentioned above.

[0048] It should be noted that the method for obtaining the registration transformation matrix in this embodiment is not unique. For example, in addition to the rigid registration method, the registration transformation matrix can also be obtained through non-rigid registration. The difference between rigid and non-rigid registration is that rigid registration usually takes the entire image as the object of registration operation, and the image can be aligned by operations such as translation, rotation, and scaling of the entire image; while non-rigid registration performs individual transformation processing on each local region (even each pixel) in the image. Based on this, the registration transformation matrix between the first medical image and the second medical image can be obtained by either rigid or non-rigid registration. This embodiment does not limit the specific method for obtaining the registration transformation matrix.

[0049] Specifically, the image information in the first space can be registered and transformed to the second space based on the registration transformation matrix. Therefore, when executing step S102, the first detection result (characterizing the anatomical structure information such as the position and size of the target lesion in the first space) can be registered and transformed to the second space based on the above registration transformation matrix, so as to obtain the initial lesion matching result of the first detection result in the second space.

[0050] S103, input the first medical image, the second medical image, the first detection result, and the initial lesion matching result into the feature extraction model, and output the lesion feature extraction result of the second medical image.

[0051] Here, in the feature extraction model, the feature extraction model is used to perform the same feature extraction operation on each type of input data to obtain the feature extraction result corresponding to each type of input data. That is, the feature extraction model can output the image feature vector of the first medical image, the image feature vector of the second medical image, the feature vector of the first detection result, and the feature vector of the initial lesion matching result.

[0052] It should be noted that, in the embodiments of this application, the initial lesion matching result obtained is equivalent to the anatomical structure information of the target lesion in the second space (e.g., location information, size information, etc.). At this time, the initial lesion matching result is unrelated to the actual detection result of the target lesion in the second medical image (e.g., the actual location and actual size of the target lesion in the second medical image, etc.).

[0053] Based on this, when the feature extraction model extracts features from the input image information (i.e., the first medical image and the second medical image), the first detection result and the initial lesion matching result are used to provide the feature extraction model with anatomical structural information about the target lesion itself. Thus, this embodiment of the application enables the feature extraction model to effectively combine the anatomical structural information of the target lesion with the medical image information, improving the accuracy of feature extraction for the same lesion in different medical images, and facilitating the improvement of the accuracy of matching and locating the same lesion in follow-up case data.

[0054] Specifically, the lesion feature extraction result mentioned above is the image feature vector of the second medical image output by the feature extraction model. The lesion feature extraction result is used at least for the lesion segmentation task of the second medical image. At this time, the lesion feature extraction result can replace the second medical image as the model input data of the lesion segmentation model. The lesion segmentation prediction result for the second medical image is obtained by outputting the lesion segmentation model. Based on the matching degree (i.e., similarity degree) between the actual detection result of the target lesion in the second medical image (i.e., the image region where the target lesion is located in the second medical image) and the lesion segmentation prediction result, the feature extraction capability of the feature extraction model and the lesion segmentation capability of the lesion segmentation model can be evaluated.

[0055] It should be noted that, in addition to the lesion segmentation task, the above-mentioned lesion feature extraction results can also be used for lesion detection tasks on second medical images (i.e., classifying pixels belonging to lesion regions in the second medical image and locating the positions of lesions in the second medical image). In this case, the above-mentioned lesion segmentation model can simply be adaptively replaced with a lesion detection model. Based on this, the specific uses of the above-mentioned output lesion feature extraction results are not limited in this application embodiment.

[0056] The specific implementation process of each of the above steps in the embodiments of this application will be described in detail below:

[0057] In this application embodiment, in addition to existing commonly used feature extraction models (such as deep learning models based on the Unet network structure or convolutional neural networks), in a preferred embodiment, by making overall structural improvements to the existing Unet network structure, this application embodiment also provides, for the feature extraction model in step S103 above, the following: Figure 2 The model structure shown is a completely new one, specifically:

[0058] Reference Figure 2 As shown, Figure 2 This illustration shows a schematic diagram of the model structure of a feature extraction model provided in an embodiment of this application. The feature extraction model employs a Unet network structure with the swin-transformer module at its core, as shown below. Figure 2 As shown, the Unet network structure includes multiple sets of symmetrical encoders 201 and decoders 202. The encoders 201 are used to downsample the input image data, and the decoders 202 are used to upsample the input image data. Each set of symmetrical encoders 201 and decoders 202 corresponds to a processing scale of image data. Different encoders 201 correspond to different processing scales, and different decoders 202 correspond to different processing scales.

[0059] Specifically, regarding the encoder part of the feature extraction model, such as Figure 2 As shown, each layer of the encoder includes at least one four-input four-output swin-transformer module. Each swin-transformer module performs the same operation for the input data of each channel. That is, when there are n swin-transformer modules in each layer of the encoder, the swin-transformer operation is repeated n times for the input data of each channel in that layer of the encoder. The specific number of swin-transformer modules included in each layer of the encoder is not limited in this embodiment.

[0060] Here, regarding the existing swin-transformer module, it should be noted that the existing swin-transformer module typically has only one input data. The main operation of the swin-transformer module is as follows: first, the input data is divided into multiple sub-data, where each sub-data corresponds to a data processing window. Then, by performing the same data processing operation on each sub-data within each data processing window, the window features corresponding to each sub-data are obtained. Finally, the window features corresponding to each sub-data are concatenated to obtain the data features corresponding to the complete input data.

[0061] Based on this, in this embodiment of the application, the swin-transformer module also contains multiple feature extraction windows. However, unlike the existing swin-transformer module, in this embodiment of the application, the swin-transformer module has four data transmission channels. Therefore, within each feature extraction window, the swin-transformer module in this embodiment of the application processes the sub-data of each of the four input data to obtain the data features corresponding to the input data of each channel.

[0062] Specifically, in the embodiments of this application, each swin-transformer module performs the same operation steps. Taking one swin-transformer module as an example, in the encoder section, the swin-transformer module is specifically used to perform the following steps a1-a3:

[0063] Step a1: Receive the output data of each channel in the upper layer encoder as the input data of the same channel in this layer encoder.

[0064] For example, for the first layer encoder, the input data of the four channels in the first layer encoder are the first medical image, the second medical image, the first detection result, and the initial lesion matching result input in step S103 above; for the second layer encoder, the input data of the four channels in the second layer encoder are the data processing results output by the first layer encoder after processing the first medical image, the second medical image, the first detection result, and the initial lesion matching result respectively.

[0065] Step a2: Divide the input data of each channel into multiple sub-data, and perform the same feature extraction and feature enhancement processing on the sub-data of different input data in each feature extraction window to obtain the enhanced features corresponding to each sub-data in each feature extraction window.

[0066] It should be noted that, in step a2, similar to the existing swin-transformer module, a feature extraction operation can be performed once for each input sub-data within each feature extraction window to obtain an initial data feature for each sub-data within the feature extraction window; based on this, the swin-transformer module in this embodiment further performs feature enhancement processing on the initial data feature of each sub-data within each feature extraction window to obtain the enhanced feature corresponding to each sub-data within that feature extraction window, thereby enhancing the feature extraction capability of the data within each feature extraction window.

[0067] Step a3: Perform splicing processing on the enhancement features of sub-data belonging to the same channel, and output the splicing processing result to the corresponding channel in the lower layer encoder.

[0068] For example, taking the first channel corresponding to the first medical image as an example, the first medical image is input into the first layer encoder through the first channel. In the first layer encoder, when the swin-transformer module includes 5 feature extraction windows, the swin-transformer module can divide the first medical image into 5 sub-data. In each feature extraction window, the same feature extraction and feature enhancement processing operations are performed to obtain the enhancement features corresponding to each sub-data in each feature extraction window. The enhancement features corresponding to the 5 sub-data are then concatenated to obtain the enhancement features of the first medical image. The obtained enhancement features are then input into the second layer encoder through the first channel. In the second layer encoder, the same operations as steps a1-a3 above are performed.

[0069] Specifically, regarding the decoder part of the feature extraction model, each pair of symmetrical encoders 201 and decoders 202 corresponds to a processing scale of image data. Based on this, each decoder 202 corresponds to a scale feature of image processing. In the same channel of each decoder 202, the decoder 202 first concatenates its corresponding scale feature with the data feature output by the decoder in the same channel of the previous stage in the feature dimension to obtain the input data corresponding to the current decoder in the same channel. Then, the reverse operation of the swin-transformer module of the encoder part is repeated for the input data corresponding to each channel. In the last encoder 202, the feature extraction result of the second medical image can be obtained from the second channel corresponding to the second medical image as the lesion feature extraction result in step S103 above.

[0070] In addition to the improvements to the overall model structure of the feature extraction model mentioned above, the specific feature extraction and enhancement operations performed within each feature extraction window are explained in detail below. Taking the first feature extraction window in the swin-transformer module of the first layer encoder as an example, the first sub-data segmented from the first medical image is used as the first input data of the first feature extraction window, the first sub-data segmented from the second medical image is used as the second input data of the first feature extraction window, the first sub-data segmented from the first detection result is used as the third input data of the first feature extraction window, and the first sub-data segmented from the initial lesion matching result is used as the fourth input data of the first feature extraction window.

[0071] Regarding the aforementioned first input data (i.e., the first sub-data segmented from the first medical image), in an optional implementation, refer to... Figure 3 As shown, Figure 3 This illustration shows a flowchart of a method for obtaining a first enhanced feature corresponding to a first input data within a first feature extraction window, according to an embodiment of this application. The method includes steps S301-S306; specifically:

[0072] S301, within the first feature extraction window, feature extraction is performed on the first input data, the second input data, the third input data, and the fourth input data respectively to obtain the first data feature of the first input data, the second data feature of the second input data, the third data feature of the third input data, and the fourth data feature of the fourth input data.

[0073] For example, taking the first input data as I1, the second input data as I2, the third input data as G1, and the fourth input data as G2 as an example, features are extracted from the first input data I1, the second input data I2, the third input data G1, and the fourth input data G2 in the first feature extraction window, respectively, to obtain the first data feature i1 of the first input data I1, the first data feature i2 of the second input data I2, the third data feature g1 of the third input data G1, and the fourth data feature g2 of the fourth input data G2.

[0074] S302, using the Q-feature matrix and K-feature matrix of the first data feature under the attention mechanism, calculate the first self-attention feature of the first data feature under the self-attention mechanism.

[0075] Here, as an optional embodiment, the first data features can be input into the first neural network, and the Q feature matrix, K feature matrix and V feature matrix of the first data features under the attention mechanism can be obtained by the output of the first neural network.

[0076] Specifically, during step S302, the first self-attention feature A11 of the first data feature i1 under the self-attention mechanism (i.e., focusing only on its own data features) can be calculated using the following formula:

[0077] ;

[0078] Wherein, Q1 represents the Q-feature matrix of the first data feature i1 under the attention mechanism;

[0079] K1 represents the K-feature matrix of the first data feature i1 under the attention mechanism;

[0080] The transpose matrix representing Q1.

[0081] S303, using the Q feature matrix of the first data feature under the attention mechanism and the K feature matrix of the second data feature under the attention mechanism, the first mutual attention feature of the first data feature under the mutual attention mechanism is calculated.

[0082] Here, as an optional embodiment, the second data features can be input into the second neural network, and the Q feature matrix, K feature matrix and V feature matrix of the second data features under the attention mechanism can be obtained by the output of the second neural network.

[0083] It should be noted that since the first data feature comes from the first medical image and the second data feature comes from the second medical image, both the first and second medical images belong to image information. That is, under the attention mechanism, the focus of attention on the first data feature and the second data feature is the same (both focus on the image information side).

[0084] Based on this, in a preferred embodiment, the second neural network may be a neural network that shares parameters with the first neural network, so as to improve the extraction accuracy of the Q feature matrix, K feature matrix and V feature matrix of the first data feature / second data feature under the attention mechanism.

[0085] Specifically, during step S303, the first mutual attention feature A12 under the mutual attention mechanism (i.e., focusing on the data features between the first and second data features on the image information side) can be calculated using the following formula:

[0086] ;

[0087] Wherein, Q1 represents the Q-feature matrix of the first data feature i1 under the attention mechanism;

[0088] K2 represents the K-feature matrix of the second data feature i2 under the attention mechanism;

[0089] The transpose matrix representing Q1.

[0090] S304, using the Q-feature matrix and K-feature matrix of the third data feature under the attention mechanism, calculate the third self-attention feature of the third data feature under the self-attention mechanism.

[0091] Here, as an optional embodiment, the third data feature can be input into the third neural network, and the Q feature matrix, K feature matrix and V feature matrix of the third data feature under the attention mechanism can be obtained by the output of the third neural network.

[0092] Specifically, during step S304, the third self-attention feature Ag11 under the self-attention mechanism (i.e., focusing only on its own data features) can be calculated using the following formula:

[0093] ;

[0094] Wherein, Qg1 represents the Q-feature matrix of the third data feature g1 under the attention mechanism;

[0095] Kg1 represents the K-feature matrix of the third data feature g1 under the attention mechanism;

[0096] The transpose matrix representing Qg1.

[0097] S305, using the Q-feature matrix of the third data feature under the attention mechanism and the K-feature matrix of the fourth data feature under the attention mechanism, the third mutual attention feature of the third data feature under the mutual attention mechanism is calculated.

[0098] Here, as an optional embodiment, the fourth data feature can be input into the fourth neural network, and the Q feature matrix, K feature matrix and V feature matrix of the fourth data feature under the attention mechanism can be obtained by the output of the fourth neural network.

[0099] It should be noted that since the third data feature is derived from the first detection result and the fourth data feature is derived from the initial lesion matching result, both the first detection result and the initial lesion matching result belong to the anatomical structure information (location information, size information, etc.) of the target lesion. That is, under the attention mechanism, the focus of attention on the third data feature and the fourth data feature is the same (both focus on the anatomical structure information side).

[0100] Based on this, in a preferred embodiment, the fourth neural network may be a neural network that shares parameters with the third neural network, so as to improve the extraction accuracy of the Q feature matrix, K feature matrix and V feature matrix of the third data feature / fourth data feature under the attention mechanism.

[0101] Specifically, during step S305, the third mutual attention feature Ag12 under the mutual attention mechanism (i.e., focusing on the data features between the third and fourth data features on the anatomical structure information side) can be calculated using the following formula:

[0102] ;

[0103] Wherein, Qg1 represents the Q-feature matrix of the third data feature g1 under the attention mechanism;

[0104] Kg2 represents the K-feature matrix of the fourth data feature g2 under the attention mechanism;

[0105] The transpose matrix representing Qg1.

[0106] S306, using the first data feature, the first self-attention feature, the first mutual attention feature, the third self-attention feature, the third mutual attention feature, the V feature matrix of the first data feature under the attention mechanism, and the V feature matrix of the second data feature under the attention mechanism, the first enhancement feature is calculated.

[0107] Specifically, during step S306, the first enhancement feature T1 can be calculated using the following formula:

[0108] ;

[0109] Where i1 represents the first data feature, A11 represents the first self-attention feature, A12 represents the first mutual attention feature, Ag11 represents the third self-attention feature, Ag12 represents the third mutual attention feature, V1 represents the V feature matrix of the first data feature i1 under the attention mechanism, and V2 represents the V feature matrix of the second data feature i2 under the attention mechanism.

[0110] It should be noted that, in calculating the first enhancement feature T1, in addition to the softmax function, other types of functions can also be used for calculation (for example, the softmax function in the above formula can also be replaced by the sigmoid function). This application does not limit the specific type of function used to calculate the first enhancement feature T1.

[0111] Regarding the second input data (i.e., the first sub-data segmented from the second medical image), in one optional implementation, refer to... Figure 4 As shown, Figure 4 This illustration shows a flowchart of a method for obtaining a second enhanced feature corresponding to second input data within a first feature extraction window, according to an embodiment of this application. The method includes steps S401-S405; specifically:

[0112] S401, using the Q-feature matrix and K-feature matrix of the second data feature under the attention mechanism, calculate the second self-attention feature of the second data feature under the self-attention mechanism.

[0113] Here, the acquisition methods of the Q-feature matrix and K-feature matrix of the second data features under the attention mechanism are the same as those described in step S303 above, and the repetitions will not be repeated here.

[0114] Specifically, during step S401, the second self-attention feature A22 under the self-attention mechanism (i.e., focusing only on its own data features) can be calculated using the following formula:

[0115] ;

[0116] Wherein, Q2 represents the Q-feature matrix of the second data feature i2 under the attention mechanism;

[0117] K2 represents the K-feature matrix of the second data feature i2 under the attention mechanism;

[0118] The transpose matrix representing Q2.

[0119] S402, using the Q feature matrix of the second data feature under the attention mechanism and the K feature matrix of the first data feature under the attention mechanism, the second mutual attention feature of the second data feature under the mutual attention mechanism is calculated.

[0120] Specifically, during step S402, the second mutual attention feature A21 under the mutual attention mechanism (i.e., focusing on the data features between the first and second data features on the image information side) can be calculated using the following formula:

[0121] ;

[0122] Wherein, Q2 represents the Q-feature matrix of the second data feature i2 under the attention mechanism;

[0123] K1 represents the K-feature matrix of the first data feature i1 under the attention mechanism;

[0124] The transpose matrix representing Q2.

[0125] S403, using the Q-feature matrix and K-feature matrix of the fourth data feature under the attention mechanism, calculate the fourth self-attention feature of the fourth data feature under the self-attention mechanism.

[0126] Here, the methods for obtaining the Q-feature matrix and K-feature matrix of the fourth data feature under the attention mechanism are the same as those described in step S305 above, and the repetitions will not be repeated here.

[0127] Specifically, during step S403, the fourth self-attention feature Ag22 under the self-attention mechanism (i.e., focusing only on its own data features) can be calculated using the following formula:

[0128] ;

[0129] Wherein, Qg2 represents the Q-feature matrix of the fourth data feature g2 under the attention mechanism;

[0130] Kg2 represents the K-feature matrix of the fourth data feature g2 under the attention mechanism;

[0131] The transpose of Qg2 is represented by this matrix.

[0132] S404, using the Q feature matrix of the fourth data feature under the attention mechanism and the K feature matrix of the third data feature under the attention mechanism, the fourth mutual attention feature of the fourth data feature under the mutual attention mechanism is calculated.

[0133] Specifically, during step S404, the fourth mutual attention feature Ag21 under the mutual attention mechanism (i.e., focusing on the data features between the third and fourth data features on the anatomical structure information side) can be calculated using the following formula:

[0134] ;

[0135] Wherein, Qg2 represents the Q-feature matrix of the fourth data feature g2 under the attention mechanism;

[0136] Kg1 represents the K-feature matrix of the third data feature g1 under the attention mechanism;

[0137] The transpose of Qg2 is represented by this matrix.

[0138] S405, using the second data feature, the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, the fourth mutual attention feature, the V feature matrix of the first data feature under the attention mechanism, and the V feature matrix of the second data feature under the attention mechanism, the second reinforcement feature is calculated.

[0139] Specifically, during step S405, the second enhancement feature T2 can be calculated using the following formula:

[0140] ;

[0141] Where i2 represents the second data feature, A22 represents the second self-attention feature, A21 represents the second mutual attention feature, Ag22 represents the fourth self-attention feature, Ag21 represents the fourth mutual attention feature, V1 represents the V feature matrix of the first data feature i1 under the attention mechanism, and V2 represents the V feature matrix of the second data feature i2 under the attention mechanism.

[0142] It should be noted that, in calculating the second enhancement feature T2, in addition to the softmax function, other types of functions can also be used for calculation (for example, the softmax function in the above formula can also be replaced by the sigmoid function). The specific function type used in this application embodiment is not limited in any way when calculating the second enhancement feature T2.

[0143] Regarding the aforementioned third input data (i.e., the first sub-data segmented from the first detection result), in one optional implementation, refer to... Figure 5 As shown, Figure 5 This illustration shows a flowchart of a method for obtaining a third enhanced feature corresponding to third input data within a first feature extraction window, according to an embodiment of this application. The method includes steps S501-S502; specifically:

[0144] S501, acquire the first self-attention feature, the first mutual attention feature, the third self-attention feature, and the third mutual attention feature.

[0145] Specifically, the methods for obtaining the first self-attention feature A11, the first mutual attention feature A12, the third self-attention feature Ag11, and the third mutual attention feature Ag12 can be found in steps S302-S305 above. Repeated steps will not be repeated here.

[0146] S502, the third reinforcement feature is calculated using the third data feature, the first self-attention feature, the first mutual attention feature, the third self-attention feature, the third mutual attention feature, the V feature matrix of the third data feature under the attention mechanism, and the V feature matrix of the fourth data feature under the attention mechanism.

[0147] Specifically, during step S502, the third enhancement feature T3 can be calculated using the following formula:

[0148] ;

[0149] Wherein, g1 represents the third data feature, A11 represents the first self-attention feature, A12 represents the first mutual attention feature, Ag11 represents the third self-attention feature, Ag12 represents the third mutual attention feature, Vg1 represents the V feature matrix of the third data feature under the attention mechanism, and Vg2 represents the V feature matrix of the fourth data feature under the attention mechanism.

[0150] It should be noted that, in calculating the third enhancement feature T3, other types of functions besides the softmax function can also be used. This application does not limit the specific function type used to calculate the third enhancement feature T3 (for example, the softmax function in the above formula can be replaced with the sigmoid function).

[0151] Regarding the fourth input data mentioned above (i.e., the first sub-data segmented from the initial lesion matching results), in one optional implementation, refer to... Figure 6 As shown, Figure 6This illustration shows a flowchart of a method for obtaining a fourth enhanced feature corresponding to fourth input data within a first feature extraction window, according to an embodiment of this application. The method includes steps S601-S602; specifically:

[0152] S601, acquire the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, and the fourth mutual attention feature.

[0153] Specifically, the methods for obtaining the second self-attention feature A22, the second mutual attention feature A21, the fourth self-attention feature Ag22, and the fourth mutual attention feature Ag21 can be found in steps S401-S404 above. Repeated steps will not be repeated here.

[0154] S602, the fourth enhancement feature is calculated using the fourth data feature, the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, the fourth mutual attention feature, the V feature matrix of the third data feature under the attention mechanism, and the V feature matrix of the fourth data feature under the attention mechanism.

[0155] Specifically, during step S602, the fourth enhancement feature T4 can be calculated using the following formula:

[0156] ;

[0157] Wherein, g2 represents the fourth data feature, A22 represents the second self-attention feature, A21 represents the second mutual attention feature, Ag22 represents the fourth self-attention feature, Ag21 represents the fourth mutual attention feature, Vg1 represents the V feature matrix of the third data feature under the attention mechanism, and Vg2 represents the V feature matrix of the fourth data feature under the attention mechanism.

[0158] It should be noted that, in calculating the fourth enhancement feature T4, in addition to the softmax function, other types of functions can also be used for calculation (for example, the softmax function in the above formula can also be replaced by the sigmoid function). The specific function type used in this application embodiment is not limited in any way when calculating the fourth enhancement feature T4.

[0159] Based on the analysis in step S103 above, the lesion feature extraction result (i.e., the image feature vector of the second medical image) of the second medical image output by the feature extraction model can replace the second medical image as the model input data of the lesion segmentation model / lesion detection model, thereby helping the lesion segmentation model to complete the lesion segmentation task of the second medical image / helping the lesion detection model to complete the lesion detection task of the second medical image.

[0160] Specifically, taking the example that the above-mentioned lesion feature extraction results can replace the second medical image as the model input data for the lesion segmentation model, in an optional implementation, when the above-mentioned lesion feature extraction results replace the second medical image as the model input data for the lesion segmentation model during the model training phase, refer to... Figure 7 As shown, Figure 7 This illustration shows a flowchart of a first method for extracting lesion features from a second medical image, as provided in an embodiment of this application. The method includes steps S701-S702; specifically:

[0161] S701, the lesion feature extraction result of the second medical image is input into the first lesion segmentation model, and the lesion segmentation prediction result of the second medical image is output.

[0162] Here, the first lesion segmentation model represents the lesion segmentation model in the training phase; however, the specific model structure of the first lesion segmentation model is not limited in this embodiment; for example, it can be a single-layer convolutional neural network structure, or other more complex multi-layer neural network structures.

[0163] Here, the lesion segmentation prediction result represents the prediction result for the image region in the second medical image where the target lesion is located; for example, the lesion segmentation prediction result can be the labeling result of the second medical image based on 0-1 labeling, wherein the image region labeled as 1 in the lesion segmentation prediction result represents the prediction result for the image region in the second medical image where the target lesion is located.

[0164] S702, based on the segmentation loss between the lesion segmentation prediction result and the second detection result of the target lesion in the second space, the model parameters of the first lesion segmentation model and the feature extraction model are adjusted to obtain the first lesion segmentation model and the feature extraction model including the adjusted parameters.

[0165] Here, the second detection result is determined based on the location and size information of the target lesion in the second medical image.

[0166] It should be noted that when calculating the segmentation loss, the cross-entropy loss function or other loss functions such as focal loss can be used. This application does not limit the specific calculation method of the segmentation loss.

[0167] Specifically, in another optional implementation, when the above-mentioned lesion feature extraction results replace the second medical image as the model input data for the lesion segmentation model during the model application stage, refer to Figure 8 As shown, Figure 8 The diagram illustrates a flowchart of a second method for extracting lesion features from a second medical image, as provided in an embodiment of this application. The method includes step S801; specifically:

[0168] S801, the lesion feature extraction result of the second medical image is input into the second lesion segmentation model, and the lesion segmentation result of the target lesion in the second medical image is output.

[0169] Here, the second lesion segmentation model represents the lesion segmentation model in the application stage, that is, the second lesion segmentation model represents the pre-trained lesion segmentation model. At this time, since the second lesion segmentation model has completed the model training process, unlike the above steps S701-S702, step S801 no longer involves adjusting the model parameters of the second lesion segmentation model and the feature extraction model.

[0170] Based on the different uses of the lesion feature extraction results of the second medical image shown in steps S701-S702 and step S801 above, it should be noted that the embodiments of this application essentially provide a data processing method for any two different medical images in follow-up case data through the execution method shown in steps S101-S103. The embodiments of this application do not limit whether the data processing results (i.e., the lesion feature extraction results of the second medical image) are specifically applied to the training stage of the model or to the application stage of the model.

[0171] The method for processing follow-up case data provided in this application involves determining a first detection result of the target lesion in a first space based on its location and size information in a first medical image. The first detection result is then transformed using a registration transformation matrix between the first and second medical images to obtain an initial lesion matching result in a second space. The first medical image, the second medical image, the first detection result, and the initial lesion matching result are input into a feature extraction model, which outputs the lesion feature extraction result of the second medical image. This method enables the model to effectively combine anatomical structural information of the lesion with medical image information, improving the accuracy of feature extraction for the same lesion in different medical images and the precision of matching and locating the same lesion in follow-up case data.

[0172] Based on the same inventive concept, this application also provides a processing device corresponding to the above-mentioned method for processing follow-up case data. Since the principle of the processing device in the embodiments of this application is similar to the above-mentioned method for processing follow-up case data in the embodiments of this application, the implementation of the processing device can refer to the implementation of the above-mentioned processing method, and the repeated parts will not be described again.

[0173] Reference Figure 9 As shown, Figure 9 This illustration shows a schematic diagram of a follow-up case data processing device provided in an embodiment of this application. The follow-up case data includes at least a first medical image and a second medical image; the first medical image and the second medical image are medical images acquired from the same object at different times; the processing device includes:

[0174] The determining module 901 is used to determine the first detection result of the target lesion in the first space based on the location information and size information of the target lesion in the first medical image; wherein, the first space represents the coordinate space where the first medical image is located;

[0175] The registration module 902 is used to transform the first detection result according to the registration transformation matrix between the first medical image and the second medical image to obtain the initial lesion matching result of the first detection result in the second space; wherein, the second space represents the coordinate space where the second medical image is located.

[0176] The processing module 903 is used to input the first medical image, the second medical image, the first detection result, and the initial lesion matching result into the feature extraction model, and output the lesion feature extraction result of the second medical image; wherein, the lesion feature extraction result is used at least for the lesion segmentation task of the second medical image.

[0177] In one optional implementation, the feature extraction model employs a Unet network structure with a swin-transformer module at its core; wherein the Unet network structure includes multiple sets of symmetrical encoders and decoders, each encoder including at least one four-input four-output swin-transformer module, the swin-transformer module containing multiple feature extraction windows, and the swin-transformer module being used for:

[0178] Receive the output data of each channel in the upper layer encoder as the input data of the same channel in this layer encoder;

[0179] The input data of each channel is divided into multiple sub-data, and the same feature extraction and feature enhancement processing is performed on the sub-data of different input data in each feature extraction window to obtain the enhanced features corresponding to each sub-data in each feature extraction window;

[0180] The enhancement features of sub-data belonging to the same channel are spliced ​​together, and the splicing result is output to the corresponding channel in the lower-level encoder.

[0181] In an optional implementation, in the feature extraction model, the first sub-data segmented from the first medical image is used as the first input data of the first feature extraction window in the swin-transformer module; the first sub-data segmented from the second medical image is used as the second input data of the first feature extraction window; the first sub-data segmented from the first detection result is used as the third input data of the first feature extraction window; and the first sub-data segmented from the initial lesion matching result is used as the fourth input data of the first feature extraction window. The swin-transformer module is used to obtain the first enhancement feature corresponding to the first input data within the first feature extraction window through the following method:

[0182] Within the first feature extraction window, feature extraction is performed on the first input data, the second input data, the third input data, and the fourth input data respectively to obtain the first data feature of the first input data, the second data feature of the second input data, the third data feature of the third input data, and the fourth data feature of the fourth input data.

[0183] Using the Q-feature matrix and K-feature matrix of the first data feature under the attention mechanism, the first self-attention feature of the first data feature under the self-attention mechanism is calculated;

[0184] Using the Q-feature matrix of the first data feature under the attention mechanism and the K-feature matrix of the second data feature under the attention mechanism, the first mutual attention feature of the first data feature under the mutual attention mechanism is calculated.

[0185] Using the Q-feature matrix and K-feature matrix of the third data feature under the attention mechanism, the third self-attention feature of the third data feature under the self-attention mechanism is calculated;

[0186] Using the Q-feature matrix of the third data feature under the attention mechanism and the K-feature matrix of the fourth data feature under the attention mechanism, the third mutual attention feature of the third data feature under the mutual attention mechanism is calculated.

[0187] The first enhancement feature is calculated using the first data feature, the first self-attention feature, the first mutual attention feature, the third self-attention feature, the third mutual attention feature, the V feature matrix of the first data feature under the attention mechanism, and the V feature matrix of the second data feature under the attention mechanism.

[0188] In an optional implementation, the swin-transformer module is used to obtain the second enhanced feature corresponding to the second input data within the first feature extraction window by the following method:

[0189] Using the Q-feature matrix and K-feature matrix of the second data feature under the attention mechanism, the second self-attention feature of the second data feature under the self-attention mechanism is calculated;

[0190] Using the Q-feature matrix of the second data feature under the attention mechanism and the K-feature matrix of the first data feature under the attention mechanism, the second mutual attention feature of the second data feature under the mutual attention mechanism is calculated.

[0191] Using the Q-feature matrix and K-feature matrix of the fourth data feature under the attention mechanism, the fourth self-attention feature of the fourth data feature under the self-attention mechanism is calculated;

[0192] Using the Q-feature matrix of the fourth data feature under the attention mechanism and the K-feature matrix of the third data feature under the attention mechanism, the fourth mutual attention feature of the fourth data feature under the mutual attention mechanism is calculated.

[0193] The second enhancement feature is calculated using the second data feature, the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, the fourth mutual attention feature, the V feature matrix of the first data feature under the attention mechanism, and the V feature matrix of the second data feature under the attention mechanism.

[0194] In an optional implementation, the swin-transformer module is used to obtain the third enhanced feature corresponding to the third input data within the first feature extraction window through the following method:

[0195] Obtain the first self-attention feature, the first mutual attention feature, the third self-attention feature, and the third mutual attention feature;

[0196] The third enhancement feature is calculated using the third data feature, the first self-attention feature, the first mutual attention feature, the third self-attention feature, the third mutual attention feature, the V feature matrix of the third data feature under the attention mechanism, and the V feature matrix of the fourth data feature under the attention mechanism.

[0197] In an optional implementation, the swin-transformer module is used to obtain the fourth enhanced feature corresponding to the fourth input data within the first feature extraction window by the following method:

[0198] Obtain the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, and the fourth mutual attention feature;

[0199] The fourth enhancement feature is calculated using the fourth data feature, the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, the fourth mutual attention feature, the V feature matrix of the third data feature under the attention mechanism, and the V feature matrix of the fourth data feature under the attention mechanism.

[0200] In one optional embodiment, the processing apparatus further includes:

[0201] The first output module is used to input the lesion feature extraction result of the second medical image into the first lesion segmentation model and output the lesion segmentation prediction result of the second medical image; wherein, the first lesion segmentation model represents the lesion segmentation model in the training stage; the lesion segmentation prediction result represents the prediction result for the image region where the target lesion is located in the second medical image;

[0202] The training module is used to adjust the model parameters of the first lesion segmentation model and the feature extraction model based on the segmentation loss between the lesion segmentation prediction result and the second detection result of the target lesion in the second space, so as to obtain the first lesion segmentation model and the feature extraction model including the adjusted parameters; wherein, the second detection result is determined based on the location information and size information of the target lesion in the second medical image.

[0203] In one optional embodiment, the processing apparatus further includes:

[0204] The second output module is used to input the lesion feature extraction result of the second medical image into the second lesion segmentation model and output the lesion segmentation result of the target lesion in the second medical image; wherein, the second lesion segmentation model represents the lesion segmentation model in the application stage.

[0205] Based on the same inventive concept, such as Figure 10 As shown, this application provides a computer device 1000 for executing the follow-up case data processing method in this application. The device includes a memory 1001, a processor 1002, and a computer program stored in the memory 1001 and executable on the processor 1002. When the processor 1002 executes the computer program, it implements the steps of the follow-up case data processing method.

[0206] Specifically, the memory 1001 and processor 1002 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 1002 runs the computer program stored in the memory 1001, it can execute the above-mentioned method for processing follow-up case data.

[0207] Corresponding to the follow-up case data processing method in this application, this application embodiment also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the steps of the above-described follow-up case data processing method.

[0208] Specifically, the storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the storage medium is run, it can execute the above-mentioned method for processing follow-up case data.

[0209] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0210] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0211] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0212] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0213] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0214] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for processing follow-up case data, characterized in that, The follow-up case data includes at least a first medical image and a second medical image; the first medical image and the second medical image are medical images collected from the same subject at different times; the processing method includes: Based on the location and size information of the target lesion in the first medical image, a first detection result of the target lesion in a first space is determined; wherein, the first space represents the coordinate space where the first medical image is located; the first detection result represents the anatomical structure information of the target lesion in the first space; the anatomical structure information includes at least location information and size information. Based on the registration transformation matrix between the first medical image and the second medical image, the first detection result is transformed to obtain the initial lesion matching result of the first detection result in the second space; wherein, the second space represents the coordinate space where the second medical image is located; the initial lesion matching result represents the anatomical structure information of the target lesion in the second space. The first medical image, the second medical image, the first detection result, and the initial lesion matching result are input into the feature extraction model, and the lesion feature extraction result of the second medical image is output; wherein, the lesion feature extraction result is used at least to replace the second medical image as the model input data of the lesion segmentation model or the lesion detection model; the first detection result and the initial lesion matching result are used to provide the feature extraction model with anatomical structure information about the target lesion itself; The feature extraction model employs a Unet network structure with a Swin-transformer module at its core. The Unet network structure includes multiple sets of symmetrical encoders and decoders, each encoder comprising at least one four-input, four-output Swin-transformer module. Each Swin-transformer module contains multiple feature extraction windows and is used for: Receive the output data of each channel in the upper layer encoder as the input data of the same channel in this layer encoder; The input data of each channel is divided into multiple sub-data, and the same feature extraction and feature enhancement processing is performed on the sub-data of different input data in each feature extraction window to obtain the enhanced features corresponding to each sub-data in each feature extraction window; The enhancement features of sub-data belonging to the same channel are spliced ​​together, and the splicing result is output to the corresponding channel in the lower-level encoder. In the feature extraction model, the first sub-data segmented from the first medical image is used as the first input data of the first feature extraction window in the swin-transformer module; the first sub-data segmented from the second medical image is used as the second input data of the first feature extraction window; the first sub-data segmented from the first detection result is used as the third input data of the first feature extraction window; and the first sub-data segmented from the initial lesion matching result is used as the fourth input data of the first feature extraction window. The second enhanced feature corresponding to the second input data within the first feature extraction window is obtained through the following method: Within the first feature extraction window, feature extraction is performed on the first input data, the second input data, the third input data, and the fourth input data respectively to obtain the first data feature of the first input data, the second data feature of the second input data, the third data feature of the third input data, and the fourth data feature of the fourth input data. Using the Q-feature matrix and K-feature matrix of the second data feature under the attention mechanism, the second self-attention feature of the second data feature under the self-attention mechanism is calculated; Using the Q-feature matrix of the second data feature under the attention mechanism and the K-feature matrix of the first data feature under the attention mechanism, the second mutual attention feature of the second data feature under the mutual attention mechanism is calculated. Using the Q-feature matrix and K-feature matrix of the fourth data feature under the attention mechanism, the fourth self-attention feature of the fourth data feature under the self-attention mechanism is calculated; Using the Q-feature matrix of the fourth data feature under the attention mechanism and the K-feature matrix of the third data feature under the attention mechanism, the fourth mutual attention feature of the fourth data feature under the mutual attention mechanism is calculated. The second enhancement feature is calculated using the second data feature, the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, the fourth mutual attention feature, the V feature matrix of the first data feature under the attention mechanism, and the V feature matrix of the second data feature under the attention mechanism.

2. The processing method according to claim 1, characterized in that, The first enhanced feature corresponding to the first input data within the first feature extraction window is obtained using the following method: Using the Q-feature matrix and K-feature matrix of the first data feature under the attention mechanism, the first self-attention feature of the first data feature under the self-attention mechanism is calculated; Using the Q-feature matrix of the first data feature under the attention mechanism and the K-feature matrix of the second data feature under the attention mechanism, the first mutual attention feature of the first data feature under the mutual attention mechanism is calculated. Using the Q-feature matrix and K-feature matrix of the third data feature under the attention mechanism, the third self-attention feature of the third data feature under the self-attention mechanism is calculated; Using the Q-feature matrix of the third data feature under the attention mechanism and the K-feature matrix of the fourth data feature under the attention mechanism, the third mutual attention feature of the third data feature under the mutual attention mechanism is calculated. The first enhancement feature is calculated using the first data feature, the first self-attention feature, the first mutual attention feature, the third self-attention feature, the third mutual attention feature, the V feature matrix of the first data feature under the attention mechanism, and the V feature matrix of the second data feature under the attention mechanism.

3. The processing method according to claim 2, characterized in that, The third enhanced feature corresponding to the third input data within the first feature extraction window is obtained using the following method: Obtain the first self-attention feature, the first mutual attention feature, the third self-attention feature, and the third mutual attention feature; The third enhancement feature is calculated using the third data feature, the first self-attention feature, the first mutual attention feature, the third self-attention feature, the third mutual attention feature, the V feature matrix of the third data feature under the attention mechanism, and the V feature matrix of the fourth data feature under the attention mechanism.

4. The processing method according to claim 1, characterized in that, The fourth enhanced feature corresponding to the fourth input data within the first feature extraction window is obtained using the following method: Obtain the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, and the fourth mutual attention feature; The fourth enhancement feature is calculated using the fourth data feature, the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, the fourth mutual attention feature, the V feature matrix of the third data feature under the attention mechanism, and the V feature matrix of the fourth data feature under the attention mechanism.

5. The processing method according to claim 1, characterized in that, The processing method further includes: The lesion feature extraction result of the second medical image is input into the first lesion segmentation model, and the lesion segmentation prediction result of the second medical image is output; wherein, the first lesion segmentation model represents the lesion segmentation model in the training stage; the lesion segmentation prediction result represents the prediction result for the image region where the target lesion is located in the second medical image; Based on the segmentation loss between the lesion segmentation prediction result and the second detection result of the target lesion in the second space, the model parameters of the first lesion segmentation model and the feature extraction model are adjusted to obtain a first lesion segmentation model and a feature extraction model including the adjusted parameters; wherein, the second detection result is determined based on the position and size information of the target lesion in the second medical image.

6. The processing method according to claim 1, characterized in that, The processing method further includes: The lesion feature extraction result of the second medical image is input into the second lesion segmentation model, and the lesion segmentation result of the target lesion in the second medical image is output; wherein, the second lesion segmentation model represents the lesion segmentation model in the application stage.

7. A device for processing follow-up case data, characterized in that, The follow-up case data includes at least a first medical image and a second medical image; the first medical image and the second medical image are medical images collected from the same subject at different times; The processing device includes: The determining module is used to determine a first detection result of the target lesion in a first space based on the location information and size information of the target lesion in the first medical image; wherein, the first space represents the coordinate space where the first medical image is located; the first detection result represents the anatomical structure information of the target lesion in the first space; the anatomical structure information includes at least location information and size information; The registration module is used to transform the first detection result according to the registration transformation matrix between the first medical image and the second medical image to obtain an initial lesion matching result of the first detection result in a second space; wherein, the second space represents the coordinate space where the second medical image is located; and the initial lesion matching result represents the anatomical structure information of the target lesion in the second space. The processing module is used to input the first medical image, the second medical image, the first detection result, and the initial lesion matching result into a feature extraction model, and output the lesion feature extraction result of the second medical image; wherein, the lesion feature extraction result is used at least to replace the second medical image as model input data for a lesion segmentation model or a lesion detection model; the first detection result and the initial lesion matching result are used to provide the feature extraction model with anatomical structure information about the target lesion itself; The feature extraction model employs a Unet network structure with a Swin-transformer module at its core. The Unet network structure includes multiple sets of symmetrical encoders and decoders, each encoder comprising at least one four-input, four-output Swin-transformer module. Each Swin-transformer module contains multiple feature extraction windows and is used for: Receive the output data of each channel in the upper layer encoder as the input data of the same channel in this layer encoder; The input data of each channel is divided into multiple sub-data, and the same feature extraction and feature enhancement processing is performed on the sub-data of different input data in each feature extraction window to obtain the enhanced features corresponding to each sub-data in each feature extraction window; The enhancement features of sub-data belonging to the same channel are spliced ​​together, and the splicing result is output to the corresponding channel in the lower-level encoder. In the feature extraction model, the first sub-data segmented from the first medical image is used as the first input data of the first feature extraction window in the swin-transformer module; the first sub-data segmented from the second medical image is used as the second input data of the first feature extraction window; the first sub-data segmented from the first detection result is used as the third input data of the first feature extraction window; and the first sub-data segmented from the initial lesion matching result is used as the fourth input data of the first feature extraction window. The swin-transformer module is used to obtain the second enhanced feature corresponding to the second input data within the first feature extraction window through the following method: Within the first feature extraction window, feature extraction is performed on the first input data, the second input data, the third input data, and the fourth input data respectively to obtain the first data feature of the first input data, the second data feature of the second input data, the third data feature of the third input data, and the fourth data feature of the fourth input data. Using the Q-feature matrix and K-feature matrix of the second data feature under the attention mechanism, the second self-attention feature of the second data feature under the self-attention mechanism is calculated; Using the Q-feature matrix of the second data feature under the attention mechanism and the K-feature matrix of the first data feature under the attention mechanism, the second mutual attention feature of the second data feature under the mutual attention mechanism is calculated. Using the Q-feature matrix and K-feature matrix of the fourth data feature under the attention mechanism, the fourth self-attention feature of the fourth data feature under the self-attention mechanism is calculated; Using the Q-feature matrix of the fourth data feature under the attention mechanism and the K-feature matrix of the third data feature under the attention mechanism, the fourth mutual attention feature of the fourth data feature under the mutual attention mechanism is calculated. The second enhancement feature is calculated using the second data feature, the second self-attention feature, the second mutual attention feature, the fourth self-attention feature, the fourth mutual attention feature, the V feature matrix of the first data feature under the attention mechanism, and the V feature matrix of the second data feature under the attention mechanism.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the processing method as described in any one of claims 1 to 6.

Images