Image recognition method and system for urinary stones

Abstract

The present application relates to a method and system for image recognition of urinary stones. Before encoding an image of a first modality and an image of a second modality, the method first inputs a first initial feature map extracted from the image of the first modality into a migration network, and migrates the first initial feature map to a second initial feature map extracted from the image of the second modality to obtain a second initial feature map having texture feature information in the first initial feature map. The second initial feature map is then input into the migration network to migrate the second initial feature map to the first initial feature map to obtain a first initial feature map having texture feature information in the second initial feature map, thereby fully realizing the complementarity of information between the image of the first modality and the image of the second modality, thereby improving the accuracy of the image segmentation model in identifying stone areas.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of medical image processing technology, and in particular to a method and system for image recognition of urinary stones. Background Art

[0002] Urinary stones are solid structures that form within the urinary system and can cause serious complications such as urinary tract inflammation and infection. Computer vision-based urinary stone image segmentation technology plays an important role in assisting doctors in treating urinary stones. By using various computer vision technologies, doctors can accurately locate urinary stones, assess their size, number, and morphology, and determine the optimal treatment plan. It can also assist doctors in monitoring the progression of stones. Therefore, computer analysis and processing of urinary stones has practical significance.

[0003] Current research on urinary stone image segmentation technology primarily focuses on specific single modalities, such as computed tomography (CT), positron emission tomography (PET), or magnetic resonance imaging (MRI). However, traditional single-modality images cannot comprehensively and accurately detect and quantify urinary stone lesions. Utilizing the complementary information between images of different modalities can help improve the accuracy of urinary stone identification.

[0004] Currently, there are several ways to combine features between multimodal images:

[0005] 1) Before inputting multimodal images into the image segmentation model, the images of different modalities are first fused into an image of a single modality, but this method results in significant information loss;

[0006] 2) Multimodal images are input into the corresponding image segmentation models separately, and fused after obtaining their respective segmentation results. However, this method does not fuse the image information of different modalities during the image segmentation process, which will lead to ignoring the information correlation between images of different modalities and resulting in poor interpretability. Summary of the Invention

[0007] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0008] The main purpose of the embodiments of the present disclosure is to provide a method and system for image recognition of urinary stones, which can more accurately identify the area of ​​stones in images of different modalities.

[0009] A first aspect of the embodiments of the present application provides an image recognition method for urinary stones, the method comprising the following steps:

[0010] Acquire a first image of a urinary stone to be identified and a second image of a urinary stone to be identified of a target patient; the first image of the urinary stone to be identified is an image of a first modality, and the second image of the urinary stone to be identified is an image of a second modality, and the first modality and the second modality are different;

[0011] Inputting the first urinary stone to-be-identified image and the second urinary stone to-be-identified image into a preset image segmentation model to obtain a urinary stone contour output by the image segmentation model;

[0012] generating a corresponding identification report according to the urinary stone outline;

[0013] The image segmentation model includes a segmentation network and a migration network, and the segmentation network includes an input layer, an encoder-decoder, and an output layer;

[0014] The training process of the image segmentation model is:

[0015] Acquire a first training urinary stone image of the first modality and a second training urinary stone image of the second modality;

[0016] Inputting the first training urinary stone image and the second training urinary stone image into the input layer to obtain a first initial feature map corresponding to the first training urinary stone image and a second initial feature map corresponding to the second training urinary stone image;

[0017] Inputting the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map; and obtaining a first intermediate feature map by migrating the second initial feature map to the first initial feature map;

[0018] Inputting the first intermediate feature map and the second intermediate feature map into the encoder-decoder to obtain a decoded feature map;

[0019] The decoded feature map is input into the output layer to obtain the output urinary stone contour.

[0020] This application proposes a method for image recognition of urinary stones, which has the following beneficial effects:

[0021] Different from the feature fusion technology in existing multimodal images, before encoding the image of the first modality and the image of the second modality, the present application first inputs the first initial feature map extracted from the image of the first modality into the migration network, and migrates the first initial feature map to the second initial feature map extracted from the image of the second modality to obtain the second initial feature map having the texture feature information in the first initial feature map. Then, the second initial feature map is input into the migration network to migrate the second initial feature map to the first initial feature map to obtain the first initial feature map having the texture feature information in the second initial feature map, thereby fully realizing the complementarity of information between the image of the first modality and the image of the second modality, thereby improving the accuracy of the image segmentation model in identifying the stone area.

[0022] In some embodiments of the present application, the codec includes a first encoder and a first decoder, and a second encoder and a second decoder; the first encoder and the second encoder have the same structure, and the first decoder and the second decoder have the same structure;

[0023] The input data of the first encoder is the first intermediate feature map, the input data of the second encoder is the second intermediate feature map, and a fusion feature map between the feature map output by the first decoder and the feature map output by the second decoder is used as the decoded feature map;

[0024] The first encoder includes 6 convolution kernels with 3 3 convolution modules and 3 downsampling modules, the first layer and the last layer of the first encoder are both the convolution modules, and there are two connected convolution modules between each two adjacent downsampling modules, the first decoder includes 6 convolution kernels of 3 3 convolution modules and 3 upsampling modules, the first layer and the last layer of the first decoder are both the convolution modules, and there are two connected convolution modules between every two adjacent upsampling modules;

[0025] The first encoder The input feature map of the jth convolution module is the first fused feature map, and the input feature map of the jth convolution module of the second encoder is the second fused feature map; wherein, the first fused feature map is a feature map fused between the output feature map of the downsampling module that is before and closest to the jth convolution module of the first encoder and the third fused feature map, and the third fused feature map is a feature map fused between the output feature map of the downsampling module that is before and closest to the jth convolution module of the first encoder and the output feature map of the downsampling module that is before and closest to the jth convolution module of the second encoder; the second fused feature map is a feature map fused between the output feature map of the downsampling module that is before and closest to the jth convolution module of the second encoder and the third fused feature map; j is 2, 4, or 6;

[0026] The input feature map of the i-th convolution module of the first encoder is the output feature map of the i-1-th convolution module, and the input feature map of the i-th convolution module of the second encoder is the output feature map of the i-1-th convolution module; i is 3 or 5;

[0027] The input feature map of the j-th convolution module of the first decoder is the fourth fused feature map, and the input feature map of the j-th convolution module of the second decoder is the fifth fused feature map; wherein the fourth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the first decoder and the sixth fused feature map, and the sixth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the first decoder and the output feature map of the upsampling module that is before and closest to the j-th convolution module of the second decoder; the fifth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the second decoder and the sixth fused feature map;

[0028] The input feature map of the i-th convolution module of the first decoder is the output feature map of the i-1-th convolution module, and the input feature map of the i-th convolution module of the second decoder is the output feature map of the i-1-th convolution module.

[0029] In some embodiments of the present application, the output end of at least one of the convolution modules in the first encoder is residually connected to the input end of the convolution module with the same feature map dimension in the first decoder, or the output end of at least one of the convolution modules in the second encoder is residually connected to the input end of the convolution module with the same feature map dimension in the second decoder.

[0030] In some embodiments of the present application, the migration network includes a region segmentation layer, a migration layer, and a fusion layer;

[0031] Inputting the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map, including:

[0032] Inputting the first initial feature map and the second initial feature map into a region segmentation layer to obtain m first segmentation regions from the first initial feature map and n second segmentation regions from the second initial feature map;

[0033] Inputting the m first segmented regions and the n second segmented regions into the migration layer, and using a migration algorithm to perform global migration and local migration from the first initial feature map to the second initial feature map based on the m first segmented regions and the n second segmented regions, to obtain global migration results and local migration results, respectively;

[0034] Inputting the global migration result and the local migration result into the fusion layer to obtain the second intermediate feature map;

[0035] The migration algorithm is used to perform global migration from the first initial feature map to the second initial feature map, and a global migration result is obtained, including:

[0036] Performing pixel-by-pixel multiplication on the first initial feature map and the second initial feature map to obtain a first pixel-by-pixel multiplication result, and inputting the first pixel-by-pixel multiplication result into an AdaIN function to obtain a global migration result;

[0037] The migration algorithm is used to locally migrate the first initial feature map to the second initial feature map to obtain a local migration result, including:

[0038] Finding the first segmented region that is most similar to each second segmented region among the m first segmented regions;

[0039] Calculating pixel-by-pixel dot products between the mask corresponding to each second segmented region and the second initial feature map one by one to obtain a second pixel-by-pixel dot product result, and calculating pixel-by-pixel dot products between the mask corresponding to the first segmented region that is most similar to each second segmented region and the first initial feature map one by one to obtain a third pixel-by-pixel dot product result;

[0040] Inputting the second pixel-by-pixel multiplication result and the third pixel-by-pixel multiplication result into the AdaIN function to obtain a migration result corresponding to each second segmented area;

[0041] The migration results corresponding to the n second segmented regions are fused to obtain a local migration result.

[0042] In some embodiments of the present application, the migration network includes a region segmentation layer, a migration layer, and a fusion layer;

[0043] Inputting the first initial feature map and the second initial feature map into the migration network to obtain a first intermediate feature map by migrating the second initial feature map to the first initial feature map, including:

[0044] Inputting the first initial feature map and the second initial feature map into a region segmentation layer to obtain m first segmentation regions from the first initial feature map and n second segmentation regions from the second initial feature map;

[0045] Inputting the m first segmented regions and the n second segmented regions into the migration layer, and using a migration algorithm to perform global migration and local migration from the second initial feature map to the first initial feature map based on the m first segmented regions and the n second segmented regions, to obtain global migration results and local migration results, respectively;

[0046] Inputting the global migration result and the local migration result into the fusion layer to obtain the first intermediate feature map;

[0047] The migration algorithm is used to perform global migration from the second initial feature map to the first initial feature map to obtain a global migration result, including:

[0048] Performing pixel-by-pixel multiplication on the first initial feature map and the second initial feature map to obtain a first pixel-by-pixel multiplication result, and inputting the first pixel-by-pixel multiplication result into an AdaIN function to obtain a global migration result;

[0049] The migration algorithm is used to locally migrate the second initial feature map to the first initial feature map to obtain a local migration result, including:

[0050] Finding the second segmented region that is most similar to each first segmented region among the n second segmented regions;

[0051] Calculating pixel-by-pixel dot products between the mask corresponding to each of the first segmented regions and the first initial feature map one by one to obtain a second pixel-by-pixel dot product result, and calculating pixel-by-pixel dot products between the mask corresponding to the second segmented region that is most similar to each of the first segmented regions and the second initial feature map one by one to obtain a third pixel-by-pixel dot product result;

[0052] Inputting the second pixel-by-pixel multiplication result and the third pixel-by-pixel multiplication result into an AdaIN function to obtain a migration result corresponding to each of the first segmented regions;

[0053] The migration results corresponding to the m first segmented regions are fused to obtain a local migration result.

[0054] In some embodiments of the present application, the first modality and the second modality are MRI and CT modalities, respectively.

[0055] To achieve the above-mentioned object, a second aspect of an embodiment of the present invention provides an image recognition system for urinary stones, the system comprising the following structure:

[0056] a data acquisition unit, configured to acquire a first image of a target patient's urinary stone to be identified and a second image of a target patient's urinary stone to be identified; the first image of the target patient's urinary stone to be identified is an image of a first modality, and the second image of the target patient's urinary stone to be identified is an image of a second modality, the first modality and the second modality being different;

[0057] a model application unit, configured to input the first image of the urinary stone to be identified and the second image of the urinary stone to be identified into a preset image segmentation model to obtain a urinary stone contour output by the image segmentation model;

[0058] A report generating unit, configured to generate a corresponding identification report according to the urinary stone outline;

[0059] The image segmentation model includes a segmentation network and a migration network, and the segmentation network includes an input layer, an encoder, a decoder, and an output layer;

[0060] The model training unit is used to perform a training process on the image segmentation model. The training process of the image segmentation model is as follows:

[0061] Acquire a first training urinary stone image of the first modality and a second training urinary stone image of the second modality;

[0062] Inputting the first training urinary stone image and the second training urinary stone image into the input layer to obtain a first initial feature map corresponding to the first training urinary stone image and a second initial feature map corresponding to the second training urinary stone image;

[0063] Inputting the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map; and obtaining a first intermediate feature map by migrating the second initial feature map to the first initial feature map;

[0064] Inputting the first intermediate feature map and the second intermediate feature map into the encoder-decoder to obtain a decoded feature map;

[0065] The decoded feature map is input into the output layer to obtain the output urinary stone contour.

[0066] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present invention provides an electronic device, comprising: at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned image recognition method for urinary stones.

[0067] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned urinary stone image recognition method.

[0068] It can be understood that the beneficial effects of the above-mentioned second to fourth aspects compared with the relevant technologies are the same as the beneficial effects of the above-mentioned first aspect compared with the relevant technologies. Please refer to the relevant description in the above-mentioned first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0070] Figure 1 Schematic diagram of the process of the image recognition method for urinary stones provided in an embodiment of the present application;

[0071] Figure 2 is a schematic diagram of the network structure provided in an embodiment of the present application;

[0072] Figure 3 is a schematic diagram of an embodiment of the image recognition system for urinary stones provided by the present application;

[0073] Figure 4 Schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0075] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0077] Urinary stones are solid structures that form within the urinary system and can cause serious complications such as urinary tract inflammation and infection. Computer vision-based urinary stone image segmentation technology plays an important role in assisting doctors in treating urinary stones. By using various computer vision technologies, doctors can accurately locate urinary stones, assess their size, number, and morphology, and determine the optimal treatment plan. It can also assist doctors in monitoring the progression of stones. Therefore, computer analysis and processing of urinary stones has practical significance.

[0078] Current research on urinary stone image segmentation technology primarily focuses on specific single modalities, such as computed tomography (CT), positron emission tomography (PET), or magnetic resonance imaging (MRI). However, traditional single-modality images cannot comprehensively and accurately detect and quantify urinary stone lesions. Utilizing the complementary information between images of different modalities can help improve the accuracy of urinary stone identification.

[0079] Currently, there are several ways to combine features between multimodal images:

[0080] 1) Before inputting multimodal images into the image segmentation model, the images of different modalities are first fused into an image of a single modality, but this method results in significant information loss;

[0081] 2) Multimodal images are input into the corresponding image segmentation models separately, and fused after obtaining their respective segmentation results. However, this method does not fuse the image information of different modalities during the image segmentation process, which will lead to ignoring the information correlation between images of different modalities and resulting in poor interpretability.

[0082] To address defects such as Figure 1 and Figure 2 The embodiment of the present application provides a method for image recognition of urinary stones, the method comprising the following steps:

[0083] Step S110, obtaining a first image of a target patient's urinary stone to be identified and a second image of a target patient's urinary stone to be identified; the first image of the target patient's urinary stone to be identified is an image of a first modality, and the second image of the target patient's urinary stone to be identified is an image of a second modality, and the first modality and the second modality are different;

[0084] Step S120, inputting the first urinary stone to-be-identified image and the second urinary stone to-be-identified image into a preset image segmentation model to obtain the urinary stone contour output by the image segmentation model;

[0085] Step S130, generating a corresponding identification report according to the urinary stone outline;

[0086] In step S110 of this embodiment, the target patient is a patient diagnosed by a doctor as having urinary stones. It should be noted that the doctor's diagnosis of the patient as having urinary stones is based on the hospital's medical methods and experience. The recognition in this application is only used to assist the doctor in locating urinary stones in medical images.

[0087] The first modality and the second modality are any one of CT, MRI, and PET, and can also be variant images of CT, MRI, or PET. For example, the image of the first modality is a CT image, and the image of the second modality is an MRI.

[0088] Images of different modalities contain complementary information. Therefore, in step S120 of this embodiment, the first image of the urinary stone to be identified and the second image of the urinary stone to be identified are input into a preset image segmentation model to obtain the urinary stone contour output by the image segmentation model. It should be noted that the image segmentation model can output the urinary stone contour corresponding to the first image of the urinary stone to be identified, and can also output the urinary stone contour corresponding to the second image of the urinary stone to be identified. During the image segmentation process, there will be an interaction process of feature information between the first image of the urinary stone to be identified and the second image of the urinary stone to be identified. For details, please refer to the subsequent introduction of the model training process.

[0089] In step S130, a corresponding identification report is generated according to the urinary stone outline, and the report includes but is not limited to content related to the size, location, number and morphology of the urinary stones.

[0090] The image segmentation model includes a segmentation network and a migration network. The segmentation network includes an input layer, an encoder-decoder, and an output layer.

[0091] The training process of the image segmentation model is:

[0092] Step S210, acquiring a first training urinary stone image of a first modality and a second training urinary stone image of a second modality;

[0093] Step S220, inputting the first training urinary stone image and the second training urinary stone image into the input layer to obtain a first initial feature map corresponding to the first training urinary stone image and a second initial feature map corresponding to the second training urinary stone image;

[0094] Step S230: Input the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map; and obtain a first intermediate feature map by migrating the second initial feature map to the first initial feature map;

[0095] Step S240, inputting the first intermediate feature map and the second intermediate feature map into the encoder-decoder to obtain a decoded feature map;

[0096] Step S250: input the decoded feature map into the output layer to obtain the output urinary stone contour.

[0097] The output of the encoder serves as the input of the decoder. The main function of the input layer is to extract the feature map; the main function of the output layer is to segment the contours of urinary stones.

[0098] In step S210, the first training urinary stone image of the first modality and the second training urinary stone image of the second modality can be derived from a hospital database. For example, by extracting the first and second modalities of the same patient's training urinary stone images from the database, M groups of first and second training urinary stone images of M patients are extracted as a training set. This is not described in detail here. It should be noted that the image preprocessing process is not described in detail here. The preprocessing can be performed based on experience, but since it is not the focus of this application, it will not be described in detail here.

[0099] In step S220, the input layer is a convolutional layer.

[0100] The first training urinary stone image is input into the input layer, and a first initial feature map corresponding to the first training urinary stone image output by the input layer is obtained.

[0101] The second training urinary stone image is input into the input layer to obtain a second initial feature map corresponding to the first training urinary stone image output by the input layer.

[0102] In step S230, the first initial feature map is input into the migration network, and then the first initial feature map is migrated to the second initial feature map to obtain the second initial feature map having the texture feature information in the first initial feature map; the second initial feature map is input into the migration network, so as to migrate the second initial feature map to the first initial feature map, that is, the first initial feature map having the texture feature information in the second initial feature map.

[0103] In step S240, the first intermediate feature map and the second intermediate feature map are input into the encoder-decoder to obtain a decoded feature map. Then, the decoded feature map is input into the output layer to obtain the output urinary stone contour.

[0104] It should be noted that when the training process is clear, the process of applying the model to the image is similar to the feature processing logic of the training process and will not be repeated here.

[0105] Unlike the feature fusion technology in existing multimodal images, before encoding the image of the first modality and the image of the second modality, the present application first inputs the first initial feature map extracted from the image of the first modality into the migration network, and migrates the first initial feature map to the second initial feature map extracted from the image of the second modality to obtain the second initial feature map (i.e., the second intermediate feature map) having the texture feature information in the first initial feature map. The second initial feature map is then input into the migration network to migrate the second initial feature map to the first initial feature map to obtain the first initial feature map (i.e., the first intermediate feature map) having the texture feature information in the second initial feature map, thereby fully realizing the complementarity of information between the image of the first modality and the image of the second modality, thereby improving the accuracy of the image segmentation model in identifying the stone area.

[0106] In some embodiments of the present application, the migration network includes a region segmentation layer, a migration layer, and a fusion layer;

[0107] In step S230 of this embodiment, the first initial feature map and the second initial feature map are input into the migration network to obtain the second intermediate feature map by migrating the first initial feature map to the second initial feature map, including the following steps S310 to S330:

[0108] Step S310: Input the first initial feature map and the second initial feature map into the region segmentation layer to obtain m first segmentation regions from the first initial feature map and n second segmentation regions from the second initial feature map;

[0109] Step S320: Input the m first segmented regions and the n second segmented regions into a migration layer, and use a migration algorithm to perform global migration and local migration from the first initial feature map to the second initial feature map based on the m first segmented regions and the n second segmented regions, thereby obtaining global migration results and local migration results respectively.

[0110] Step S330: input the global migration result and the local migration result into the fusion layer to obtain a second intermediate feature map;

[0111] The migration algorithm is used to perform global migration from the first initial feature map to the second initial feature map, and the global migration results are obtained, including:

[0112] The first initial feature map and the second initial feature map are multiplied pixel by pixel to obtain a first pixel-by-pixel multiplication result, and the first pixel-by-pixel multiplication result is input into the AdaIN function to obtain a global migration result.

[0113] The migration algorithm is used to perform local migration from the first initial feature map to the second initial feature map, and the local migration results are obtained, including:

[0114] Find the first segmented region that is most similar to each second segmented region among the m first segmented regions;

[0115] Calculate the pixel-by-pixel multiplication between the mask corresponding to each second segmented region and the second initial feature map one by one to obtain a second pixel-by-pixel multiplication result, and calculate the pixel-by-pixel multiplication between the mask corresponding to the most similar first segmented region corresponding to each second segmented region and the first initial feature map one by one to obtain a third pixel-by-pixel multiplication result;

[0116] Input the second pixel-by-pixel multiplication result and the third pixel-by-pixel multiplication result into the AdaIN function to obtain the migration result corresponding to each second segmented area;

[0117] The migration results corresponding to the n second segmentation regions are fused to obtain a local migration result.

[0118] First, let's introduce the transfer network. The output of the segmentation network's input layer serves as the input to the transfer network, i.e., the region segmentation layer. The region segmentation layer primarily extracts the first segmented region from the first initial feature map. Note that this is related to the preset labels, as region labeling is a fundamental operation on training data in this field and will not be detailed here. The region segmentation layer also extracts the second segmented region from the second initial feature map.

[0119] The purposes of the migration layer include:

[0120] 1) Migrating features from the first initial feature map to the second initial feature map;

[0121] 2) Migrating the features in the second initial feature map to the first initial feature map;

[0122] In this embodiment, the processes 1) and 2) both include global migration and local migration. Here, local migration is added on top of global migration to enhance the consistency between local parts and improve the image segmentation effect.

[0123] The purpose of the fusion layer is to fuse the local migration results with the global migration results.

[0124] In step S310 of this embodiment, m first segmented regions are obtained from the first initial feature map and n second segmented regions are obtained from the second initial feature map according to the preset labels, where the number nm is determined by the labels.

[0125] It should be noted that the first segmented area is equal to the pixel-by-pixel product between the corresponding mask and the first initial feature map, and the second segmented area is equal to the pixel-by-pixel product between the corresponding mask and the second initial feature map.

[0126] In step S320 of this embodiment, the m first segmented regions and the n second segmented regions are input to the migration layer, so that a migration algorithm is used to perform global migration from the first initial feature map to the second initial feature map based on the m first segmented regions and the n second segmented regions.

[0127] The migration algorithm is used to perform global migration from the first initial feature map to the second initial feature map, and the global migration result is obtained, including:

[0128] Perform pixel-by-pixel multiplication on the first initial feature map and the second initial feature map to obtain a first pixel-by-pixel multiplication result, and input the first pixel-by-pixel multiplication result into the AdaIN function to obtain a global migration result;

[0129] AdaIN is a new adaptive instance normalization (AdaIN) layer that aligns the mean and variance of content features with the mean and variance of style features, such as the first and second initial feature maps of content features, respectively, depending on the object being transferred.

[0130] Global migration refers to performing pixel-by-pixel multiplication on the first initial feature map and the second initial feature map, and then inputting them into the AdaIN function to obtain a result.

[0131] The migration algorithm is used to perform local migration from the first initial feature map to the second initial feature map, and the local migration result is obtained, including:

[0132] Find the first segmented region that is most similar to each second segmented region among the m first segmented regions;

[0133] Calculate the pixel-by-pixel multiplication between the mask corresponding to each second segmented region and the second initial feature map one by one to obtain a second pixel-by-pixel multiplication result, and calculate the pixel-by-pixel multiplication between the mask corresponding to the most similar first segmented region corresponding to each second segmented region and the first initial feature map one by one to obtain a third pixel-by-pixel multiplication result;

[0134] Input the second pixel-by-pixel multiplication result and the third pixel-by-pixel multiplication result into the AdaIN function to obtain the migration result corresponding to each second segmented area;

[0135] The migration results corresponding to the n second segmentation regions are fused to obtain a local migration result.

[0136] Taking the jth second segmentation area among n second segmentation areas as an example, it is necessary to find the first segmentation area that is most similar to the jth second segmentation area among the m first segmentation areas. The purpose of finding this most similar area is to perform local migration between the jth second segmentation area and the corresponding first segmentation area.

[0137] In order to find the j-th second segmentation region corresponding to a first segmentation region, a pre-trained model and preset labels can be used to find the most similar region between the two modalities.

[0138] Then, the pixel-by-pixel multiplication between the mask corresponding to each second segmentation area and the second initial feature map is calculated one by one to obtain a second pixel-by-pixel multiplication result, and the pixel-by-pixel multiplication between the mask corresponding to the most similar first segmentation area and the first initial feature map is calculated one by one to obtain a third pixel-by-pixel multiplication result; the second pixel-by-pixel multiplication result and the third pixel-by-pixel multiplication result are input into the AdaIN function to obtain the migration result corresponding to each second segmentation area; the migration results corresponding to n second segmentation areas are fused to obtain a local migration result.

[0139] In some embodiments of the present application, the migration network includes a region segmentation layer, a migration layer, and a fusion layer;

[0140] In step S230, the first initial feature map and the second initial feature map are input into the migration network to obtain a first intermediate feature map by migrating the second initial feature map to the first initial feature map, including the following steps S410 to S430:

[0141] Step S410: Input the first initial feature map and the second initial feature map into a region segmentation layer to obtain m first segmentation regions from the first initial feature map and n second segmentation regions from the second initial feature map;

[0142] Step S420: Input the m first segmented regions and the n second segmented regions into a migration layer, and use a migration algorithm to perform global migration and local migration from the second initial feature map to the first initial feature map based on the m first segmented regions and the n second segmented regions, thereby obtaining global migration results and local migration results respectively.

[0143] Step S430: input the global migration result and the local migration result into the fusion layer to obtain a first intermediate feature map;

[0144] The migration algorithm is used to perform global migration from the second initial feature map to the first initial feature map, and the global migration results are obtained, including:

[0145] Perform pixel-by-pixel multiplication on the first initial feature map and the second initial feature map to obtain a first pixel-by-pixel multiplication result, and input the first pixel-by-pixel multiplication result into the AdaIN function to obtain a global migration result;

[0146] The migration algorithm is used to locally migrate the second initial feature map to the first initial feature map, and the local migration results are obtained, including:

[0147] Find the second segmented region that is most similar to each first segmented region among the n second segmented regions;

[0148] Calculate the pixel-by-pixel multiplication between the mask corresponding to each first segmented area and the first initial feature map one by one to obtain a second pixel-by-pixel multiplication result, and calculate the pixel-by-pixel multiplication between the mask corresponding to the second segmented area most similar to each first segmented area and the second initial feature map one by one to obtain a third pixel-by-pixel multiplication result;

[0149] Input the second pixel-by-pixel multiplication result and the third pixel-by-pixel multiplication result into the AdaIN function to obtain the migration result corresponding to each first segmented area;

[0150] The migration results corresponding to the m first segmented regions are fused to obtain a local migration result.

[0151] It should be noted that this embodiment is similar to the above embodiment. The process of migrating the second initial feature map to the first initial feature map to obtain the first intermediate feature map introduced in this embodiment is similar to the above-mentioned process of migrating the first initial feature map to the second initial feature map, and will not be described in detail here.

[0152] This embodiment innovatively uses a migration algorithm compared to the above embodiments. While still retaining the two recognition routes of separately recognizing the image of the first modality and the image of the second modality, before encoding the image of the first modality and the image of the second modality, global migration and local migration are used in the migration network in the process of migrating the first initial feature map to the second initial feature map. By fusing the global information and the local information, the second initial feature map can have the global information of the first initial feature map, and the second initial feature map can have the local information of the first initial feature map, thereby enhancing the consistency between the local information and realizing the image of different modalities. The texture features of the details between the images of different modalities are supplemented; through the migration network, global migration and local migration are used in the process of migrating the second initial feature map to the first initial feature map. Through the fusion of global information and local information, the first initial feature map can have the global information of the second initial feature map, and the first initial feature map can have the local information of the second initial feature map, thereby enhancing the consistency between the local information and realizing the supplement of the texture features of the details between images of different modalities; fully realizing the complementarity of information between the images of the first modality and the images of the second modality, thereby improving the accuracy of the image segmentation models in identifying the stone areas of the images of different modalities.

[0153] like Figure 2 As shown, in some embodiments, the codec includes a first encoder and a first decoder, and a second encoder and a second decoder; the first encoder and the second encoder have the same structure, and the first decoder and the second decoder have the same structure;

[0154] The fused feature map between the feature map output by the first decoder and the feature map output by the second decoder is used as the decoded feature map.

[0155] The first encoder includes 6 convolution kernels with a value of 3 3 convolution modules and 3 downsampling modules (the figure shows the downsampling layer or upsampling layer). The first layer and the last layer of the first encoder are both convolution modules, and there are two connected convolution modules between each two adjacent downsampling modules. The first decoder includes 6 convolution kernels with a value of 3. 3 convolution modules and 3 upsampling modules. The first and last layers of the first decoder are both convolution modules, and there are two connected convolution modules between every two adjacent upsampling modules.

[0156] The first encoder The input feature map of the jth convolution module is the first fused feature map, and the input feature map of the jth convolution module of the second encoder is the second fused feature map; wherein, the first fused feature map is a feature map fused between the output feature map of a downsampling module that is before and closest to the jth convolution module of the first encoder and the third fused feature map, and the third fused feature map is a feature map fused between the output feature map of a downsampling module that is before and closest to the jth convolution module of the first encoder and the output feature map of a downsampling module that is before and closest to the jth convolution module of the second encoder; the second fused feature map is a feature map fused between the output feature map of a downsampling module that is before and closest to the jth convolution module of the second encoder and the third fused feature map; j is 2, 4, or 6;

[0157] The input feature map of the i-th convolution module of the first encoder is the output feature map of the i-1-th convolution module, and the input feature map of the i-th convolution module of the second encoder is the output feature map of the i-1-th convolution module; i is 3 or 5;

[0158] The input feature map of the j-th convolution module of the first decoder is the fourth fused feature map, and the input feature map of the j-th convolution module of the second decoder is the fifth fused feature map; wherein, the fourth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the first decoder and the sixth fused feature map, and the sixth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the first decoder and the output feature map of the upsampling module that is before and closest to the j-th convolution module of the second decoder; the fifth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the second decoder and the sixth fused feature map;

[0159] The input feature map of the i-th convolution module of the first decoder is the output feature map of the i-1-th convolution module, and the input feature map of the i-th convolution module of the second decoder is the output feature map of the i-1-th convolution module.

[0160] The structure of this embodiment includes two identical structures, namely, a first encoder-first decoder and a second encoder-second decoder, wherein the first encoder or the second encoder extracts features through convolution and pooling operations, and the first decoder restores image resolution through upsampling and convolution operations. In this embodiment, the two sets of encoding-decoding structures can simultaneously realize the separate feature learning of two different modality images of CT and MRI, and the two sets of encoding-decoding structures again realize information sharing and interaction (i.e., the first intermediate feature map and the second intermediate feature map after migration), which can further improve the learning of the information sharing and interaction between the second initial feature map (i.e., the second intermediate feature map) having texture feature information in the first initial feature map and the first initial feature map (i.e., the first intermediate feature map) having texture feature information in the second initial feature map, thereby improving the accuracy of stone area recognition.

[0161] In some embodiments, the output of at least one convolutional module in the first encoder is residually connected to the input of a convolutional module with the same feature map dimensions in the first decoder, or the output of at least one convolutional module in the second encoder is residually connected to the input of a convolutional module with the same feature map dimensions in the second decoder. The feature map of the encoder and the feature map of the decoder are spliced ​​together through a skip connection, which not only preserves detail information but also incorporates contextual information, helping to restore image details and improve segmentation accuracy.

[0162] It should be noted that the image segmentation model is completed after backpropagation based on the loss function until convergence. This is common knowledge in the field and will not be repeated here.

[0163] In this embodiment, in order to improve the accuracy of the model in identifying stones, this embodiment designs the loss function of the segmentation network:

[0164] The loss function of the segmentation network can be:

[0165]

[0166] in, represents the loss function; Indicates similarity; represents the i-th first intermediate feature map sample; represents the jth second intermediate feature map sample; Represents a set of similar sample pairs; Represents a set of dissimilar sample pairs; and It is determined during data acquisition, that is, the acquisition process of the first training urinary stone image and the second training urinary stone image.

[0167] The following describes the process of obtaining similarity:

[0168] (1) dividing the first intermediate feature map and the second intermediate feature map extracted from the first training urinary stone image of the first modality and the second training urinary stone image of the second modality into different groups of sample pairs;

[0169] Because there are multiple first training urinary stone images and multiple second training urinary stone images, multiple first intermediate feature maps and second intermediate feature maps will be obtained, so they can be divided into multiple groups. One group includes a first intermediate feature map and a second intermediate feature map. Assuming there are m first intermediate feature maps and n second intermediate feature maps, there are mn groups.

[0170] (2) Calculate the similarity of each pair of samples;

[0171] The calculation process includes:

[0172] 1)

[0173] in, is a feature map formed by concatenating the i-th first intermediate feature map sample and the j-th second intermediate feature map sample; is the weight value; is the bias term; is a non-linear activation function.

[0174] 2)

[0175] in, for similarity between is the weight value; is the bias term, is the Sigmoid activation function.

[0176] This embodiment uses Characterization A collection of .

[0177] The benefits of this processing are:

[0178] The segmentation network can be constrained to make the feature maps of similar samples closer and the feature maps of dissimilar samples more separated, thereby improving the fusion effect of the segmentation network on the feature maps.

[0179] This embodiment can be applied to the following scenarios:

[0180] The target patient was diagnosed with urinary stones and was advised to undergo regular monitoring. The patient underwent MRI and CT scans of the stones at different hospitals.

[0181] In order to assist in the treatment of patients, doctors use the image segmentation model provided in this application to identify the contours of the urinary stone area in MRI images and CT images respectively, and identify the contours of the urinary stone area in the MRI image and the contours of the urinary stone area in the CT image; in the process of image processing, the information complementarity of the two modal images is achieved, that is, when processing one modality image, the information in the other modality image will be migrated in, so that when processing one modality, the information of its own modality and the information of other modalities can be extracted, thereby realizing the interaction of feature information.

[0182] Finally, an MRI or CT urinary stone report is generated.

[0183] like Figure 3 As shown, one embodiment of the present application provides a method and system for image recognition of urinary stones, the system including the following structure:

[0184] The data acquisition unit 1100 is configured to acquire a first image of a target patient's urinary stone to be identified and a second image of a target patient's urinary stone to be identified; the first image of the target patient's urinary stone to be identified is an image of a first modality, and the second image of the target patient's urinary stone to be identified is an image of a second modality, and the first modality and the second modality are different;

[0185] The model application unit 1200 is used to input the first urinary stone to be identified image and the second urinary stone to be identified image into a preset image segmentation model to obtain the urinary stone contour output by the image segmentation model;

[0186] The report generating unit 1300 is used to generate a corresponding identification report according to the urinary stone outline;

[0187] Among them, the image segmentation model includes a segmentation network and a migration network. The segmentation network includes an input layer, an encoder, a decoder, and an output layer;

[0188] The model training unit 1400 is used to train the image segmentation model. The training process of the image segmentation model is as follows:

[0189] Acquire a first training urinary stone image of a first modality and a second training urinary stone image of a second modality;

[0190] Inputting the first training urinary stone image and the second training urinary stone image into the input layer, obtaining a first initial feature map corresponding to the first training urinary stone image and a second initial feature map corresponding to the second training urinary stone image;

[0191] Inputting the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map; and obtaining a first intermediate feature map by migrating the second initial feature map to the first initial feature map;

[0192] Inputting the first intermediate feature map and the second intermediate feature map into an encoder-decoder to obtain a decoded feature map;

[0193] The decoded feature map is input into the output layer to obtain the output urinary stone contour.

[0194] It should be noted that the urinary stone image recognition system provided in this embodiment and the above-mentioned urinary stone image recognition method are based on the same inventive concept. Therefore, the relevant content of the above-mentioned urinary stone image recognition method is also applicable to the content of the urinary stone image recognition system. Therefore, it will not be repeated here.

[0195] like Figure 4 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0196] at least one memory;

[0197] at least one processor;

[0198] at least one program;

[0199] The programs are stored in the memory, and the processor executes at least one program to implement the image recognition method for urinary stones described above.

[0200] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0201] The electronic device according to the embodiment of the present application is described in detail below.

[0202] The processor 1600 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0203] Memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in memory 1700 and is called by processor 1600 to execute the urinary stone image recognition method of the embodiment of the present invention.

[0204] Input / output interface 1800, used for information input and output;

[0205] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0206] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0207] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0208] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned urinary stone image recognition method.

[0209] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0210] The embodiments described in the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0211] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0212] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0213] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0214] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0215] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0216] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0217] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0218] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0219] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0220] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A method for image recognition of urinary stones, characterized in that: The method comprises the following steps: Acquire a first image of a urinary stone to be identified and a second image of a urinary stone to be identified of a target patient; the first image of the urinary stone to be identified is an image of a first modality, the second image of the urinary stone to be identified is an image of a second modality, and the first modality and the second modality are different; Inputting the first urinary stone to-be-identified image and the second urinary stone to-be-identified image into a preset image segmentation model to obtain a urinary stone contour output by the image segmentation model; generating a corresponding identification report according to the urinary stone outline; The image segmentation model includes a segmentation network and a migration network, and the segmentation network includes an input layer, an encoder-decoder, and an output layer; The training process of the image segmentation model is: Acquire a first training urinary stone image of the first modality and a second training urinary stone image of the second modality; Inputting the first training urinary stone image and the second training urinary stone image into the input layer to obtain a first initial feature map corresponding to the first training urinary stone image and a second initial feature map corresponding to the second training urinary stone image; Inputting the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map; and obtaining a first intermediate feature map by migrating the second initial feature map to the first initial feature map; the migration network includes a region segmentation layer, a migration layer, and a fusion layer; Inputting the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map, including: Inputting the first initial feature map and the second initial feature map into a region segmentation layer to obtain m first segmentation regions from the first initial feature map and n second segmentation regions from the second initial feature map; Inputting the m first segmented regions and the n second segmented regions into the migration layer, and using a migration algorithm to perform global migration and local migration from the first initial feature map to the second initial feature map based on the m first segmented regions and the n second segmented regions, respectively obtaining a global migration result and a local migration result corresponding to the second initial feature map, and inputting the global migration result and the local migration result corresponding to the second initial feature map into the fusion layer to obtain the second intermediate feature map; Inputting the first initial feature map and the second initial feature map into the migration network to obtain a first intermediate feature map by migrating the second initial feature map to the first initial feature map, including: Inputting the m first segmented regions and the n second segmented regions into the migration layer, and using a migration algorithm to perform global migration and local migration from the second initial feature map to the first initial feature map based on the m first segmented regions and the n second segmented regions, respectively obtaining global migration results and local migration results corresponding to the first initial feature map, and inputting the global migration results and local migration results corresponding to the first initial feature map into the fusion layer to obtain the first intermediate feature map; Inputting the first intermediate feature map and the second intermediate feature map into the encoder-decoder to obtain a decoded feature map; The decoded feature map is input into the output layer to obtain the output urinary stone contour.

2. The urinary stone image recognition method and system according to claim 1, characterized in that: The input layer and the output layer are both convolutional layers.

3. The urinary stone image recognition method and system according to claim 2, characterized in that: The codec includes a first encoder and a first decoder, and a second encoder and a second decoder; the first encoder and the second encoder have the same structure, and the first decoder and the second decoder have the same structure; The input data of the first encoder is the first intermediate feature map, the input data of the second encoder is the second intermediate feature map, and a fusion feature map between the feature map output by the first decoder and the feature map output by the second decoder is used as the decoded feature map; The first encoder includes 6 convolution kernels with 3 3 convolution modules and 3 downsampling modules, the first layer and the last layer of the first encoder are both the convolution modules, and there are two connected convolution modules between each two adjacent downsampling modules, the first decoder includes 6 convolution kernels of 3 3 convolution modules and 3 upsampling modules, the first layer and the last layer of the first decoder are both the convolution modules, and there are two connected convolution modules between every two adjacent upsampling modules; The first encoder The input feature map of the jth convolution module is the first fused feature map, and the input feature map of the jth convolution module of the second encoder is the second fused feature map; wherein, the first fused feature map is a feature map fused between the output feature map of the downsampling module that is before and closest to the jth convolution module of the first encoder and the third fused feature map, and the third fused feature map is a feature map fused between the output feature map of the downsampling module that is before and closest to the jth convolution module of the first encoder and the output feature map of the downsampling module that is before and closest to the jth convolution module of the second encoder; the second fused feature map is a feature map fused between the output feature map of the downsampling module that is before and closest to the jth convolution module of the second encoder and the third fused feature map; j is 2, 4, or 6; The input feature map of the i-th convolution module of the first encoder is the output feature map of the i-1-th convolution module, and the input feature map of the i-th convolution module of the second encoder is the output feature map of the i-1-th convolution module; i is 3 or 5; The input feature map of the j-th convolution module of the first decoder is the fourth fused feature map, and the input feature map of the j-th convolution module of the second decoder is the fifth fused feature map; wherein the fourth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the first decoder and the sixth fused feature map, and the sixth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the first decoder and the output feature map of the upsampling module that is before and closest to the j-th convolution module of the second decoder; the fifth fused feature map is a feature map fused between the output feature map of the upsampling module that is before and closest to the j-th convolution module of the second decoder and the sixth fused feature map; The input feature map of the i-th convolution module of the first decoder is the output feature map of the i-1-th convolution module, and the input feature map of the i-th convolution module of the second decoder is the output feature map of the i-1-th convolution module.

4. The urinary stone image recognition method and system according to claim 3, characterized in that: The output end of at least one of the convolution modules in the first encoder is residually connected to the input end of the convolution module with the same feature map dimension in the first decoder, or the output end of at least one of the convolution modules in the second encoder is residually connected to the input end of the convolution module with the same feature map dimension in the second decoder.

5. The urinary stone image recognition method and system according to claim 1, characterized in that: A migration algorithm is used to perform global migration from the first initial feature map to the second initial feature map, to obtain a global migration result, including: Performing pixel-by-pixel multiplication on the first initial feature map and the second initial feature map to obtain a first pixel-by-pixel multiplication result, and inputting the first pixel-by-pixel multiplication result into an AdaIN function to obtain a global migration result; The migration algorithm is used to locally migrate the first initial feature map to the second initial feature map to obtain a local migration result, including: Finding the first segmented region that is most similar to each second segmented region among the m first segmented regions; Calculating pixel-by-pixel dot products between the mask corresponding to each second segmented region and the second initial feature map one by one to obtain a second pixel-by-pixel dot product result, and calculating pixel-by-pixel dot products between the mask corresponding to the first segmented region that is most similar to each second segmented region and the first initial feature map one by one to obtain a third pixel-by-pixel dot product result; Inputting the second pixel-by-pixel multiplication result and the third pixel-by-pixel multiplication result into the AdaIN function to obtain a migration result corresponding to each second segmented area; The migration results corresponding to the n second segmented regions are fused to obtain a local migration result.

6. The urinary stone image recognition method and system according to claim 1, characterized in that: A migration algorithm is used to perform global migration from the second initial feature map to the first initial feature map, to obtain a global migration result, including: Performing pixel-by-pixel multiplication on the first initial feature map and the second initial feature map to obtain a first pixel-by-pixel multiplication result, and inputting the first pixel-by-pixel multiplication result into an AdaIN function to obtain a global migration result; The migration algorithm is used to locally migrate the second initial feature map to the first initial feature map to obtain a local migration result, including: Finding the second segmented region that is most similar to each first segmented region among the n second segmented regions; Calculating pixel-by-pixel dot products between the mask corresponding to each of the first segmented regions and the first initial feature map one by one to obtain a second pixel-by-pixel dot product result, and calculating pixel-by-pixel dot products between the mask corresponding to the second segmented region that is most similar to each of the first segmented regions and the second initial feature map one by one to obtain a third pixel-by-pixel dot product result; Inputting the second pixel-by-pixel multiplication result and the third pixel-by-pixel multiplication result into an AdaIN function to obtain a migration result corresponding to each of the first segmented regions; The migration results corresponding to the m first segmented regions are fused to obtain a local migration result.

7. The image recognition method for urinary stones according to claim 1, characterized in that: The first modality and the second modality are MRI and CT modalities, respectively.

8. A urinary stone image recognition method system, characterized in that: The system includes the following structure: a data acquisition unit, configured to acquire a first image of a target patient's urinary stone to be identified and a second image of a target patient's urinary stone to be identified; the first image of the target patient's urinary stone to be identified is an image of a first modality, and the second image of the target patient's urinary stone to be identified is an image of a second modality, the first modality and the second modality being different; a model application unit, configured to input the first image of the urinary stone to be identified and the second image of the urinary stone to be identified into a preset image segmentation model to obtain a urinary stone contour output by the image segmentation model; A report generating unit, configured to generate a corresponding identification report according to the urinary stone outline; The image segmentation model includes a segmentation network and a migration network, and the segmentation network includes an input layer, an encoder-decoder, and an output layer; The model training unit is used to perform a training process on the image segmentation model. The training process of the image segmentation model is as follows: Acquire a first training urinary stone image of the first modality and a second training urinary stone image of the second modality; Inputting the first training urinary stone image and the second training urinary stone image into the input layer to obtain a first initial feature map corresponding to the first training urinary stone image and a second initial feature map corresponding to the second training urinary stone image; Inputting the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map; and obtaining a first intermediate feature map by migrating the second initial feature map to the first initial feature map; the migration network includes a region segmentation layer, a migration layer, and a fusion layer; Inputting the first initial feature map and the second initial feature map into the migration network to obtain a second intermediate feature map by migrating the first initial feature map to the second initial feature map, including: Inputting the first initial feature map and the second initial feature map into a region segmentation layer to obtain m first segmentation regions from the first initial feature map and n second segmentation regions from the second initial feature map; Inputting the m first segmented regions and the n second segmented regions into the migration layer, and using a migration algorithm to perform global migration and local migration from the first initial feature map to the second initial feature map based on the m first segmented regions and the n second segmented regions, respectively obtaining a global migration result and a local migration result corresponding to the second initial feature map, and inputting the global migration result and the local migration result corresponding to the second initial feature map into the fusion layer to obtain the second intermediate feature map; Inputting the first initial feature map and the second initial feature map into the migration network to obtain a first intermediate feature map by migrating the second initial feature map to the first initial feature map, including: Inputting the m first segmented regions and the n second segmented regions into the migration layer, and using a migration algorithm to perform global migration and local migration from the second initial feature map to the first initial feature map based on the m first segmented regions and the n second segmented regions, respectively obtaining global migration results and local migration results corresponding to the first initial feature map, and inputting the global migration results and local migration results corresponding to the first initial feature map into the fusion layer to obtain the first intermediate feature map; Inputting the first intermediate feature map and the second intermediate feature map into the encoder-decoder to obtain a decoded feature map; The decoded feature map is input into the output layer to obtain the output urinary stone contour.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the image recognition method for urinary stones according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the urinary stone image recognition method according to any one of claims 1 to 7.

Images