Medical image recognition methods, medical model training methods, devices and equipment

Abstract

This application relates to a medical image recognition method, a medical model training method, an apparatus, and a computer device. The method includes: acquiring a medical image to be recognized; performing multi-size feature extraction on the current input feature map to obtain a current original-size medical feature map and current medical feature maps corresponding to each target size; fusing the current original-size medical feature map and the current medical feature maps corresponding to each target size to obtain a current fused feature map; convolving the current fused feature map to obtain a current target feature map; merging the current original-size medical feature map and the current target feature map to obtain a next input feature map; and using the next input feature map as the current input feature map to obtain a medical target prediction result corresponding to the medical image. This method can improve the accuracy of medical image recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a medical image recognition method, a medical model training method, an apparatus, and a computer device. Background Technology

[0002] Currently, deep learning methods are used to segment tissue and organ regions in MRI images, marking the approximate outlines of tissues and organs in cross-sections of patient MRI image slices, which can achieve automatic segmentation of normal tissue and organ shapes.

[0003] However, the MRI images taken by patients vary greatly in shape and size, and may even show deformed tissues and organs. For example, in patients with hyperplasia, the tissues and organs may appear sharp, forked, and protrude into the bladder. Deep learning trains models using a data-driven approach, and these images of abnormal tissues and organs can make it difficult for the model to converge during training, affecting the accurate identification of the contours of abnormal areas. Summary of the Invention

[0004] Therefore, it is necessary to provide a medical image recognition method, medical model training method, device, and computer equipment that can improve the accuracy of medical image recognition in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a medical image recognition method, the method comprising:

[0006] Multi-size feature extraction is performed on the current input feature map to obtain the current original size medical feature map and the current medical feature maps corresponding to each target size. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map, wherein the initial input feature map is the medical image to be identified.

[0007] The current fused feature map is convolved to obtain the current target feature map. The current original-size medical feature map and the current target feature map are then merged to obtain the next input feature map.

[0008] The next input feature map is used as the current input feature map, and multi-size feature extraction is performed on the current input feature map to obtain the current original size medical feature map and the current medical feature map corresponding to each target size, so as to obtain the location region of the medical target prediction result corresponding to the medical image.

[0009] In one embodiment, the multi-size feature extraction of the current input feature map, including the current original-size medical feature map and the current medical feature maps corresponding to each target size, includes:

[0010] The current input feature map is downsampled to obtain downsampled feature maps of different sizes;

[0011] Convolution is performed on the downsampled feature maps of different sizes to obtain first feature maps of different sizes, and convolution is performed on the feature maps of the original size to obtain second feature maps;

[0012] The first feature map is upsampled step by step to obtain upsampled feature maps of different sizes, wherein the second number is less than the first number;

[0013] After merging the upsampled feature map with the first feature map and the second feature map of the corresponding size, the merged feature map is convolved to obtain the third feature map, wherein the third feature map of the original size is the current original size medical feature map;

[0014] Upsampling is performed on the third feature map that is not of the original size to obtain the current medical feature map of the original size corresponding to each target size.

[0015] In one embodiment, the medical target prediction result is obtained by predicting using a pre-trained medical model; wherein the medical model includes a multi-size feature extraction and fusion network and a skip connection network.

[0016] The process of extracting multi-size features from the current input feature map, including the current original-size medical feature map and the current medical feature maps corresponding to each target size, and fusing the current original-size medical feature map and the current medical feature maps corresponding to each target size to obtain the current fused feature map, includes:

[0017] The multi-size feature extraction and fusion network is used to extract multi-size features from the current input feature map, including the current original size medical feature map and the current medical feature maps corresponding to each target size. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map.

[0018] The process of convolving the current fused feature map to obtain the current target feature map, merging the current original-size medical feature map and the current target feature map to obtain the next input feature map includes:

[0019] The current target feature map is obtained by convolving the current fused feature map through the skip connection network. The current original-size medical feature map and the current target feature map are then merged to obtain the next input feature map.

[0020] In one embodiment, the multi-size feature extraction and fusion network includes a downsampling module, a first convolutional module, a first upsampling module, a merging module, a second convolutional module, a second upsampling module, and a fusion module;

[0021] The process of extracting multi-size features from the current input feature map, including the current original-size medical feature map and the current medical feature maps corresponding to each target size, and fusing the current original-size medical feature map and the current medical feature maps corresponding to each target size to obtain the current fused feature map, includes:

[0022] The current input feature map is processed sequentially through the downsampling module, the first convolution module, the first upsampling module, the merging module, the second convolution module, the second upsampling module, and the fusion module to extract multi-size features from the current input feature map. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map.

[0023] In one embodiment, the method further includes: obtaining the location region of the medical target to be identified based on the medical target prediction result; the step of obtaining the location region of the medical target to be identified based on the medical target prediction result includes:

[0024] Obtain the original size output result from the predicted medical target;

[0025] The original size output result is processed based on the probability and probability threshold of each pixel in the original size output result;

[0026] Extract the location region of the medical target to be identified from the processed original-size output.

[0027] In one embodiment, after obtaining the location region of the medical target to be identified based on the medical target prediction result, the process includes:

[0028] The image coordinates of the outer contour of the target are obtained by performing coordinate transformation on the location region of the medical target to be identified and collecting the boundary coordinate points.

[0029] The medical target to be identified is displayed in the medical image based on the image coordinates of the target's outer contour.

[0030] In one embodiment, the step of collecting the image coordinates of the outer contour of the target by performing coordinate transformation on the boundary coordinate points of the location region of the medical target to be identified includes:

[0031] Obtain the transformation relationship between the pixel coordinate system and the world coordinate system;

[0032] Based on the transformation relationship, the image coordinates of the outer contour of the medical target to be identified are transformed to obtain the world coordinates of the outer contour of the target.

[0033] Secondly, this application also provides a medical model training method, the medical model training method comprising:

[0034] Acquire medical sample data, which includes sample medical images and corresponding target labels;

[0035] The medical model uses a multi-size feature extraction and fusion network to extract multi-size features from the current input sample feature map, including the current original size sample medical feature map and the current sample medical feature maps corresponding to each target size. The current original size sample medical feature map and the current sample medical feature maps corresponding to each target size are then fused to obtain the current fused sample feature map, where the initial input sample feature map is the sample medical image.

[0036] The current fused sample feature map is obtained by convolving the current target sample feature map through the skip connection network of the medical model. The current original size sample medical feature map and the current target sample feature map are then merged to obtain the next input sample feature map.

[0037] The next input sample feature map is used as the current input sample feature map, and the multi-size feature extraction and fusion network of the medical model is used to extract multi-size features from the current input sample feature map, including the current original size sample medical feature map and the current sample medical feature map corresponding to each target size, to obtain the prediction results of sample medical targets under different sizes.

[0038] The medical model is optimized based on the prediction results of the medical targets of the samples at different sizes and the target labels to obtain a trained medical model.

[0039] In one embodiment, optimizing the medical model based on the prediction results of the medical targets of the samples at different sizes and the target labels to obtain the trained medical model includes:

[0040] The loss function value is obtained based on the prediction results of the medical target of the sample under different sizes and the target label;

[0041] The medical model is optimized based on the loss function value until the loss function value meets the requirements, thus obtaining a fully trained medical model.

[0042] In one embodiment, obtaining the loss function value based on the prediction results of the medical target of the sample at different sizes and the target label includes:

[0043] Obtain the first prediction results of the sample medical target at different sizes output by the last multi-size feature fusion network;

[0044] Obtain the second prediction result corresponding to the original-size sample medical feature map output by the target multi-size feature fusion network in the medical model;

[0045] Obtain the first weights corresponding to the first prediction result and the second prediction result, wherein the first weight corresponding to the second prediction result is less than the first weight corresponding to the first prediction result;

[0046] The loss function value is calculated based on the first weight, the difference between the first prediction result and the target label, and the difference between the second prediction result and the target label.

[0047] In one embodiment, calculating the loss function value based on the weights, the difference between the first prediction result and the target label, and the difference between the second prediction result and the target label includes:

[0048] Obtain the second weights corresponding to the first prediction results of different sizes, and the second weight of the first prediction result of the larger size is greater than the second weight of the first prediction result of the smaller size;

[0049] The first loss term is obtained based on the second weight and the difference between each of the first prediction results and the target label;

[0050] A second loss term is obtained based on the second prediction result and the target label;

[0051] The loss function value is calculated based on the first weight, the first loss term, and the second loss term.

[0052] In one embodiment, the step of extracting multi-size features from the current input sample feature map using a multi-size feature extraction fusion network of a medical model, including the current original-size sample medical feature map and the current sample medical feature maps corresponding to each target size, includes:

[0053] By using the multi-size feature fusion networks of the medical model, the current input sample feature map is downsampled to obtain downsampled sample feature maps of different sizes;

[0054] Convolution is performed on the downsampled sample feature maps of different sizes to obtain first sample feature maps of different sizes, and convolution is performed on the feature maps of the original size to obtain second sample feature maps;

[0055] The first sample feature map is upsampled step by step to obtain upsampled sample feature maps of different sizes;

[0056] The upsampled sample feature map is merged with the first sample feature map and the second sample feature map of the corresponding size. The merged sample feature map is then convolved to obtain the third sample feature map, wherein the third sample feature map of the original size is the original size sample medical feature map.

[0057] Upsampling is performed on the third sample feature map that is not of the original size to obtain the current medical sample feature map of the original size corresponding to each target size.

[0058] Thirdly, this application also provides a medical image recognition device, the device comprising:

[0059] The medical image acquisition module is used to acquire the medical image to be identified.

[0060] The first feature extraction and fusion module is used to extract multi-size features from the current input feature map, including the current original size medical feature map and the current medical feature maps corresponding to each target size. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map, wherein the initial input feature map is the medical image to be identified.

[0061] The first skip connection module is used to convolve the current fused feature map to obtain the current target feature map, and merge the current original-size medical feature map and the current target feature map to obtain the next input feature map;

[0062] The first target prediction result acquisition module is used to take the next input feature map as the current input feature map, and continue to extract multi-size features from the current input feature map, including the current original size medical feature map and the current medical feature maps corresponding to each target size, to obtain the medical target prediction result corresponding to the medical image.

[0063] Fourthly, this application also provides a medical model training device, the medical model training device comprising:

[0064] A medical sample data acquisition module is used to acquire medical sample data, which includes sample medical images and corresponding target labels;

[0065] The second feature extraction and fusion module is used to extract multi-size features from the current input sample feature map through a multi-size feature extraction and fusion network of a medical model. This extracts the current original-size sample medical feature map and the current sample medical feature maps corresponding to each target size. The module then fuses the current original-size sample medical feature map and the current sample medical feature maps corresponding to each target size to obtain the current fused sample feature map. The initial current input sample feature map is the sample medical image.

[0066] The second skip connection module is used to convolve the current fused sample feature map through the skip connection network of the medical model to obtain the current target sample feature map, and merge the current original size sample medical feature map and the current target sample feature map to obtain the next input sample feature map;

[0067] The second target prediction result acquisition module is used to take the next input sample feature map as the current input sample feature map, and continue to extract multi-size features from the current input sample feature map through the multi-size feature extraction and fusion network of the medical model, so as to obtain the prediction results of the sample medical target under different sizes.

[0068] The training module is used to optimize the medical model based on the prediction results of the medical targets of the samples at different sizes and the target labels, so as to obtain the trained medical model.

[0069] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0070] The aforementioned medical image recognition method, medical model training method, device, and computer equipment improve the ability to capture features of targets of different shapes and sizes in medical images by extracting and fusing features of multiple sizes. In addition, the current target feature map is obtained by convolving the current fused feature map. The current original-size medical feature map and the current target feature map are merged to obtain the next input feature map. In this way, the current input feature map incorporates the features of the previous input feature maps, that is, it retains the original-size medical features, reduces information loss caused during sampling, reduces the risk of gradient vanishing due to excessive convolution layers, enriches spatial information, and improves the ability of convolution to capture the overall features of targets in medical images, thereby improving the accuracy of medical image recognition. Attached Figure Description

[0071] Figure 1 This is an application environment diagram of a medical image recognition method in one embodiment;

[0072] Figure 2 This is a flowchart illustrating a medical image recognition method in one embodiment;

[0073] Figure 3 This is a schematic diagram of the structure of a medical model in one embodiment;

[0074] Figure 4 This is a schematic diagram of a cascading connection network in one embodiment;

[0075] Figure 5This is a schematic diagram of the structure of a multi-size feature extraction and fusion network in one embodiment;

[0076] Figure 6 A flowchart for prostate MRI image recognition in another embodiment;

[0077] Figure 7 This is a segmentation result of the prostate region in one embodiment;

[0078] Figure 8 This is a schematic diagram of the prostate edge contour delineation process in one embodiment;

[0079] Figure 9 This is a schematic diagram illustrating the principle of multi-scale features in one embodiment;

[0080] Figure 10 A contour view of the prostate gland in one embodiment;

[0081] Figure 11 In one embodiment, the surgical robot software interface displays a prostate contour map;

[0082] Figure 12 This is a flowchart illustrating a medical model training method in one embodiment;

[0083] Figure 13 This is a schematic diagram of a loss function value calculation method in one embodiment;

[0084] Figure 14 This is a flowchart illustrating a medical model training method in another embodiment;

[0085] Figure 15 This is a schematic diagram of the labels in a real area in one embodiment;

[0086] Figure 16 This is a schematic diagram illustrating the prediction probability in one embodiment;

[0087] Figure 17 This is a graph showing the changes in training loss and metrics in one embodiment;

[0088] Figure 18 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0090] The medical image recognition method and medical model training method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with medical imaging device 104 via a network. Terminal 102 can receive the medical image to be identified scanned by medical imaging device 104, perform multi-size feature extraction on the current input feature map, extracting the current original-size medical feature map and the current medical feature maps corresponding to each target size, and fuse the current original-size medical feature map and the current medical feature maps corresponding to each target size to obtain the current fused feature map; convolve the current fused feature map to obtain the current target feature map, merge the current original-size medical feature map and the current target feature map to obtain the next input feature map; use the next input feature map as the current input feature map to obtain the medical target prediction result corresponding to the medical image; subsequently, the location region of the medical target to be identified can be obtained based on the medical target prediction result. By extracting and fusing features of different sizes, the ability to capture features of targets of different shapes and sizes in medical images to be identified can be improved. In addition, the current fused feature map is convolved to obtain the current target feature map. The current original-size medical feature map and the current target feature map are merged to obtain the next input feature map. In this way, the current input feature map incorporates the features of the previous input feature maps, that is, it retains the original-size medical features, reduces information loss caused by sampling, reduces the risk of gradient vanishing due to excessive convolution layers, enriches spatial information, and improves the ability of convolution to capture the overall features of targets in the image to be identified, thereby improving the accuracy of medical image recognition.

[0091] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The medical imaging device 104 includes, but is not limited to, various imaging devices, such as CT imaging devices (CT: Computed Tomography, which uses precisely collimated X-ray beams and highly sensitive detectors to perform a series of cross-sectional scans around a part of the human body, and can reconstruct precise three-dimensional images of tumors, etc.), magnetic resonance imaging devices (a type of tomographic imaging that uses magnetic resonance to obtain electromagnetic signals from the human body and reconstruct human body information images), positron emission tomography (PET / MR) devices, positron emission tomography / magnetic resonance imaging systems, etc. The medical image to be identified can be an image sequence acquired by the medical imaging device 104, such as a T2W sequence of MRI images.

[0092] In one embodiment, such as Figure 2 As shown, a medical image recognition method is provided, which can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0093] S202: Perform multi-size feature extraction on the current input feature map. Extract the current original size medical feature map and the current medical feature maps corresponding to each target size. Fuse the current original size medical feature map and the current medical feature maps corresponding to each target size to obtain the current fused feature map.

[0094] S204: Convolve the current fused feature map to obtain the current target feature map, merge the current original-size medical feature map and the current target feature map to obtain the next input feature map.

[0095] S206: Use the next input feature map as the current input feature map to obtain the medical target prediction result corresponding to the medical image.

[0096] Specifically, medical images are images that require target identification. They can be image sequences acquired by medical imaging equipment, such as T2W sequences of MRI images. For example, when identifying the prostate, medical images of the prostate are acquired using medical imaging equipment. In other embodiments, these medical images can be medical images corresponding to other tissues or organs. The corresponding medical target, i.e., tissues or organs, can be extracted from the medical image through the processing described below. This allows for subsequent surgical planning based on the identification results, thereby improving surgical efficiency.

[0097] The current input feature map refers to the feature map for multi-size feature extraction. In this application, multiple multi-size feature extraction fusion networks are used to extract multi-size features from their respective current input feature maps, resulting in the current original-size medical feature map and the current medical feature maps corresponding to each target size. The current original-size medical feature map is the feature map corresponding to the medical image to be recognized without changing its dimensions. The current medical feature maps corresponding to each target size can be obtained by downsampling the original-size current medical feature map to obtain downsampled feature maps for each target size, and then performing feature extraction on the downsampled feature maps.

[0098] The input to the first multi-size feature extraction and fusion network is obtained from medical images, while the input to the other multi-size feature extraction and fusion networks is the current input feature map obtained from the output of the previous multi-size feature extraction and fusion network.

[0099] The subsequent current input feature map can be obtained by merging the current original-size medical feature map and the current target feature map. For example, the current fused feature map output by the previous multi-size feature extraction and fusion network is convolved to obtain the current target feature map. Then, the original-size medical feature map output by the previous multi-size feature extraction and fusion network and the current target feature map are merged to obtain the next input feature map, which is then used as the input of the next multi-size feature extraction and fusion network.

[0100] In the medical field, taking prostate MRI images as an example, a multi-size feature extraction and fusion network is used to downsample and obtain prostate feature maps of different scales within the same convolutional layer. These feature maps of different scales are then output to the next convolutional layer. The prostate feature map with unchanged scale is used as the output of the backbone network, while the prostate feature map with changed scale is used as an auxiliary network. This multiple feature fusions at different scales improve the model's ability to capture prostate features of different shapes and sizes in MRI images. Furthermore, the feature maps from previous layers are indirectly reused. The convolutional outputs of the backbone network are subjected to intermittent convolutional layer feature fusion, continuously fusing features of different scales from previous convolutional layers to enrich spatial information and improve the ability of 3D convolution to capture the overall features of the prostate in MRI images.

[0101] In the above embodiments, the deep learning model's ability to capture prostate region features of different shapes and sizes in T2W MRI image sequences is enhanced by multi-scale spatial feature fusion. This enables precise segmentation and delineation of malformed prostate regions, improving the shortcomings of existing methods in delineating malformed prostates in T2W MRI image sequences, thereby reducing the time doctors spend on preoperative planning and improving the efficiency of prostate puncture.

[0102] Optionally, the above method further includes: obtaining the location region of the medical target to be identified based on the medical target prediction results.

[0103] In this embodiment, the location region of the medical target to be identified is obtained by recognizing the original size feature map output at the end.

[0104] The aforementioned medical image recognition method improves the ability to capture features of targets of different shapes and sizes in medical images by extracting and fusing features of multiple sizes. In addition, the current fused feature map is convolved to obtain the current target feature map. The current original-size medical feature map and the current target feature map are merged to obtain the next input feature map. In this way, the current input feature map incorporates the features of the previous input feature maps, thus preserving the original-size medical features, reducing information loss during sampling, reducing the risk of gradient vanishing due to excessive convolution layers, enriching spatial information, and improving the ability of convolution to capture the overall features of targets in medical images, thereby improving the accuracy of medical image recognition.

[0105] In one embodiment, multi-size feature extraction is performed on the current input feature map to obtain the current original-size medical feature map and the current medical feature maps corresponding to each target size, including: downsampling the current input feature map to obtain downsampled feature maps of different sizes; convolving the downsampled feature maps of different sizes to obtain first feature maps of different sizes, and convolving the original-size feature map to obtain second feature maps; upsampling the first feature map step by step to obtain upsampled feature maps of different sizes; merging the upsampled feature maps with the corresponding first and second feature maps, and convolving the merged feature maps to obtain third feature maps, wherein the original-size third feature map is the current original-size medical feature map; and upsampling the non-original-size third feature map to obtain the original-size current medical feature map corresponding to each target size.

[0106] Specifically, downsampling is performed on the current input feature map. For example, the current input feature map can be downsampled sequentially. The number of downsampling operations can be preset or determined according to the size of the medical image to be recognized, such as three times. Convolving downsampled feature maps of different sizes with the original feature map extracts effective information from the downsampled feature maps of different sizes and the original feature map, such as prostate characterization information. Subsequently, the first feature map is upsampled step by step to obtain an upsampled feature map. In this embodiment, the number of upsampling operations is less than the number of downsampling operations. For example, downsampling is performed 3 times, meaning that the current input feature map is downsampled 3 times. The number of upsampling operations can be 1, meaning that only one upsampling operation is performed on each different size of the first feature map to obtain upsampled feature maps of different sizes. Then, the upsampled feature maps of the same size are merged with the corresponding first and second feature maps, and the merged feature maps are convolved to obtain the third feature map. The original size of the third feature map is the current original size medical feature map. Upsampling is performed on the third feature maps that are not of the original size to obtain the current original size medical feature maps corresponding to each target size, so that all feature maps of different sizes are converted to the original size, and the converted feature maps can be merged.

[0107] In the above embodiments, prostate feature maps of different scales in the same convolutional layer are obtained by downsampling. The prostate feature maps of different scales are output to the next convolutional layer. The prostate feature maps with unchanged scales are used as the output of the backbone network, and the prostate feature maps with changed scales are used as the auxiliary network. By fusing features of different scales multiple times, the model's ability to capture prostate features of different shapes and sizes in MRI images is improved.

[0108] In one embodiment, the medical target prediction result is obtained by a pre-trained medical model; wherein the medical model includes a multi-size feature extraction and fusion network and a skip-level connection network; multi-size feature extraction is performed on the current input feature map to obtain the current original-size medical feature map and the current medical feature maps corresponding to each target size; the current original-size medical feature map and the current medical feature maps corresponding to each target size are fused to obtain the current fused feature map, including: multi-size feature extraction is performed on the current input feature map to obtain the current original-size medical feature map and the current medical feature maps corresponding to each target size through the multi-size feature extraction and fusion network; the current original-size medical feature map and the current medical feature maps corresponding to each target size are fused to obtain the current fused feature map; convolution is performed on the current fused feature map to obtain the current target feature map; the current original-size medical feature map and the current target feature map are merged to obtain the next input feature map, including: convolution is performed on the current fused feature map through the skip-level connection network to obtain the current target feature map; the current original-size medical feature map and the current target feature map are merged to obtain the next input feature map.

[0109] Specifically, in combination Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a medical model in one embodiment. In this embodiment, it includes multiple multi-size feature extraction and fusion networks and a skip-level connection network. The multi-size feature fusion network is used to process the current input feature map to obtain the current fused feature map, and the skip-level connection network is used to convolve the current fused feature map to obtain the current target feature map. The current original-size medical feature map and the current target feature map are merged to obtain the next input feature map.

[0110] The multi-size feature fusion network first uses downsampling to obtain feature maps of different sizes from the input features. Then, it performs convolution on the feature maps of different scales to obtain prostate representations of different sizes and shapes. Next, it upsamples the small-sized features after convolution step by step and fuses them to reduce the information loss caused by downsampling. Finally, it upsamples the small-sized feature maps to the original size and splices them together to enhance the model's ability to capture features of prostates of different sizes and shapes.

[0111] Skip-connection networks are used to connect features before and after multi-scale fusion. They add the original size features that have not been spliced ​​and fused to the spliced, fused and convolutional features, and then continue to perform multi-scale fusion operations on the added features. By skipping-connection to the output of the previous layer, the original size information of the unspliced ​​and fused features can be increased, spatial feature information can be enriched, information loss caused during sampling can be reduced, the risk of gradient vanishing due to excessive convolution layers can be reduced, and the model's ability to extract prostate features can be improved.

[0112] In one embodiment, combined Figure 4 As shown, Figure 4 This is a schematic diagram of a skip-connection network in one embodiment. In this embodiment, the feature information is enhanced by using the output of the previous layer through skip connections, reducing information loss during the sampling process and also reducing the risk of gradient vanishing due to excessively deep convolutional layers.

[0113] Among them, such as Figure 4 It includes multiple multi-scale extraction and fusion networks. Adjacent multi-scale extraction and fusion networks are connected by a skip-level connection network. This skip-level connection network is used to obtain the current fusion feature map output by the previous multi-scale extraction and fusion network, and convolve it to obtain the current target feature map. The current target feature map is then merged with the current original-size medical feature map output by the previous multi-scale extraction and fusion network, thus obtaining the input of the next multi-scale extraction and fusion network.

[0114] like Figure 4 In the middle, the current original size medical feature map ( Figure 3 The W[W[X]+UP(W1[X1])]) in the current fused feature is compared with the output of the previous multi-scale extraction fusion network. Figure X After addition, multi-scale feature fusion is performed on the current input feature map after addition. This skip-level feature fusion of previous layer features is repeated throughout the convolutional neural network. By skipping the previous layer output, the original size information of the unjoined and fused layers can be increased, spatial feature information can be enriched, information loss caused during sampling can be reduced, the risk of gradient vanishing due to excessive convolutional layers can be reduced, and the model's ability to extract prostate features can be improved.

[0115] In one embodiment, the multi-size feature extraction and fusion network includes a downsampling module, a first convolution module, a first upsampling module, a merging module, a second convolution module, a second upsampling module, and a fusion module. The process involves extracting a current original-size medical feature map and current medical feature maps corresponding to each target size from the current input feature map, and then fusing the current original-size medical feature map and the current medical feature maps corresponding to each target size to obtain a current fused feature map. This includes sequentially passing the downsampling module, the first convolution module, the first upsampling module, the merging module, the second convolution module, the second upsampling module, and the fusion module to extract a current original-size medical feature map and current medical feature maps corresponding to each target size from the current input feature map, and then fusing the current original-size medical feature map and the current medical feature maps corresponding to each target size to obtain the current fused feature map.

[0116] Specifically, downsampling the current input feature map to obtain downsampled feature maps of different sizes includes: downsampling the current input feature map using a downsampling module to obtain downsampled feature maps of different sizes; convolving the downsampled feature maps of different sizes to obtain first feature maps of different sizes, and convolving the original feature map to obtain a second feature map, including: convolving the downsampled feature maps of different sizes using a first convolution module to obtain first feature maps of different sizes, and convolving the original feature map to obtain a second feature map; and progressively upsampling the first feature map to obtain upsampled feature maps of different sizes, including: upsampling the first feature map using a first upsampling module. Upsampling is performed stepwise to obtain upsampled feature maps of different sizes; the upsampled feature maps are then merged with the corresponding first and second feature maps, including: merging the upsampled feature maps with the corresponding first and second feature maps using a merging module; convolution is performed on the merged feature maps to obtain a third feature map, including convolution is performed on the merged feature maps using a second convolution module; upsampling is performed on the third feature maps of non-original sizes to obtain the original size current medical feature maps corresponding to each target size, including: upsampling the third feature maps of non-original sizes using a second upsampling module to obtain the original size current medical feature maps corresponding to each target size.

[0117] Specifically, in combination Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a multi-size feature extraction and fusion network in one embodiment. The multi-size feature extraction and fusion network includes a downsampling module, a first convolution module, a first upsampling module, a merging module, a second convolution module, a second upsampling module, and a fusion module.

[0118] Combination Figure 5The current input feature map is first downsampled by a downsampling module. For example, the input feature X from the previous layer is downsampled by a factor of 2 to obtain downsampled feature maps X1, X2, and X3. Then, the first convolution module convolves the three feature maps of different sizes to obtain the first feature maps W1[X1], W2[X2], and W3[X3]. The original feature map is then convolved to obtain the second feature map W[X]. The first upsampling module then upsamples the first feature map step by step to obtain upsampled feature maps of different sizes. For example, the smaller feature map is upsampled by a factor of 1 to obtain upsampled feature maps UP(W1[X1]), UP(W2[X2]), and UP(W3[X3]) to reduce the loss after the first sampling convolution. The sampled feature maps and the convolutional feature maps are then added together at the same positions using a merging module to obtain merged feature maps: W[X]+UP(W1[X1]), W1[X1]+UP(W2[X2]), and W2[X2]+UP(W3[X3]). Finally, these feature maps are convolved separately and then upsampled back to their original size to obtain third feature maps: W[W[X]+UP(W1[X1]), W1[W1[X1]+UP(W2[X2]), W2[W2[X2]+UP(W3[X3]), and W3[W3[X3]]. The original-size third feature map is W[W[X]+UP(W1[X1]), and the non-original-size third feature maps are W1[W1[X1]+UP(W2[X2]), W2[W2[X2]+UP(W3[X3]), and W3[W3[X3]]. The third feature map, which is not of the original size, is then upsampled by the second upsampling module to obtain the original size current medical feature map corresponding to each target size: UP(W1[W1[X1]+UP(W2[X2])), UP(W2[W2[X2]+UP(W3[X3])), and UP(W3[W3[X3]]). Finally, the fusion module merges all the feature maps to obtain the fused current feature map, i.e., Y = concat{W[W[X]+UP(W1[X1])], UP(W1[W1[X1]+UP(W2[X2])), UP(W2[W2[X2]+UP(W3[X3])), UP(W3[W3[X3]])}.

[0119] In the above embodiments, prostate feature maps of different scales are obtained by downsampling in the same convolutional layer. The prostate feature maps of different scales are output to the next convolutional layer. The prostate feature maps with unchanged scales are used as the output of the backbone network, and the prostate feature maps with changed scales are used as the auxiliary network. By fusing features of different scales multiple times, the model's ability to capture prostate features of different shapes and sizes in MRI images is improved.

[0120] In one embodiment, obtaining the location region of the medical target to be identified based on the medical target prediction result includes: obtaining the original size output result in the medical target prediction result; processing the original size output result based on the probability of each pixel and the probability threshold in the original size output result; and extracting the location region of the medical target to be identified from the processed original size output result.

[0121] Combination Figure 6 As shown, Figure 6 The flowchart for prostate MRI image recognition in another embodiment is as follows: Specifically, a prostate MRI image is input, converted into image pixel values, and then the image pixel values ​​are processed and converted into tensor form. The image pixel tensor form is then loaded into the model parameters obtained during training in a convolutional neural network. The original-size output image of the model is selected, where the original-size output image can be a probability matrix. In this way, the probability matrix is ​​set so that pixels with a prediction probability greater than 0.6 are 1 and the rest are 0, that is, pixels with a prediction probability greater than 0.6 are set to belong to the prostate (0, 1 matrix). The coordinates of the edge pixel points of region 1 in the image are calculated, that is, the pixel coordinates of the contour line, so as to display the contour line on the input prostate image.

[0122] Combination Figure 7 As shown, where Figure 7 In the diagram, 'a' represents the manually labeled real prostate region, 'b' represents the binary image output by the model overlaid on the MRI image, and the output of the convolutional neural network is the probability value of the prostate region. Setting the probability value greater than a certain value as 1 will yield the binary image of prostate segmentation.

[0123] In one embodiment, after obtaining the location region of the medical target to be identified based on the medical target prediction result, the process includes: performing coordinate transformation on the location region of the medical target to be identified, collecting the boundary coordinate points to obtain the image coordinates of the target's outer contour; and displaying the medical target to be identified in the medical image based on the image coordinates of the target's outer contour.

[0124] In one embodiment, the image coordinates of the outer contour of the target are obtained by collecting boundary coordinate points of the location region of the medical target to be identified through coordinate transformation, including: obtaining the transformation relationship between the pixel coordinate system and the world coordinate system; and transforming the image coordinates of the outer contour of the medical target to be identified according to the transformation relationship to obtain the world coordinates of the outer contour of the target.

[0125] Specifically, in combination Figure 8 As shown, after importing the prostate MRI image, the pixel coordinates of the prostate contour image are obtained through a medical recognition model. Then, the coordinates are converted to world coordinates through the origin, spacing, and direction of the MRI image. Finally, the world coordinates are written into the MRI image, and the prostate and its contour lines can be displayed on the interface.

[0126] Specifically, this application utilizes multi-scale feature fusion to enhance the deep learning model's ability to capture features of the prostate region. For example... Figure 9 The image shown is the same image at different sizes. The white rectangles represent convolution operators. The original-sized image has a smaller range in each convolution operation, meaning it extracts a smaller range of the prostate region. However, the same convolution operator at the smaller size extracts a larger range of the prostate region. By fusing these features, information from different regions can be superimposed at the same location, enriching the representation of the prostate region.

[0127] Specifically, this application converts the image coordinates obtained from the model output into world coordinates to support the software's contour delineation of the prostate, such as reading MRI image information. Here, Voxel Spacing represents the voxel spacing, Origin represents the original coordinates, and orientation indicates the image's storage direction (affecting the world coordinate conversion; combinations include left / right L / R, front / back A / P, and up / down S / I, with different combinations resulting in different image orientations and different world coordinate conversion processes). Voxel coordinates represent the image coordinates, with Z values ​​representing slices. For example:

[0128] World coordinate X=Spacing X*(Voxel X-1)–Origin X

[0129] World coordinate Y=Spacing Y*(Voxel Y-1)–Origin Y

[0130] World coordinate Z=Spacing Z*(Voxel Z-1)+Origin Z

[0131] See Figure 10 As shown, Figure 10 To obtain the prostate contour map, this application extracts the edge lines from the binary image output by the model.

[0132] Specifically, in combination Figure 11 As shown, one embodiment of this application's module is loaded onto a surgical robot, wherein... Figure 10 It can obtain the edge contour coordinates, then convert the image coordinates to world coordinates, and optimize the world coordinates to make the contour lines as smooth as possible. After importing the MRI image sequence, wait 10 seconds to outline the prostate contour on the MRI image.

[0133] The prostate segmentation model designed in this application is adapted to the surgical robot process. After importing the image, clicking the automatic outlining button will cause the machine to start calculating the contour coordinates and write the contour coordinates into the imported MRI image. The interface displays the prostate and its contour line, which takes about 10 seconds.

[0134] In one embodiment, see Figure 12 As shown, this application also provides a medical model training method, which may include:

[0135] S1202: Obtain medical sample data, which includes sample medical images and corresponding target labels.

[0136] Specifically, the sample medical image is an image that requires target recognition. It can be an image sequence acquired by a medical imaging device, such as a T2W sequence of MRI images. For example, when identifying the prostate, a prostate medical image is acquired using a medical imaging device. In other embodiments, the sample medical image can be a medical image corresponding to other tissues or organs, and the corresponding sample medical target can be extracted from the sample medical image through the processing described below. The target label is the location of the sample medical target, etc., pre-annotated manually.

[0137] S1204: The multi-size feature extraction fusion network of the medical model is used to extract the medical feature map of the current input sample from the original size sample and the medical feature map of the current sample corresponding to each target size. The original size sample medical feature map and the medical feature map of the current sample corresponding to each target size are fused to obtain the current fused sample feature map, where the initial input sample feature map is the sample medical image.

[0138] S1206: The current target sample feature map is obtained by convolving the current fused sample feature map through the skip connection network of the medical model. The current original size sample medical feature map and the current target sample feature map are merged to obtain the next input sample feature map.

[0139] S1208: Use the next input sample feature map as the current input sample feature map, and continue to extract multi-size features from the current input sample feature map through the multi-size feature extraction fusion network of the medical model, including the current original size sample medical feature map and the current sample medical feature map corresponding to each target size, until the prediction results of sample medical targets under different sizes are obtained.

[0140] Specifically, the method for obtaining the prediction results of the sample medical goals can be found in the prediction method of the medical goal prediction results above, and will not be repeated here.

[0141] S1210: Optimize the medical model based on the prediction results of medical targets for samples of different sizes and the target labels to obtain the trained medical model.

[0142] The process of optimizing the medical model based on the prediction results of the medical targets of the samples at different sizes and the target labels to obtain a trained medical model includes: obtaining a loss function value based on the prediction results of the medical targets of the samples at different sizes and the target labels; optimizing the medical model based on the loss function value until the loss function value meets the requirements, thus obtaining a trained medical model.

[0143] Specifically, in combination Figure 13 As shown, this application utilizes multiple loss forms to train the convolutional neural network, achieving better parameter weight updates. The final backpropagation loss value used by the convolutional neural network is calculated as: LOSS = a(Loss1 + 0.8 * Loss2 + 0.4 * Loss3 + 0.2 * Loss4) + (1 - a)Loss5, where a ∈ [0,1], used to control the proportion of the output of the intermediate layer of the network. Since the output of the intermediate layer does not extract features well, the weight of Loss5 is set to be small, and the proportion of the loss Loss1, Loss2, Loss3, and Loss4 in the final output of the model is high. The output maps of different scales are used to calculate the loss with labels of different sizes, because the labels have also been resized.

[0144] In one embodiment, the loss function value is obtained based on the prediction results of medical targets of different sizes and the target labels, including: obtaining the first prediction results of medical targets of different sizes output by the last multi-size feature fusion network; obtaining the second prediction result corresponding to the original size sample medical feature map output by the target multi-size feature fusion network in the medical model; obtaining the first weights corresponding to the first prediction result and the second prediction result, wherein the first weight corresponding to the second prediction result is less than the first weight corresponding to the first prediction result; and calculating the loss function value based on the first weights, the difference between the first prediction result and the target label, and the difference between the second prediction result and the target label.

[0145] In one embodiment, the loss function value is calculated based on the weights, the difference between the first prediction result and the target label, and the difference between the second prediction result and the target label. This includes: obtaining the second weights corresponding to the first prediction results of different sizes, wherein the second weight of the larger first prediction result is greater than the second weight of the smaller first prediction result; obtaining a first loss term based on the second weights and the difference between each first prediction result and the target label; obtaining a second loss term based on the second prediction result and the target label; and calculating the loss function value based on the first weights, the first loss term, and the second loss term.

[0146] To optimize multi-scale feature extraction capabilities, output maps of different scales are output in the final output of the model. At the same time, intermediate layer outputs are added to standardize training when the network layers are deep. The loss values ​​calculated from these outputs are weighted and summed (given the high proportion of the final output, the loss calculated at different scales is weighted according to the size), and they work together to optimize the parameter updates during model training.

[0147] In this model, the loss calculated based on the original scale of the last multi-scale feature fusion during network output is denoted as Loss1, and the losses calculated based on the three different scales are denoted as Loss2, loss1, loss2, and loss3, respectively. The loss calculated by the intermediate layers of the network is Loss3. Therefore, the final output is given a larger weight, and the intermediate layers are given a smaller weight. Based on the weights, the loss used by the entire model during training is Loss = α * (Loss1 + Loss2) + (1 - α) * Loss3.

[0148] For the outputs of non-intermediate layers, the weights are reduced sequentially according to their size. For example, the weight of Loss1 is 1, and the weights of loss1 to loss3 are 0.8, 0.4 and 0.2 respectively. In other embodiments, other values ​​can also be set, and no specific limitation is made here.

[0149] In this application, the weights are first set according to the different networks, and then according to the size, which can adapt to the convolutional neural network to update the network parameters.

[0150] Specifically, in combination Figure 14 As shown, the input is a prostate MRI image, which is converted into image pixel values. Then, the image pixel values ​​are processed and converted into tensor form. This image pixel tensor form is then used to train a 3D convolutional neural network, which outputs prostate prediction maps of different scales and output maps of intermediate network layers. The loss values ​​are calculated with the labels, and after being weighted and added, they are fed back to the model to update the network parameters. Subsequently, the evaluation index is calculated on the test set, and the optimal model parameters are saved.

[0151] Combination Figure 15 and 16 This application utilizes the DICE value, commonly used in medical image segmentation, as a judgment index to save model parameters and select models. The principle is to compare the similarity between two images; higher similarity results in higher prediction outcomes. The calculation formula is as follows:

[0152]

[0153] Where T is the value of the true region and P is the prediction probability, the numerator only retains the predicted value of the target region, that is, the closer the numerator is to the denominator, the more accurate the prediction result is.

[0154] Combination Figure 17 As shown in the graphs, the loss value and evaluation index change with the number of training iterations during the training process of this application. It can be seen that the convolutional neural network model has a good convergence effect, tends to stabilize after 800 iterations, and its performance on the test set is basically consistent with that on the training set.

[0155] In the above embodiments, the proportion of the auxiliary network output in the loss calculation is adjusted, and the loss proportion among the backbone network, auxiliary network, and intermediate output layer is controlled to optimize the gradient update direction and further improve the segmentation ability of the prostate region in MRI images.

[0156] In one embodiment, a multi-size feature extraction fusion network of a medical model is used to extract multi-size features from the current input sample feature map, including the current original-size sample medical feature map and the current sample medical feature maps corresponding to each target size. This includes: downsampling the current input sample feature map using the multi-size feature fusion networks of the medical model to obtain downsampled sample feature maps of different sizes; convolving the downsampled sample feature maps of different sizes to obtain first sample feature maps of different sizes, and convolving the original-size feature map to obtain second sample feature maps; progressively upsampling the first sample feature maps to obtain upsampled sample feature maps of different sizes; merging the upsampled sample feature maps with the corresponding first and second sample feature maps, and convolving the merged sample feature maps to obtain a third sample feature map, wherein the original-size third sample feature map is the original-size sample medical feature map; and upsampling the non-original-size third sample feature map to obtain the original-size current medical sample feature map corresponding to each target size.

[0157] Specifically, the limitations of the multi-size feature extraction and fusion network in this embodiment can be found above, and will not be repeated here.

[0158] In the above embodiments, the multi-scale fusion method forces the deep learning model to effectively learn and capture features of prostate regions of different shapes and sizes, resulting in more accurate segmentation of abnormally shaped regions. This saves doctors the time spent manually marking prostate locations before surgery, improving surgical efficiency. Furthermore, by enriching spatial feature information and implementing loss adjustment mechanisms, the model can be trained better, accelerating convergence and reducing training time, thus advancing the application of deep learning in the medical industry.

[0159] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0160] Based on the same inventive concept, this application also provides a medical image recognition device and a medical model training device for implementing the medical image recognition method and medical model training method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the medical image recognition device and medical model training device provided below can be found in the limitations of the medical image recognition method and medical model training method described above, and will not be repeated here.

[0161] In one embodiment, a medical image recognition device is provided, comprising:

[0162] The first feature extraction and fusion module is used to extract multi-size features from the current input feature map, including the current original size medical feature map and the current medical feature maps corresponding to each target size. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map, where the initial input feature map is the medical image to be identified.

[0163] The first skip connection module is used to convolve the current fused feature map to obtain the current target feature map, and merge the current original-size medical feature map and the current target feature map to obtain the next input feature map;

[0164] The first target prediction result acquisition module is used to take the next input feature map as the current input feature map, and continue to extract multi-size features from the current input feature map to obtain the current original-size medical feature map and the current medical feature map corresponding to each target size, so as to obtain the medical target prediction result corresponding to the medical image. In one embodiment, the above-mentioned first feature extraction and fusion module is further used to downsample the current input feature map to obtain downsampled feature maps of different sizes; convolve the downsampled feature maps of different sizes to obtain first feature maps of different sizes, and convolve the original-size feature map to obtain second feature maps; upsample the first feature maps step by step to obtain upsampled feature maps of different sizes; merge the upsampled feature maps with the corresponding first and second feature maps, and convolve the merged feature maps to obtain third feature maps, wherein the original-size third feature map is the current original-size medical feature map; upsample the non-original-size third feature map to obtain the original-size current medical feature map corresponding to each target size.

[0165] In one embodiment, the medical target prediction result is obtained by using a pre-trained medical model; wherein the medical model includes a multi-size feature extraction and fusion network and a skip-level connection network;

[0166] The first feature extraction and fusion module described above is also used to extract the current original size medical feature map and the current medical feature map corresponding to each target size from the current input feature map through a multi-size feature extraction and fusion network, and to fuse the current original size medical feature map and the current medical feature map corresponding to each target size to obtain the current fused feature map;

[0167] The first feature extraction and fusion module is also used to convolve the current fused feature map through a skip connection network to obtain the current target feature map, and to merge the current original-size medical feature map and the current target feature map to obtain the next input feature map.

[0168] In one embodiment, the multi-size feature extraction and fusion network includes a downsampling module, a first convolutional module, a first upsampling module, a merging module, a second convolutional module, a second upsampling module, and a fusion module;

[0169] The aforementioned first feature extraction and fusion module is further configured to: downsample the current input feature map using a downsampling module to obtain downsampled feature maps of different sizes; convolve the downsampled feature maps of different sizes using a first convolution module to obtain first feature maps of different sizes, and convolve the original feature map to obtain a second feature map; upsample the first feature map stepwise using a first upsampling module to obtain upsampled feature maps of different sizes; merge the upsampled feature maps with the corresponding first and second feature maps using a merging module; convolve the merged feature maps using a second convolution module to obtain a third feature map; upsample the non-original third feature map using a second upsampling module to obtain the original size current medical feature map corresponding to each target size; and fuse the current original size medical feature map and the original size current medical feature maps corresponding to each target size using a fusion module to obtain the current fused feature map.

[0170] In one embodiment, the apparatus further includes a target recognition module, configured to obtain the location region of the medical target to be identified based on the medical target prediction result. The target recognition module is further configured to acquire the original-size output result from the medical target prediction result; process the original-size output result based on the probability and probability threshold of each pixel in the original-size output result; and extract the location region of the medical target to be identified from the processed original-size output result.

[0171] In one embodiment, the target recognition module is further configured to perform coordinate transformation on the location area of ​​the medical target to be identified, collect the boundary coordinate points to obtain the image coordinates of the target's outer contour, and display the medical target to be identified in the medical image based on the image coordinates of the target's outer contour.

[0172] In one embodiment, the target recognition module is further configured to obtain the transformation relationship between the pixel coordinate system and the world coordinate system; and to perform coordinate transformation on the image coordinates of the outer contour of the medical target to be recognized according to the transformation relationship to obtain the world coordinates of the outer contour of the target.

[0173] In one embodiment, a medical model training device is provided, comprising:

[0174] The medical sample data acquisition module is used to acquire medical sample data, which includes medical images of the samples and corresponding target labels.

[0175] The second feature extraction and fusion module is used to extract multi-size features from the current input sample feature map through the multi-size feature extraction and fusion network of the medical model. The current original size sample medical feature map and the current sample medical feature map corresponding to each target size are extracted. The current original size sample medical feature map and the current sample medical feature map corresponding to each target size are fused to obtain the current fused sample feature map. The current input sample feature map at the beginning is the sample medical image.

[0176] The second skip connection module is used to convolve the current fused sample feature map through the skip connection network of the medical model to obtain the current target sample feature map, and merge the current original size sample medical feature map and the current target sample feature map to obtain the next input sample feature map;

[0177] The second target prediction result acquisition module is used to take the next input sample feature map as the current input sample feature map, and continue to extract multi-size features from the current input sample feature map through the multi-size feature extraction and fusion network of the medical model, so as to obtain the prediction results of the sample medical target under different sizes.

[0178] The training module is used to optimize the medical model based on the prediction results of medical targets for samples of different sizes and the target labels, so as to obtain the trained medical model.

[0179] In one embodiment, the training module includes: a loss function value generation unit for obtaining loss function values ​​based on the prediction results of the medical target of the sample at different sizes and the target label;

[0180] The training unit is used to optimize the medical model based on the loss function value until the loss function value meets the requirements, thus obtaining a fully trained medical model.

[0181] In one embodiment, the loss function value generation unit is further configured to obtain the first prediction result of the medical target sample at different sizes output by the last multi-size feature fusion network; obtain the second prediction result corresponding to the original size sample medical feature map output by the target multi-size feature fusion network in the medical model; obtain the first weights corresponding to the first prediction result and the second prediction result, wherein the first weight corresponding to the second prediction result is less than the first weight corresponding to the first prediction result; and calculate the loss function value based on the first weight, the difference between the first prediction result and the target label, and the difference between the second prediction result and the target label.

[0182] In one embodiment, the loss function value generation unit is further configured to obtain second weights corresponding to first prediction results of different sizes, wherein the second weight of the first prediction result of a larger size is greater than the second weight of the first prediction result of a smaller size; obtain a first loss term based on the second weights and the difference between each first prediction result and the target label; obtain a second loss term based on the second prediction result and the target label; and calculate the loss function value based on the first weights, the first loss term, and the second loss term.

[0183] In one embodiment, the second feature extraction and fusion module is further configured to: downsample the current input sample feature map to obtain downsampled sample feature maps of different sizes through the multi-size feature fusion networks of the medical model; convolve the downsampled sample feature maps of different sizes to obtain first sample feature maps of different sizes, and convolve the feature map of the original size to obtain second sample feature maps; upsample the first sample feature maps stepwise to obtain upsampled sample feature maps of different sizes; merge the upsampled sample feature maps with the first sample feature maps and second sample feature maps of the corresponding sizes, and convolve the merged sample feature maps to obtain third sample feature maps, wherein the third sample feature map of the original size is the original size sample medical feature map; and upsample the third sample feature map of the non-original size to obtain the current medical sample feature map of the original size corresponding to each target size.

[0184] The modules in the aforementioned image recognition device and medical model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0185] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 18As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image recognition device and a medical model training method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0186] Those skilled in the art will understand that Figure 18 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0188] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0189] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0190] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0191] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0192] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A medical image recognition method, characterized in that, The method includes: Multi-size feature extraction is performed on the current input feature map to obtain the current original size medical feature map and the current medical feature maps corresponding to each target size. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map. The current fused feature map is convolved to obtain the current target feature map. The current original-size medical feature map and the current target feature map are then merged to obtain the next input feature map. The next input feature map is used as the current input feature map, and multi-size feature extraction is performed on the current input feature map to obtain the current original size medical feature map and the current medical feature map corresponding to each target size, so as to obtain the medical target prediction result corresponding to the medical image; The step of extracting multi-size features from the current input feature map, including the current original-size medical feature map and the current medical feature maps corresponding to each target size, includes: The current input feature map is downsampled to obtain downsampled feature maps of different sizes; Convolution is performed on the downsampled feature maps of different sizes to obtain first feature maps of different sizes, and convolution is performed on the feature maps of the original size to obtain second feature maps; The first feature map is upsampled step by step to obtain upsampled feature maps of different sizes; After merging the upsampled feature map with the first feature map and the second feature map of the corresponding size, the merged feature map is convolved to obtain the third feature map, wherein the third feature map of the original size is the current original size medical feature map; Upsampling is performed on the third feature map that is not of the original size to obtain the current medical feature map of the original size corresponding to each target size.

2. The medical image recognition method according to claim 1, characterized in that, The medical target prediction result is obtained by using a pre-trained medical model; wherein the medical model includes a multi-size feature extraction and fusion network and a skip connection network; The process of extracting multi-size features from the current input feature map, including the current original-size medical feature map and the current medical feature maps corresponding to each target size, and fusing the current original-size medical feature map and the current medical feature maps corresponding to each target size to obtain the current fused feature map, includes: The multi-size feature extraction and fusion network is used to extract multi-size features from the current input feature map, including the current original size medical feature map and the current medical feature maps corresponding to each target size. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map. The process of convolving the current fused feature map to obtain the current target feature map, merging the current original-size medical feature map and the current target feature map to obtain the next input feature map includes: The current target feature map is obtained by convolving the current fused feature map through the skip connection network. The current original-size medical feature map and the current target feature map are then merged to obtain the next input feature map.

3. The medical image recognition method according to claim 2, characterized in that, The multi-size feature extraction and fusion network includes a downsampling module, a first convolutional module, a first upsampling module, a merging module, a second convolutional module, a second upsampling module, and a fusion module; The process of extracting multi-size features from the current input feature map, including the current original-size medical feature map and the current medical feature maps corresponding to each target size, and fusing the current original-size medical feature map and the current medical feature maps corresponding to each target size to obtain the current fused feature map, includes: The current input feature map is processed sequentially through the downsampling module, the first convolution module, the first upsampling module, the merging module, the second convolution module, the second upsampling module, and the fusion module to extract multi-size features from the current input feature map. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map.

4. The medical image recognition method according to any one of claims 1 to 3, characterized in that, The method further includes: obtaining the location region of the medical target to be identified based on the medical target prediction results; The process of obtaining the location region of the medical target to be identified based on the medical target prediction results includes: Obtain the original size output result from the predicted medical target; The original size output result is processed based on the probability and probability threshold of each pixel in the original size output result; Extract the location region of the medical target to be identified from the processed original-size output.

5. The medical image recognition method according to claim 4, characterized in that, After obtaining the location region of the medical target to be identified based on the medical target prediction results, the process includes: The image coordinates of the outer contour of the target are obtained by collecting the boundary coordinate points of the location region of the medical target to be identified. The medical target to be identified is displayed in the medical image based on the image coordinates of the target's outer contour.

6. The medical image recognition method according to claim 5, characterized in that, The coordinate transformation of the image coordinate region of the outer contour of the medical target to be identified is performed to obtain the world coordinates of the outer contour of the target, including: Obtain the transformation relationship between the pixel coordinate system and the world coordinate system; Based on the transformation relationship, the image coordinates of the outer contour of the medical target to be identified are transformed to obtain the world coordinates of the outer contour of the target.

7. A method for training a medical model, characterized in that, The medical model training method includes: Acquire medical sample data, which includes sample medical images and corresponding target labels; The medical model uses a multi-size feature extraction and fusion network to extract multi-size features from the current input sample feature map, including the current original size sample medical feature map and the current sample medical feature maps corresponding to each target size. The current original size sample medical feature map and the current sample medical feature maps corresponding to each target size are then fused to obtain the current fused sample feature map, where the initial input sample feature map is the sample medical image. The current fused sample feature map is obtained by convolving the current target sample feature map through the skip connection network of the medical model. The current original-size sample medical feature map and the current target sample feature map are then merged to obtain the next input sample feature map. The next input sample feature map is used as the current input sample feature map, and the multi-size feature extraction and fusion network of the medical model is used to extract multi-size features from the current input sample feature map, including the current original size sample medical feature map and the current sample medical feature map corresponding to each target size, to obtain the prediction results of sample medical targets under different sizes. The medical model is optimized based on the prediction results of the medical targets of the samples at different sizes and the target labels to obtain a trained medical model. The process of extracting multi-size features from the current input sample feature map using a multi-size feature extraction fusion network based on a medical model includes extracting the current original-size sample medical feature map and the current sample medical feature maps corresponding to each target size. By using the multi-size feature fusion networks of the medical model, the current input sample feature map is downsampled to obtain downsampled sample feature maps of different sizes; Convolution is performed on the downsampled sample feature maps of different sizes to obtain first sample feature maps of different sizes, and convolution is performed on the feature maps of the original size to obtain second sample feature maps; The first sample feature map is upsampled step by step to obtain upsampled sample feature maps of different sizes; The upsampled sample feature map is merged with the first sample feature map and the second sample feature map of the corresponding size. The merged sample feature map is then convolved to obtain the third sample feature map, wherein the third sample feature map of the original size is the original size sample medical feature map. Upsampling is performed on the third sample feature map that is not of the original size to obtain the current medical sample feature map of the original size corresponding to each target size.

8. The medical model training method according to claim 7, characterized in that, The step of optimizing the medical model based on the prediction results of the medical targets of the samples at different sizes and the target labels to obtain the trained medical model includes: The loss function value is obtained based on the prediction results of the medical target of the sample under different sizes and the target label; The medical model is optimized based on the loss function value until the loss function value meets the requirements, thus obtaining a fully trained medical model.

9. The method according to claim 8, characterized in that, The step of obtaining the loss function value based on the prediction results of the medical target of the sample at different sizes and the target label includes: Obtain the first prediction results of the sample medical target at different sizes output by the last multi-size feature fusion network; Obtain the second prediction result corresponding to the original-size sample medical feature map output by the target multi-size feature fusion network in the medical model; Obtain the first weights corresponding to the first prediction result and the second prediction result, wherein the first weight corresponding to the second prediction result is less than the first weight corresponding to the first prediction result; The loss function value is calculated based on the first weight, the difference between the first prediction result and the target label, and the difference between the second prediction result and the target label.

10. The medical model training method according to claim 9, characterized in that, The loss function value is calculated based on the weights, the difference between the first prediction result and the target label, and the difference between the second prediction result and the target label, including: Obtain the second weights corresponding to the first prediction results of different sizes, and the second weight of the first prediction result of the larger size is greater than the second weight of the first prediction result of the smaller size; The first loss term is obtained based on the second weight and the difference between each of the first prediction results and the target label; A second loss term is obtained based on the second prediction result and the target label; The loss function value is calculated based on the first weight, the first loss term, and the second loss term.

11. A medical image recognition device, characterized in that, The device includes: The first feature extraction and fusion module is used to extract multi-size features from the current input feature map, including the current original size medical feature map and the current medical feature maps corresponding to each target size. The current original size medical feature map and the current medical feature maps corresponding to each target size are then fused to obtain the current fused feature map, wherein the initial input feature map is the medical image to be identified. The first skip connection module is used to convolve the current fused feature map to obtain the current target feature map, and merge the current original-size medical feature map and the current target feature map to obtain the next input feature map; The first target prediction result acquisition module is used to take the next input feature map as the current input feature map, and continue to extract multi-size features from the current input feature map, including the current original size medical feature map and the current medical feature map corresponding to each target size, to obtain the medical target prediction result corresponding to the medical image. The first feature extraction and fusion module is further configured to downsample the current input feature map to obtain downsampled feature maps of different sizes; convolve the downsampled feature maps of different sizes to obtain first feature maps of different sizes, and convolve the feature map of the original size to obtain second feature maps; upsample the first feature map step by step to obtain upsampled feature maps of different sizes; merge the upsampled feature maps with the first and second feature maps of corresponding sizes, and convolve the merged feature maps to obtain third feature maps, wherein the third feature map of the original size is the current original size medical feature map; upsample the third feature map of the non-original size to obtain the current medical feature map of the original size corresponding to each target size.

12. A medical model training device, characterized in that, The medical model training device includes: A medical sample data acquisition module is used to acquire medical sample data, which includes sample medical images and corresponding target labels; The second feature extraction and fusion module is used to extract multi-size features from the current input sample feature map through a multi-size feature extraction and fusion network of a medical model. This extracts the current original-size sample medical feature map and the current sample medical feature maps corresponding to each target size. The module then fuses the current original-size sample medical feature map and the current sample medical feature maps corresponding to each target size to obtain the current fused sample feature map. The initial current input sample feature map is the sample medical image. The second skip connection module is used to convolve the current fused sample feature map through the skip connection network of the medical model to obtain the current target sample feature map, and merge the current original size sample medical feature map and the current target sample feature map to obtain the next input sample feature map; The second target prediction result acquisition module is used to take the next input sample feature map as the current input sample feature map, and continue to extract multi-size features from the current input sample feature map through the multi-size feature extraction and fusion network of the medical model, so as to obtain the prediction results of the sample medical target under different sizes. The training module is used to optimize the medical model based on the prediction results of the medical targets of the samples at different sizes and the target labels, so as to obtain the trained medical model. The second feature extraction and fusion module is further configured to: downsample the current input sample feature map to obtain downsampled sample feature maps of different sizes through the multi-size feature fusion networks of the medical model; convolve the downsampled sample feature maps of different sizes to obtain first sample feature maps of different sizes, and convolve the original size feature map to obtain second sample feature maps; upsample the first sample feature map step by step to obtain upsampled sample feature maps of different sizes; merge the upsampled sample feature maps with the corresponding first and second sample feature maps, and convolve the merged sample feature maps to obtain third sample feature maps, wherein the original size third sample feature map is the original size sample medical feature map; and upsample the non-original size third sample feature map to obtain the original size current medical sample feature map corresponding to each target size.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6 or 7 to 10.

Images