Neck metastatic lymph node identification method and device, electronic equipment and storage medium

By performing non-local filtering and feature map fusion in the frequency domain, the recognition effect of neck metastatic lymph nodes is improved, the problem of poor recognition effect in the prior art is solved, and the recognition accuracy and efficiency are improved.

CN119027394AInactive Publication Date: 2024-11-26脉得智能科技(无锡)有限公司

Patent Information

Application Number
CN202411121304.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing ultrasound images have poor effect on the recognition of metastatic lymph nodes in the neck based on doctor's experience, and are greatly affected by ultrasound equipment and doctor's experience level, resulting in inaccurate recognition results.

Method used

By extracting the initial feature map of the ultrasound image of the neck lymph node in the spatial domain, and performing non-local filtering in the frequency domain, the modulated spectrum is obtained, and then converted to the spatial domain fusion feature map, it is used to improve the recognition effect of metastatic lymph nodes.

Benefits of technology

It improves the recognition accuracy and calculation efficiency of neck metastatic lymph nodes, reduces the calculation overhead of non-local filtering, enhances the expression ability of features, and improves the reliability of recognition results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027394A_ABST
    Figure CN119027394A_ABST
Patent Text Reader

Abstract

The invention provides a neck metastatic lymph node identification method and device, electronic equipment and a storage medium, and the method comprises the steps: extracting an initial feature map of a neck lymph node ultrasonic image; converting the initial feature map to a frequency domain, and obtaining an initial frequency spectrum of the initial feature map; performing non-local filtering on the initial frequency spectrum to obtain a modulation frequency spectrum; converting the modulation spectrum to a spatial domain, and obtaining a target feature map; and performing metastatic lymph node identification by using the target feature map to obtain an identification result. According to the scheme, the initial feature map of the ultrasonic image of the neck lymph node is extracted in the spatial domain, and then the initial frequency spectrum of the initial feature map is subjected to non-local filtering in the frequency domain to obtain the target feature map capable of more comprehensively representing the state information of the neck lymph node, so that the recognition effect of the metastatic neck lymph node is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of image analysis, and in particular to a method, device, electronic device and storage medium for identifying metastatic lymph nodes in the neck. Background Art

[0002] Lymph nodes are very important immune organs in the human body and are the places where antigen stimulation produces immune responses. Once tumors throughout the body invade the lymphatic system, they are likely to metastasize to the cervical lymph nodes, especially primary tumors in the head and neck area are very likely to cause cervical lymph node metastasis. Relevant studies have shown that if cervical lymph node metastasis occurs, the patient's 5-year survival rate will decrease by 5%. Therefore, timely detection of abnormal cervical lymph nodes and correct identification of the nature of the lymph nodes are of great clinical significance for the diagnosis and treatment of related diseases, especially for the determination of the staging of patients with malignant tumors, the formulation of subsequent treatment plans, and the tracking of efficacy.

[0003] At present, the related technology of identifying metastatic lymph nodes in the neck based on ultrasound images is usually based on the doctor's experience. The identification result is greatly affected by the ultrasound equipment and the doctor's experience level, resulting in poor metastatic lymph node identification effect. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for identifying cervical metastatic lymph nodes, so as to improve the recognition effect of metastatic lymph nodes.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying metastatic lymph nodes in the neck, the method comprising: extracting an initial feature map of an ultrasonic image of the neck lymph nodes; converting the initial feature map to the frequency domain to obtain an initial spectrum of the initial feature map; performing non-local filtering on the initial spectrum to obtain a modulation spectrum; converting the modulation spectrum to the spatial domain to obtain a target feature map; and using the target feature map to identify metastatic lymph nodes to obtain an identification result.

[0006] In the implementation process of the above scheme, the initial feature map of the cervical lymph node ultrasound image is extracted in the spatial domain, and then the initial spectrum of the initial feature map is non-locally filtered in the frequency domain to obtain a target feature map that can more comprehensively characterize the cervical lymph node status information, thereby improving the recognition effect of cervical metastatic lymph nodes.

[0007] In an implementation of the first aspect, the non-local filtering of the initial spectrum to obtain the modulated spectrum includes: using a non-local filter to non-locally filter the initial spectrum to obtain the modulated spectrum; wherein the non-local filter is used to integrate the global information in the initial feature map in the frequency domain.

[0008] In the implementation process of the above scheme, the initial spectrum is non-locally filtered by a non-local filter to obtain a modulated spectrum that integrates the global information in the initial feature map, which is beneficial to reducing the computational overhead of non-local filtering, and further beneficial to improving the computational efficiency of the above-mentioned cervical metastatic lymph node identification method.

[0009] In an implementation of the first aspect, the non-local filtering of the initial spectrum to obtain the modulated spectrum includes: using a non-local filter to non-locally filter the initial spectrum to obtain the modulated spectrum; wherein the non-local filter is used to integrate the global information in the initial feature map in the frequency domain.

[0010] In the implementation process of the above scheme, the initial spectrum is non-locally filtered by a pre-trained non-local filter. On the one hand, the use of a filter for filtering can reduce the computational overhead of non-local filtering, which is beneficial to improving the computational efficiency of the above-mentioned cervical metastatic lymph node recognition method; on the other hand, by pre-training the non-local filter, the interaction between different features in the frequency domain spatial position can be learned more efficiently, so that the extracted features can contain more comprehensive global information, thereby improving the cervical metastatic lymph node recognition effect of the above-mentioned cervical metastatic lymph node recognition method.

[0011] In an implementation of the first aspect, converting the modulation spectrum into the spatial domain to obtain a target feature map includes: converting the modulation spectrum into the spatial domain to obtain a modulation feature map; and fusing the modulation feature map and the initial feature map to obtain a target feature map.

[0012] In the implementation process of the above scheme, the modulation feature map and the initial feature map are fused to obtain a target feature map containing rich global information and rich local information. The target feature map is used to identify cervical metastatic lymph nodes, which is beneficial to improve the cervical metastatic lymph node recognition effect of the above cervical metastatic lymph node recognition method.

[0013] In an implementation of the first aspect, the initial feature map includes a first feature map and a second feature map, and extracting the initial feature map of the cervical lymph node ultrasound image includes: extracting the first feature map of the cervical lymph node ultrasound image; wherein the first feature map is a multi-scale feature map; performing multiple adaptive convolution operations on the first feature map with the smallest scale to obtain the second feature map; wherein the second feature map includes at least one of lymph node morphology features, lymph node boundary features, lymph node internal echo intensity features, lymph gate status features, and lymph node calcification features.

[0014] In the implementation process of the above scheme, at least one of the lymph node morphological features, lymph node boundary features, lymph node internal echo intensity features, lymph node gate status features and lymph node calcification features is extracted through adaptive convolution operation, thereby enhancing the expression ability of the extracted features, thereby improving the cervical metastatic lymph node recognition effect of the above-mentioned cervical metastatic lymph node recognition method.

[0015] In an implementation of the first aspect, extracting the first feature map of any scale of a cervical lymph node ultrasound image includes: using multiple feature extraction blocks fused with attention weights to extract the first feature map of the cervical lymph node ultrasound image; wherein the attention weight is used to indicate the importance of the feature map channel.

[0016] In the implementation process of the above scheme, when extracting the feature map, by fusing the attention weights, the feature map is made to pay more attention to the feature channels that are more important to the task, thereby improving the feature extraction capability, and further improving the cervical metastatic lymph node recognition effect of the above cervical metastatic lymph node recognition method.

[0017] In an implementation of the first aspect, performing multiple adaptive convolution operations on the first feature map with the smallest scale to obtain the second feature map includes: performing multiple adaptive convolution operations on the first feature map with the smallest scale; and splicing the feature maps obtained by the multiple adaptive convolution operations by channel to obtain the second feature map.

[0018] In the implementation process of the above scheme, the feature maps obtained by multiple adaptive convolution operations are spliced ​​by channel to fuse feature information of different dimensions, which is beneficial to improve the expression ability of the features and further improve the cervical metastatic lymph node recognition effect of the above cervical metastatic lymph node recognition method.

[0019] In a second aspect, an embodiment of the present application provides a device for identifying metastatic lymph nodes in the neck, the device comprising:

[0020] An initial feature map extraction unit, used for extracting an initial feature map of a cervical lymph node ultrasound image;

[0021] An initial spectrum acquisition unit, used for converting the initial feature map into a frequency domain to acquire an initial spectrum of the initial feature map;

[0022] A modulation spectrum acquisition unit, used for performing non-local filtering on the initial spectrum to acquire a modulation spectrum;

[0023] A target feature map acquisition unit, used for converting the modulation spectrum into a spatial domain to acquire a target feature map;

[0024] The recognition result acquisition unit is used to use the target feature map to perform metastatic lymph node recognition and obtain a recognition result.

[0025] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other through the communication bus; the memory stores computer program instructions that can be executed by the processor, and when the computer program instructions are read and run by the processor, the method provided by the first aspect or any possible implementation of the first aspect is executed.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the method provided by the first aspect or any possible implementation of the first aspect is executed.

[0027] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A schematic diagram of the process of identifying cervical metastatic lymph nodes provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the structure of a cervical metastatic lymph node identification model provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the structure of the Module_2 module in the model provided in the embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of the Module_3 module in the model provided in the embodiment of the present application;

[0033] Figure 5 A schematic diagram of the structure of the adaptive convolution ACM in the model provided in the embodiment of the present application;

[0034] Figure 6 A schematic diagram of the structure of the Module_4 module in the model provided in the embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of a cervical metastatic lymph node identification device provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0038] With the continuous updating and advancement of technology, imaging diagnosis has been widely used in predicting the status, staging and metastasis of cervical lymph nodes. It provides a non-invasive way to identify lymph nodes, which can reduce unnecessary lymph node biopsies or lymph node dissections for patients. Ultrasound imaging technology has the advantages of real-time, non-invasive, radiation-free and low cost, making it the preferred examination method for lymph node diseases.

[0039] At present, in the related technology, the identification of metastatic lymph nodes in the neck based on ultrasound images is usually performed based on the doctor's experience. The identification result is greatly affected by the ultrasound equipment and the doctor's experience level, resulting in poor metastatic lymph node identification effect.

[0040] Based on this, an embodiment of the present application provides a method for identifying metastatic cervical lymph nodes, which extracts an initial feature map of an ultrasound image of cervical lymph nodes in the spatial domain, and then performs non-local filtering on the initial spectrum of the initial feature map in the frequency domain to obtain a target feature map that can more comprehensively characterize the cervical lymph node status information, thereby improving the recognition effect of metastatic cervical lymph nodes.

[0041] The following is an introduction to the above-mentioned method of identifying metastatic cervical lymph nodes. Figure 1 The present application embodiment provides a method for identifying metastatic lymph nodes in the neck, the method comprising:

[0042] Step S110: extracting an initial feature map of the cervical lymph node ultrasound image;

[0043] Step S120: converting the initial feature map into the frequency domain to obtain an initial frequency spectrum of the initial feature map;

[0044] Step S130: performing non-local filtering on the initial spectrum to obtain a modulated spectrum;

[0045] Step S140: converting the modulation spectrum into the spatial domain to obtain a target feature map;

[0046] Step S150: using the target feature map to identify metastatic lymph nodes and obtain identification results.

[0047] It is understandable that the cervical lymph node ultrasound image in the above step S110 may be an image directly captured by the electronic device that executes the above cervical metastatic lymph node identification method, or may be an image sent by other electronic devices (such as an ultrasound image acquisition device) to the electronic device that executes the above cervical metastatic lymph node identification method. That is, the cervical lymph node ultrasound image input to the above cervical metastatic lymph node identification model 200 may be an ultrasound image directly captured by the electronic device that deploys the above cervical metastatic lymph node identification model 200, or may be an ultrasound image sent by other electronic devices to the electronic device that deploys the above cervical metastatic lymph node identification model 200.

[0048] The following is an introduction to the scheme of extracting the initial feature map in step S110:

[0049] As an optional implementation of the above-mentioned cervical metastatic lymph node identification method, the above-mentioned initial feature map may include a first feature map and a second feature map. In this case, step S110 includes: extracting a first feature map of the cervical lymph node ultrasound image; wherein the first feature map is a multi-scale feature map; performing multiple adaptive convolution operations on the first feature map with the smallest scale to obtain a second feature map; wherein the second feature map includes at least one of lymph node morphological features, lymph node boundary features, lymph node internal echo intensity features, lymph gate status features, and lymph node calcification features.

[0050] It can be understood that the "first" in the first feature map and the "second" in the second feature map do not refer to the number of feature maps, but to the distinction between different ways of obtaining feature maps. Among them, the first feature map can be a multi-scale feature map extracted by a feature extraction block, and the feature maps contained in the first feature map are, for example: a feature map with a size of 64×64, a feature map with a size of 32×32, and at least one feature map with a size of 16×16. The second feature map is a feature map extracted using adaptive convolution. Of course, the first feature map may contain one or more feature maps, and the second feature map may also contain one or more feature maps.

[0051] It can be understood that the above-mentioned adaptive convolution is an improved method of the traditional convolution operation, which better matches the local features of the input image by dynamically adjusting certain features of the convolution kernel. Through adaptive convolution, at least one of the lymph node morphological features, lymph node boundary features, lymph node internal echo intensity features, lymph node gate status features and lymph node calcification features can be extracted, thereby enhancing the expression ability of the extracted features.

[0052] The above scheme extracts at least one of the lymph node morphological features, lymph node boundary features, lymph node internal echo intensity features, lymph node gate status features and lymph node calcification features through adaptive convolution operation, thereby enhancing the expression ability of the extracted features, and further improving the cervical metastatic lymph node recognition effect of the above cervical metastatic lymph node recognition method.

[0053] As an optional implementation of the above-mentioned method for identifying metastatic cervical lymph nodes, the above-mentioned extraction of the first feature map of any scale of the cervical lymph node ultrasound image includes: using multiple feature extraction blocks fused with attention weights to extract the first feature map of the cervical lymph node ultrasound image; wherein the attention weight is used to indicate the importance of the feature map channel.

[0054] It is understandable that the above-mentioned multiple feature extraction blocks can be repeatedly stacked, for example, 24 times, that is, the input image of the feature extraction block is subjected to 24 consecutive feature extraction operations to obtain a feature map of a certain depth. The attention weight is used to characterize the importance of the feature map channel. By superimposing the attention weight on the feature map, the feature map can pay more attention to the feature channels that are more important to the task.

[0055] When extracting the feature map, the above scheme fuses the attention weights so that the feature map pays more attention to the feature channels that are more important to the task, thereby improving the feature extraction capability and further improving the cervical metastatic lymph node recognition effect of the above cervical metastatic lymph node recognition method.

[0056] As an optional implementation of the above-mentioned cervical metastatic lymph node identification method, multiple adaptive convolution operations are performed on the first feature map with the smallest scale to obtain a second feature map, including: performing multiple adaptive convolution operations on the first feature map with the smallest scale; and splicing the feature maps obtained by the multiple adaptive convolution operations according to channels to obtain the second feature map.

[0057] It can be understood that multiple adaptive convolution operations will obtain multiple feature maps, each of which may contain feature information of different dimensions. Therefore, by splicing the feature maps obtained by multiple adaptive convolution operations by channel, the feature information of these different dimensions can be fused, thereby improving the expressiveness of the features.

[0058] The above scheme splices the feature maps obtained by multiple adaptive convolution operations by channels, thereby fusing feature information of different dimensions, which is beneficial to improving the expression ability of the features, and further improving the cervical metastatic lymph node recognition effect of the above cervical metastatic lymph node recognition method.

[0059] It can be understood that in addition to concatenating multiple feature maps obtained by multiple adaptive convolution operations, one or more feature maps that are most useful for the task can also be selected, which is also conducive to the accurate expression of features.

[0060] The following is an introduction to the scheme of obtaining the initial spectrum in step S120:

[0061] The above step S120 can perform a two-dimensional discrete Fourier transform on the initial feature map along the spatial direction, so as to convert the feature map into the frequency domain, thereby obtaining an initial frequency spectrum containing all components in the initial feature map.

[0062] It is understandable that, since the initial feature map may include multiple feature maps, step S120 may extract the initial spectrum of each feature map respectively.

[0063] The following is an introduction to the scheme of performing non-local filtering on the initial spectrum in step S130 above:

[0064] As an optional implementation of the above-mentioned cervical metastatic lymph node identification method, the above-mentioned step S130 includes: using a non-local filter to perform non-local filtering on the initial spectrum to obtain a modulated spectrum; wherein the non-local filter is used to integrate the global information in the initial feature map in the frequency domain.

[0065] It can be understood that the above non-local filter can be a low-pass filter, a high-pass filter, a band-pass filter, etc. The non-local filter can filter different frequency components by superimposing different weights on different frequency components.

[0066] The above scheme performs non-local filtering on the initial spectrum through a non-local filter, thereby obtaining a modulated spectrum that integrates the global information in the initial feature map, which is beneficial to reducing the computational overhead of non-local filtering, and further beneficial to improving the computational efficiency of the above-mentioned cervical metastatic lymph node identification method.

[0067] As an optional implementation of the above-mentioned cervical metastatic lymph node identification method, the above-mentioned non-local filtering of the initial spectrum using a non-local filter includes: performing non-local filtering of the initial spectrum using a pre-trained non-local filter.

[0068] It can be understood that the above-mentioned non-local filter that can be pre-trained can be a learnable filter, and the learnable filter can directly learn the weights of each frequency component in the feature map from the original data set.

[0069] In addition, prior knowledge does not need to be introduced when training the learnable filter, so that the learnable filter can directly learn the weight distribution of frequency components from the data set, which is conducive to improving the generalization ability of the learnable filter.

[0070] The above scheme performs non-local filtering on the initial spectrum through a pre-trained non-local filter. On the one hand, using the filter for filtering can reduce the computational overhead of non-local filtering, which is beneficial to improving the computational efficiency of the above-mentioned cervical metastatic lymph node recognition method; on the other hand, by pre-training the non-local filter, the interaction between different features in the frequency domain spatial position can be learned more efficiently, so that the extracted features can contain more comprehensive global information, thereby improving the cervical metastatic lymph node recognition effect of the above-mentioned cervical metastatic lymph node recognition method.

[0071] The following is an introduction to the scheme of obtaining the target feature map in step S140:

[0072] As an optional implementation of the above-mentioned cervical metastatic lymph node identification method, the above-mentioned step S140 includes: converting the modulation spectrum into the spatial domain to obtain a modulation feature map; fusing the modulation feature map and the initial feature map to obtain a target feature map.

[0073] It is understandable that the modulation spectrum can be converted back to the spatial domain using a two-dimensional discrete Fourier transform. The above two-dimensional discrete Fourier transform and two-dimensional discrete Fourier transform are both relatively mature technologies in image processing technology. Please refer to the relevant technology for specific implementation methods, and the embodiments of this application will not be repeated.

[0074] It can also be understood that due to the non-local filtering, the above-mentioned modulation feature map contains relatively rich global information, while the initial feature map contains relatively rich local information. After fusing the modulation feature map with the initial feature map, a target feature map containing both relatively rich local information and relatively rich global information can be obtained.

[0075] The above scheme fuses the modulation feature map and the initial feature map to obtain a target feature map containing rich global information and rich local information, and uses the target feature map to identify cervical metastatic lymph nodes, which is beneficial to improve the cervical metastatic lymph node identification effect of the above cervical metastatic lymph node identification method.

[0076] In addition, it can be understood that since the initial feature map may contain multiple feature maps, steps S120 to S140 can obtain the corresponding target feature map for each feature map, and finally fuse the target feature map corresponding to each feature map in the initial feature map, and then use the fused target feature map to identify cervical metastatic lymph nodes to obtain the identification result.

[0077] The recognition result obtained in the above step S150 may include the location of the lymph nodes and their properties, and the properties of the lymph nodes may include, for example, metastatic lymph nodes and non-metastatic lymph nodes.

[0078] See also Figure 2 The above-mentioned cervical metastatic lymph node identification method also relates to a cervical metastatic lymph node identification model 200, which includes: an initial feature map extraction module 210, a non-local filtering module 220 and a recognition result acquisition module 230, wherein:

[0079] The initial feature map extraction module 210 is used to perform the initial feature map extraction step (i.e., step S110) in the above-mentioned cervical metastatic lymph node identification method;

[0080] The non-local filtering module 220 is used to execute the initial spectrum acquisition step (i.e., step S120), the modulation spectrum acquisition step (i.e., step S130), and the target feature map acquisition step (i.e., step S140) in the above-mentioned cervical metastatic lymph node identification method;

[0081] The recognition result acquisition module 230 is used to execute the recognition result acquisition step (ie, step S150) in the above-mentioned cervical metastatic lymph node recognition method.

[0082] It is understandable that after acquiring the ultrasonic image of the cervical lymph nodes, the ultrasonic image may be preprocessed first, and the preprocessing may include: adjusting the size of the ultrasonic image to 512×512; and normalizing the ultrasonic image to scale the pixel values ​​to between [0,1].

[0083] The initial feature map extraction module 210 is introduced as follows:

[0084] See also Figure 1 The initial feature map extraction module 210 includes modules Module_1, Module_2 and Module_3. The following describes these three modules respectively:

[0085] (1) Module_1: This module consists of a convolution layer and a pooling layer. The convolution layer includes three operations: convolution, batch normalization, and activation. The convolution kernel size of the convolution layer can be set to 7×7 and the step size is set to 2. The pooling layer uses the maximum pooling with a kernel size of 3×3 and a step size of 2. The convolution layer and the pooling layer reduce the size of the feature map of the input Module_1 module by 1 / 2, and finally obtain a feature map F with a size of 128×128. 1-1 .

[0086] (2) Module_2: There are three modules in total. The three Module_2 modules are connected in sequence. The input image of the first Module_2 module is the feature map F. 1-1 The three Module_2 modules obtain feature maps F of size 64×64 respectively 2-1 , feature map F with size 32×32 2-2 and a feature map F of size 16×16 2-3 .

[0087] See also Figure 3 In the Module_2 module, a repeatedly stacked feature extraction block is set, and the number of repeated stacking times can be set to 24.

[0088] In the feature extraction block, the input feature map is first subjected to feature fusion of the parameters of each channel by a convolution layer with a convolution kernel size of 1×1 to obtain the feature map F b-1 , feature map F b-1 Input to the attention weight vector generation module Att to generate the weight vector L representing the importance of each channel. Att contains three operations: global average pooling, convolution, and sigmoid activation. b-1 Multiply, that is, multiply the element value of vector L by the feature map F b-1 Multiply each element on the corresponding channel to get the same value as F b-1 Feature maps F of the same size b-2 . Feature map F b-2 Then a convolution layer with a convolution kernel size of 1×1 performs feature fusion on the parameters of each channel to obtain the feature map F b-3 Then the input feature map and the feature map F b-3 Addition, that is, add the parameter values ​​of the corresponding channels and corresponding index positions on the feature map, and use the activation function to activate it to obtain the output of a feature extraction block.

[0089] At the end of Module_2, there is a convolution layer with a convolution kernel size of 3×3 and a step size of 2, which is used to downsample the feature map and reduce the size of the feature map by 1 / 2.

[0090] The output feature map F of the last group of Module_2 modules 2-3 It is then input into Module_3 for feature extraction.

[0091] (3)Module_3: Please refer to Figure 4 Module_3 contains several parallel adaptive convolution operations ACM. By arranging multiple adaptive convolution operations of different scales in parallel, the feature map F is captured. 2-3 The different features that characterize the properties of lymph nodes include lymph node morphology, lymph node boundary, internal echo intensity of lymph nodes, lymph node gate status and lymph node calcification, so as to enhance the adaptability of the network to different features.

[0092] See also Figure 5 In adaptive convolution ACM, a convolution layer with a convolution kernel size of 3×3 is first used to map the feature map F 2-3 Perform channel adjustment to reduce the number of channels of the feature map and obtain the feature map F 3-1 , the height, width, and number of channels of the feature map are represented as h, w, and c respectively.

[0093] Use an adaptive average pooling operation (Adaptive Pooling) to the feature map F 2-3 Adaptive average pooling is performed to obtain a feature map of size k×k generated based on regional pooling, which is then processed by a convolution layer with a convolution kernel size of 1×1 to generate an adaptive convolution kernel. The height, width, and number of channels of the convolution kernel are denoted as k, k, and c, respectively.

[0094] The feature map F 3-1 Perform deep convolution operation with adaptive convolution kernel, that is, the feature map F 3-1 The parameters of each channel are convolved with the kernel parameters of the corresponding channel of the adaptive convolution kernel to obtain the feature map F 3-2 A convolutional layer is then used to fuse the information of each channel and adjust the number of channels of the output feature map.

[0095] The amount of data of the adaptive convolution operation ACM and the size of each convolution kernel are adaptively determined by the model during the training process. The outputs of multiple adaptive convolution operation ACMs representing different features are spliced ​​by channel, and a convolution layer with a step length of 2 is used to fuse the information output by each ACM, reduce the number of channels of the feature map, and reduce the size of the feature map. Finally, the final output feature map F of the Module_3 module is obtained. 3-3 , the size of the feature map is 8×8.

[0096] The non-local filtering module 220 is introduced below:

[0097] In the above-mentioned cervical metastatic lymph node recognition model 200, the non-local filtering module 220 includes four Module_4 modules, and the input feature maps of the four Module_4 modules are feature maps F 2-1 、F 2-2 、F 2-3 and F 3-3 , the four Module_4 modules respectively perform the feature map F 2-1 、F 2-2 、F 2-3 and F 3-3 Perform non-local filtering.

[0098] See also Figure 6 Module_4 mainly includes discrete Fourier transform FFT, spectrum adjustment and inverse discrete Fourier transform. Module_4 first converts the feature map from the spatial domain to the frequency domain and modulates the frequency information using a learnable filter. The learnable filter is learned directly from the original data set without introducing human prior knowledge, which allows the model to more efficiently learn the interaction between different features in the frequency domain spatial position, so that the module can use global information to enhance the input feature map and improve the performance of the model.

[0099] The input feature map of Module_4 is obtained by adjusting the number of channels through a convolutional layer Conv to obtain the feature map F 4-1 , along the spatial direction of the feature map F 4-1 Perform a two-dimensional discrete Fourier transform FFT to convert the feature map to the frequency domain and obtain a feature map F 4-1 The spectrum X of all frequency components.

[0100] The spectrum is modulated by multiplying the parameters of the learnable filter with the spectrum X. It is understandable that when the learnable filter is trained in the training phase, the dimension of the parameter K of the learnable filter needs to be set to the same dimension as the spectrum X.

[0101] Then Module_4 uses the two-dimensional discrete Fourier inverse transform IFFT to convert the filter-adjusted spectrum back to the spatial domain to obtain the feature map F 4-2 , due to the use of learnable filter K, the feature map F 4-2 Contains the global information of the ultrasound image. Then the feature map F 4-1 and F 4-2 Add and use a convolutional layer to fuse the local information in the spatial domain and the global information after frequency domain processing to obtain the output feature map F of module Module_4 4-3 .

[0102] The recognition result acquisition module 230 is introduced as follows:

[0103] In the above-mentioned cervical metastatic lymph node recognition model 200, the recognition result acquisition module includes Module_5 module and Module_6 module. The following introduces these two modules respectively:

[0104] (1) Module_5: It consists of a convolutional layer and an upsampling layer. The convolutional layer is used to fuse the information of the extracted feature map, and the upsampling layer is used to expand the size of the feature map. The upsampling is implemented by bilinear interpolation. The output of each Module_4 module is processed step by step by Module_5 to obtain a feature map F with a size of 256×256. 5-1 .

[0105] (2) Module_6: It consists of a convolutional layer and an upsampling layer. Feature map F 5-1 After extracting features through the convolutional layer, the number of channels is reduced to obtain a feature map F with 3 channels and a size of 256×256. 6-1 Then, bilinear interpolation is used to convert F 6-1 The size is expanded to 512×512, and the feature map F with 3 channels and a size of 512×512 is obtained. 6-2 . Feature map F 6-2 The parameter values ​​of the three channels represent the probability values ​​of the pixel being the metastatic lymph node area, the non-metastatic lymph node area, and the pixel in other areas. Module_6 compares the feature map by channel F 6-2 If the parameter value of the first channel is greater than that of the other two channels, the treatment is 2; if the parameter value of the second channel is greater than that of the other two channels, the treatment is 1; if the parameter value of the third channel is greater than that of the other two channels, the treatment is 0. Finally, each pixel value is multiplied by 128 and converted into a grayscale image to obtain the recognition result image representing the location and nature of the lymph nodes.

[0106] The following is an introduction to the training method of the above-mentioned cervical metastatic lymph node recognition model 200, which includes:

[0107] Step 1: Obtain training data;

[0108] Ultrasound images of the patient's cervical lymph nodes are obtained using ultrasound equipment; professionals annotate the collected ultrasound images, and the annotations are divided into lymph node outline annotations and property annotations. The properties of the lymph nodes are determined by the corresponding pathological diagnosis results and can be divided into two categories: metastatic lymph nodes and non-metastatic lymph nodes.

[0109] The labeled data is converted into a grayscale image by writing a program, where the grayscale value of the metastatic lymph node area is set to 255, the grayscale value of the non-metastatic lymph node area is set to 128, and the grayscale value of other areas is set to 0;

[0110] Since the image sizes collected from different ultrasound devices are not uniform, all images are resized to 512×512 before being input into the model. The images are then standardized and the pixel values ​​are scaled to between [0,1]. The standardized images are used as input images for the diagnostic model.

[0111] Step 2: Conduct model training;

[0112] The training dataset of cervical lymph node ultrasound images was divided into training set, validation set, and test set in a ratio of 8:1:1. The model was trained using the training set data. The Adam optimizer was used during the training process, and the weighted cross entropy loss function was used as the loss function. During the training process, the test set data was used to evaluate the performance of the saved diagnostic model. When the evaluation result reached the preset target, the model training was completed; if the evaluation result did not meet expectations, the parameters needed to be adjusted and the model training repeated.

[0113] See also Figure 7 Based on the same inventive concept, the present application also provides a cervical metastatic lymph node identification device 300, which includes:

[0114] An initial feature map extraction unit 310 is used to extract an initial feature map of a cervical lymph node ultrasound image;

[0115] An initial spectrum acquisition unit 320, configured to convert the initial feature map into a frequency domain to acquire an initial spectrum of the initial feature map;

[0116] A modulation spectrum acquisition unit 330, configured to perform non-local filtering on the initial spectrum to acquire a modulation spectrum;

[0117] A target feature map acquisition unit 340 is used to convert the modulation spectrum into a spatial domain to acquire a target feature map;

[0118] The recognition result acquisition unit 350 is used to use the target feature map to perform metastatic lymph node recognition and obtain a recognition result.

[0119] As an optional implementation of the above-mentioned cervical metastatic lymph node identification device, the above-mentioned modulation spectrum acquisition unit 330 is specifically used to: use a non-local filter to non-locally filter the initial spectrum to obtain a modulation spectrum; wherein the non-local filter is used to integrate the global information in the initial feature map in the frequency domain.

[0120] As an optional implementation of the above-mentioned cervical metastatic lymph node identification device, the above-mentioned modulation spectrum acquisition unit 330 is specifically used to: use a pre-trained non-local filter to perform non-local filtering on the initial spectrum.

[0121] As an optional implementation of the above-mentioned cervical metastatic lymph node identification device, the above-mentioned target feature map acquisition unit 340 is specifically used to: convert the modulation spectrum into the spatial domain to obtain a modulation feature map; and fuse the modulation feature map and the initial feature map to obtain a target feature map.

[0122] As an optional implementation of the above-mentioned cervical metastatic lymph node identification device, the above-mentioned initial feature map includes a first feature map and a second feature map, and the initial feature map extraction unit 310 is specifically used to: extract the first feature map of the cervical lymph node ultrasound image; wherein the first feature map is a multi-scale feature map; perform multiple adaptive convolution operations on the first feature map with the smallest scale to obtain the second feature map; wherein the second feature map includes at least one of lymph node morphology features, lymph node boundary features, lymph node internal echo intensity features, lymph gate status features, and lymph node calcification features.

[0123] As an optional implementation of the above-mentioned cervical metastatic lymph node identification device, the above-mentioned initial feature map extraction unit 310 is specifically used to: when extracting the first feature map of any scale of the cervical lymph node ultrasound image, use multiple feature extraction blocks fused with attention weights to extract the first feature map of the cervical lymph node ultrasound image; wherein the attention weight is used to indicate the importance of the feature map channel.

[0124] As an optional implementation of the above-mentioned cervical metastatic lymph node identification device, the above-mentioned initial feature map extraction unit 310 is specifically used to: perform multiple adaptive convolution operations on the first feature map with the smallest scale; splice the feature maps obtained by the multiple adaptive convolution operations according to channels to obtain the second feature map.

[0125] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present application. Figure 8 The electronic device 400 includes: a processor 410, a memory 420 and a communication interface 430. These components are interconnected and communicate with each other through a communication bus 440 and / or other forms of connection mechanisms (not shown).

[0126] The memory 420 includes one or more (only one is shown in the figure), which may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The processor 410 and other possible components may access the memory 420, and read and / or write data therein.

[0127] The processor 410 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 410 can be a general-purpose processor, including a central processing unit (CPU), a micro control unit (MCU), a network processor (NP) or other conventional processors; it can also be a special-purpose processor, including a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0128] The communication interface 430 includes one or more (only one is shown in the figure), which can be used to communicate directly or indirectly with other devices to exchange data. For example, the communication interface 430 can be an Ethernet interface; it can be a mobile communication network interface, such as an interface of a 3G, 4G, or 5G network; or it can be other types of interfaces with data transceiving functions.

[0129] One or more computer program instructions may be stored in the memory 420 , and the processor 410 may read and execute these computer program instructions to implement the cervical metastatic lymph node identification method and other desired functions provided in the embodiments of the present application.

[0130] Understandably, Figure 8 The structure shown is for illustration only. The electronic device 400 may also include Figure 8 More or fewer components as shown, or with Figure 8 Different configurations are shown. Figure 8 Each component shown in the figure can be implemented by hardware, software or a combination thereof. For example, the electronic device 400 can be a single server (or other device with computing processing capabilities), a combination of multiple servers, a cluster of a large number of servers, etc., and can be both a physical device and a virtual device.

[0131] The present application also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are read and executed by a computer processor, the method for identifying metastatic cervical lymph nodes provided in the present application is executed. For example, the computer-readable storage medium can be implemented as Figure 8 The memory 420 in the electronic device 400.

[0132] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0133] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0135] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0136] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0137] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0138] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0139] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying metastatic lymph nodes in the neck, It is characterized in that The method comprises: Extracting the initial feature map of the cervical lymph node ultrasound image; Converting the initial feature map to the frequency domain to obtain an initial frequency spectrum of the initial feature map; Performing non-local filtering on the initial spectrum to obtain a modulation spectrum; Convert the modulation spectrum into the spatial domain to obtain a target feature map; The target feature map is used to identify metastatic lymph nodes and obtain an identification result.

2. The method for identifying cervical metastatic lymph nodes according to claim 1, It is characterized in that The performing non-local filtering on the initial spectrum to obtain a modulated spectrum includes: The initial spectrum is non-locally filtered using a non-local filter to obtain a modulated spectrum; wherein the non-local filter is used to integrate the global information in the initial feature map in the frequency domain.

3. The method for identifying cervical metastatic lymph nodes according to claim 2, It is characterized in that The non-local filtering of the initial spectrum by using a non-local filter comprises: The initial spectrum is subjected to non-local filtering using a pre-trained non-local filter.

4. The method for identifying cervical metastatic lymph nodes according to claim 1, It is characterized in that The step of converting the modulation spectrum into a spatial domain to obtain a target feature map includes: Convert the modulation spectrum into the spatial domain to obtain a modulation feature map; The modulation feature map and the initial feature map are fused to obtain a target feature map.

5. The method for identifying cervical metastatic lymph nodes according to claim 1, It is characterized in that The initial feature map includes a first feature map and a second feature map, and the initial feature map of the cervical lymph node ultrasound image is extracted, including: Extracting the first feature map of the cervical lymph node ultrasound image; wherein the first feature map is a multi-scale feature map; Perform multiple adaptive convolution operations on the first feature map with the smallest scale to obtain the second feature map; wherein the second feature map includes at least one of lymph node morphology feature, lymph node boundary feature, lymph node internal echo intensity feature, lymph node gate state feature and lymph node calcification feature.

6. The method for identifying cervical metastatic lymph nodes according to claim 5, It is characterized in that Extracting the first feature map of any scale of the cervical lymph node ultrasound image includes: The first feature map of the cervical lymph node ultrasound image is extracted by using multiple feature extraction blocks fused with attention weights; wherein the attention weights are used to indicate the importance of the feature map channels.

7. The method for identifying cervical metastatic lymph nodes according to claim 5, It is characterized in that The step of performing multiple adaptive convolution operations on the first feature map with the smallest scale to obtain the second feature map includes: Performing multiple adaptive convolution operations on the first feature map with the smallest scale; The feature maps obtained by multiple adaptive convolution operations are spliced ​​by channel to obtain the second feature map.

8. A device for identifying metastatic lymph nodes in the neck, It is characterized in that The device comprises: An initial feature map extraction unit, used for extracting an initial feature map of a cervical lymph node ultrasound image; An initial spectrum acquisition unit, used for converting the initial feature map into a frequency domain to acquire an initial spectrum of the initial feature map; A modulation spectrum acquisition unit, used for performing non-local filtering on the initial spectrum to acquire a modulation spectrum; A target feature map acquisition unit, used for converting the modulation spectrum into a spatial domain to acquire a target feature map; The recognition result acquisition unit is used to use the target feature map to perform metastatic lymph node recognition and obtain a recognition result.

9. An electronic device, It is characterized in that include: A processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 7 by calling the program instructions.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Unsupervised video target segmentation method based on frequency domain global filtering

    CN116071748A

  • Cancerous metastasis neck lymph node identification system based on multi-scale feature fusion

    CN117115613A

  • Thyroid nodule ultrasound image segmentation method

    CN118172557A

Cited By

  • Feature fusion method, image segmentation method, electronic equipment and storage medium

    CN120147807A