Cervical lymph node identification method and device

Abstract

The present application relates to a method and device for identifying cervical lymph nodes; obtaining a CT image to be identified; respectively extracting detail features and spatial features of the CT image to be identified to obtain detail features and spatial features of the CT image to be identified; fusing the detail features and spatial features to obtain target fusion features; identifying cervical lymph nodes on the CT image to be identified based on the target fusion features to obtain an identification result; based on this, the detail information and overall spatial dependency information of the CT image to be identified can be fully considered, and then by fusing the detail features and spatial features, the network can be helped to capture information of multiple scales from coarse to fine; identifying cervical lymph nodes on the CT image to be identified based on the target fusion features can effectively avoid the problem of incorrect identification when the cervical lymph node structure changes are large, thereby improving the accuracy of the identification result.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for identifying cervical lymph nodes. Background Art

[0002] Thyroid cancer is the most common endocrine malignancy of the head and neck, most of which are papillary thyroid carcinoma (PTC), accounting for about 85.0%~90.0%. PTC mainly metastasizes through cervical lymph nodes. Although the most common is central lymph node metastasis, lateral cervical lymph node metastasis is an important risk factor for local recurrence and secondary surgery in PTC patients, and its harm is more serious than the primary lesion. When providing precision medicine to patients, it is crucial to clarify the scope of intraoperative lymph node dissection and accurately and effectively identify cervical lymph node metastasis, especially lateral cervical lymph node metastasis.

[0003] In the prior art, an automatic lymph node delineation algorithm is usually used to realize lymph node identification. However, in actual applications, when the neck tissue structure varies greatly, the existing automatic lymph node delineation algorithm is prone to misidentification, resulting in low lymph node identification accuracy.

[0004] There is currently no effective solution to the problem of low accuracy in cervical lymph node identification in the prior art. Summary of the invention

[0005] Based on this, it is necessary to provide a method and device for identifying cervical lymph nodes in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for identifying cervical lymph nodes, the method comprising:

[0007] Acquire a CT image to be identified;

[0008] Respectively extracting detail features and spatial features of the CT image to be identified to obtain detail features and spatial features of the CT image to be identified;

[0009] Fusing the detail feature and the spatial feature to obtain a target fusion feature;

[0010] Based on the target fusion feature, cervical lymph nodes are identified in the CT image to be identified to obtain an identification result.

[0011] In a second aspect, the present application also provides a cervical lymph node identification device, the device comprising:

[0012] An acquisition module, used for acquiring a CT image to be identified;

[0013] A feature extraction module, used to extract detail features and spatial features of the CT image to be identified, respectively, to obtain detail features and spatial features of the CT image to be identified;

[0014] A feature fusion module, used for fusing the detail feature and the spatial feature to obtain a target fusion feature;

[0015] The recognition module is used to identify the cervical lymph nodes in the CT image to be identified based on the target fusion feature to obtain a recognition result.

[0016] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the embodiments of the first aspect are implemented.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the embodiments of the first aspect above are implemented.

[0018] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect above.

[0019] The above-mentioned cervical lymph node identification method and device can fully consider the detail information and overall spatial dependency information of the CT image to be identified by respectively performing detail feature extraction and spatial feature extraction on the CT image to be identified; and then fuse the detail features and spatial features to obtain the target fusion feature; based on the target fusion feature, the cervical lymph node identification is performed on the CT image to be identified, and the local details and overall structure of the CT image can be comprehensively considered, so as to better identify the cervical lymph nodes and improve the accuracy of the cervical lymph node identification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a diagram of an application environment of a cervical lymph node identification method in one embodiment;

[0022] Figure 2 is a schematic flow chart of a method for identifying cervical lymph nodes in one embodiment;

[0023] Figure 3 is a schematic diagram of a convolutional neural network module in one embodiment;

[0024] Figure 4 is a schematic diagram of a Transformer module in one embodiment;

[0025] Figure 5 A schematic diagram of a flow chart of a step of obtaining target fusion features in one embodiment;

[0026] Figure 6 is a schematic diagram of a feature fusion upsampling module in one embodiment;

[0027] Figure 7 A schematic diagram of obtaining target fusion features in one embodiment;

[0028] Figure 8 It is a schematic flow chart of the step of obtaining the global features of the CT image to be identified in one embodiment;

[0029] Fig. 9 is a schematic diagram of a global context guidance module in one embodiment;

[0030] Fig.10 is a schematic diagram of a segmentation module in one embodiment;

[0031] Fig.11 A schematic diagram of a lymph node identification process in one embodiment;

[0032] Fig.12 A schematic diagram of a lymph node classification process in one embodiment;

[0033] Fig.13 is a structural block diagram of a cervical lymph node identification device in one embodiment;

[0034] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] The cervical lymph node identification method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers and laptops; the server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0037] At present, imaging examination is an important method for evaluating cervical lymph node metastasis in PTC patients, including computed tomography (CT) and magnetic resonance imaging. Among them, CT examination does not rely on operator experience, and the image quality is relatively stable; and the examination imaging range is wide, which is conducive to observing lymph nodes and other structures in the neck and adjacent areas. At present, in clinical practice, metastatic lymph nodes are mainly manually outlined by doctors on CT images, which usually takes 1-2 hours for one case. In addition, due to the individual structural differences of different cases, the outline results of different doctors are highly subjective. Therefore, this method is time-consuming and heavily dependent on the clinical experience of doctors. To solve the above problems, this embodiment provides a cervical lymph node recognition method that can effectively improve the recognition results.

[0038] In an exemplary embodiment, Figure 2 As shown, Figure 2 FIG. 1 is a flow chart of a method for identifying cervical lymph nodes in an embodiment; the method is applied to Figure 1 The terminal in the example is used for explanation. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0039] Step S201, obtaining a CT image to be identified.

[0040] The CT image to be identified may be, but is not limited to, a CT image of a patient with papillary thyroid carcinoma to be identified.

[0041] Step S202 , respectively extracting detail features and spatial features of the CT image to be identified, to obtain detail features and spatial features of the CT image to be identified.

[0042] Among them, detail feature extraction is used to extract detail features in the CT image to be identified; detail features may include but are not limited to edge information, texture information, etc.; spatial feature extraction is used to extract spatial features in the CT image to be identified; spatial features may include but are not limited to spatial dependency information; it can be understood that based on the spatial dependency information, the position, shape, contour and other information of the cervical lymph nodes in the CT image to be identified can be determined.

[0043] Step S203: fuse the detail features and the spatial features to obtain target fused features.

[0044] Exemplarily, detail feature extraction and spatial feature extraction are simultaneously performed on the input CT image to be identified to obtain detail features and spatial features corresponding to the CT image to be identified; further, feature fusion is performed on the detail features and spatial features to obtain target fusion features. It should be noted that the method of feature fusion of detail features and spatial features can be, but is not limited to, feature splicing, which is not specifically limited here.

[0045] Step S204: performing cervical lymph node recognition on the CT image to be recognized based on the target fusion features to obtain a recognition result.

[0046] Among them, cervical lymph nodes are lymphatic tissue structures located in the neck. They are an important part of the human immune system, responsible for filtering lymph and participating in immune responses. Among them, the recognition result is represented by a binary mask image; 1 in the binary mask image represents the area where the cervical lymph nodes are located, and 0 represents the background area.

[0047] In an exemplary embodiment, the cervical lymph nodes of the CT image to be identified are identified based on the target fusion feature, and the identification result can be obtained by: performing dimensionality reduction processing on the target fusion feature so that the channel dimension corresponding to the target fusion feature is reduced to 1; based on the Sigmoid activation function, each value in the target feature vector is mapped to the (0, 1) interval to obtain the prediction probability corresponding to each pixel in the CT image to be identified; based on the preset threshold and the prediction probability corresponding to each pixel, a corresponding binary mask image is generated to obtain the corresponding identification result. Among them, the preset threshold needs to be set according to the actual identification requirements and is not specifically limited here.

[0048] In this embodiment, by performing detail feature extraction and spatial feature extraction on the CT image to be identified, the detail information and spatial dependency information in the CT image to be identified can be accurately captured; then, by fusing the detail features and spatial features, the cervical lymph nodes are identified on the CT image to be identified based on the target fusion features, and the detail information and the overall spatial structure of the cervical lymph nodes are fully considered. Based on this, the problem of misidentification when the cervical lymph node structure changes greatly can be effectively avoided, thereby improving the accuracy of the cervical lymph node identification results.

[0049] In one embodiment, respectively extracting detail features and spatial features from the CT image to be identified to obtain detail features and spatial features of the CT image to be identified includes the following steps:

[0050] Step 1: extract detail features of the CT image to be identified through the detail branch feature extractor of the image segmentation model to obtain the detail features of the CT image to be identified.

[0051] Step 2: Extract spatial features of the CT image to be identified by using the spatial branch feature extractor of the image segmentation model to obtain the spatial features of the CT image to be identified.

[0052] Among them, the image segmentation model refers to a trained image segmentation model; wherein the image segmentation model includes a detail-space-dependent dual-branch feature extractor; wherein the detail-space-dependent dual-branch feature extractor includes a detail branch feature extractor and a space branch feature extractor; it can be understood that the detail branch feature extractor and the space branch feature extractor are two parallel feature extractors, which are used to simultaneously perform corresponding feature extraction on the CT image to be identified.

[0053] Exemplarily, the data of the CT image to be identified is input into the image segmentation model, and the detail-spatial dependency dual-branch feature extractor of the image segmentation model, i.e., the detail branch feature extractor and the spatial branch feature extractor, performs detail feature extraction on the CT image to be identified through the detail branch feature extractor to obtain the detail features of the CT image to be identified; and performs spatial feature extraction on the CT image to be identified through the spatial branch feature extractor to obtain the spatial features of the CT image to be identified, based on which the detail information and spatial dependency information of the CT image to be identified can be successfully captured; wherein the detail information includes edge information, texture information, etc.; the spatial dependency information is the spatial relationship and interdependence between pixels or voxels in the CT image to be identified.

[0054] In this embodiment, based on the detail branch feature extractor and the space branch feature extractor, the detail information and spatial dependency information of the cervical lymph nodes can be successfully captured, laying a foundation for improving the accuracy of the cervical lymph node recognition results.

[0055] In one embodiment, the detail branch feature extractor includes N stages of convolutional neural network modules. The detail branch feature extractor of the image segmentation model extracts detail features of the CT image to be identified, and obtains detail features of the CT image to be identified, including the following steps:

[0056] Step 1: For the convolutional neural network module of the first stage, the CT image to be identified is input into the convolutional neural network module of the first stage to obtain the detailed features of the first stage.

[0057] Among them, the detailed features are represented in the form of feature maps.

[0058] Step 2: For each convolutional neural network module in the convolutional neural network modules from the second stage to the Nth stage, the detail features output by the convolutional neural network module of the previous stage are input into the convolutional neural network module of the next stage to obtain the detail features of the corresponding stage.

[0059] The convolutional neural network module is composed of a preset number of first convolutional network blocks and second convolutional network blocks with residual connections stacked together; the preset number is denoted as c stage , preset quantity c stage , it needs to be adaptively adjusted according to the network depth, which is not specifically limited here; the first convolutional network block is composed of a 3D convolutional layer with a convolution kernel size of 1×1×1 and 3×3×3, a batch normalization layer (Batch Normalization, BN) and a PReLU activation function layer. The second convolutional network block is composed of a 3D convolutional layer with a convolution kernel size of 1×1×1 and 3×3×3, a group normalization layer (GroupNormalization, GN) and a PReLU activation function layer. It should be noted that since the normalization of the group normalization layer is performed within the group, it can ensure that the performance of the second convolutional network block is more stable under different batch sizes, so that the network remains stable during training, accelerates the training speed, and also has a certain regularization effect. By way of example, the schematic diagram of the convolutional neural network module is shown in the figure. Figure 3 shown.

[0060] In an exemplary embodiment, the detail branch feature extractor includes four stages of convolutional neural network modules connected in sequence; the input of the convolutional neural network module of the first stage is the CT image to be identified, the output of the convolutional neural network module of the first stage is the input of the convolutional neural network module of the second stage; the output of the convolutional neural network module of the second stage is the input of the convolutional neural network module of the third stage; the output of the convolutional neural network module of the third stage is the input of the convolutional neural network module of the fourth stage. The preset number c of the convolutional neural network modules of the four stages is stageThey are [4, 4, 2, 2] respectively; among them, the convolutional neural network modules in the first and second stages are responsible for extracting low-level features of the cervical lymph nodes in the CT images to be identified, such as edge information and texture information of the cervical lymph nodes. For the convolutional neural network modules in the third and fourth stages, as the network depth increases, the convolutional neural network modules begin to extract more complex semantic information. Through a large receptive field, the network can take into account the overall information of the CT image to be identified. Among them, the receptive field will increase accordingly with each additional layer of the network.

[0061] It should be noted that for the last convolutional layer in the convolutional neural network module of each stage, the step size of the convolutional layer is set to 2 to achieve downsampling, reduce the resolution of the feature map, and increase the receptive field so that the network can fully consider the overall information; it can be understood that setting the step size of the last convolutional layer in the convolutional neural network module of each stage to 2 means that the convolution kernel moves a distance of two pixels each time, so that the length, width and height of the feature map are halved, thereby reducing the resolution of the feature map.

[0062] It should be noted that the detail features corresponding to each stage are represented in the form of feature maps. The resolution of the detail features of each stage, that is, the resolution of the feature map, gradually decreases with the increasing number of stages of the convolutional neural network module, and the number of channels of the detail features of each stage, that is, the number of channels of the feature map, gradually increases with the increasing number of stages of the convolutional neural network module.

[0063] Exemplarily, assuming that the dimension of the CT image to be identified is 96×96×96, for the convolutional neural network module of the first stage, the CT image to be identified is input into the convolutional neural network module of the first stage to obtain the detail features of the first stage, and the dimension of the detail features of the first stage is 48×48×48×12; further, the detail features of the first stage are input into the convolutional neural network module of the second stage to obtain the detail features of the second stage, and the dimension of the detail features of the second stage is 24×24×24×24; for the convolutional neural network modules of each stage after the second stage, the detail features output by the convolutional neural network module of the previous stage are input into the convolutional neural network module of the next stage, and so on, the dimension of the detail features of the third stage is 12×12×12×48, and the dimension of the detail features of the fourth stage is 6×6×6×96.

[0064] In this embodiment, the convolutional neural network module based on multiple stages can fully extract the detail features in the CT image to be identified, thereby ensuring that more overall detail information can be captured, laying the foundation for improving the accuracy of cervical lymph node identification results.

[0065] In one embodiment, the spatial branch feature extractor includes N stages of Transformer modules; the spatial feature extractor of the CT image to be identified is used to extract the spatial features of the CT image to be identified, and the spatial features of the CT image to be identified are obtained, including the following steps:

[0066] Step 1: For the Transformer module of the first stage, the CT image to be identified is input into the Transformer module of the first stage to obtain the spatial features corresponding to the first stage.

[0067] Among them, the spatial features are represented in the form of feature maps.

[0068] Step 2: For each Transformer module in the Transformer modules from the second stage to the Nth stage, the spatial features output by the Transformer module of the previous stage are input into the Transformer module of the next stage to obtain the spatial features of the corresponding stage.

[0069] Exemplarily, assuming that the dimension of the CT image to be identified is 96×96×96, for the Transformer module of the first stage, the CT image to be identified is input into the Transformer module of the first stage to obtain the spatial features of the first stage, and the dimension of the spatial features of the first stage is 48×48×48×12; further, the spatial features of the first stage are input into the Transformer module of the second stage to obtain the spatial features of the second stage, and the dimension of the spatial features of the second stage is 24×24×24×24; for each Transformer module in each stage after the second stage, the spatial features output by the Transformer module of the previous stage are input into the Transformer module of the next stage, and so on, the dimension of the spatial features of the third stage is 12×12×12×48, and the dimension of the spatial features of the fourth stage is 6×6×6×96.

[0070] Preferably, step 1, for the Transformer module of the first stage, inputting the CT image to be identified into the Transformer module of the first stage to obtain the spatial features corresponding to the first stage, includes the following steps:

[0071] Step 1.1, divide the CT image to be identified into multiple image blocks.

[0072] Step 1.2, perform dimensionality reduction processing on multiple image blocks to obtain corresponding block embeddings.

[0073] Step 1.3, perform nonlinear transformation on the block embedding to obtain the spatial features corresponding to the first stage.

[0074] Similarly, step 2, for each Transformer module in the second stage to the Nth stage, the spatial features output by the Transformer module of the previous stage are input into the Transformer module of the next stage to obtain the spatial features of the corresponding stage, including the following steps:

[0075] Step 2.1, divide the spatial features of the previous stage to obtain multiple image blocks.

[0076] Step 2.2, perform dimensionality reduction processing on multiple image blocks to obtain corresponding block embeddings.

[0077] Step 2.3, perform nonlinear transformation on the block embedding to obtain the spatial features corresponding to the current stage.

[0078] Among them, the Transformer module includes a linear projection module (Linear Projection), a block embedding module (Patch Embedding) and a transformation module (Transformer Layer), such as Figure 4 shown.

[0079] The linear projection module is used to divide the input of each stage into multiple image blocks. For example, based on the linear projection module, the input of each stage is divided into N 3D Patch image blocks according to the preset position sequence. , the superscript i represents the serial number of the image block, and the dimension of the image block is P×P×P. The dimension of the image block needs to be set according to the actual segmentation requirements, and is not specifically limited here. For example, the dimension of the image block is 12×12×12. Each Patch image block after segmentation It can be expressed by formula (1).

[0080] (1)

[0081] Formula (1) represents the image block Belongs to the C channel The size of the matrix space is: C represents the Cth channel of the feature, and N is the number of image blocks; N is determined by formula (2).

[0082] (2)

[0083] in, , respectively represent the length, width and height of the CT image or spatial feature to be identified input by the Transformer module at each stage. Further, the 3D Patch image blocks are sequentially transformed into vectors of length P×P×P , that is, the Patch image block with a dimension of 12×12×12 is transformed into a vector of length 1728.

[0084] The block embedding module is used to perform dimensionality reduction processing on multiple image blocks to obtain corresponding block embeddings. For example, based on the block embedding module, the 3D Patch image blocks are The transformed vector , mapped to a vector space with an embedding dimension of D; the embedding dimension needs to be set according to the model performance and computational cost, and is not specifically limited here. For example, if the embedding dimension D is 1024, the block embedding module can embed N vectors of length 1728 , mapped to N embedding vectors of length 1024. Preferably, in order to encode the block spatial position information, a position embedding with dimension D is introduced , and add it to the space of the block embedding module to retain the position information. Based on this, we can get the first block embedding z of (N+1) dimensions D. The first block embedding z is shown in formula (3).

[0085] (3)

[0086] in, represents the block embedding projection transformation, represents the positional embedding of dimension D.

[0087] The transformation module includes a preset number of transformation layers (Transformer Layer), which is used to perform nonlinear transformation processing on the first embedding block to obtain the spatial features corresponding to the current stage; the preset number of layers is recorded as , preset number of layers , needs to be set according to the actual spatial feature extraction requirements, and is not specifically limited here. Among them, the Transformer layer includes a multi-head self-attention layer (MSA) and a multi-layer perceptron (MLP). The multi-layer perceptron MLP is composed of two fully connected layers and a nonlinear activation function to improve the training stability and performance of the model.

[0088] For example, for the transformation module in the Transformer module of each stage, the first block of embedding z is input into the transformation module. The first block of embedding z is first normalized by the LN (Layer Normalization) layer, that is, the data points in each layer of the first block of embedding z are subtracted from the mean of the layer and divided by the standard deviation of the layer. Based on the multi-head self-attention layer MSA, the first block of embedding z after layer normalization is converted into three matrices of Query, Key and Value. Then, by calculating the matching degree between Query and Key, the corresponding attention weight is determined, and the Value is weighted averaged based on the attention weight to obtain the second block of embedding, which is recorded as . Embed the second piece Sum with the first embedding z to get the third embedding , the third block is embedded As shown in formula (4).

[0089] (4)

[0090] Furthermore, the third block is embedded based on the LN layer After normalization, the normalized third block is embedded , after nonlinear processing by the multi-layer perception layer MLP, the fourth embedding is obtained , and embed the fourth block Embedded with the third block Sum and get the patch embedding, which is recorded as , As shown in formula (5).

[0091] (5)

[0092] It should be noted that for each stage of the Transformer module, the patch needs to be embedded into the Transform into the spatial dimension size corresponding to the input data of the corresponding stage; for example, for the first stage Transformer module, if the spatial dimension of the input CT image to be identified is 96×96×96, the patch needs to be embedded The dimension changes to 96×96×96.

[0093] Furthermore, each stage of the Transformer module also includes a third convolutional network block. The structure of the third convolutional network block is the same as that of the first convolutional network block, which will not be described in detail here. The convolutional layer step size in the third convolutional network block is set to 2 to reduce the feature resolution and increase the feature channel dimension. For example, the spatial branch feature extractor includes four stages of Transformer modules. For the first stage Transformer module, based on the third convolutional network block, the dimensionality-transformed patch is embedded By performing dimensionality reduction processing, the spatial features of the first stage can be obtained. The dimension of the spatial features of the first stage is 48×48×48×12. According to the method described in the above embodiment, by analogy, the dimension of the spatial features of the second stage is 24×24×24×24, the dimension of the spatial features of the third stage is 12×12×12×48, and the dimension of the spatial features of the fourth stage is 6×6×6×96.

[0094] In this embodiment, the Transformer module based on N stages can fully extract the spatial features in the CT image to be identified, thereby ensuring the successful capture of the spatial dependency information corresponding to the cervical lymph nodes in the CT image to be identified, laying the foundation for improving the accuracy of the cervical lymph node recognition results.

[0095] In one embodiment, Figure 5 As shown, Figure 5 The flowchart of the step of obtaining the target fusion feature in one embodiment is as follows; the detail feature and the spatial feature are fused to obtain the target fusion feature, including the following steps:

[0096] Step S501, for the convolutional neural network modules and Transformer modules with the same number of stages, the detail features and spatial features of the corresponding stages are fused to obtain the first intermediate fusion features of the corresponding stages.

[0097] The method for fusing the detail features and spatial features of the corresponding stages may be, but is not limited to, feature splicing.

[0098] It should be noted that the dimensions of the detail features output by the convolutional neural network module and the Transformer module with the same number of stages are the same as the dimensions of the spatial features; illustratively, taking the convolutional neural network module of the first stage and the Transformer module of the first stage as examples, when the dimension of the CT image to be identified is 96×96×96, the convolutional neural network module of the first stage performs detail feature extraction, and the dimension of the obtained detail features is 48×48×48×12; the Transformer module of the first stage performs spatial feature extraction, and the dimension of the obtained spatial features is 48×48×48×12, and then the detail features and spatial features with dimensions of 48×48×48×12 are spliced ​​to obtain the first intermediate fusion features of the first stage.

[0099] Step S502, performing feature fusion on the first intermediate fusion features of multiple stages through the fusion module of the image segmentation model to obtain the target fusion feature.

[0100] It should be noted that since the CT image to be identified is downsampled through N stages of convolutional neural network modules and N stages of Transformer modules, in order to ensure the accuracy of the target fusion features, it is necessary to perform feature fusion on the first intermediate fusion features of multiple stages based on the fusion module to help restore the details lost in the downsampling process and help the network capture information at multiple scales from coarse to fine.

[0101] Preferably, step S502, performing feature fusion on the first intermediate fusion features of multiple stages through a fusion module of the image segmentation model to obtain a target fusion feature, includes the following steps:

[0102] Step 1: perform fusion upsampling processing on the first intermediate fusion feature of the Nth stage to obtain the corresponding second intermediate fusion feature.

[0103] Step 2: In the order from stage N-1 to the first stage, for each stage, the second intermediate fusion feature obtained in the previous stage and the first intermediate fusion feature corresponding to the current stage are fused and upsampled until the first intermediate fusion feature of the first stage completes the fusion and upsampling process to obtain the target fusion feature.

[0104] It should be noted that the image segmentation model also includes a fusion module; the fusion module includes N feature fusion upsampling modules. Each feature fusion upsampling module includes splicing, residual unit and transposed convolution. The schematic diagram of the feature fusion upsampling module is as follows Figure 6As shown; the splicing in the feature fusion upsampling module is used to realize feature fusion and obtain intermediate fusion features, which include first intermediate fusion features and second intermediate fusion features; the residual unit is used to transfer low-level features to high-level features and reduce the channel dimension at the same time; the transposed convolution is used to realize upsampling of the intermediate fusion features of the corresponding stage and gradually restore the spatial resolution of the intermediate fusion features; wherein the size of the transposed convolution is 3×3×3.

[0105] For example, see Figure 7 , this embodiment takes the output of the four-stage convolutional neural network module and the four-stage Transformer module as an example for explanation, and the output of the four-stage convolutional neural network module and the output of the four-stage Transformer module are jump-connected to the feature fusion upsampling modules of the corresponding order; for example, the output of the first-stage convolutional neural network module and the output of the first-stage Transformer module are jump-connected to the corresponding feature fusion upsampling module 1, and so on. If the dimension of the CT image to be identified is 96×96×96, the dimension of the detail features of the first stage is 48×48×48×12; the dimension of the detail features of the second stage is 24×24×24×24, the dimension of the detail features of the third stage is 12×12×12×48, and the dimension of the detail features of the fourth stage is 6×6×6×96. The dimension of the spatial features of the first stage is 48×48×48×12, the dimension of the spatial features of the second stage is 24×24×24×24, the dimension of the spatial features of the third stage is 12×12×12×48, and the dimension of the spatial features of the fourth stage is 6×6×6×96.

[0106] Furthermore, based on the feature fusion upsampling module 4, the detail features with a dimension of 6×6×6×96 and the spatial features with a dimension of 6×6×6×96 of the fourth stage are firstly concatenated to obtain the first intermediate fusion features with a dimension of 6×6×6×192 of the fourth stage; further, based on the residual block unit and the transposed convolution, the first intermediate fusion features with a dimension of 6×6×6×192 are upsampled, that is, the channel dimension is reduced and the resolution is increased, so as to obtain the second intermediate fusion features corresponding to the fourth stage. The dimension of the second intermediate fusion features of the fourth stage is 1 2×12×12×48, that is, the dimension of the second intermediate fusion feature output by the feature fusion upsampling module 4 is 12×12×12×48; in the order of the third to the first stage, for each stage, the second intermediate fusion feature obtained in the previous stage and the first intermediate fusion feature corresponding to the current stage are fused and upsampled until the first intermediate fusion feature of the first stage completes the fusion and upsampling process to obtain the target fusion feature, wherein the output of the feature fusion upsampling module 1 is the target fusion feature, and the dimension of the target fusion feature is 96×96×96×6.

[0107] In this embodiment, for the convolutional neural network modules and Transformer modules with the same number of stages, the detail features and spatial features of the corresponding stages are fused to accurately obtain the first intermediate fusion features of the corresponding stages; then, based on the fusion module, the first intermediate fusion features of multiple stages are fused, which can help restore the details lost in the downsampling process and help the network capture information at multiple scales from coarse to fine, laying the foundation for improving the accuracy of cervical lymph node recognition.

[0108] In one embodiment, the cervical lymph node identification method further comprises the following steps:

[0109] Step 1: extract global features of the CT image to be identified to obtain global features of the CT image to be identified.

[0110] The method of extracting global features of the CT image to be identified may be, but is not limited to, extracting global features of the CT image to be identified through a global context guidance module in an image segmentation model to obtain global features of the CT image to be identified.

[0111] The global features may include, but are not limited to, spatial features corresponding to small-sized cervical lymph nodes.

[0112] Step 2: Fuse the detail features, spatial features, and global features to obtain the target fusion features.

[0113] It should be noted that in order to avoid the loss of global relative position information of small-sized cervical lymph nodes during the downsampling process, this embodiment can effectively enhance the expression of spatial information of small-sized cervical lymph nodes in the anatomical structure by performing global feature extraction on the CT image to be identified. Based on this, the problem of missed detection of small-sized cervical lymph nodes can be effectively avoided.

[0114] Exemplarily, a CT image to be identified is obtained, and detail feature extraction, spatial feature extraction and global feature extraction are performed on the CT image to be identified, respectively, to obtain detail features, spatial features and global features of the CT image to be identified; the detail features, spatial features and global features are fused to obtain target fusion features; based on the target fusion features, cervical lymph nodes are identified on the CT image to be identified to obtain an identification result; based on this, the multi-scale feature information corresponding to the CT image to be identified can be fully considered, thereby further improving the accuracy of the cervical lymph node identification result.

[0115] In one embodiment, Figure 8 As shown, Figure 8The flowchart of the step of obtaining the global features of the CT image to be identified in one embodiment is as follows; extracting the global features of the CT image to be identified to obtain the global features of the CT image to be identified includes the following steps:

[0116] Step S801 : extracting local features and neighborhood features of the CT image to be identified respectively, and obtaining local features and neighborhood features of the CT image to be identified.

[0117] It should be noted that the image segmentation model includes a global context guidance module, and the global context guidance module includes a local feature extractor and a surrounding context extractor.

[0118] Among them, the local feature extractor includes a convolution layer for extracting local features of the CT image to be identified; preferably, the size of the convolution layer is 3×3×3; the surrounding context extractor includes multiple consecutive dilated convolution layers for extracting neighborhood features of the CT image to be identified, and effectively learning the surrounding environment with a relatively large receptive field; preferably, the surrounding context extractor includes three consecutive dilated convolution layers, and the dilation coefficients of the dilated convolution layers are 1, 2, and 3, respectively.

[0119] Step S802: Fusing local features and neighborhood features to obtain joint features.

[0120] It should be noted that the global context guidance module also includes a feature fuser, which is used to fuse local features and neighborhood features to obtain corresponding joint features.

[0121] Step S803, determining the weight coefficient corresponding to each channel of the joint feature.

[0122] Among them, the weight coefficient is used to characterize the importance of each channel.

[0123] Step S804: Based on the weight coefficients corresponding to the channels, scale change processing is performed on the joint features to obtain global features of the CT image to be identified.

[0124] It should be noted that the global context guidance module also includes a global context extractor. The global context extractor is used to extract the global context information of the CT image to be identified to improve the joint features and obtain the global features of the CT image to be identified. The global context extractor includes a global average pooling layer and a fully connected layer. By introducing channel attention in the global context extractor, the weight coefficient, i.e., the importance, of each channel feature map in the joint features can be accurately obtained, so that the network pays more attention to the feature maps with heavy weights. The schematic diagram of the global context guidance module is shown in FIG. Fig. 9 shown.

[0125] It should be noted that based on the global average pooling layer and the fully connected layer, the global context of the corresponding channel can be aggregated. Finally, the learned weight coefficients of each channel are multiplied by all elements of the corresponding channel through the scale transformation operation to enhance important features and weaken unimportant features.

[0126] Exemplarily, the local feature extractor is instantiated as a 3×3×3 convolution layer, and the surrounding context extractor is instantiated as three consecutive dilated convolution layers, and the dilation coefficients of the dilated convolution layers are 1, 2, and 3, respectively. The CT image to be identified passes through the local feature extractor, and the output is a local feature with a dimension of 96×96×96×3. The CT image to be identified passes through the surrounding context extractor, and the output is a neighborhood feature with a dimension of 96×96×96×3. Based on the feature fusion device, the local features and neighborhood features are concatenated to obtain joint features, and the dimension of the joint features is 96×96×96×6. It should be noted that the joint features also need to be batch normalized BN and PReLU activated to improve the model performance. Based on the global context guidance module, the weight coefficients corresponding to each channel of the joint feature are determined; based on the weight coefficients corresponding to each channel, the joint features are scaled to obtain the global features of the CT image to be identified, and the dimension of the global features is 96×96×96×6.

[0127] In this embodiment, by obtaining the local features and neighborhood features of the CT image to be identified, fusing the local features and neighborhood features to obtain the joint features, and further improving the joint features to obtain the corresponding global features, the spatial position information of the cervical lymph nodes can be effectively enhanced, and the important position information of the cervical lymph nodes lost in the four-stage downsampling process of the detail-space-dependent dual-branch feature extractor can be reduced, laying the foundation for further improving the accuracy of the cervical lymph node recognition results.

[0128] In a specific embodiment, the image segmentation model also includes a segmentation module, which includes a convolution layer of size 3×3×3, a batch normalization layer, a PReLU activation function layer, a convolution layer of size 1×1×1, and a Sigmoid activation function layer. The schematic diagram of the segmentation module is as follows: Fig.10 The convolution layer in the segmentation module is used to gradually reduce the channel dimension of the target fusion feature to 1, and the Sigmoid activation function layer is used to convert the output of the convolution layer into the predicted probability of the cervical lymph node region category, and obtain the corresponding binary mask image, i.e., the recognition result, through the thresholding operation.

[0129] Preferably, performing cervical lymph node recognition on the CT image to be recognized based on the target fusion feature to obtain the recognition result includes the following steps:

[0130] Step 1: Perform dimensionality reduction processing on the target fusion features to obtain the corresponding target feature vector.

[0131] Step 2: Map the target feature vector to obtain the prediction probability corresponding to each pixel in the CT image to be identified.

[0132] Step 3: Determine the recognition result corresponding to the CT image to be recognized according to the preset threshold and multiple prediction probabilities.

[0133] Exemplarily, the target fusion feature is subjected to dimensionality reduction processing to obtain the corresponding target feature vector, so that the channel dimension corresponding to the target fusion feature is reduced to 1, and then based on the Sigmoid activation function, each value in the target feature vector is mapped to the (0, 1) interval to obtain the prediction probability corresponding to each pixel in the CT image to be identified; further, based on the preset threshold, the pixels with a prediction probability higher than the preset threshold are set to 1, and the pixels with a prediction probability not higher than the preset threshold are set to 0. Based on this, the corresponding binary mask image can be obtained, and the binary mask image is the recognition result corresponding to the CT image to be identified; wherein, 1 in the binary mask image represents the cervical lymph node area, and 0 represents the background area. It should be noted that the preset threshold needs to be set according to the actual recognition requirements, and is not specifically limited here. For example, the preset threshold is 0.5.

[0134] In an exemplary embodiment, a schematic diagram of the cervical lymph node identification process is shown in FIG. Fig.11 As shown. Obtain the CT image of the cervical lymph node to be identified and the image segmentation model; based on the four-stage convolutional neural network module, the four-stage Transformer module and the global context guidance module, respectively extract the detail feature, spatial feature and global feature of the CT image of the cervical lymph node to be identified, and obtain the detail feature, spatial feature and global feature of the CT image of the cervical lymph node to be identified; further, based on the comprehensive feature fusion module, fuse the detail feature, spatial feature and global feature to obtain the target fusion feature; based on the segmentation module, perform cervical lymph node recognition on the CT image of the cervical lymph node to be identified according to the target fusion feature, and obtain the recognition result.

[0135] In a specific embodiment, before obtaining a CT image of the cervical lymph nodes to be identified and an image segmentation model, it includes: obtaining a lymph node segmentation data set and an initial image segmentation model; training the initial image segmentation model according to the lymph node segmentation data set until the target loss function meets the preset convergence condition, thereby obtaining a trained image segmentation model.

[0136] Optionally, this embodiment is implemented on the Ubuntu 20.04.6 operating system, using Python (3.8) and PyTorch (version 1.12.1) deep learning framework to train the initial image segmentation model on an NVIDIA GeFore RTX 3090 GPU. The AdamW optimizer is used for training, with an initial learning rate of 1e-4, a momentum of 0.95, and a weight decay of 1e-5. The target loss function uses the Dice loss combined with the cross-entropy loss, that is, , where the weighting coefficient The Dice loss function and cross entropy loss formula are shown in (6)-(7).

[0137] (6)

[0138] (7)

[0139] in, , indicating that the i-th category of the true label is c pixels, , indicating that the i-th category of the predicted label is c pixels; C, indicates the number of categories; N, indicates the number of samples.

[0140] Furthermore, the segmentation performance of the image segmentation model is evaluated based on the average Dice Similarity Coefficient (DSC) and the average Hausdorff Distance (AVD). The Dice coefficient, also known as the overlap index, is the most commonly used metric for verifying medical body segmentation. The Dice coefficient is shown in formula (8).

[0141] (8)

[0142] in, Represents the true segmentation three-dimensional set of a single case sample in the test set, represents the three-dimensional set of model predictions for a single case sample in the test set, and the average DSC is defined as the mean of the Dice values ​​of all samples.

[0143] Among them, the average Hausdorff distance AVD is the average Hausdorff distance (HD) of all points. The average Hausdorff distance AVD is considered to be stable and less sensitive to outliers than the Hausdorff distance HD. The average Hausdorff distance AVD is shown in formula (9).

[0144] (9)

[0145] in, , represents the one-way Hausdorff distance from set A to set B, As shown in formula (10).

[0146] (10)

[0147] in, Represents the Euclidean distance from any point a in set A to any point b in set B; N represents the number of elements in set A.

[0148] In this embodiment, the image segmentation model is verified on the test set to obtain an average DSC of 0.785 and an average Hausdorff distance of 5.87, and the performance of the image segmentation model is excellent.

[0149] Optionally, obtaining a lymph node segmentation dataset comprises the following steps:

[0150] Step 1: Obtain an original sample data set; the original sample data set includes multiple original CT image samples.

[0151] Exemplarily, clinical data and original CT image data of patients diagnosed with papillary thyroid carcinoma by surgical pathology were collected. According to the inclusion and exclusion criteria, and to avoid the influence of any form of treatment on lymph node morphology information, 206 patients diagnosed with PTC for the first time were finally included, including 149 females and 57 males, with an age range of 16 to 78 years. Each original CT image consists of 168 to 328 slices of 512×512 pixels, with a voxel spatial resolution of ([0.48~0.54]×[0.48~0.54]×[0.98~1.21]) mm 3 .

[0152] Step 2: annotate multiple original CT image samples to obtain corresponding multiple original annotated images.

[0153] For example, two radiologists with more than three years of clinical experience used ITK-SNAP Version 3.8.0 interactive software to perform voxel segmentation and delineation of lymph nodes with a short axis diameter greater than 2 mm in all layers of the cervical zone II to V, combining semi-automatic delineation and manual correction delineation, where the pixel value of the lymph node area was set to 1 and the pixel value of the background area was set to 0. The results were reviewed by another radiologist with six years of CT imaging experience. The valid data finally included in the image segmentation model included 17,544 valid slices of 206 PTC patients and 4,691 lymph nodes with a short axis diameter of 2 mm to 18 mm with pixel-level annotation.

[0154] Step 3: preprocess the multiple original CT image samples and the original annotated images to obtain the corresponding multiple CT image samples and the multiple annotated images.

[0155] Preferably, step 3 comprises the following steps:

[0156] Step 3.1, performing image intensity normalization processing on a plurality of original CT image samples to obtain a corresponding plurality of first image samples.

[0157] For example, in clinical practice, a soft tissue window is set to observe the morphology of lymph nodes in cervical CT images. The window position (Window Width, WW) of the soft tissue window is usually set to 20~60HU. In order to identify lymph node details and improve brightness, the window position is set to 50HU, and a smaller window width of 400 (Window Center, WC) is set to improve image contrast. The minimum value of the original CT image sample pixel value before image intensity normalization processing and maximum value , the specific calculation formulas are shown in formulas (11) and (12).

[0158] (11)

[0159] (12)

[0160] in, , represents the minimum value of the original CT image sample pixel value; , represents the maximum value of the original CT image sample pixel value; WC, represents the window width; WW, represents the window position.

[0161] Furthermore, the original pixel value range of -150HU~250HU corresponding to the original CT image sample is linearly transformed to 0~1 to obtain the corresponding multiple first image samples; the linear transformation formula is shown in formulas (13)-(14).

[0162] (13)

[0163] (14)

[0164] in, Represents the new pixel value of each pixel after transformation. Equal to the original pixel value Subtract the original pixel value The minimum value (i.e. -250HU), then multiplied by the conversion coefficient , and finally add the minimum value of the new pixel value , here , Indicates the pixel value range after transformation is 0~1, Indicates the pixel value range before transformation is -150HU~250HU. It should be noted that unifying the intensity range of the image to a smaller range helps the model converge faster and improves the stability of the model.

[0165] Step 3.2: performing image size normalization processing on the plurality of first image samples to obtain a corresponding plurality of second image samples.

[0166] It should be noted that different scanning devices or scanning parameters may lead to inconsistent pixel spacing of the image. At present, the image size standardization processing methods usually include nearest neighbor interpolation and linear interpolation. However, nearest neighbor interpolation is prone to cause obvious block effects, and the function generated by linear interpolation is not necessarily smooth, which cannot meet the transformation requirements of complex boundary shape images such as lymph nodes. In this embodiment, three-dimensional B-spline interpolation is performed for each pixel (x, y, z) in the first image sample, and the interpolation transformation formula is shown in formula (15).

[0167] (15)

[0168] in, , represents the interpolation function in three-dimensional space; represents the pixel value of the first image sample at coordinates (i, j, k), , and are B-spline basis functions of order 3 in the x, y, and z directions, for example, As shown in formula (16).

[0169] (16)

[0170] in, is an indicator function. When |xx is satisfied i |<1, the value is 1, otherwise the value is 0. The pixel spacing of the first image sample can be adjusted to a uniform target value by 3D B-spline interpolation to ensure that the obtained second image sample has a consistent size. Based on the above image size normalization process, the pixel spacing of the first image sample and the original annotated image is adjusted in the x, y and z directions at the same time, so that the target pixel spacing is 1mm, 1mm and 1.5mm.

[0171] Step 3.3: crop the plurality of second image samples and the plurality of original annotated images according to a preset cropping ratio to obtain a corresponding plurality of CT image samples and a plurality of annotated images.

[0172] It should be noted that since the detection and segmentation of lymph nodes is mainly concentrated in the cervical lymph nodes and the surrounding areas, the diagnostic value of other areas of the CT image is small. If cropping is not performed, unnecessary calculation will be increased.

[0173] Exemplarily, the distribution range of lymph nodes of all samples in the data set is statistically analyzed, and the size distribution of the covered lymph nodes is determined to be 68×68×42~92×92×88. Considering that cropping a larger size requires more computing resources, and at the same time, to meet the requirements of the image segmentation model input size of 2 to the power of n, multiple second image samples and multiple original annotated images are cropped according to the preset cropping ratio of 96×96×96 (in pixels), and multiple corresponding CT image samples and multiple annotated images are obtained. It should be noted that the original annotated images in this embodiment have completed the image size standardization process.

[0174] Step 4: construct a lymph node segmentation dataset based on multiple CT image samples and multiple annotated images.

[0175] It can be understood that the CT image to be identified in any of the above embodiments is image data obtained by performing corresponding preprocessing on the original CT image to be identified in the manner described in the above embodiments, and the specific processing process will not be described in detail here.

[0176] In some other embodiments, based on the recognition result corresponding to the CT image to be recognized, the benign and malignant classification and recognition of cervical lymph nodes can be further realized. It should be noted that in this embodiment, the benign and malignant classification and recognition of cervical lymph nodes is realized based on the image classification model.

[0177] Among them, the image classification model is a trained image classification model; the backbone feature extraction network of the image classification model adopts a 3D dual path network (DPN); it should be noted that the DPN structure combines the advantages of the classification network ResNet and DenseNet, and combines the characteristics of jump connections and dense connections. DPN has achieved good performance through effective feature reuse and feature exploration capabilities. This structure not only retains the advantage of ResNet in solving deep network training problems through jump connections, but also inherits the advantages of DenseNet in feature reuse and transmission efficiency, achieving higher parameter efficiency and lower computational complexity. In addition, the design of DPN also reduces memory consumption and makes the network easier to optimize, which enables DPN to show excellent performance and efficiency when processing complex deep learning tasks.

[0178] Exemplarily, the method for classifying and identifying benign and malignant cervical lymph nodes includes: extracting features of the CT image to be identified based on the recognition result corresponding to the CT image to be identified, so as to obtain morphological features of the CT image to be identified; and classifying the benign and malignant cervical lymph nodes based on the morphological features to obtain a classification result.

[0179] In some other embodiments, when the recognition result corresponding to the CT image to be recognized is applied to the classification and recognition of benign and malignant cervical lymph nodes, after obtaining the original sample data set, the following step is further included:

[0180] From the original sample data set, multiple original classified CT image samples corresponding to lymph nodes with a short axis diameter greater than 5mm are screened, and multiple original classified CT image samples are annotated as benign or malignant to obtain multiple corresponding original classified annotated images; for example, lymph nodes with a short axis diameter greater than 5mm are outlined, where the pixel value of the benign nodule area is set to 1, and the pixel value of the malignant lymph node is set to 2, and 206 PTC patients finally obtain 185 metastatic lymph nodes (malignant lymph nodes) and 627 non-metastatic lymph nodes (benign lymph nodes). Further, the original classified CT image samples are preprocessed to obtain corresponding classified CT image samples; based on multiple classified CT image samples, a lymph node classification data set is constructed.

[0181] Exemplarily, the preprocessing of the original classified CT image samples may include, but is not limited to: 3D lymph node region cropping, image standardization, and data enhancement. Specifically, the 3D lymph node region cropping includes: traversing a single lymph node mask region according to the 3D lymph node region outlined by the original classified annotated image, cropping the corresponding region of interest of the original classified CT image sample, and saving the cropped single lymph node image region into a .npy file containing a voxel matrix.

[0182] It should be noted that since the image sizes of the cropped three-dimensional lymph nodes are different, and the input to the image classification model requires a fixed size, image standardization is required. The image standardization process is as follows: traverse the lymph nodes one by one, and the size distribution of the lymph nodes is statistically 12×12×22~28×28×30. Considering the image size of the lymph nodes and the required computing resources, the cropped lymph nodes are bilinearly interpolated and scaled to 32×32×32. And the voxel values ​​are normalized to zero mean and unit standard deviation. It should be noted that image standardization can reduce the differences between data, prevent the image classification model from being affected by data offset, and help improve the generalization ability of the model, so that it performs better on different data sets.

[0183] It should be noted that since the number of lymph nodes is relatively small, in order to achieve a certain number of training scales, data enhancement processing is required for the lymph nodes. Specifically, the data enhancement processing is to randomly flip, rotate within 30 degrees, and translate the cropped area of ​​the malignant lymph nodes along three axes, and save it to a new .npy file.

[0184] In some other embodiments, considering the order of magnitude of the classification samples and the training parameter computing resources of the model, the number of DPN stacks of the feature extraction backbone network of the image classification model is set to 3, 4, 20, and 3 in sequence to learn deeper features. Fig.12 , combined with mask annotation, the original classified CT image samples are subjected to 3D lymph node region cropping to obtain 3D lymph node images. Furthermore, the cropped 3D lymph node images are preprocessed by image normalization and other preprocessing before being input into the image classification model. Among them, the dimension of a single lymph node image after preprocessing is 32×32×32×1. When a single lymph node image is input into the image classification model, it will first be processed by a convolutional layer of size 3×3×3, and the output feature map dimension is 32×32×32×64. Next, the feature map is input into four dual-path blocks in sequence, and the feature map dimensions change to 32×32×32×64, 32×32×32×320, 16×16×16×672, 8×8×8×1528, and 4×4×4×2560. Furthermore, a three-dimensional average pooling layer is used to reduce the feature dimension to 2560, and the corresponding morphological features are obtained. Among them, the three-dimensional average pooling layer has the advantages of reducing the computational complexity of the model and increasing the robustness to small feature translations. Finally, a binary logistic regression layer is used to convert the input morphological features into the probability that the original classified CT image sample is a malignant class, and the corresponding classification results are obtained.

[0185] In some other embodiments, the SGD optimizer is used to train the image classification model, with an initial learning rate of 0.01, a momentum of 0.9, and a weight decay coefficient of 1e-5. The batch size is set to 32, the number of training rounds is set to 100 rounds, and the loss function uses Focal Loss, as shown in formula (17).

[0186] (17)

[0187] in, represents the probability that the model predicts the positive class, The modulation factor can be expressed by Control the imbalance of the number of simple and difficult samples, The weight factor can suppress the imbalance of the number of positive and negative samples. Optionally, according to the ratio of the number of positive and negative samples in training, is 1.2, is 0.75.

[0188] Optionally, the classification performance of the image classification model is evaluated by accuracy (Acc), recall, precision, F1Score and AUC value, as shown in formulas (18)-(21).

[0189] (18)

[0190] (19)

[0191] (20)

[0192] (twenty one)

[0193] Among them, TP (true positive): indicates that the lymph node is predicted to be positive and is actually positive; FP (false positive): indicates that the lymph node is predicted to be positive and is actually negative; FN (false negative): indicates that the lymph node is predicted to be negative and is actually positive; TN (true negative): indicates that the lymph node is predicted to be negative and is actually negative. In this example, the classification indicators of benign and malignant lymph nodes on the test set are Acc=0.84, Recall=0.86, Precision=0.77, F1=0.81, and AUC=0.91.

[0194] In a specific embodiment, the image segmentation model and the image classification model are comprehensively evaluated based on the average localization rate, the malignant lymph node recall rate, and the average number of false positives.

[0195] Among them, the average localization rate LR refers to the number of accurately located lymph nodes in the predicted lymph nodes to the number of real lymph nodes, that is, the average localization rate, which is equal to the ratio of the number of correctly located predicted lymph nodes to the number of real lymph nodes. The condition for determining whether the predicted lymph node location is accurate is: if the IoU (intersection over union) coefficient between a predicted lymph node and a real lymph node is greater than 0.5, then the lymph node is considered to be accurately located.

[0196] Among them, the malignant lymph node recall rate refers to the ratio of the number of predicted malignant lymph nodes TP to the number of real malignant lymph nodes, that is, the malignant lymph node recall rate is equal to the ratio of TP to the number of real malignant lymph nodes.

[0197] Among them, the average number of false positives refers to the ratio of the number of FP predicted to be malignant lymph nodes that are actually benign lymph nodes to the number of CT cases in the test set, that is, the average number of false positives is equal to the ratio of FP to the number of CT cases in the test set.

[0198] Specifically, the verification results of the two stages of joint segmentation and classification on the test set are as follows: The first stage is the segmentation stage. According to the recognition results corresponding to the CT image samples predicted by the image segmentation model, that is, the mask file, the CT image samples are cropped to obtain the classification data .npy file. 370 lymph nodes were predicted by the image segmentation model, of which 302 predicted lymph nodes were accurately located.

[0199] The second stage is the classification stage. The real benign and malignant lymph nodes in the test set are annotated with a total of 398 lymph nodes, including 225 benign lymph nodes and 173 malignant lymph nodes. The image segmentation stage predicts 370 lymph nodes, of which 188 are predicted as malignant lymph nodes and 114 are predicted as benign lymph nodes among the 302 lymph nodes with accurate positioning. Among them, 128 of the predicted malignant lymph nodes are actually malignant lymph nodes, a total of 45 malignant lymph nodes are missed, and there are 60 false positive lymph nodes among the predicted malignant lymph nodes.

[0200] Finally, the indicators of the two stages on the test set were an average localization rate of 0.76, a malignant lymph node recall rate of 0.74, and an average number of false positives of 1.40. The performance reached the expected level and can meet practical applications.

[0201] The above-mentioned cervical lymph node recognition method, on the one hand, can accurately capture the detail information and spatial dependency information in cervical lymph node recognition by extracting detail features and spatial features from the CT image to be identified; and then by fusing detail features and spatial features, it can help the network capture information of multiple scales from coarse to fine, laying a foundation for improving the accuracy of cervical lymph node recognition. Based on the target fusion feature, the cervical lymph node recognition of the CT image to be identified fully considers the detail information and overall spatial structure of the cervical lymph nodes, based on which the problem of misidentification in the case of large differences in cervical lymph node structural changes can be effectively avoided, thereby improving the accuracy of cervical lymph node recognition results. On the other hand, by obtaining the local features and neighborhood features of the CT image to be identified, and fusing the local features and neighborhood features to obtain the joint features, the joint features are further improved to obtain the corresponding global features, based on which the spatial position information of the cervical lymph nodes can be effectively enhanced, laying a foundation for further improving the accuracy of cervical lymph node recognition results.

[0202] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0203] Based on the same inventive concept, the embodiment of the present application also provides a cervical lymph node identification device for implementing the cervical lymph node identification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more cervical lymph node identification device embodiments provided below can refer to the limitations of the cervical lymph node identification method above, and will not be repeated here.

[0204] In an exemplary embodiment, Fig.13 As shown, a cervical lymph node identification device is provided, comprising: an acquisition module 1301, a feature extraction module 1302, a feature fusion module 1303 and an identification module 1304;

[0205] An acquisition module 1301 is used to acquire a CT image to be identified;

[0206] The feature extraction module 1302 is used to extract detail features and spatial features of the CT image to be identified, respectively, to obtain detail features and spatial features of the CT image to be identified;

[0207] The feature fusion module 1303 is used to fuse the detail feature and the spatial feature to obtain the target fusion feature;

[0208] The recognition module 1304 is used to recognize the cervical lymph nodes on the CT image to be recognized based on the target fusion feature to obtain a recognition result.

[0209] The above-mentioned cervical lymph node identification device can fully consider the detail information and overall spatial dependency information of the cervical lymph nodes by respectively extracting detail features and spatial features of the CT image to be identified; and then fuses the detail features and spatial features to obtain the target fusion features; based on the target fusion features, the cervical lymph nodes are identified on the CT image to be identified, and the local details and overall structure of the cervical lymph nodes can be comprehensively considered, so as to better identify the CT image to be identified and improve the accuracy of the cervical lymph node identification results.

[0210] In one embodiment, the feature extraction module 1302 is also used to

[0211] The detail feature extractor of the CT image to be identified is used to extract the detail feature of the CT image to be identified, thereby obtaining the detail feature of the CT image to be identified;

[0212] The spatial feature extractor of the CT image to be identified is used to extract the spatial features of the CT image to be identified, thereby obtaining the spatial features of the CT image to be identified.

[0213] In one embodiment, the detail branch feature extractor includes N stages of convolutional neural network modules; the feature extraction module 1302 is also used to

[0214] For the convolutional neural network module of the first stage, the CT image to be identified is input into the convolutional neural network module of the first stage to obtain the detailed features of the first stage;

[0215] For each convolutional neural network module in the convolutional neural network modules from the second stage to the Nth stage, the detail features output by the convolutional neural network module of the previous stage are input into the convolutional neural network module of the next stage to obtain the detail features of the corresponding stage.

[0216] In one embodiment, the spatial branch feature extractor includes N stages of Transformer modules; the feature extraction module 1302 is also used to

[0217] For the Transformer module of the first stage, the CT image to be identified is input into the Transformer module of the first stage to obtain the spatial features corresponding to the first stage;

[0218] For each Transformer module in the Transformer modules from the second stage to the Nth stage, the spatial features output by the Transformer module of the previous stage are input into the Transformer module of the next stage to obtain the spatial features of the corresponding stage.

[0219] In one embodiment, the feature extraction module 1302 is also used to

[0220] Divide the CT image to be identified to obtain multiple image blocks;

[0221] Perform dimensionality reduction on multiple image blocks to obtain corresponding block embeddings;

[0222] The block embedding is processed by nonlinear transformation to obtain the spatial features corresponding to the first stage.

[0223] In one embodiment, the feature fusion module 1303 is also used to

[0224] For the convolutional neural network modules and Transformer modules with the same number of stages, the detail features and spatial features of the corresponding stages are fused to obtain the first intermediate fusion features of the corresponding stages;

[0225] Through the fusion module of the image segmentation model, the first intermediate fusion features of multiple stages are fused to obtain the target fusion features.

[0226] In one embodiment, the feature fusion module 1303 is also used to

[0227] Perform fusion upsampling processing on the first intermediate fusion feature of the Nth stage to obtain the corresponding second intermediate fusion feature;

[0228] In the order from the N-1th stage to the first stage, for each stage, the second intermediate fusion feature obtained in the previous stage and the first intermediate fusion feature corresponding to the current stage are fused and up-sampled until the first intermediate fusion feature of the first stage completes the fusion and up-sampling process to obtain the target fusion feature.

[0229] In one embodiment, the feature extraction module 1302 is also used to

[0230] Performing global feature extraction on the CT image to be identified to obtain the global feature of the CT image to be identified;

[0231] Feature fusion module 1303 is also used for

[0232] The detail features, spatial features and global features are fused to obtain the target fusion features.

[0233] In one embodiment, the feature extraction module 1302 is also used to

[0234] Performing local feature extraction and neighborhood feature extraction on the CT image to be identified respectively, and obtaining local features and neighborhood features of the CT image to be identified;

[0235] Fuse local features and neighborhood features to obtain joint features;

[0236] Determine the weight coefficient corresponding to each channel of the joint feature;

[0237] Based on the weight coefficient corresponding to each channel, the joint feature is scaled to obtain the global feature of the CT image to be identified.

[0238] Each module in the above-mentioned cervical lymph node identification device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0239] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.14 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to cervical lymph node identification. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for identifying cervical lymph nodes is implemented.

[0240] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0241] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0242] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0243] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0244] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0245] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0246] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0247] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying cervical lymph nodes, It is characterized in that The method comprises: Acquire a CT image to be identified; the CT image to be identified includes a CT image of a patient with papillary thyroid carcinoma to be identified; Extracting detail features and spatial features of the CT image to be identified respectively, to obtain detail features and spatial features of the CT image to be identified; the detail features include edge information and texture information of the cervical lymph nodes; the spatial features include spatial dependency information; the spatial dependency information is used to determine the position, shape and contour of the cervical lymph nodes in the CT image to be identified; Fusing the detail feature and the spatial feature to obtain a target fusion feature; Based on the target fusion feature, the cervical lymph nodes are identified on the CT image to be identified, and an identification result is obtained; The method further comprises: Performing global feature extraction on the CT image to be identified to obtain global features of the CT image to be identified; the global features include spatial features corresponding to small-sized cervical lymph nodes; Fusing the detail feature, the spatial feature and the global feature to obtain a target fused feature; The extracting detail features and spatial features of the CT image to be identified respectively to obtain detail features and spatial features of the CT image to be identified includes: The detail feature extractor of the CT image to be identified is used to extract detail features of the CT image to be identified, so as to obtain detail features of the CT image to be identified; wherein the detail branch feature extractor includes a convolutional neural network module of N stages; the convolutional neural network module is composed of a preset number of first convolutional network blocks and second convolutional network blocks with residual connections stacked together; the first convolutional network block is composed of a 3D convolutional layer with convolution kernel sizes of 1×1×1 and 3×3×3, a batch normalization layer, and a PReLU activation function layer; the second convolutional network block is composed of a 3D convolutional layer with convolution kernel sizes of 1×1×1 and 3×3×3, a group normalization layer, and a PReLU activation function layer; The spatial feature extractor of the CT image to be identified is used to extract spatial features of the CT image to be identified, so as to obtain the spatial features of the CT image to be identified; wherein the spatial branch feature extractor includes N stages of Transformer modules; the Transformer module includes a linear projection module, a block embedding module and a transformation module; the linear projection module is used to divide the input of each stage to obtain multiple image blocks; the block embedding module is used to perform dimensionality reduction processing on the multiple image blocks to obtain corresponding block embeddings; the transformation module is used to perform nonlinear transformation processing on the block embeddings to obtain spatial features corresponding to the corresponding stages.

2. The method according to claim 1, It is characterized in that The detail branch feature extractor of the image segmentation model extracts detail features of the CT image to be identified to obtain detail features of the CT image to be identified, including: For the convolutional neural network module of the first stage, the CT image to be identified is input into the convolutional neural network module of the first stage to obtain the detail features of the first stage; For each convolutional neural network module in the convolutional neural network modules from the second stage to the Nth stage, the detail features output by the convolutional neural network module of the previous stage are input into the convolutional neural network module of the next stage to obtain the detail features of the corresponding stage.

3. The method according to claim 2, It is characterized in that The spatial branch feature extractor of the image segmentation model extracts spatial features of the CT image to be identified to obtain the spatial features of the CT image to be identified, including: For the Transformer module of the first stage, the CT image to be identified is input into the Transformer module of the first stage to obtain the spatial features corresponding to the first stage; For each Transformer module in the Transformer modules from the second stage to the Nth stage, the spatial features output by the Transformer module of the previous stage are input into the Transformer module of the next stage to obtain the spatial features of the corresponding stage.

4. The method according to claim 3, It is characterized in that The Transformer module for the first stage inputs the CT image to be identified into the Transformer module of the first stage to obtain the spatial features corresponding to the first stage, including: Dividing the CT image to be identified to obtain a plurality of image blocks; Performing dimensionality reduction processing on the plurality of image blocks to obtain corresponding block embeddings; The block embedding is subjected to nonlinear transformation processing to obtain the spatial features corresponding to the first stage.

5. The method according to claim 3, It is characterized in that The step of fusing the detail feature and the spatial feature to obtain a target fusion feature includes: For the convolutional neural network module and the Transformer module with the same number of stages, the detail features and spatial features of the corresponding stages are fused to obtain the first intermediate fused features of the corresponding stages; The first intermediate fusion features of multiple stages are fused through a fusion module of the image segmentation model to obtain target fusion features.

6. The method according to claim 5, It is characterized in that The step of fusing the first intermediate fusion features of the plurality of stages through the fusion module of the image segmentation model to obtain the target fusion features includes: Perform fusion upsampling processing on the first intermediate fusion feature of the Nth stage to obtain the corresponding second intermediate fusion feature; In the order from the N-1th stage to the first stage, for each stage, the second intermediate fusion feature obtained in the previous stage and the first intermediate fusion feature corresponding to the current stage are fused and up-sampled until the first intermediate fusion feature of the first stage completes the fusion and up-sampling process to obtain the target fusion feature.

7. The method according to claim 1, It is characterized in that The extracting global features of the CT image to be identified to obtain the global features of the CT image to be identified includes: Respectively performing local feature extraction and neighborhood feature extraction on the CT image to be identified to obtain local features and neighborhood features of the CT image to be identified; Fusing the local features and the neighborhood features to obtain joint features; Determine the weight coefficient corresponding to each channel of the joint feature; Based on the weight coefficient corresponding to each channel, the joint feature is scaled to obtain the global feature of the CT image to be identified.

8. A cervical lymph node identification device, It is characterized in that The device comprises: An acquisition module, used for acquiring a CT image to be identified; the CT image to be identified includes a CT image of a patient with papillary thyroid carcinoma to be identified; A feature extraction module, used to extract detail features and spatial features of the CT image to be identified, respectively, to obtain detail features and spatial features of the CT image to be identified; the detail features include edge information and texture information of the cervical lymph nodes; the spatial features include spatial dependency information; the spatial dependency information is used to determine the position, shape and contour of the cervical lymph nodes in the CT image to be identified; A feature fusion module, used for fusing the detail feature and the spatial feature to obtain a target fusion feature; A recognition module, used for performing cervical lymph node recognition on the CT image to be recognized based on the target fusion feature to obtain a recognition result; The feature extraction module is further used to extract global features of the CT image to be identified to obtain global features of the CT image to be identified; the global features include spatial features corresponding to small-sized cervical lymph nodes; The feature fusion module is further used to fuse the detail feature, the spatial feature and the global feature to obtain a target fusion feature; The feature extraction module is further used to extract detail features of the CT image to be identified through the detail branch feature extractor of the image segmentation model to obtain the detail features of the CT image to be identified; wherein the detail branch feature extractor includes a convolutional neural network module of N stages; the convolutional neural network module is composed of a preset number of first convolutional network blocks and second convolutional network blocks stacked with residual connections; the first convolutional network block is composed of a 3D convolutional layer with convolution kernel sizes of 1×1×1 and 3×3×3, a batch normalization layer and a PReLU activation function layer; the second convolutional network block is composed of a 3D convolutional layer with convolution kernel sizes of 1×1×1 and 3×3×3, a group normalization layer and a PReLU activation function layer; The spatial feature extractor of the CT image to be identified is used to extract spatial features of the CT image to be identified, so as to obtain the spatial features of the CT image to be identified; wherein the spatial branch feature extractor includes N stages of Transformer modules; the Transformer module includes a linear projection module, a block embedding module and a transformation module; the linear projection module is used to divide the input of each stage to obtain multiple image blocks; the block embedding module is used to perform dimensionality reduction processing on the multiple image blocks to obtain corresponding block embeddings; the transformation module is used to perform nonlinear transformation processing on the block embeddings to obtain spatial features corresponding to the corresponding stages.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images