A method and device for classification and management of nasopharyngoscope images

Abstract

The embodiment of the specification discloses a kind of for nasopharyngoscope image classification method, comprising: obtaining available image;By image classification model, the target feature map corresponding to available image is obtained, to determine the image classification result of available image;Target feature map includes: carrying out multiple rounds of feature extraction operation;First round of feature extraction operation includes: after available image is subjected to one or more convolution processing, feature map is obtained, the feature extraction result of this round of feature extraction operation is obtained by one or more parallel convolution processing to the feature map obtained;Each non-first round of feature extraction operation includes: the feature extraction result of previous round of feature extraction operation is obtained after one or more convolution processing, feature map is obtained, the feature extraction result of this round of feature extraction operation is obtained by one or more parallel convolution processing to the feature map obtained;Parallel convolution processing is used to extract different range of features;The feature extraction result of last round of feature extraction operation is used as the target feature map corresponding to available image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for classifying and managing nasopharyngeal laryngoscope images. Background Technology

[0002] In the field of nasopharyngoscopy, a large number of images need to be acquired for processing and analysis. Among these, how to accurately classify nasopharyngoscopy images and manage them based on this classification is a crucial issue. Summary of the Invention

[0003] This specification provides a method and apparatus for classifying and managing nasopharyngeal laryngoscope images, in order to solve the technical problem of how to classify and manage images more effectively.

[0004] To address the aforementioned technical problems, the embodiments in this specification provide the following technical solutions: This specification provides an embodiment of a classification method for nasopharyngeal laryngoscopy images, the method comprising: Get available images; The available image is input into the image classification model, the image classification model obtains the target feature map corresponding to the available image, and the image classification result of the available image is determined based on the target feature map; Obtaining the target feature map corresponding to the available image includes: Perform multiple rounds of feature extraction operations; The first round of feature extraction in the multi-round feature extraction operation includes: processing the available image through one or more convolutions to obtain a feature map, and processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of this round of feature extraction; the parallel convolution processing is used to extract features of different ranges. Each non-first round feature extraction operation in the multi-round feature extraction operation includes: obtaining a feature map by performing one or more convolution processes on the feature extraction result obtained from the previous round feature extraction operation; and obtaining the feature map by performing one or more parallel convolution processes on the obtained feature map to obtain the feature extraction result of the current round feature extraction operation; the parallel convolution process is used to extract features of different ranges. The feature extraction result obtained from the last round of feature extraction is used as the target feature map corresponding to the available image.

[0005] Optionally, the parallel convolution processing is performed by a parallel convolution module, which includes a large kernel DSConv and a small kernel convolution.

[0006] Optionally, the parallel convolution processing includes: The feature map channels are divided into n branches. The first branch is subjected to convolution with a kernel size of 3×3 and a stride of 1 for feature extraction. The remaining n-1 branches are subjected to large kernel DSConv for feature extraction. Among them, n≥4, the large kernel DSConv used for feature extraction of the remaining branches has a kernel size of not less than 7, and the kernel sizes of each other are different and the kernel size interval is not less than 4. The feature extraction results of each branch are concatenated, then convolved, and the convolved result is output through the specified output channel, and then passed through the BatchNorm layer and the Silu activation function; The result after passing through the Silu activation function is added to the input features processed by parallel convolution, and then output after passing through the BatchNorm layer.

[0007] Optionally, for any branch, feature extraction for that branch includes: This branch is then processed through convolution or a large kernel DSConv for feature extraction.

[0008] Optionally, if parallel convolution is performed multiple times in a single feature extraction operation, the result of the previous parallel convolution is used as the input for the next parallel convolution.

[0009] Optionally, the convolution processing includes: sequentially performing convolutional layer, BatchNorm layer, and Silu activation function processing.

[0010] Optionally, if convolution is performed multiple times in a single feature extraction operation, the result of the previous convolution is used as the input for the next convolution.

[0011] Optionally, the image classification result of the usable image determined based on the target feature map includes: The target feature map is flattened and then passed through a fully connected layer to determine the image classification result of the available image.

[0012] Optionally, the available image is a pre-processed image, the pre-processing including cropping borders and / or angular rotation.

[0013] This specification provides an embodiment of a method for managing nasopharyngeal laryngoscopy images, the method comprising: The image classification results of the available images are determined by the above image classification method. The image classification results are used to characterize the confidence level of the available images belonging to one or more categories. For any category, if the confidence level of an available image belonging to that category reaches a preset condition, then that available image is determined as the standard image of that category.

[0014] Optionally, the method further includes: If a standard image cannot be identified for a certain category, a corresponding prompt will be given.

[0015] This specification provides an embodiment of a classification device for nasopharyngeal laryngoscopy images, the device comprising: The image acquisition module is used to acquire available images; The image processing module is used to input the available image into the image classification model, obtain the target feature map corresponding to the available image from the image classification model, and determine the image classification result of the available image based on the target feature map; Obtaining the target feature map corresponding to the available image includes: Perform multiple rounds of feature extraction operations; The first round of feature extraction in the multi-round feature extraction operation includes: processing the available image through one or more convolutions to obtain a feature map, and processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of this round of feature extraction; the parallel convolution processing is used to extract features of different ranges. Each non-first round feature extraction operation in the multi-round feature extraction operation includes: obtaining a feature map by performing one or more convolution processes on the feature extraction result obtained from the previous round feature extraction operation; and obtaining the feature map by performing one or more parallel convolution processes on the obtained feature map to obtain the feature extraction result of the current round feature extraction operation; the parallel convolution process is used to extract features of different ranges. The feature extraction result obtained from the last round of feature extraction is used as the target feature map corresponding to the available image.

[0016] This specification provides an embodiment of a device for managing nasopharyngeal laryngoscopy images, the device comprising: The image classification module is used to determine the image classification result of the available image using the image classification method described above. The image classification result is used to characterize the confidence level of the available image belonging to one or more categories. The image determination module is used to determine an available image as the standard image of any category if the confidence level of an available image belonging to that category reaches a preset condition.

[0017] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By using parallel convolution processing to take into account and extract features of different ranges, it can achieve excellent extraction results for features of different ranges and enable the nasopharyngoscopy image classification model to have a larger receptive field, thereby effectively improving the feature extraction effect and thus improving the classification and management effect of nasopharyngoscopy images. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments of this specification or the prior art will be briefly described below. Obviously, the drawings used in some embodiments of this application are only described below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the image classification method provided in the first embodiment of this specification.

[0020] Figure 2 This is a schematic diagram of model training in the first embodiment of this specification.

[0021] Figure 3 This is a schematic diagram of the classification model in the first embodiment of this specification.

[0022] Figure 4 This is a schematic diagram of the PLKC module in the first embodiment of this specification.

[0023] Figure 5 This is a flowchart illustrating the image management method provided in the second embodiment of this specification.

[0024] Figure 6 This is a schematic diagram of the architecture of the image classification device provided in the third embodiment of this specification.

[0025] Figure 7 This is a schematic diagram of the architecture of the image management device provided in the fourth embodiment of this specification. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments involved in the specific implementation are only a part of the embodiments of this application, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in the specific implementation without creative effort should fall within the protection scope of this application.

[0027] The first embodiment of this specification (hereinafter referred to as "Embodiment 1") provides a classification method for nasopharyngeal endoscopy images. The executing entity of Embodiment 1 includes, but is not limited to, a terminal, a server, an operating system, or an application program; that is, the executing entity can be diverse and can be set, used, or changed as needed. Alternatively, a third-party application can assist the executing entity in performing Embodiment 1. For example, a server can execute the image classification method in Embodiment 1, and a corresponding application program can be installed on a terminal (which may be held by a user). Data transmission can occur between the terminal or application program and the server, thereby assisting the server in performing the image classification method in Embodiment 1.

[0028] like Figure 1 As shown, the classification method for nasopharyngeal laryngoscopy images provided in Embodiment 1 includes: S101: Obtain available images; Example 1: Images can be acquired through various suitable methods, and the content and source of the images are not limited.

[0029] The acquired image can be preprocessed, and the preprocessed image can be used as a usable image, thus obtaining a usable image. Preprocessing may include cropping borders and / or angular rotation.

[0030] If the acquired image does not require preprocessing, it can be used directly as a usable image.

[0031] S103: Input the available image into an image classification model, obtain the target feature map corresponding to the available image from the image classification model, and determine the image classification result of the available image based on the target feature map; wherein, obtaining the target feature map corresponding to the available image includes: performing multiple rounds of feature extraction operations; the first round of feature extraction operations in the multiple rounds of feature extraction operations includes: processing the available image through one or more convolutions to obtain a feature map, and processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of this round of feature extraction operations; the parallel convolution processing is used to extract features of different ranges; each non-first round of feature extraction operations in the multiple rounds of feature extraction operations includes: processing the feature extraction result obtained from the previous round of feature extraction operations through one or more convolutions to obtain a feature map, and processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of this round of feature extraction operations; the parallel convolution processing is used to extract features of different ranges; the feature extraction result obtained from the last round of feature extraction operations is used as the target feature map corresponding to the available image.

[0032] In Example 1, for any available image, the available image can be input into the image classification model, the image classification model can obtain the target feature map corresponding to the available image, and the image classification result of the available image can be determined based on the target feature map corresponding to the available image.

[0033] The process of obtaining the target feature map corresponding to the available image may include: performing multiple rounds of feature extraction operations on the available image, and using the feature extraction result obtained from the last round of feature extraction operations as the target feature map corresponding to the available image.

[0034] For any available image, the first round of feature extraction may include: processing the available image through one or more convolutions to obtain a feature map, and then processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of this round of feature extraction. The parallel convolution processing is used to extract features of different ranges.

[0035] Feature extraction can be performed in two or more rounds. For the second and subsequent rounds of feature extraction (if any), each round (i.e., each non-first round of feature extraction) may include: processing the feature extraction result from the previous round of feature extraction through one or more convolutions to obtain a feature map; and processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of the current round of feature extraction. The parallel convolution processing is used to extract features of different ranges.

[0036] In Example 1, a single convolutional process may include: sequentially performing convolutional layers, BatchNorm layers, and Silu activation function processing. The BatchNorm layer refers to a given mini-batch where the mean and variance of all samples in that mini-batch are calculated, and these statistics are then used to normalize each feature. The Silu activation function, also known as the Swish function, is a self-gated activation function that adaptively adjusts its output based on the input value. In a single feature extraction operation, if multiple convolutional processes are performed, the result of the previous convolutional process is used as the input for the next convolutional process. Here, "previous" and "next" refer to two consecutive processes, and the same applies below.

[0037] The following section provides further explanation of each round of feature extraction operations: 1. First round of feature extraction operation For any available image, in the first round of feature extraction, the available image undergoes a convolution operation, which means that the available image is passed through a convolutional layer, a BatchNorm layer, and a Silu activation function in sequence. Specifically, the result obtained after processing the available image through the convolutional layer is then processed through the BatchNorm layer, and the result obtained after processing through the BatchNorm layer is then processed through the Silu activation function. The result obtained after the Silu activation function is the feature extraction result of this convolution operation.

[0038] If multiple convolution processes are required in the first round of feature extraction, the result of the previous convolution process (i.e., the feature extraction result) is used as the input for the next convolution process.

[0039] After one or more convolution processes, the feature map of the usable image (or the feature map corresponding to the usable image) can be obtained. By performing one or more parallel convolution processes on the feature map of the usable image, the feature extraction result of the first round of feature extraction can be obtained.

[0040] 2. Non-first-round feature extraction operations In each subsequent round of feature extraction, the feature extraction results from the previous round are subjected to one or more convolutional processes. One convolutional process involves sequentially passing the feature extraction results through a convolutional layer, a BatchNorm layer, and a Silu activation function. Specifically, the result after the convolutional layer is processed by a BatchNorm layer, and then by a Silu activation function. The final result after the Silu activation function is the feature extraction result for this convolutional operation.

[0041] If multiple convolution operations are required, the result of the previous convolution process (i.e., the feature extraction result) is used as the input for the next convolution process.

[0042] After one or more convolution processes, a feature map (still a feature map of the usable image) can be obtained. By performing one or more parallel convolution processes on the newly obtained feature map, the feature extraction result of this round of feature extraction can be obtained.

[0043] In each of the above feature extraction rounds, a feature map is obtained after each convolution process. Each round of feature extraction also yields a feature extraction result (which can be a feature map).

[0044] The parallel convolution processing in each round of feature extraction is explained below: Each parallel convolution process can be executed by a parallel convolution module, also known as the PLKC (Parallel Large Kernal Convolution) module, which is used to extract features of different ranges.

[0045] A single parallel convolutional module can include multiple convolutions, some of which are large-kernel convolutions and others are small-kernel convolutions. The kernel size of the large-kernel convolutions is larger than that of the small-kernel convolutions. The large-kernel convolutions can be large-kernel DSConv (depthseparable convolution). Preferably, the kernel size of the large-kernel DSConv is not less than 7, and the kernel sizes of each large-kernel DSConv are different, with a kernel size interval of not less than 4. Small-kernel convolutions can be selected as needed, for example, a convolution with a kernel size of 3×3 and a stride of 1.

[0046] In Example 1, the single parallel convolution module also includes a BatchNorm layer and a Silu activation function.

[0047] In Example 1, a single parallel convolution process may include: The feature map channels are divided into multiple branches (e.g., n branches, with the ratio of the number of channels in each branch being C1:C2:C3:...:Cn, preferably n>=4, and the number of branches is the same as the number of convolutions (including large kernel convolution and small kernel convolution) in a single parallel convolution module. Here, n, C1, C2, C3,...Cn are all hyperparameters that can be determined or adjusted based on runtime and model performance). Some branches are extracted using convolutions (i.e., small kernel convolutions in the parallel convolution module), while the remaining branches are extracted using large kernel DSConv (i.e., large kernel DSConv in the parallel convolution module). For example, the first branch is extracted using a 3×3 kernel with a stride of 1, and the remaining n-1 branches are extracted using large kernel DSConv (i.e., each of the remaining n-1 branches is extracted using its corresponding large kernel DSConv). Preferably, the major kernels DSConv used for feature extraction of the remaining n-1 branches satisfy the following: the convolution kernel of a single major kernel DSConv is not less than 7, the convolution kernel sizes of the major kernels DSConv are different from each other, and the kernel size interval between any two major kernels DSConv is not less than 4. The feature extraction results of each branch are concatenated, then convolved, and the convolved result is output through the specified output channel, and then passed through the BatchNorm layer and the Silu activation function; The result after the Silu activation function (i.e., the feature map) is added to the input feature map of this parallel convolution process, and then passed through the BatchNorm layer before outputting (the output is also a feature map).

[0048] For any branch, feature extraction can include passing the branch through a convolution or a large kernel DSConv to extract features.

[0049] As mentioned earlier, in a single feature extraction operation, the feature map undergoes one or more parallel convolution processes, thus involving one or more parallel convolution modules. If multiple parallel convolution processes are performed in a single feature extraction operation, the result of the previous parallel convolution process is used as the input for the next parallel convolution process.

[0050] In Example 1, the feature extraction result obtained from the last round of feature extraction is used as the target feature map corresponding to the usable image, and the image classification result of the usable image is determined based on the target feature map. Determining the image classification result of the usable image based on the target feature map may include: performing a flattening operation on the target feature map, followed by passing it through a fully connected layer to determine the image classification result of the usable image. The flattening operation is an operation that converts a multidimensional tensor into a one-dimensional vector. During the flattening operation, the target feature map can be convolved to change its number of channels.

[0051] After performing multiple rounds of feature extraction, the target feature map can be convolved as needed. Then, the image classification result of the usable image is determined based on the target feature map after convolution. Example 1 is not limited to this. Since the usable image may be obtained by preprocessing the original image, the image classification result of the usable image can also be equivalent to the image classification result of the original image.

[0052] In Example 1, the image classification model includes a backbone network and a classification module. The multi-round feature extraction operations described above can be performed by the backbone network; that is, the convolutional layers, BatchNorm layers, Silu activation functions, and parallel convolutional modules in each round of feature extraction can all be included in the backbone network. The image classification result, which determines the usable image based on the target feature map, can be performed by the classification module. The classification module can include a classification head.

[0053] The first few layers of the backbone network (which can be preset and are not limited in specific terms) can perform convolutional processing in each round of feature extraction, while subsequent layers perform parallel convolutional operations in each round of feature extraction.

[0054] Image classification models can be pre-built. For example, a network based on the aforementioned parallel large-kernel convolution can be constructed, and training, testing, and validation sets can be established with corresponding dataset labels. The model is then trained using a training pair, tested and validated, and the model weights that meet the requirements are saved to obtain the image classification model, such as... Figure 2 As shown.

[0055] The graphic classification method provided in Embodiment 1 is further illustrated below with specific examples.

[0056] Laryngeal diseases, including infections, foreign bodies, injuries, and tumors, seriously affect people's health and lives. The larynx is a vital organ for respiration and voice production, and larynx diseases can impact these functions to varying degrees. Furthermore, larynx diseases can cause systemic pathological changes and may also be a reaction to systemic diseases; therefore, the significance of examining and treating larynx diseases extends far beyond simply addressing the larynx itself. Due to the larynx's unique location and complex physiological structure, direct visualization is often impossible. When diagnosing larynx diseases, doctors typically obtain internal information by inserting a nasopharyngoscope into the larynx to take images, which then informs diagnosis and treatment. Both doctors and patients hope that efficient and accurate nasopharyngoscopy techniques can help diagnose conditions and facilitate timely treatment.

[0057] Routine nasopharyngoscopy is a technical procedure directly related to the operator's experience and sense of responsibility. Standardized nasopharyngoscopy is an important prerequisite for avoiding missed or misdiagnosed lesions. If the examination and image acquisition process lacks external supervision, creating blind spots and failing to cover all anatomical areas of the nasopharynx and larynx, early malignant tumors hidden in the narrow anatomical regions of the nasopharynx and larynx can easily be missed, causing patients to miss the opportunity for early diagnosis and treatment of tumors.

[0058] To address the aforementioned issues, the image classification method for nasopharyngeal laryngoscopy images provided in Example 1 can more effectively classify nasopharyngeal laryngoscopy images, helping to improve the standardization of nasopharyngeal laryngoscopy examinations and image acquisition, thereby enhancing the diagnostic accuracy of nasopharyngeal diseases. The following example illustrates this method: Based on the location of the nasopharyngeal laryngoscopy images, they can be divided into 21 categories (20 locations + background), including left nasal cavity, right nasal cavity, left nasopharynx, right nasopharynx, oropharynx and hypopharynx (distant view), left pharyngeal folds, right pharyngeal folds, root of tongue and vallecula, soft palate (midline), left tonsils, right tonsils, hypopharynx and larynx (midline view, diaphragmatic phase), hypopharynx (midline view, exposing the postcricoid region), left pyriform fossa, right pyriform fossa, panoramic view of the larynx (inspiratory phase), close-up view of both vocal cords (inspiratory phase), panoramic view of the oral cavity, hard palate, floor of mouth, and background.

[0059] It can be obtained through appropriate means (including existing means). NasopharynxThe laryngoscope images can be preprocessed as needed, such as cropping the original images with black borders (the original images typically contain invalid information like dates) and random rotation. Based on the location of the laryngoscope image, it can be classified (or preprocessed if necessary) and assigned corresponding labels (training, testing, and validation sets) to establish training, testing, and validation sets. A network based on the parallel large-kernel convolutions described above is constructed, trained, tested, and validated, and the model weights that meet the requirements are saved to obtain the image classification model.

[0060] In practical applications, nasopharyngeal laryngoscope images are acquired as usable images. If preprocessing is required, the image obtained after preprocessing is used as the usable image. The usable image is input into an image classification model, which obtains the target feature map corresponding to the usable image. The image classification result of the usable image is determined based on the target feature map. The image classification result can include which category the usable image belongs to, for example, which of the aforementioned 20 body parts it belongs to, and give the probability. Images that do not belong to the aforementioned 20 body parts are marked as background images, such as images outside the human body.

[0061] refer to Figure 3 The working process of the image classification model in this example may include the following: Step 1: The input available image is subjected to a convolution with a kernel size of 3×3 and a stride of 2. After the convolution, it is passed through a BatchNorm layer and then through the Silu activation function to obtain a feature map. The feature map is then subjected to a convolution with a kernel size of 3×3 and a stride of 2. After the convolution, it is passed through a BatchNorm layer and then through the Silu activation function to obtain a feature map.

[0062] Step 2: Pass the feature map obtained in Step 1 through a convolution with a kernel size of 3×3 and a stride of 1. After the convolution, pass it through a BatchNorm layer and then through the Silu activation function to obtain the feature map. Pass the feature map through a convolution with a kernel size of 3×3 and a stride of 1 again. After the convolution, pass it through a BatchNorm layer and then through the Silu activation function to obtain the feature map.

[0063] Steps 1 and 2 above involve performing four convolutional processes on the available image to obtain the processed feature map. The stride used in the first two convolutional processes is different from that used in the last two. Step 3: Pass the feature map obtained in Step 2 through a set of parallel convolutional modules (in this example, this set of parallel convolutional modules consists of n1 PLKC modules). The output of the previous parallel convolutional module is used as the input of the next parallel convolutional module to obtain the feature map (i.e., the feature extraction result). In this example, steps 1 to 3 belong to the first round of feature extraction operation, which involves four convolution processes and n1 parallel convolution processes.

[0064] Step 4: The feature map obtained in Step 3 is processed by a convolution with a kernel size of 3×3 and a stride of 2, then passed through a BatchNorm layer, and then through the Silu activation function to obtain the feature map. Step 5: Pass the feature map obtained in Step 4 through a set of parallel convolutional modules (in this example, this set of parallel convolutional modules consists of n² PLKC modules). The output of the previous parallel convolutional module is used as the input of the next parallel convolutional module to obtain the feature map (i.e., the feature extraction result). In this example, steps 4 and 5 belong to the second round of feature extraction operations, which includes one convolutional process and n² parallel convolutional processes.

[0065] Step 6: Pass the feature map obtained in Step 5 through a convolution with a kernel size of 3×3 and a stride of 2, then through a BatchNorm layer, and then through the Silu activation function to obtain the feature map. Step 7: Pass the feature map obtained in Step 6 through a set of parallel convolutional modules (in this example, this set of parallel convolutional modules consists of n3 PLKC modules). The output of the previous parallel convolutional module is used as the input of the next parallel convolutional module to obtain the feature map (i.e., the feature extraction result). In this example, steps 6 and 7 belong to the third round of feature extraction, which involves one convolution and n3 parallel convolutions.

[0066] Step 8: Pass the feature map obtained in Step 7 through a convolution with a kernel size of 3×3 and a stride of 2, then through a BatchNorm layer, and then through the Silu activation function to obtain the feature map; Step 9: Pass the feature map obtained in Step 8 through a set of parallel convolutional modules (in this example, this set of parallel convolutional modules consists of n4 PLKC modules). The output of the previous parallel convolutional module is used as the input of the next parallel convolutional module to obtain the feature map (i.e., the feature extraction result).

[0067] In this example, steps 8 and 9 belong to the fourth round of feature extraction, which involves one convolution and n4 parallel convolutions.

[0068] In this example, the feature map obtained from the fourth round of feature extraction can be used as the target feature map; Step 10: The target feature map obtained in Step 9 is subjected to a 1×1 convolution to change the number of channels to 1280, then flattened along the second dimension, and finally passed through a fully connected layer to obtain the image classification result and output it.

[0069] In the above steps, n1, n2, n3, and n4 are hyperparameters of the image classification model, which can be determined or adjusted based on the running time and accuracy. For example, hyperparameters n1, n2, n3, and n4 are 2, 4, 6, and 6 respectively (for example only).

[0070] refer to Figure 4 In this example, a single parallel convolution module may include a convolution with a kernel size of 3×3 and a stride of 1, and three large kernels DSConv. The kernel sizes of the three large kernels DSConv can be 19×19, 23×23, and 27×27, respectively. Figure 4 In this example, "k" represents the convolution kernel. The hyperparameters n, C1, C2, C3, and C4 are 4, 0.5, 0.25, 0.125, and 0.125 respectively (for example only), which means that the channels of the feature map input to this parallel convolution module are divided into four branches of 0.5x, 0.25x, 0.125x, and 0.125x.

[0071] refer to Figure 4 The operation of a single parallel convolution module is as follows: Step 1: After the feature map is input into the parallel convolutional module, the first branch with 0.5 times the number of channels goes through a 3×3 convolution with a stride of 1; the second branch with 0.25 times the number of channels goes through a 19×19 large kernel DSConv with a stride of 1; the third branch with 0.125 times the number of channels goes through a 23×23 large kernel DSConv with a stride of 1; and the fourth branch with 0.125 times the number of channels goes through a 27×27 large kernel DSConv with a stride of 1. After each branch convolution, it goes through a BatchNorm layer, and then through the Silu activation function to obtain the processed feature map. Step 2: Concat the channels of the results from each branch in Step 1, then perform a 1×1 convolution to adjust the number of channels to the specified output channels, then pass it through the BatchNorm layer, and finally pass it through the Silu activation function to obtain the feature map; Step 3: Add the feature map obtained in Step 2 to the feature map input to the parallel convolutional module, and then pass it through the BatchNorm layer to output the feature map.

[0072] The above steps enable accurate classification of nasopharyngeal laryngoscopy images in real-world situations.

[0073] Example 1 can achieve the following beneficial effects: In Example 1, feature extraction is performed on each branch through parallel convolution processing. This parallel convolution processing can simultaneously use both large-kernel and small-kernel convolutions, which work together to extract features. Large-kernel convolutions offer stronger feature extraction capabilities, possess a larger receptive field, and can extract features over a wider range. Furthermore, using a large-kernel DSConv reduces the number of parameters and computational cost of the image classification model. Small-kernel convolutions take into account smaller features and can fully learn the key and detailed information of the training data (including the training images) during the model building phase. This approach allows for the extraction of features of varying sizes (especially smaller features), achieving excellent extraction results for different feature ranges, and enabling the image classification model to have a larger receptive field. This effectively improves feature extraction capabilities and results, thereby enhancing image classification performance and increasing classification accuracy.

[0074] In Example 1, performing convolution processing before parallel convolution processing can improve the extraction effect of shallow features of the image, reduce the size of the feature map, and increase the number of channels, which is beneficial to improving the processing effect of parallel convolution processing, thereby improving the image classification effect and accuracy.

[0075] The constructed image classification model consists of a backbone network and a classification module. The backbone network is used to extract image features, and the classification module includes a classification head that outputs the category of the usable image. Since the backbone network is built based on parallel large-kernel convolution, and parallel convolution modules mainly using parallel channels large-kernel depthwise separable convolution are used for feature extraction, the image classification model has higher robustness and better image classification performance.

[0076] Example 1 has broad application prospects. Continuing with the example above, Example 1 can more accurately classify nasopharyngeal endoscopic images, allowing doctors to make disease diagnoses based on the classified images, which helps improve diagnostic efficiency and accuracy.

[0077] like Figure 5 As shown, the second embodiment of this specification (hereinafter referred to as "Embodiment Two") provides an image management method for nasopharyngeal laryngoscope images, the method comprising: S202: The image classification result of the available image is determined by the method described in Example 1, wherein the image classification result is used to characterize the confidence level of the available image belonging to one or more categories; The method described in Example 1 can determine the image classification result of the available images, and the image classification result can be used to characterize the confidence level of the available images belonging to one or more categories.

[0078] Following the example in Example 1, the image classification results of nasopharyngoscopy images can be used to characterize the confidence level of whether a nasopharyngoscopy image belongs to one of 21 categories. For instance, if image classification determines that a nasopharyngoscopy image belongs to the left pyriform fossa category, then the confidence level for belonging to the left pyriform fossa category will be high, while the confidence level for belonging to the other 20 categories will be low. The specific setting of the confidence level can be chosen according to needs, such as calculating the confidence level for each category based on the image classification results.

[0079] In reality, multiple images may belong to the same category, but their quality (e.g., sharpness) may differ, meaning their image features vary. Since image classification results are based on image features, the classification results for multiple images of the same category can contain richer information beyond just the category itself, such as the image's features. Because confidence can be represented by the image classification results, even multiple images of the same category can have different confidence scores (i.e., confidence in belonging to that category) if their image features or quality differ. Generally, higher image confidence indicates better image quality or more prominent features.

[0080] S204: For any category, if the confidence level of an available image belonging to that category reaches a preset condition, then that available image is determined as the standard image of that category.

[0081] Since new usable images can be continuously acquired, new images may be continuously identified as belonging to any given category. As mentioned above, each image belonging to a category has its own confidence level. If the confidence level of a usable image belonging to a category reaches a preset condition, that usable image can be identified as the standard image for that category. The preset condition can be set as needed; for example, it could be a confidence level higher than a certain standard, or the image with the highest confidence level in any category could be used as the standard image for that category.

[0082] Since standard images are generally of higher quality, in addition to achieving the effects of Example 1, Example 2 also realizes image quality control and management for various types of images.

[0083] In practice, after image acquisition or a phase is completed, one or more categories may still lack a standard image. If a standard image for a certain category cannot be determined, a corresponding prompt can be given. This allows for further image acquisition targeting that category, and through the above-mentioned methods, image classification and standard image determination for that category can be achieved. This helps narrow down the image acquisition scope, improves the accuracy, comprehensiveness, and efficiency of image acquisition, and ensures that standard images can be determined for each category.

[0084] The following uses the example from Example 1 to illustrate the application of Example 2.

[0085] In nasopharyngoscopy, the quality of the images captured varies, making image quality control difficult. Since image quality affects the efficiency and accuracy of disease diagnosis, there is a need for quality control of nasopharyngoscopy images. Example 2 addresses the current issues with nasopharyngoscopy quality control and achieves the desired image quality control effect.

[0086] Specifically, for the acquired nasopharyngeal laryngoscope images, the image classification result of the laryngoscope images is determined using the method in Example 1, thereby determining which category the nasopharyngeal laryngoscope image belongs to. For any category, if the confidence level of a usable image belonging to that category reaches a preset condition, then that usable image is determined as the standard image for that category. For example, during the acquisition of nasopharyngeal laryngoscope images of a patient, multiple images of the left pyriform fossa may be acquired. The image classification results of these multiple images are all classified as the left pyriform fossa category, but the quality of these images varies. Then, the confidence level of these multiple images belonging to the left pyriform fossa category can be determined. If the confidence level of an image belonging to the left pyriform fossa category reaches a preset condition, it indicates that the image capture quality is better or the features of the left pyriform fossa are more obvious, and then this image can be used as the standard image for the left pyriform fossa category. In this way, the standard image corresponding to the category of each part can be determined, realizing image quality control and management for the category corresponding to each part.

[0087] As can be seen, Example 2 can not only be used for classifying nasopharyngeal endoscopic images, but also achieve image quality control for each category of each part, thereby determining the standard image for each category. Standard images facilitate a clearer view of each part, allowing doctors to make diagnoses based on the standard images corresponding to each category. This helps doctors make more efficient and accurate diagnoses, reduces doctors' workload, alleviates patient suffering, and improves diagnostic efficiency and accuracy, thus providing strong support for the medical field.

[0088] Furthermore, if a standard image cannot be determined for a certain area (it may be that no relevant images have been collected for that area, or that the quality of all images in that area is substandard), a prompt can be given, allowing for further image acquisition (in order to determine the standard image) or examination of that area. This further narrows down the image acquisition range, increases the image acquisition speed, and helps improve the speed, efficiency, and comprehensiveness of diagnosis, ensuring diagnostic accuracy and effectiveness.

[0089] like Figure 6 As shown, the third embodiment of this specification provides a classification device for nasopharyngeal laryngoscopy images corresponding to the method described in Embodiment 1. The device includes: Image acquisition module 301 is used to acquire available images; Image processing module 303 is used to input the available image into an image classification model, obtain a target feature map corresponding to the available image from the image classification model, and determine the image classification result of the available image based on the target feature map; Obtaining the target feature map corresponding to the available image includes: Perform multiple rounds of feature extraction operations; The first round of feature extraction in the multi-round feature extraction operation includes: processing the available image through one or more convolutions to obtain a feature map, and processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of this round of feature extraction; the parallel convolution processing is used to extract features of different ranges. Each non-first round feature extraction operation in the multi-round feature extraction operation includes: obtaining a feature map by performing one or more convolution processes on the feature extraction result obtained from the previous round feature extraction operation; and obtaining the feature map by performing one or more parallel convolution processes on the obtained feature map to obtain the feature extraction result of the current round feature extraction operation; the parallel convolution process is used to extract features of different ranges. The feature extraction result obtained from the last round of feature extraction is used as the target feature map corresponding to the available image.

[0090] Optionally, the parallel convolution processing is performed by a parallel convolution module, which includes a large kernel DSConv and a small kernel convolution.

[0091] Optionally, the parallel convolution processing includes: The feature map channels are divided into n branches. The first branch is subjected to convolution with a kernel size of 3×3 and a stride of 1 for feature extraction. The remaining n-1 branches are subjected to large kernel DSConv for feature extraction. Among them, n≥4, the large kernel DSConv used for feature extraction of the remaining branches has a kernel size of not less than 7, and the kernel sizes of each other are different and the kernel size interval is not less than 4. The feature extraction results of each branch are concatenated, then convolved, and the convolved result is output through the specified output channel, and then passed through the BatchNorm layer and the Silu activation function; The result after passing through the Silu activation function is added to the input features processed by parallel convolution, and then output after passing through the BatchNorm layer.

[0092] Optionally, for any branch, feature extraction for that branch includes: This branch is then processed through convolution or a large kernel DSConv for feature extraction.

[0093] Optionally, if parallel convolution is performed multiple times in a single feature extraction operation, the result of the previous parallel convolution is used as the input for the next parallel convolution.

[0094] Optionally, the convolution processing includes: sequentially performing convolutional layer, BatchNorm layer, and Silu activation function processing.

[0095] Optionally, if convolution is performed multiple times in a single feature extraction operation, the result of the previous convolution is used as the input for the next convolution.

[0096] Optionally, the image classification result of the usable image determined based on the target feature map includes: The target feature map is flattened and then passed through a fully connected layer to determine the image classification result of the available image.

[0097] Optionally, the available image is a pre-processed image, the pre-processing including cropping borders and / or angular rotation.

[0098] Example 3 achieves the same beneficial effects as Example 1.

[0099] like Figure 7 As shown, the fourth embodiment of this specification provides an image management device for nasopharyngeal laryngoscope images corresponding to the method described in Embodiment 2. The device includes: Image classification module 402 is used to determine the image classification result of an available image using the method described in Embodiment 1 or Embodiment 2, wherein the image classification result is used to characterize the confidence level of the available image belonging to one or more categories; The image determination module 404 is used to determine an available image as the standard image of any category if the confidence level of an available image belonging to that category reaches a preset condition.

[0100] Optionally, the image determination module 404 is further configured to provide a corresponding prompt if a standard image for a certain category is not determined.

[0101] Example 4 achieves the same beneficial effects as Example 2.

[0102] The above embodiments can be used in combination. The above descriptions are merely embodiments of this specification and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A classification method for nasopharyngeal laryngoscopy images, characterized in that, The method includes: Get available images; The available image is input into the image classification model, the image classification model obtains the target feature map corresponding to the available image, and the image classification result of the available image is determined based on the target feature map; Obtaining the target feature map corresponding to the available image includes: Perform multiple rounds of feature extraction operations; The first round of feature extraction in the multi-round feature extraction operation includes: processing the available image through one or more convolutions to obtain a feature map, and processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of this round of feature extraction; the parallel convolution processing is used to extract features of different ranges. Each non-first round feature extraction operation in the multi-round feature extraction operation includes: obtaining a feature map by performing one or more convolution processes on the feature extraction result obtained from the previous round feature extraction operation; and obtaining the feature map by performing one or more parallel convolution processes on the obtained feature map to obtain the feature extraction result of the current round feature extraction operation; the parallel convolution process is used to extract features of different ranges. The feature extraction result obtained from the last round of feature extraction is used as the target feature map corresponding to the available image.

2. The method as described in claim 1, characterized in that, The parallel convolution processing is performed by a parallel convolution module, which includes a large kernel DSConv and a small kernel convolution.

3. The method as described in claim 1 or 2, characterized in that, The parallel convolution processing includes: The feature map channels are divided into n branches. The first branch is subjected to convolution with a kernel size of 3×3 and a stride of 1 for feature extraction. The remaining n-1 branches are subjected to large kernel DSConv for feature extraction. Among them, n≥4, the large kernel DSConv used for feature extraction of the remaining branches has a kernel size of not less than 7, and the kernel sizes of each other are different and the kernel size interval is not less than 4. The feature extraction results of each branch are concatenated, then convolved, and the convolved result is output through the specified output channel, and then passed through the BatchNorm layer and the Silu activation function; The result after passing through the Silu activation function is added to the input features processed by parallel convolution, and then output after passing through the BatchNorm layer.

4. The method as described in claim 3, characterized in that, For any branch, feature extraction for that branch includes: This branch is then processed through convolution or a large kernel DSConv for feature extraction.

5. The method according to any one of claims 1 to 4, characterized in that, In a single feature extraction operation, if parallel convolution is performed multiple times, the result of the previous parallel convolution is used as the input for the next parallel convolution.

6. The method as described in claim 1, characterized in that, The convolution process includes sequentially performing convolutional layer, BatchNorm layer, and Silu activation function processing.

7. The method as described in claim 1 or 6, characterized in that, In a single feature extraction operation, if the convolution process is performed multiple times, the result of the previous convolution process is used as the input for the next convolution process.

8. The method as described in claim 1, characterized in that, The image classification results of the available images determined based on the target feature map include: The target feature map is flattened and then passed through a fully connected layer to determine the image classification result of the available image.

9. The method as described in claim 1, characterized in that, The available image is a pre-processed image, which includes cropping borders and / or angular rotation.

10. A method for managing nasopharyngeal laryngoscope images, characterized in that, The method includes: The image classification result of the available image is determined by the method of any one of claims 1 to 9, wherein the image classification result is used to characterize the confidence level of the available image belonging to one or more categories; For any category, if the confidence level of an available image belonging to that category reaches a preset condition, then that available image is determined as the standard image of that category.

11. The method as described in claim 10, characterized in that, The method further includes: If a standard image cannot be identified for a certain category, a corresponding prompt will be given.

12. A classification device for nasopharyngeal laryngoscope images, characterized in that, The device includes: The image acquisition module is used to acquire available images; The image processing module is used to input the available image into the image classification model, obtain the target feature map corresponding to the available image from the image classification model, and determine the image classification result of the available image based on the target feature map; Obtaining the target feature map corresponding to the available image includes: Perform multiple rounds of feature extraction operations; The first round of feature extraction in the multi-round feature extraction operation includes: processing the available image through one or more convolutions to obtain a feature map, and processing the obtained feature map through one or more parallel convolutions to obtain the feature extraction result of this round of feature extraction; the parallel convolution processing is used to extract features of different ranges. Each non-first round feature extraction operation in the multi-round feature extraction operation includes: obtaining a feature map by performing one or more convolution processes on the feature extraction result obtained from the previous round feature extraction operation; and obtaining the feature map by performing one or more parallel convolution processes on the obtained feature map to obtain the feature extraction result of the current round feature extraction operation; the parallel convolution process is used to extract features of different ranges. The feature extraction result obtained from the last round of feature extraction is used as the target feature map corresponding to the available image.

13. A device for managing images from a nasopharyngoscope, characterized in that, The device includes: An image classification module is configured to determine an image classification result of an available image using the method described in any one of claims 1 to 9, wherein the image classification result is used to characterize the confidence level of the available image belonging to one or more categories; The image determination module is used to determine an available image as the standard image of any category if the confidence level of an available image belonging to that category reaches a preset condition.

Images