A method and device for selecting optimal frames of a fiber nasopharyngoscope video based on CNN and transformer

Abstract

The application provides a CNN and Transformer-based optimal frame selection method and device for a fiberoptic nasopharyngoscope video, the method comprising: acquiring a fiberoptic nasopharyngoscope video of a single subject; the fiberoptic nasopharyngoscope video comprising at least one breathing cycle of the subject; splitting the fiberoptic nasopharyngoscope video into a plurality of continuously arranged image frames; selecting one from the image frames as an optimal frame for the fiberoptic nasopharyngoscope video based on a pre-trained frame selection network model; and the frame selection network model comprising at least a CNN structure and a Transformer structure. In the application, the CNN structure and the Transformer structure are used to identify the image frames, thereby selecting the optimal frame from the split image frames, and solving the problem of accurate selection of the optimal frame for the fiberoptic nasopharyngoscope video.

Description

Technical Field

[0001] This application relates to the field of medical image recognition technology, and more specifically, to a method and apparatus for selecting the best frame of fiberoptic nasopharyngoscopy video based on CNN and Transformer. Background Technology

[0002] The adenoids, also known as the pharyngeal tonsils, are rich in lymphatic tissue located at the junction of the top and posterior wall of the nasopharynx, between the pharyngeal recesses on both sides. They can elicit local or systemic immune responses and play an important defensive and protective role in the digestive and respiratory tracts, but they are also susceptible to bacterial invasion and often cause inflammation.

[0003] Computer-aided / autonomous assessment is the intended direction of current medical development. However, adenoids are usually assessed by obtaining video through fiberoptic nasopharyngoscopy, from which a key frame is selected for observation and evaluation. But fiberoptic nasopharyngoscopy video contains multiple image frames, and how to select the best frame is a problem that needs to be solved at this stage. Summary of the Invention

[0004] The problem addressed in this application is how to select the optimal frame from a fiberoptic nasopharyngoscopy video.

[0005] To address the aforementioned problems, the first aspect of this application provides a method for selecting the optimal frame of fiberoptic nasopharyngoscopy video based on CNN and Transformer, comprising:

[0006] Acquire fiberoptic nasopharyngoscopy video of a single subject; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the subject;

[0007] The fiberoptic nasopharyngoscope video is split into multiple consecutively set image frames;

[0008] Based on a pre-trained frame selection network model, one frame is selected from the image frames as the best frame of the fiberoptic nasopharyngoscopy video; the frame selection network model includes at least a CNN structure and a Transformer structure.

[0009] A second aspect of this application provides a fiberoptic nasopharyngoscope video optimal frame selection device based on CNN and Transformer, comprising:

[0010] A video acquisition module is used to acquire fiberoptic nasopharyngoscopy video of a single object; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the object.

[0011] An image splitting module is used to split the fiberoptic nasopharyngoscope video into multiple consecutively set image frames;

[0012] The model processing module is used to select one frame from the image frames as the best frame of the fiberoptic nasopharyngoscopy video based on a pre-trained frame selection network model; the frame selection network model includes at least a CNN structure and a Transformer structure.

[0013] A third aspect of this application provides an electronic device comprising: a memory and a processor;

[0014] The memory is used to store programs;

[0015] The processor, coupled to the memory, is used to execute the program for:

[0016] Acquire fiberoptic nasopharyngoscopy video of a single subject; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the subject;

[0017] The fiberoptic nasopharyngoscope video is split into multiple consecutively set image frames;

[0018] Based on a pre-trained frame selection network model, one frame is selected from the image frames as the best frame of the fiberoptic nasopharyngoscopy video; the frame selection network model includes at least a CNN structure and a Transformer structure.

[0019] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the above-described method for optimal frame selection of fiberoptic nasopharyngoscopy video based on CNN and Transformer.

[0020] In this application, CNN and Transformer structures are used to identify image frames, thereby selecting the best frame from the image frames at the split point, thus solving the problem of accurately selecting the best frame in fiberoptic nasopharyngoscopy video. Attached Figure Description

[0021] Figure 1 This is a model architecture diagram of the fiberoptic nasopharyngoscopy video optimal frame selection method based on CNN and Transformer according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of the fiberoptic nasopharyngoscopy video optimal frame selection method based on CNN and Transformer according to embodiments of this application;

[0023] Figure 3 This is a flowchart of the model processing of the fiberoptic nasopharyngoscopy video optimal frame selection method based on CNN and Transformer according to embodiments of this application;

[0024] Figure 4This is a flowchart illustrating the training of the fiberoptic nasopharyngoscopy video optimal frame selection method model based on CNN and Transformer according to embodiments of this application;

[0025] Figure 5 This is a structural block diagram of a fiberoptic nasopharyngoscope video optimal frame selection device based on CNN and Transformer according to an embodiment of this application;

[0026] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] To make the above-mentioned objects, features, and advantages of this application more apparent and understandable, specific embodiments of this application will be described in detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0028] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0029] To address the aforementioned issues, this application provides a novel optimal frame selection scheme for fiberoptic nasopharyngoscopy videos based on CNN and Transformer. This scheme utilizes CNN and Transformer structures to identify image frames and select the optimal frame, thus solving the problem of selecting the best frame for fiberoptic nasopharyngoscopy videos.

[0030] This application provides a method for selecting the best frame from fiberoptic nasopharyngoscopy video based on CNN and Transformer. The specific scheme of this method is as follows: Figures 1-4 As shown, this method can be executed by a CNN and Transformer-based optimal frame selection device for fiberoptic nasopharyngoscopy videos. This CNN and Transformer-based optimal frame selection device for fiberoptic nasopharyngoscopy videos can be integrated into electronic devices such as computers, servers, computer clusters, and data centers. Combined with... Figure 1 , Figure 2 The diagram shows a flowchart of a fiberoptic nasopharyngoscopy video optimal frame selection method based on CNN and Transformer according to an embodiment of this application; wherein, the fiberoptic nasopharyngoscopy video optimal frame selection method based on CNN and Transformer includes:

[0031] S100, acquire fiberoptic nasopharyngoscopy video of a single subject; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the subject;

[0032] In this application, the object is used to refer to the patient, and the fiberoptic nasopharyngoscopy video can be a video obtained by filming the patient.

[0033] In this application, the fiberoptic nasopharyngoscopy video is acquired through a fiberoptic nasopharyngoscope. The fiberoptic nasopharyngoscopy video can be a video stream in which every frame of the video image contains the adenoids, or it can be a video stream in which only a portion of the video images contain the adenoids.

[0034] In this application, each frame of video image contains a video stream of the adenoids, which can be obtained by doctors or other personnel directly shooting only the adenoids, or by cutting off the part that does not contain the adenoids after shooting.

[0035] In this application, the video stream to be evaluated is acquired in real time, thereby directly providing the evaluation result and improving real-time performance and timeliness.

[0036] In this application, the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the subject, which refers to the morphological changes of the adenoids in the fiberoptic nasopharyngoscopy video during one breath.

[0037] S200, the fiberoptic nasopharyngoscope video is split into multiple consecutively set image frames;

[0038] In this application, the facial video consists of image frames arranged sequentially according to time steps. The facial video can be split into multiple image frames according to time steps, and each image frame can have an image sequence number to represent its corresponding order or time step.

[0039] S300, based on a pre-trained frame selection network model, selects one frame from the image frames as the best frame of the fiberoptic nasopharyngoscopy video; the frame selection network model includes at least a CNN structure and a Transformer structure.

[0040] In this application, CNN and Transformer structures are used to identify image frames, thereby selecting the best frame from the image frames at the split point, thus solving the problem of accurately selecting the best frame in fiberoptic nasopharyngoscopy video.

[0041] In one implementation, combined with Figure 3 As shown, S300, based on a pre-trained frame selection network model, selects one frame from the image frames as the optimal frame for the fiberoptic nasopharyngoscopy video, including:

[0042] S301, Select a first predetermined number of image frames from the image frames;

[0043] In one implementation, the image frame is selected randomly.

[0044] In this way, by using random selection to increase the differences between input image frames, the situation of inaccurate selection of the best image frame caused by the extreme similarity between adjacent image frames is avoided.

[0045] S302, input the first predetermined number of image frames into the pre-trained frame selection network model to obtain a better image frame;

[0046] S303, Select a second predetermined number of image frames from the remaining image frames;

[0047] In one embodiment, the sum of the number of the second predetermined number of image frames and the number of the preferred image frames is the first predetermined number.

[0048] In this application, N = M + 1 is set to ensure that the number of image frames processed by the network model in each filtering / frame selection is consistent, thus avoiding filtering errors caused by the number of input images.

[0049] S304, input the second predetermined number of the image frames and the preferred image frames into the frame selection network model to obtain another preferred image frame;

[0050] S305, the image frame selection action and the frame selection network model processing action are executed repeatedly until all the image frames are selected; the final better image frame is the best frame of the fiberoptic nasopharyngoscopy video.

[0051] In this application, the image frame selection action and the frame selection network model processing action are executed cyclically, that is, the action of "selecting a second predetermined number of image frames from the remaining image frames; inputting the second predetermined number of image frames and the better image frame obtained in the previous cycle into the frame selection network model to obtain another better image frame as the better image frame for this cycle" is executed cyclically.

[0052] In this application, each filtering / frame selection network model process yields a better image frame, and the best image frame obtained from the last filtering / frame selection network model process is the best frame of the fiberoptic nasopharyngoscope video.

[0053] In this application, a first predetermined number of N frames and a second predetermined number of M frames are set. After splitting the fiberoptic nasopharyngoscopy video into multiple consecutively set image frames, N image frames are selected from these image frames (they will be deleted from the consecutively set image frames after selection and will not be selected again in subsequent selections) as the input images for the first selection. A frame selection network model selects one of these N image frames as the preferred image frame. Then, M image frames are selected from the remaining image frames (of the multiple consecutively set image frames), and these M image frames, along with the preferred image frame selected in the first selection, are used as the input images for the second selection. A frame selection network model selects one of these M+1 image frames as the preferred image frame for the second selection. Then, M image frames are selected from the remaining image frames (of the multiple consecutively set image frames), and these M image frames, along with the preferred image frame selected in the second selection, are used as the input images for the third selection. ... This selection process is repeated until all of the multiple consecutively set image frames have been selected once.

[0054] In one implementation, if there are fewer than M remaining image frames in the last selection, the remaining image frames can be supplemented to M by selecting from the deleted image frames or copying the remaining image frames.

[0055] This ensures that the number of image frames selected each time is the same, avoiding filtering errors caused by the number of input images.

[0056] In one implementation, combined with Figure 4 As shown, the training process of the frame selection network model includes:

[0057] S401, Obtain a fiberoptic nasopharyngoscope video sample, wherein the fiberoptic nasopharyngoscope video sample is labeled with multiple tag frames;

[0058] In this application, each fiberoptic nasopharyngoscopy video sample can be divided into multiple consecutively set image frames. Among these image frames, one frame is the best image frame, and there are multiple image frames that are very similar to the best image frame. These are used as label frames to achieve the best training effect.

[0059] S402, Based on the fiberoptic nasopharyngoscope video samples, generate multiple sets of training data; each set of training data has a corresponding label frame;

[0060] In this application, each set of training data contains N data points, including one of the label frames.

[0061] S403, input each set of training data into the frame selection network model to obtain the predicted image frame;

[0062] S404, Calculate the overall loss of the frame selection network model based on the labeled frame and the predicted image frame;

[0063] The formula for calculating the overall loss is as follows:

[0064]

[0065] Where L is the overall loss, N is the number of training data sets, i is the group number of the training data set, and G(z) is the overall loss. i Let x be the pixel value of the predicted image frame from the i-th training data set. i Let be the pixel value of the label frame of the i-th training data group, and α and β be the assigned weights.

[0066] In this application, the difference between the predicted image frame and the label frame is estimated by the overall loss, thereby iterating the frame selection network model.

[0067] S405, iterate the frame selection network model based on the overall loss until the loss converges.

[0068] In one implementation, step S402 generates multiple sets of training data based on the fiberoptic nasopharyngoscope video samples, including:

[0069] The fiberoptic nasopharyngoscope video is split into multiple consecutively set image frames; the tag frame is the split image frame.

[0070] The split image frames are divided into negative samples and positive samples, and the label frames are the positive samples;

[0071] A second predetermined number of image frames are selected from the negative sample image frames, and one of the labeled frames is randomly selected to form a set of training data; the labeled frame is used as the annotation for the set of training data.

[0072] Repeat the combined actions until a predetermined number of training data are selected.

[0073] In this application, repeatedly performing the combination action means repeatedly selecting a second predetermined number of image frames from the negative sample image frames, randomly selecting one from the label frames, combining them into a set of training data, and using the label frames as annotations for the set of training data.

[0074] In this application, the label frame is a part of the split image frames. Each fiberoptic nasopharyngoscopy video sample can be split into multiple consecutively set image frames. Among these image frames, one frame is the best image frame, and there are multiple image frames that are very similar to the best image frame. These are used as label frames to achieve the best training effect.

[0075] In this application, the label frame is separated from the image frame by segmentation, and the remaining image frame is used as negative sample data.

[0076] In this application, a set of training data consists of N image frames, including M negative samples and one positive sample. The negative sample is selected from the remaining image frames, and this selection can be random to avoid training interference caused by adjacent image frames being extremely similar. The positive sample is selected from the label frame, thus combining them into a set of training data.

[0077] In this application, the process continues until a predetermined number of training data sets are selected, that is, until a sufficient number of training data sets are selected.

[0078] In one implementation, combined with Figure 1 As shown, in step S403, each set of training data is input into the frame selection network model to obtain predicted image frames, including:

[0079] Based on the CNN structure, the training data is downsampled multiple times to obtain the downsampled frame features;

[0080] The frame features are input into the Transformer structure to obtain the frame features after feature fusion;

[0081] The frame features are processed using a hidden layer to obtain the predicted image frame.

[0082] In this application, a Transformer structure is used to perform global feature extraction on the downsampled feature map, thereby fusing and extracting the features.

[0083] It should be noted that, Figure 1 The diagram does not show all fully connected layers, ReLU layers, etc.; only the main architecture is shown, and some details have been omitted. In actual processing, corresponding processing layers can be added based on requirements.

[0084] In this application, the Transformer structure includes N Transformer layers, each Transformer layer including a linear embedding layer, a first reshaping layer, an even number of Swing Transformer modules, and a second reshaping layer;

[0085] The linear embedding layer changes the dimension of the input image data;

[0086] The first reshaping layer changes the number of channels in the image data;

[0087] Multiple Swin Transformer modules perform feature extraction on the image data to obtain feature information;

[0088] The second reshaping layer rearranges the output of the Transformer Layer to obtain frame features.

[0089] In this application, the linear embedding layer is a Linear Embedding layer, which converts the dimension of the input vector into a preset dimension value, i.e., a dimension value that the Swin Transformer module can receive; the first reshaping layer is a Reshape layer, which is used to transform a specified matrix into a matrix of a specific dimension, while keeping the number of elements in the matrix unchanged. It can readjust the number of rows, columns, and dimensions of the matrix. In this application, it is used to adjust the number of channels of the input vector to convert the number of channels of the input vector into a preset number of channels, i.e., a number of channels that the Swin Transformer module can receive; the multiple Swin Transformer modules are Swin Transformer Block×N layers, which are used to extract features from the input image data, extract image characteristics, and obtain feature information / feature maps; the second reshaping layer is a Reshape layer after the Swin Transformer module, which is used to reshape the output of the Transformer Layer to obtain frame feature information.

[0090] In this application, there is an even number of Swin Transformer modules. Two adjacent Swin Transformer modules have similar structures and form a combination, which is exactly the same as the structure of the combination of the other two Swin Transformer modules.

[0091] In one implementation, the Swing Transformer module includes an MLP module and a displacement window-based MSA module, both of which have a normalization layer before them and residual connections after them.

[0092] Among them, the two consecutive Swing Transformer Blocks have similar architectures, but the MSA modules are different. The first Swing Transformer Block has a W-MSA module, and the second Swing Transformer Block has a SW-MSA module. The rest of the modules are the same.

[0093] The MLP module is a two-layer MLP module with a GeLU nonlinearity sandwiched in between, and the LayerNorm (LN) layer is the normalization layer.

[0094] In one implementation, before inputting each set of training data into the frame selection network model to obtain the predicted image frame in step S403, the method further includes:

[0095] Each set of training data is arranged according to a preset strategy to form a near-square image; a preset sliding window is generated, the length and width of which are integer multiples of the image frame; the sliding window is slid along the diagonal of the near-square image at intervals to generate multiple large-size images, each with the same size as the sliding window; the generated large-size images are downsampled to obtain corresponding downsampled images, the downsampled images having the same size as the image frame; the downsampled images are added to the set of training data, keeping the label frame unchanged.

[0096] In this application, this part of the technical solution is set before step S403, which inputs each set of training data into the frame selection network model to obtain a predicted image frame, and before the step of inputting a first predetermined number of the image frames into the pre-trained frame selection network model to obtain a better image frame.

[0097] In this application, the preset strategy can be: based on the number of image frames (including positive and negative samples) in the training data, determine the nearest N value to the squared value; arrange the image frames in N rows sequentially until the arrangement is complete, and obtain an approximately square image.

[0098] Among them, the difference (absolute value) between the square of N and the number of image frames is the smallest, smaller than the difference (absolute value) between the square of (N-1) and the number of image frames, and smaller than the difference (absolute value) between the square of (N+1) and the number of image frames.

[0099] The approximate square image is determined by the number of image frames arranged (regardless of the shape of the image frames). For example, an arrangement of 3×3 image frames is considered a square. If the number of image frames is insufficient, the lower right corner of the square is missing; if the number of image frames is excessive, the lower left corner of the square is redundant.

[0100] For example, if the length and width of an image frame are HW, then the length and width of the sliding window can be 2H and 2W, or 3H and 3W, etc.

[0101] In this application, the sliding window along the diagonal of the near-square image means that the sliding window slides along the diagonal from the lower left corner to the upper right corner of the square, thereby achieving maximum data acquisition.

[0102] In this application, interval sliding refers to sliding with a step size greater than 1, thereby avoiding the acquisition of too many images of different sizes. The specific sliding step size can be determined according to the actual situation.

[0103] In this application, by combining and then using a sliding window acquisition method, the global and local features of the large-size image can be increased while ensuring the image size, and the randomness and correspondence with the image frame can be increased.

[0104] In this application, if some data in the sliding window is missing, it can be filled with 0.

[0105] In this application, by generating a large-size image and then downsampling it to obtain a downsampled image, the downsampled image has corresponding homology and similarity with the image frame and the label frame, thereby avoiding feature fragmentation between the negative samples of the image frame and the label frame and improving the accuracy of the judgment.

[0106] In this application, the specific process of inputting each set of training data into the frame selection network model to obtain the predicted image frame is the same as the specific process of inputting a first predetermined number of image frames into the pre-trained frame selection network model to obtain a better image frame, and will not be described again in this application.

[0107] This application provides a fiberoptic nasopharyngoscope video optimal frame selection device based on CNN and Transformer, used to execute the fiberoptic nasopharyngoscope video optimal frame selection method based on CNN and Transformer described above. The following is a detailed description of the fiberoptic nasopharyngoscope video optimal frame selection device based on CNN and Transformer.

[0108] like Figure 5 As shown, the fiberoptic nasopharyngoscope video optimal frame selection device based on CNN and Transformer includes:

[0109] The video acquisition module 101 is used to acquire fiberoptic nasopharyngoscopy video of a single object; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the object.

[0110] Image splitting module 102 is used to split the fiberoptic nasopharyngoscope video into multiple consecutively set image frames;

[0111] The model processing module 103 is used to select one frame from the image frames as the best frame of the fiberoptic nasopharyngoscopy video based on a pre-trained frame selection network model; the frame selection network model includes at least a CNN structure and a Transformer structure.

[0112] In one implementation, the model processing module 103 is further configured to:

[0113] A first predetermined number of image frames are selected from the image frames; the first predetermined number of image frames are input into a pre-trained frame selection network model to obtain a better image frame; a second predetermined number of image frames are selected from the remaining image frames; the second predetermined number of image frames and the better image frame are input into the frame selection network model to obtain another better image frame; the image frame selection action and the frame selection network model processing action are executed repeatedly until all image frames are selected; the finally obtained better image frame is the best frame of the fiberoptic nasopharyngoscopy video.

[0114] In one implementation, the image frame is selected randomly.

[0115] In one embodiment, the sum of the number of the second predetermined number of image frames and the number of the preferred image frames is the first predetermined number.

[0116] In one implementation, a model training module is also included, which is used for:

[0117] A fiberoptic nasopharyngoscopy video sample is acquired, wherein the fiberoptic nasopharyngoscopy video sample is labeled with multiple tag frames; multiple sets of training data are generated based on the fiberoptic nasopharyngoscopy video sample; each set of training data has a corresponding tag frame; each set of training data is input into the frame selection network model to obtain a predicted image frame; the overall loss of the frame selection network model is calculated based on the tag frame and the predicted image frame; the frame selection network model is iterated based on the overall loss until the loss converges.

[0118] In one implementation, the model training module is further used for:

[0119] The fiberoptic nasopharyngoscopy video is split into multiple consecutively set image frames; the label frame is the split image frame; the split image frames are divided into negative samples and positive samples, and the label frame is the positive sample; a second predetermined number of image frames are selected from the negative sample image frames, and one of the label frames is randomly selected to combine them into a set of training data; the label frame is used as the label for the set of training data; the combination action is repeated until a predetermined number of training data are selected.

[0120] In one implementation, the model training module is further used for:

[0121] The training data is downsampled multiple times based on the CNN structure to obtain downsampled frame features; the frame features are input into the Transformer structure to obtain frame features after feature fusion; the frame features are processed by a hidden layer to obtain the predicted image frame.

[0122] The fiberoptic nasopharyngoscope video optimal frame selection device based on CNN and Transformer provided in the above embodiments of this application and the fiberoptic nasopharyngoscope video optimal frame selection method based on CNN and Transformer provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by their stored applications.

[0123] The above describes the internal functions and structure of a fiber optic nasopharyngoscope video optimal frame selection device based on CNN and Transformer, such as... Figure 6 As shown, in practice, this fiber optic nasopharyngoscope video optimal frame selection device based on CNN and Transformer can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0124] Memory 301 can be configured to store a program.

[0125] Additionally, memory 301 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0126] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0127] Processor 303, coupled to memory 301, is used to execute programs in memory 301 for:

[0128] Acquire fiberoptic nasopharyngoscopy video of a single subject; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the subject;

[0129] The fiberoptic nasopharyngoscope video is split into multiple consecutively set image frames;

[0130] Based on a pre-trained frame selection network model, one frame is selected from the image frames as the best frame of the fiberoptic nasopharyngoscopy video; the frame selection network model includes at least a CNN structure and a Transformer structure.

[0131] In one implementation, the processor 303 is further configured to:

[0132] A first predetermined number of image frames are selected from the image frames; the first predetermined number of image frames are input into a pre-trained frame selection network model to obtain a better image frame; a second predetermined number of image frames are selected from the remaining image frames; the second predetermined number of image frames and the better image frame are input into the frame selection network model to obtain another better image frame; the image frame selection action and the frame selection network model processing action are executed repeatedly until all image frames are selected; the finally obtained better image frame is the best frame of the fiberoptic nasopharyngoscopy video.

[0133] In one implementation, the image frame is selected randomly.

[0134] In one embodiment, the sum of the number of the second predetermined number of image frames and the number of the preferred image frames is the first predetermined number.

[0135] In one implementation, the processor 303 is further configured to:

[0136] A fiberoptic nasopharyngoscopy video sample is acquired, wherein the fiberoptic nasopharyngoscopy video sample is labeled with multiple tag frames; multiple sets of training data are generated based on the fiberoptic nasopharyngoscopy video sample; each set of training data has a corresponding tag frame; each set of training data is input into the frame selection network model to obtain a predicted image frame; the overall loss of the frame selection network model is calculated based on the tag frame and the predicted image frame; the frame selection network model is iterated based on the overall loss until the loss converges.

[0137] In one implementation, the processor 303 is further configured to:

[0138] The fiberoptic nasopharyngoscopy video is split into multiple consecutively set image frames; the label frame is the split image frame; the split image frames are divided into negative samples and positive samples, and the label frame is the positive sample; a second predetermined number of image frames are selected from the negative sample image frames, and one of the label frames is randomly selected to combine them into a set of training data; the label frame is used as the label for the set of training data; the combination action is repeated until a predetermined number of training data are selected.

[0139] In one implementation, the processor 303 is further configured to:

[0140] The training data is downsampled multiple times based on the CNN structure to obtain downsampled frame features; the frame features are input into the Transformer structure to obtain frame features after feature fusion; the frame features are processed by a hidden layer to obtain the predicted image frame.

[0141] In this application, Figure 6 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 6 The components shown.

[0142] The electronic device provided in this embodiment is based on the same inventive concept as the fiber optic nasopharyngoscope video optimal frame selection method based on CNN and Transformer provided in this application embodiment, and has the same beneficial effects as the methods adopted, run or implemented by the application stored therein.

[0143] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0148] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0149] This application also provides a computer-readable storage medium corresponding to the fiberoptic nasopharyngoscopy video optimal frame selection method based on CNN and Transformer provided in the foregoing embodiments, wherein a computer program (i.e., a program product) is stored thereon, and the computer program, when run by a processor, executes the fiberoptic nasopharyngoscopy video optimal frame selection method based on CNN and Transformer provided in any of the foregoing embodiments.

[0150] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0151] The computer-readable storage medium provided in the above embodiments of this application and the fiberoptic nasopharyngoscopy video optimal frame selection method based on CNN and Transformer provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0152] It should be noted that numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for selecting the optimal frame in fiberoptic nasopharyngoscopy video based on CNN and Transformer, characterized in that, include: Acquire fiberoptic nasopharyngoscopy video of a single subject; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the subject; The fiberoptic nasopharyngoscope video is split into multiple consecutively set image frames; Based on a pre-trained frame selection network model, one frame is selected from the image frames as the best frame of the fiberoptic nasopharyngoscopy video. The frame selection network model includes at least a CNN structure and a Transformer structure; The pre-trained frame selection network model selects one frame from the image frames as the best frame for the fiberoptic nasopharyngoscopy video, including: Select a first predetermined number of image frames from the image frames; The first predetermined number of image frames are input into a pre-trained frame selection network model to obtain a better image frame. Select a second predetermined number of image frames from the remaining image frames; The second predetermined number of image frames and the preferred image frames are input into the frame selection network model to obtain another preferred image frame. The image frame selection action and the frame selection network model processing action are executed repeatedly until all the image frames are selected; the final better image frame is the best frame of the fiberoptic nasopharyngoscopy video. The training process of the frame selection network model includes inputting each set of training data into the frame selection network model to obtain predicted image frames. The step of inputting each set of training data into the frame selection network model to obtain predicted image frames includes: Based on the CNN structure, the training data is downsampled multiple times to obtain the downsampled frame features; The frame features are input into the Transformer structure to obtain the frame features after feature fusion; The frame features are processed using a hidden layer to obtain the predicted image frame.

2. The optimal frame selection method for fiberoptic nasopharyngoscopy video based on CNN and Transformer according to claim 1, characterized in that, The image frames are selected randomly.

3. The optimal frame selection method for fiberoptic nasopharyngoscopy video based on CNN and Transformer according to claim 1, characterized in that, The sum of the second predetermined number of image frames and the preferred number of image frames is the first predetermined number.

4. The optimal frame selection method for fiberoptic nasopharyngoscopy video based on CNN and Transformer according to any one of claims 1-3, characterized in that, The training process of the frame selection network model includes: Acquire fiberoptic nasopharyngoscopy video samples, wherein the fiberoptic nasopharyngoscopy video samples are labeled with multiple tag frames; Based on the fiberoptic nasopharyngoscope video samples, multiple sets of training data are generated; each set of training data has a corresponding label frame. Each set of training data is input into the frame selection network model to obtain the predicted image frame; Based on the labeled frame and the predicted image frame, calculate the overall loss of the frame selection network model; The frame selection network model is iterated based on the overall loss until the loss converges.

5. The optimal frame selection method for fiberoptic nasopharyngoscopy video based on CNN and Transformer according to claim 4, characterized in that, The process involves generating multiple sets of training data based on the fiberoptic nasopharyngoscopy video samples, including: The fiberoptic nasopharyngoscope video is split into multiple consecutively set image frames; the tag frame is the split image frame. The split image frames are divided into negative samples and positive samples, and the label frames are the positive samples; A second predetermined number of image frames are selected from the negative sample image frames, and one of the labeled frames is randomly selected to form a set of training data; the labeled frame is used as the annotation for the set of training data. Repeat the combined actions until a predetermined number of training data are selected.

6. A fiberoptic nasopharyngoscope video optimal frame selection device based on CNN and Transformer, characterized in that, include: A video acquisition module is used to acquire fiberoptic nasopharyngoscopy video of a single object; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the object. An image splitting module is used to split the fiberoptic nasopharyngoscope video into multiple consecutively set image frames; The model processing module is used to select one frame from the image frames as the best frame of the fiberoptic nasopharyngoscopy video based on a pre-trained frame selection network model; the frame selection network model includes at least a CNN structure and a Transformer structure. The pre-trained frame selection network model selects one frame from the image frames as the best frame for the fiberoptic nasopharyngoscopy video, including: Select a first predetermined number of image frames from the image frames; The first predetermined number of image frames are input into a pre-trained frame selection network model to obtain a better image frame. Select a second predetermined number of image frames from the remaining image frames; The second predetermined number of image frames and the preferred image frames are input into the frame selection network model to obtain another preferred image frame. The image frame selection action and the frame selection network model processing action are executed repeatedly until all the image frames are selected; the final better image frame is the best frame of the fiberoptic nasopharyngoscopy video. The training process of the frame selection network model includes inputting each set of training data into the frame selection network model to obtain predicted image frames. The step of inputting each set of training data into the frame selection network model to obtain predicted image frames includes: Based on the CNN structure, the training data is downsampled multiple times to obtain the downsampled frame features; The frame features are input into the Transformer structure to obtain the frame features after feature fusion; The frame features are processed using a hidden layer to obtain the predicted image frame.

7. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is used to execute the program for: Acquire fiberoptic nasopharyngoscopy video of a single subject; the fiberoptic nasopharyngoscopy video includes at least one respiratory cycle of the subject; The fiberoptic nasopharyngoscope video is split into multiple consecutively set image frames; Based on a pre-trained frame selection network model, one frame is selected from the image frames as the best frame of the fiberoptic nasopharyngoscopy video. The frame selection network model includes at least a CNN structure and a Transformer structure; The pre-trained frame selection network model selects one frame from the image frames as the best frame for the fiberoptic nasopharyngoscopy video, including: Select a first predetermined number of image frames from the image frames; The first predetermined number of image frames are input into a pre-trained frame selection network model to obtain a better image frame. Select a second predetermined number of image frames from the remaining image frames; The second predetermined number of image frames and the preferred image frames are input into the frame selection network model to obtain another preferred image frame. The image frame selection action and the frame selection network model processing action are executed repeatedly until all the image frames are selected; the final better image frame is the best frame of the fiberoptic nasopharyngoscopy video. The training process of the frame selection network model includes inputting each set of training data into the frame selection network model to obtain predicted image frames. The step of inputting each set of training data into the frame selection network model to obtain predicted image frames includes: Based on the CNN structure, the training data is downsampled multiple times to obtain the downsampled frame features; The frame features are input into the Transformer structure to obtain the frame features after feature fusion; The frame features are processed using a hidden layer to obtain the predicted image frame.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the optimal frame selection method for fiberoptic nasopharyngoscopy video based on CNN and Transformer as described in any one of claims 1-5.

Images