Endoscope cleaning behavior detection method, electronic equipment, storage medium and computer program product

Abstract

The invention provides an endoscope cleaning behavior detection method, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: dividing a to-be-detected video combination according to a specified duration, and obtaining a plurality of sub-video combinations; wherein the to-be-detected video combination comprises to-be-detected videos shot at a plurality of visual angles for the cleaning behavior of the to-be-detected endoscope; performing feature extraction on each sub-video combination to obtain a feature vector sequence corresponding to the sub-video combination; adding a position code to each feature vector sequence to obtain a to-be-processed vector sequence; and inputting the to-be-processed vector sequence corresponding to each sub-video combination into a trained detection model to obtain a step category and a quality score corresponding to each sub-video combination. According to the scheme of the invention, the method is used for processing the to-be-detected video combination based on the trained detection model, and accurately achieves the recognition of each step in the cleaning process and the evaluation of the cleaning quality.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for detecting endoscope cleaning behavior, electronic equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Digestive endoscopes have been widely used in hospitals at all levels. With the continuous advancement of endoscopic imaging and auxiliary treatment technology, it has gradually evolved from a simple diagnostic tool to a minimally invasive surgical instrument that combines refined diagnosis and precise treatment, playing an important role in the diagnosis of digestive system diseases. At the same time, multi-field clinical applications and complex and changeable endoscopic structures have also put forward higher requirements for the cleaning, disinfection and maintenance of digestive endoscopes. Therefore, it is very important to detect the cleaning behavior of the endoscope to determine the quality of the endoscope cleaning, and there is an urgent need for a solution that can effectively detect the cleaning behavior of the endoscope. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method for detecting endoscope cleaning behavior and an electronic device, a storage medium, and a computer program product, which are used to process the video combination to be detected based on a trained detection model, accurately realize the identification of each step in the cleaning process and the evaluation of the cleaning quality.

[0004] On the one hand, the present application provides a method for detecting endoscope cleaning behavior, comprising:

[0005] The video combination to be detected is divided according to the specified time length to obtain multiple sub-video combinations; wherein the video combination to be detected includes the video to be detected shot at multiple viewing angles of the endoscope cleaning behavior to be detected;

[0006] Extract features from each sub-video combination to obtain a feature vector sequence corresponding to the sub-video combination;

[0007] Add position code to each feature vector sequence to obtain a vector sequence to be processed;

[0008] The vector sequences to be processed corresponding to each sub-video combination are respectively input into the trained detection model to obtain the step category and quality score corresponding to each sub-video combination.

[0009] In one embodiment, extracting features from each sub-video combination to obtain a feature vector sequence corresponding to the sub-video combination includes:

[0010] For each sub-video in each sub-video combination, each video frame in the sub-video is normalized, and an initial feature vector is extracted from each normalized video frame through a trained feature extraction model; wherein a corresponding feature extraction model is provided for each sub-video of each viewing angle;

[0011] The initial feature vectors of multiple video frames at the same time point in the multiple sub-videos are fused to obtain a fused feature vector at the time point;

[0012] The fused feature vectors of multiple time points are constructed as a feature vector sequence.

[0013] In one embodiment, adding a position code to each feature vector sequence to obtain a vector sequence to be processed includes:

[0014] For each fused feature vector in each feature vector sequence, construct the position information of each element based on the order of the fused feature vector in the feature vector sequence and the sequence number of each element in the fused feature vector;

[0015] Construct a position encoding vector according to the position information of all elements in each fused feature vector;

[0016] Each fused feature vector is fused with its corresponding position encoding vector to obtain a vector to be processed, and a sequence of vectors to be processed is constructed according to a plurality of vectors to be processed corresponding to each feature vector sequence.

[0017] In one embodiment, the detection model is constructed based on a Transformer model, and the detection model includes a first output layer and a second output layer, the first output layer performs a classification task, and the second output layer performs a regression task;

[0018] The training steps of the feature extraction model and the detection model include:

[0019] Input each video frame of each sample sub-video in the sample sub-video combination in the sample data set into the corresponding feature extraction model to obtain an initial feature vector corresponding to each video frame; wherein each sample sub-video combination carries a step category label and a quality score label;

[0020] Using the initial feature vector corresponding to each video frame in each sample sub-video combination, construct a vector sequence to be processed corresponding to the sample sub-video combination;

[0021] Input the to-be-processed vector sequence corresponding to each sample sub-video combination into the detection model respectively, and obtain the prediction step category and prediction quality score corresponding to each sample sub-video combination;

[0022] Adjusting network parameters of the feature extraction model and the detection model according to the difference between the predicted step category and the step category label, and the difference between the predicted quality score and the quality score label for each sample sub-video combination;

[0023] The above process is repeated until the feature extraction model and the detection model converge, thereby obtaining a trained feature extraction model and a trained detection model.

[0024] In one embodiment, the step of acquiring the sample data set includes:

[0025] Obtain multiple sub-video combinations, and perform normalization processing on each video frame in each sub-video combination;

[0026] In response to the labeling instruction, a step category label and a quality score label are added to each sub-video combination after normalization to obtain a plurality of sample sub-video combinations;

[0027] A sample dataset is constructed based on a combination of multiple sample sub-videos.

[0028] In one embodiment, the method further comprises:

[0029] After obtaining multiple step categories corresponding to the endoscope cleaning behavior to be detected and the quality score of each step category, a comprehensive evaluation result of the endoscope cleaning behavior to be detected is determined according to the multiple step categories and the quality score of each step category.

[0030] In one embodiment, determining the comprehensive evaluation result of the endoscope cleaning behavior to be detected according to the multiple step categories and the quality score of each step category includes:

[0031] If there is at least one group of identical step categories among the multiple step categories, and the number of groups of identical step categories reaches a preset group number threshold, or the number of any group of identical step categories reaches a preset number threshold, the comprehensive evaluation result of the endoscope cleaning behavior to be tested is determined to be unqualified.

[0032] On the other hand, the present application provides an electronic device, the electronic device comprising:

[0033] processor;

[0034] a memory for storing processor-executable instructions;

[0035] Wherein, the processor is configured to execute the above-mentioned endoscope cleaning behavior detection method.

[0036] Furthermore, the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can be executed by a processor to complete the above-mentioned endoscope cleaning behavior detection method.

[0037] In addition, the present application provides a computer program product, including a computer program / instruction, which implements the above-mentioned endoscope cleaning behavior detection method when executed by a processor.

[0038] The present application scheme can effectively identify and evaluate the quality of each step in the endoscope cleaning behavior to be detected with the help of the trained detection model; since the video combination to be detected is constructed by using the videos to be detected shot from multiple perspectives, the video combination to be detected can characterize the hand movements, tool usage and operation details of the endoscope cleaning behavior under multiple perspectives. Therefore, with the help of the video combination to be detected, the feature vector serial number corresponding to each step can be extracted, and then a vector sequence to be processed is generated, which contains a large number of cleaning features to ensure the accuracy of step identification and quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0040] Figure 1 A schematic diagram of an application scenario of a method for detecting endoscope cleaning behavior provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0042] Figure 3 A schematic diagram of a flow chart of a method for detecting endoscope cleaning behavior provided in one embodiment of the present application;

[0043] Figure 4 A schematic diagram of a video frame captured by a top-view camera provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of a flow chart of a method for constructing a feature vector sequence provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of a flow chart of a method for constructing a vector sequence to be processed provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of the overall architecture of a detection solution for endoscope cleaning behavior provided in one embodiment of the present application;

[0047] Figure 8 A flowchart of a model training method provided in one embodiment of the present application;

[0048] Fig. 9 A schematic diagram of a flow chart of a method for constructing a sample data set provided in an embodiment of the present application;

[0049] Fig.10 A block diagram of an endoscope cleaning behavior detection device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0051] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0052] Figure 1 Schematic diagram of an application scenario of the method for detecting endoscope cleaning behavior provided in an embodiment of the present application. Figure 1 As shown, the application scenario includes multiple clients 20 and servers 30; the client 20 can be a camera device, deployed at multiple locations in the endoscope cleaning area, can collect video data from multiple perspectives, and can pass the video data collected by itself to the server 30; the server 30 can be a host, a server, a server cluster or a cloud computing center, and can execute a detection method for endoscope cleaning behavior based on video data from multiple perspectives.

[0053] like Figure 2 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 2 A processor 11 is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 so that the electronic device 1 can execute all or part of the process of the method in the following embodiment. In one embodiment, the electronic device 1 can be the above-mentioned server 30, which is used to execute the detection method of endoscope cleaning behavior.

[0054] The memory 12 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0055] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by the processor 11 to complete the endoscope cleaning behavior detection method provided in the present application.

[0056] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for detecting endoscope cleaning behavior provided in the present application.

[0057] See also Figure 3 , is a flow chart of a method for detecting endoscope cleaning behavior provided in an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps 310 to 340.

[0058] Step 310: Divide the video combination to be detected according to the specified time length to obtain multiple sub-video combinations; wherein the video combination to be detected includes the video to be detected shot at multiple viewing angles of the endoscope cleaning behavior to be detected.

[0059] Cameras can be deployed at multiple locations in the endoscope cleaning area to collect videos to be tested from multiple perspectives during the endoscope cleaning process. For example, three cameras are set up, which are installed above the cleaning area (to collect videos to be tested from top to bottom), on the left side of the cleaning area (to collect videos to be tested from the side perspective), and in front of the cleaning area (to collect videos to be tested from the front perspective). The collected videos to be tested can capture hand movements, tool usage, and operation details during the cleaning process from multiple perspectives.

[0060] See also Figure 4 , is a schematic diagram of a video frame captured by a top-view camera according to an embodiment of the present application, such as Figure 4 As shown, the hand movements, tool usage and operation details during cleaning can be seen in the video frames of the video to be inspected captured by the top-view camera.

[0061] The designated duration can be configured as needed. For the endoscope cleaning process, it can be pre-divided into multiple steps with the same processing duration, and the processing duration is used as the designated duration. For example, the complete endoscope cleaning process takes 2 minutes, includes 12 steps, and the designated duration is 10 seconds.

[0062] Each time to be detected in the video combination to be detected is divided into specified time lengths, and sub-videos of multiple perspectives corresponding to the same time divided are constructed into a sub-video combination, thereby obtaining multiple sub-video combinations.

[0063] Step 320: extract features from each sub-video combination to obtain a feature vector sequence corresponding to the sub-video combination.

[0064] For each sub-video combination, feature vectors may be extracted from video frames within the sub-videos of the sub-video combination, and a feature vector sequence may be constructed based on the extracted feature vectors of consecutive video frames.

[0065] Step 330: Add position codes to each feature vector sequence to obtain a vector sequence to be processed.

[0066] Among them, positional encoding is used to indicate the position information of the vector in the sequence and the position information of the elements in the vector.

[0067] A position code is generated for each feature vector sequence, and the position code is added to the feature vector sequence to obtain a vector sequence to be processed.

[0068] Step 340: Input the to-be-processed vector sequences corresponding to each sub-video combination into the trained detection model to obtain the step category and quality score corresponding to each sub-video combination.

[0069] The vector sequence to be processed corresponding to each sub-video combination is input into the trained detection model respectively. The vector sequence to be processed of the sub-video combination is processed by the detection model to obtain the step category and quality score corresponding to the sub-video combination.

[0070] In this case, the trained detection model can output the corresponding step category and quality score for each sub-video combination, where the step category represents the cleaning step corresponding to the sub-video combination, and the quality score indicates the cleaning quality of the cleaning step.

[0071] Ideally, the multiple sub-video combinations divided from the video combination to be tested should correspond to the various steps pre-divided in the endoscope cleaning process, and the quality scores should all meet the standards.

[0072] Through the above measures, the trained detection model can be used to effectively identify and evaluate the quality of each step in the endoscope cleaning behavior to be detected; since the video combination to be detected is constructed by using the videos to be detected shot from multiple perspectives, the video combination to be detected can characterize the hand movements, tool usage and operation details of the endoscope cleaning behavior under multiple perspectives, therefore, with the help of the video combination to be detected, the feature vector serial number corresponding to each step can be extracted, and then a vector sequence to be processed is generated, which contains a large number of cleaning features to ensure the accuracy of step identification and quality assessment.

[0073] In one embodiment, see Figure 5 , is a flow chart of a method for constructing a feature vector sequence provided in an embodiment of the present application, such as Figure 5 As shown, the method may include the following steps 510 to 530.

[0074] Step 510: For each sub-video in each sub-video combination, each video frame in the sub-video is normalized, and an initial feature vector is extracted from each normalized video frame through a trained feature extraction model; wherein a corresponding feature extraction model is provided for each sub-video of each viewing angle.

[0075] For each sub-video in each sub-video combination, each video frame in the sub-video may be normalized so that the pixel value of each pixel in the video frame is within a fixed range [0, 1].

[0076] A corresponding normalization processing method can be selected based on the color format of the video frame. For example, if the video frame is a grayscale image with a pixel value between 0 and 255, the pixel value of each pixel can be divided by 255, so that the pixel value of each pixel in the video frame is within [0, 1], and normalization is completed.

[0077] After normalization, each normalized video frame can be forward propagated through the trained feature extraction model to obtain a feature map, and then the feature map is reduced in dimension through global average pooling to obtain an initial feature vector. In this solution, the feature extraction model can be built as needed. Exemplarily, the feature extraction model is a combination of ResNet-50 and a global pooling layer. The last convolution layer of ResNet-50 can output a feature map of 2048*7*7. After the global pooling layer, an initial feature vector of 2048*1*1 can be obtained.

[0078] A corresponding feature extraction model can be set in advance for the sub-video of each perspective, so that each normalized video frame of the sub-video of any perspective is input into its corresponding feature extraction model, and the initial feature vector is extracted for its corresponding video frame through each feature extraction model.

[0079] Step 520: fuse the initial feature vectors of multiple video frames at the same time point in multiple sub-videos to obtain a fused feature vector at the time point.

[0080] The initial feature vectors of multiple video frames at the same time point in multiple sub-videos in the same sub-video combination can be fused by concatenation or splicing to obtain a fused feature vector at the time point.

[0081] Exemplarily, the serial fusion method is represented by the following formula (1):

[0082] F total =concat(F′ top ,F′ left ,F′ front ) (1)

[0083] Among them, F′ top represents the initial feature vector of the video frame in the sub-video captured by the top-view camera; F′ left represents the initial feature vector of the video frame in the sub-video captured by the left-view camera; F′ front represents the initial feature vector of the video frame in the sub-video captured by the front-view camera; F total Represents the fused feature vector.

[0084] If the size of each initial feature vector is 2048*1*1, the three initial feature vectors are fused to obtain a fused feature vector with a size of 6144*1*1.

[0085] In this case, if the frame rate of each sub-video is 24 Hz, the initial feature vectors of each sub-video within one second can construct 24 fused feature vectors.

[0086] Step 530: constructing the fused feature vectors of multiple time points into a feature vector sequence.

[0087] The initial feature vectors of each sub-video in a sub-video combination can be used to construct fused feature vectors at multiple time points. By arranging the multiple fused feature vectors in chronological order, a feature vector sequence can be constructed.

[0088] Through this measure, a corresponding feature vector sequence can be constructed for each sub-video combination.

[0089] In one embodiment, see Figure 6 , is a flow chart of a method for constructing a vector sequence to be processed provided in an embodiment of the present application, such as Figure 6 As shown, the method may include the following steps 610 to 630.

[0090] Step 610: For each fused feature vector in each feature vector sequence, construct the position information of each element based on the order of the fused feature vector in the feature vector sequence and the sequence number of each element in the fused feature vector.

[0091] In this solution, the number of fused feature vectors in each feature vector sequence can be the same, because the sub-video combination is determined based on the same specified time division. Each fused feature vector has a corresponding order in the feature vector sequence, and the sequence number of each element in any fused feature vector can represent the position of the element in the fused feature vector.

[0092] In this case, the position information of each element can be constructed by using the order of the fused feature vector in the feature vector sequence and the sequence number of each element in the fused feature vector.

[0093] Exemplarily, the construction method of the location information can be expressed by the following formula (2) and formula (3):

[0094]

[0095]

[0096] Among them, PE (pos,2i) Indicates the position information of the element with an even number; PE (pos,2i+1) Indicates the position information of the element with an odd sequence number; pos indicates the order of the fused feature vector in the feature vector sequence, and its value range is from 0 to N, where N+1 is the total number of vectors in the feature vector sequence; d model is the dimension of the fused feature vector, that is, the total number of elements in the fused feature vector; the value of i ranges from 0 to d model / 2.

[0097] Step 620: construct a position encoding vector according to the position information of all elements in each fused feature vector.

[0098] After calculating the position information of all elements in any fused feature vector, multiple position information can be sorted based on the arrangement order of the elements corresponding to the position information in the fused feature vector, so as to construct a position encoding vector. Among them, the elements in the position encoding vector correspond one-to-one to the elements in the fused feature vector, and the former is the position information of the latter.

[0099] Step 630: Fuse each fused feature vector with its corresponding position encoding vector to obtain a vector to be processed, and construct a vector sequence to be processed according to a plurality of vectors to be processed corresponding to each feature vector sequence.

[0100] After generating a corresponding position encoding vector for any fused feature vector, the two can be fused, where the fusion method can be to add the elements with the same sequence number in the fused feature vector and the position encoding vector. The vector to be processed can be obtained by fusion.

[0101] The multiple to-be-processed vectors corresponding to the multiple fused feature vectors in each feature vector sequence can construct a to-be-processed vector sequence. Here, the arrangement order of the multiple to-be-processed vectors in the to-be-processed vector sequence is the same as the arrangement order of the corresponding fused feature vectors.

[0102] See also Figure 7 , is a schematic diagram of the overall architecture of the endoscope cleaning behavior detection solution provided in an embodiment of the present application, such as Figure 7As shown, a top view camera, a left view camera and a front view camera are set in the endoscope cleaning area to collect top view video data, front view video data and left view video data respectively. After the top view video data, the front view video data and the left view video data are divided into sub-video combinations, each video frame is normalized, and each normalized video frame is input into the corresponding feature extraction model. The initial feature vector is extracted by each feature extraction model, and the three initial feature vectors corresponding to the same time point are constructed as a fused feature vector. After adding position coding to the fused feature vector to generate a feature vector to be processed, the sequence of vectors to be processed constructed by multiple feature vectors to be processed can be input into the detection model, so as to obtain the step category and quality score corresponding to the sub-video combination.

[0103] In one embodiment, in this solution, the detection model can be constructed based on the Transformer model, and the detection model includes a first output layer and a second output layer, the first output layer performs a classification task, and the second output layer performs a regression task. Exemplarily, the first output layer may include a fully connected layer and a softmax function; the second output layer may include a fully connected layer.

[0104] In this solution, the detection model and feature extraction model can be trained through end-to-end training to obtain better training results.

[0105] See also Figure 8 , is a flow chart of a model training method provided in an embodiment of the present application, such as Figure 8 As shown, the method may include the following steps 810 to 850.

[0106] Step 810: Input each video frame of each sample sub-video in the sample sub-video combination in the sample data set into the corresponding feature extraction model to obtain an initial feature vector corresponding to each video frame; wherein each sample sub-video combination carries a step category label and a quality score label.

[0107] In this solution, in order to reduce the processing steps of the sample sub-video combination in the sample data set during the training phase, each video frame of the sample sub-video can be normalized when constructing the sample data set so that the pixel value of each pixel of the video frame is between 0 and 1.

[0108] The step category labels indicate the actual steps represented by the sample sub-video combination. The quality score labels indicate the execution quality of the cleaning steps within the sample sub-video combination.

[0109] Each video frame of each sample sub-video in the sample sub-video combination can be input into a feature extraction model corresponding to the sample sub-video, so that the video frame is processed by the feature extraction model to obtain an initial feature vector.

[0110] Step 820: construct a vector sequence to be processed corresponding to each sample sub-video combination using the initial feature vector corresponding to each video frame in each sample sub-video combination.

[0111] For each sample sub-video combination, based on the initial feature vectors of the video frames at the same time point in each sub-video, a fused feature vector at that time point can be constructed, and a feature vector sequence can be constructed with the fused feature vectors of multiple time points. Further, by adding position coding to the feature vector sequence, a vector sequence to be processed corresponding to the sample sub-video combination can be obtained. Here, the method of adding position coding can be referred to the relevant description above, which will not be repeated here.

[0112] Step 830: Input the to-be-processed vector sequences corresponding to each sample sub-video combination into the detection model respectively, and obtain the prediction step category and prediction quality score corresponding to each sample sub-video combination.

[0113] After obtaining the vector sequences to be processed corresponding to each sample sub-video combination, each vector sequence to be processed can be output to the detection model respectively. After the vector sequence to be processed is processed by the detection model, the prediction step category is output by the first output layer and the prediction quality score is output by the second output layer.

[0114] Exemplarily, the calculation process of the first output layer is represented by the following formula (4):

[0115] Step = softmax(F transformer W step + b step ) (4)

[0116] Among them, Step represents the prediction step category; F transformer Represents the features of the last layer output of the Transformer model; W step represents the weight of the fully connected layer in the first output layer; b step represents the bias of the fully connected layer in the first output layer.

[0117] Exemplarily, the calculation process of the second output layer is represented by the following formula (5):

[0118] Score = F transformer W score +b score (5)

[0119] Among them, Score represents the prediction quality score; F transformer Represents the features of the last layer output of the Transformer model; W score represents the weight of the fully connected layer in the second output layer; b score represents the bias of the fully connected layer in the second output layer.

[0120] Step 840: Adjust the network parameters of the feature extraction model and the detection model according to the difference between the predicted step category and the step category label, and the difference between the predicted quality score and the quality score label for each sample sub-video combination.

[0121] Step 850: Repeat the above process until the feature extraction model and the detection model converge to obtain a trained feature extraction model and a trained detection model.

[0122] After obtaining the predicted step category and predicted quality score for each sample sub-video combination, the difference between the predicted step category and the step category label, and the difference between the predicted quality score and the quality score label can be evaluated through a preset loss function, and the network parameters of multiple feature extraction models and detection models can be adjusted based on the evaluation results.

[0123] After adjusting the network parameters, the process can return to step 810 to start a new round of training. After repeated iterations, when the function value of the loss function tends to be stable or the training round reaches a preset round threshold, it can be considered that the feature extraction model and the detection model have converged. At this time, a trained feature extraction model and a trained detection model are obtained.

[0124] In one embodiment, before training the model, a sample dataset may be constructed. Fig. 9 , is a flow chart of a method for constructing a sample data set provided in an embodiment of the present application, such as Fig. 9 As shown, the method may include the following steps 910 to 930.

[0125] Step 910: Obtain multiple sub-video combinations, and perform normalization processing on each video frame in each sub-video combination.

[0126] The sub-video combination includes multiple sub-videos, and the multiple sub-videos are collected from multiple viewing angles at the same time by multiple camera devices deployed in the endoscope cleaning area.

[0127] For each video frame in each video frame combination, a normalization process may be performed so that the pixel value of the pixel in the video frame is between 0 and 1.

[0128] Step 920: In response to the labeling instruction, a step category label and a quality score label are added to each sub-video combination after normalization to obtain multiple sample sub-video combinations.

[0129] Step 930: construct a sample data set based on a combination of multiple sample sub-videos.

[0130] The labeling instructions are used to indicate the step category labels and quality score labels corresponding to the sub-video combinations.

[0131] After obtaining the labeling instruction, a step category label and a quality score label can be added to the normalized sub-video combination indicated by the labeling instruction, thereby obtaining a sample sub-video combination.

[0132] After labeling each normalized sub-video combination, multiple sample sub-video combinations can be obtained.

[0133] After obtaining multiple sample sub-video combinations, a sample data set may be constructed based on the multiple sample sub-video combinations.

[0134] To ensure good training results in subsequent training, the number of sample sub-video combinations corresponding to different step category labels should be close, and the number of sample sub-video combinations corresponding to different quality score labels should be close.

[0135] In one embodiment, after obtaining multiple step categories corresponding to the endoscope cleaning behavior to be detected and the quality score of each step category, a comprehensive evaluation result of the endoscope cleaning behavior to be detected can be determined according to the multiple step categories and the quality score of each step category. Here, the comprehensive evaluation result can evaluate the cleaning quality of the endoscope cleaning behavior to be detected as a whole.

[0136] In one embodiment, when determining the comprehensive evaluation result of the endoscope cleaning behavior to be detected based on multiple step categories and the quality score of each step category, it can be determined whether there are the same step categories in the multiple step categories. If so, the same step categories can be grouped together.

[0137] On the one hand, if there is at least one group of the same step categories in multiple step categories, it means that the cleaning step indicated by the step category lasts for a long time. In this case, if the number of groups of the same step category reaches the preset group number threshold, it means that there are multiple cleaning steps that last for a long time, and it can be determined that the comprehensive evaluation result of the endoscope cleaning behavior to be tested is unqualified. Here, the group number threshold can be configured as needed. For example, the group number threshold can be any one of 1, 2, 3, 4, 5, etc.

[0138] Alternatively, if the number of any group of the same step categories reaches a preset number threshold, it means that the cleaning step time indicated by the step category is too long, and at this time, the comprehensive evaluation result of the endoscope cleaning behavior to be tested can be determined to be unqualified. Here, the number threshold can be configured as needed, and illustratively, the number threshold can be any one of 2, 3, 4, 5, 6, etc.

[0139] On the other hand, if there is not at least one group of identical step categories in the multiple step categories, it can be checked whether the lowest score of the quality score of each step category reaches a preset score threshold. Here, the score threshold can be configured as needed. Exemplarily, taking a 10-point scale as an example, the score threshold can be 6 points. If the lowest score reaches the score threshold, the comprehensive evaluation result can be considered qualified. If the lowest score does not reach the score threshold, the comprehensive evaluation result can be considered unqualified.

[0140] Fig.10 is a block diagram of a detection device for endoscope cleaning behavior according to an embodiment of the present invention, such as Fig.10 As shown, the device may include:

[0141] The division module 1010 is used to divide the video combination to be detected according to the specified time length to obtain multiple sub-video combinations; wherein the video combination to be detected includes the video to be detected shot at multiple viewing angles of the endoscope cleaning behavior to be detected;

[0142] An extraction module 1020 is used to extract features from each sub-video combination to obtain a feature vector sequence corresponding to the sub-video combination;

[0143] An adding module 1030 is used to add a position code to each feature vector sequence to obtain a vector sequence to be processed;

[0144] The input module 1040 is used to input the to-be-processed vector sequence corresponding to each sub-video combination into the trained detection model to obtain the step category and quality score corresponding to each sub-video combination.

[0145] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned endoscope cleaning behavior detection method, and will not be repeated here.

[0146] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0148] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

Claims

1. A method for detecting endoscope cleaning behavior, characterized in that: include: The video combination to be detected is divided according to the specified time length to obtain multiple sub-video combinations; wherein the video combination to be detected includes the video to be detected shot at multiple viewing angles of the endoscope cleaning behavior to be detected; Extract features from each sub-video combination to obtain a feature vector sequence corresponding to the sub-video combination; Add position code to each feature vector sequence to obtain a vector sequence to be processed; The vector sequences to be processed corresponding to each sub-video combination are respectively input into the trained detection model to obtain the step category and quality score corresponding to each sub-video combination.

2. The method according to claim 1, characterized in that: The step of extracting features from each sub-video combination to obtain a feature vector sequence corresponding to the sub-video combination includes: For each sub-video in each sub-video combination, each video frame in the sub-video is normalized, and an initial feature vector is extracted from each normalized video frame through a trained feature extraction model; wherein a corresponding feature extraction model is provided for each sub-video of each viewing angle; The initial feature vectors of multiple video frames at the same time point in the multiple sub-videos are fused to obtain a fused feature vector at the time point; The fused feature vectors of multiple time points are constructed as a feature vector sequence.

3. The method according to claim 1, characterized in that The adding of position codes to each feature vector sequence to obtain a vector sequence to be processed includes: For each fused feature vector in each feature vector sequence, construct the position information of each element based on the order of the fused feature vector in the feature vector sequence and the sequence number of each element in the fused feature vector; Construct a position encoding vector according to the position information of all elements in each fused feature vector; Each fused feature vector is fused with its corresponding position encoding vector to obtain a vector to be processed, and a sequence of vectors to be processed is constructed according to a plurality of vectors to be processed corresponding to each feature vector sequence.

4. The method according to claim 2, characterized in that: The detection model is constructed based on the Transformer model, and the detection model includes a first output layer and a second output layer, the first output layer performs a classification task, and the second output layer performs a regression task; The training steps of the feature extraction model and the detection model include: Input each video frame of each sample sub-video in the sample sub-video combination in the sample data set into the corresponding feature extraction model to obtain an initial feature vector corresponding to each video frame; wherein each sample sub-video combination carries a step category label and a quality score label; Using the initial feature vector corresponding to each video frame in each sample sub-video combination, construct a vector sequence to be processed corresponding to the sample sub-video combination; Input the to-be-processed vector sequence corresponding to each sample sub-video combination into the detection model respectively, and obtain the prediction step category and prediction quality score corresponding to each sample sub-video combination; Adjusting network parameters of the feature extraction model and the detection model according to the difference between the predicted step category and the step category label, and the difference between the predicted quality score and the quality score label for each sample sub-video combination; The above process is repeated until the feature extraction model and the detection model converge, thereby obtaining a trained feature extraction model and a trained detection model.

5. The method according to claim 4, characterized in that The step of obtaining the sample data set includes: Obtain multiple sub-video combinations, and perform normalization processing on each video frame in each sub-video combination; In response to the labeling instruction, a step category label and a quality score label are added to each sub-video combination after normalization to obtain a plurality of sample sub-video combinations; A sample dataset is constructed based on a combination of multiple sample sub-videos.

6. The method according to claim 1, characterized in that The method further comprises: After obtaining multiple step categories corresponding to the endoscope cleaning behavior to be detected and the quality score of each step category, a comprehensive evaluation result of the endoscope cleaning behavior to be detected is determined according to the multiple step categories and the quality score of each step category.

7. The method according to claim 6, characterized in that Determining the comprehensive evaluation result of the endoscope cleaning behavior to be detected according to the multiple step categories and the quality score of each step category includes: If there is at least one group of identical step categories among the multiple step categories, and the number of groups of identical step categories reaches a preset group number threshold, or the number of any group of identical step categories reaches a preset number threshold, the comprehensive evaluation result of the endoscope cleaning behavior to be tested is determined to be unqualified.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the method for detecting endoscope cleaning behavior as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the method for detecting endoscope cleaning behavior as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for detecting endoscope cleaning behavior described in any one of claims 1 to 7 is implemented.

Images