Endoscopic examination evaluation system and method based on video understanding network

Abstract

The present invention relates to an endoscopic examination evaluation system and method based on a video understanding network. The system comprises: a behavior and action recognition module, which is configured to perform, on the basis of a pre-trained video understanding network model based on dual-cascade spatio-temporal attention, scene action recognition on an endoscopic examination video frame image collected by a video acquisition module, so as to obtain scene action recognition data; a cleaning-event recognition module, which is configured to perform association and mapping between a scene to be cleaned and a cleaning action in the scene action recognition data, so as to obtain a cleaning-event recognition data set; and an endoscopic examination recording module, which is configured to determine evaluation parameters of an endoscopic examination process on the basis of the cleaning-event recognition data set. In the present invention, semantic action classification and detection are performed on an endoscopic examination operation and a corresponding video image, and semantic action data and stipulated endoscopic examination action data are used to form comparable data to perform auxiliary prompting and index quantification, thereby promoting the standardization of endoscopic examinations.

Description

Endoscopic Examination Evaluation System and Method Based on Video Understanding Network

[0001] Cross-references

[0002] This application claims priority to Chinese Patent Application No. 202411186475.4, filed with the Chinese Patent Office no earlier than August 28, 2024, entitled "Endoscopic Examination and Evaluation System and Method Based on Video Understanding Network", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to the field of artificial intelligence, and in particular to an endoscopic examination and evaluation system and method based on video understanding networks. Background Technology

[0004] Colorectal carcinoma (CRC) is one of the most common malignant tumors. Multiple studies have reported that early-stage CRC has a high survival rate after treatment; therefore, early screening plays a crucial role in the diagnosis and treatment of CRC.

[0005] Colonoscopy, as the gold standard for diagnosing colorectal polyps (CP), is a core component of the screening process. However, studies have shown that missed diagnoses of colorectal adenoma (CRA) during colonoscopy are generally related to many factors, including the patient's bowel preparation, the time required to withdraw the endoscope, the physician's experience, and the use and type of auxiliary imaging techniques. Withdrawal time is a crucial indicator of colonoscopy quality. However, clinically, during colonoscopy withdrawal, physicians perform non-observational procedures such as bowel irrigation and surgical interventions. This limitation means that withdrawal time alone cannot accurately reflect the time spent observing the intestines to locate lesions. Therefore, current techniques urgently need improvement. Summary of the Invention

[0006] This invention provides an endoscopic examination evaluation system and method based on video understanding networks to at least partially solve the above-mentioned problems.

[0007] In a first aspect, the present invention provides an endoscopic examination evaluation system based on a video understanding network, the endoscopic examination evaluation system based on a video understanding network includes a video acquisition module, a behavior and action recognition module, a cleaning event recognition module, and an endoscopic examination recording module.

[0008] The behavior and action recognition module is used to perform scene action recognition on the endoscopic examination video frame images acquired by the video acquisition module based on a pre-trained video understanding network model based on dual-cascaded spatiotemporal attention, and obtain scene action recognition data.

[0009] The cleaning event recognition module is used to associate and map the scene to be cleaned and the cleaning action in the scene action recognition data to obtain a cleaning event recognition data set.

[0010] The endoscopy examination record module is used to determine the evaluation parameters of the endoscopy examination process based on the cleaning event identification data set.

[0011] Optionally, the behavior recognition module is further configured to train a pre-constructed dual-cascade spatiotemporal attention model based on pre-acquired endoscopic examination video samples to obtain the video understanding network model based on dual-cascade spatiotemporal attention; the dual-cascade spatiotemporal attention model is obtained by connecting a normalization module and multiple sets of dual-cascade unit blocks and a prediction head module in series.

[0012] Optionally, the standardization module is used to convert the endoscopic examination video frame images acquired by the video acquisition module into two multi-dimensional input data by downsampling according to multiple preset dimensions.

[0013] Optionally, the preset dimensions include the number of video frames, the original width of the video frame, the original height of the video frame, and the number of channels per video frame.

[0014] Optionally, the multiple sets of dual-cascaded unit blocks are connected in series. For any set of dual-cascaded unit blocks, they are used to perform convolution operations on the two multi-dimensional input data using two sets of spatially separable convolutional blocks to obtain a first feature map and a second feature map, respectively; and are used to downsample the first feature map using a lateral fusion module to align the dimensions of the first feature map with the second feature map and concatenate them to obtain a third feature map; and are used to use the first feature map and the third feature map as input to a connection module; the connection module is the next set of dual-cascaded unit blocks or a prediction head module connected in series.

[0015] Optionally, the prediction head module is used to perform SOFTMAX operation on the first feature map and the third feature map of multiple sets of double-cascaded unit block inputs, average the obtained scores, output the scene action recognition probability, and obtain scene action recognition data.

[0016] Optionally, the video understanding network-based endoscopic examination evaluation system further includes an auxiliary prompting module and a client module;

[0017] The auxiliary prompting module is used to provide operation prompts corresponding to the scene action recognition data based on the scene action recognition data.

[0018] The client module is used to display the evaluation parameters and the operation prompts.

[0019] Secondly, the present invention provides an endoscopic examination evaluation method based on a video understanding network, the endoscopic examination evaluation method based on a video understanding network comprising:

[0020] Based on a pre-trained video understanding network model with dual-cascaded spatiotemporal attention, scene action recognition is performed on the acquired endoscopic examination video frame images to obtain scene action recognition data.

[0021] The scene to be cleaned and the cleaning action in the scene action recognition data are associated and mapped to obtain a set of cleaning event recognition data.

[0022] Based on the cleaning event identification data set, evaluation parameters for the endoscopic examination process are determined.

[0023] Thirdly, the present invention provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor;

[0024] When the computer program is executed by the processor, it implements the steps of the endoscopic examination evaluation method based on video understanding network as described in any of the preceding claims.

[0025] Fourthly, the present invention provides a computer-readable storage medium storing an endoscopic examination evaluation program based on a video understanding network, wherein when the endoscopic examination evaluation program based on a video understanding network is executed by a controller, the steps of the endoscopic examination evaluation method based on a video understanding network as described in any of the preceding claims are implemented.

[0026] The various embodiments of this invention can accurately reflect the time spent observing the intestines to locate lesions by evaluating parameters. This invention can perform semantic action classification and detection on endoscopic examination procedures and corresponding image scenes, and form comparable data with the prescribed endoscopic examination action data for auxiliary prompts and indicator quantification. This promotes the standardization of endoscopic examinations and improves the lesion detection rate. This invention can effectively evaluate the endoscopic examination process and improve examination quality. Attached Figure Description

[0027] Figure 1 is a block diagram of an endoscopic examination evaluation system based on a video understanding network according to an embodiment of the present invention;

[0028] Figure 2 is a flowchart of an endoscopic examination evaluation method based on a video understanding network according to an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0030] Example 1

[0031] This invention provides an endoscopic examination evaluation system based on a video understanding network, as shown in Figure 1. The endoscopic examination evaluation system based on a video understanding network includes an S1 video acquisition module, an S2 behavior and action recognition module, an S4 cleaning event recognition module, and an S5 endoscopic examination recording module.

[0032] The behavior and action recognition module is used to perform scene action recognition on the endoscopic examination video frame images acquired by the video acquisition module based on a pre-trained video understanding network model based on dual-cascaded spatiotemporal attention, and obtain scene action recognition data.

[0033] The cleaning event recognition module is used to associate and map the scene to be cleaned and the cleaning action in the scene action recognition data to obtain a cleaning event recognition data set.

[0034] The endoscopy examination record module is used to determine the evaluation parameters of the endoscopy examination process based on the cleaning event identification data set.

[0035] The scene action recognition data includes recognition data for scene actions such as normal observation, biopsy, snare removal, electrocautery excision, scene to be cleaned (feces present), immersion suction, hidden suction, saline rinsing, lens approaching mucosa, and lens stationary.

[0036] Cleaning procedures include immersion absorbing, concealed absorbing, and saline rinsing.

[0037] The evaluation parameters include intestinal cleanliness, cleaning time, time spent in uncleaned intestinal segments, operation time, surgical procedures, and precise withdrawal time of the endoscope.

[0038] To reduce the rate of missed diagnoses during endoscopy, the withdrawal time must first be greater than 6 minutes to ensure more thorough and clear observation of the intestinal mucosa and more careful examination of potential lesion locations, thereby reducing the incidence of CRC. However, factors such as physician experience and individual differences in intestinal conditions can affect the quality of endoscopy. During the examination, if the patient's bowel preparation is insufficient and the field of vision is blurred, the endoscopist needs to perform bowel irrigation, and the efficiency of irrigation is related to the physician's operating technique. Different physicians' scores in bowel cleaning during withdrawal are correlated with the detection of adenomas and polyps. Therefore, this time should be excluded from the withdrawal time to avoid prolonged withdrawal time due to insufficient bowel preparation and to remove confounding factors caused by differences in physician operation. There are two concepts regarding the optimal withdrawal time: Visual withdrawal time (VWT) and Overall withdrawal time (OWT). VWT excludes biopsy collection time and treatment time, retaining only the diagnostic time, which helps to more accurately analyze the withdrawal process. Therefore, the withdrawal time should accurately exclude the time for surgical treatment and biopsy. In addition, during the microscopic examination, doctors observe for a relatively long time to avoid missing diagnoses. However, spending too much time in areas where there are no lesions will not only increase the examination time, but also have no substantial significance in improving ADR. Therefore, the time spent stationary with the lens should be accurately excluded.

[0039] Currently, there are no tools to accurately analyze the semantics of scenarios such as normal observation, biopsy, snare removal, electrocautery resection, presence of feces, immersion aspiration, concealed aspiration, saline irrigation, lens approaching the mucosa, and lens stillness. Therefore, it is impossible to accurately give the withdrawal time and intestinal cleanliness, resulting in the current situation where colonoscopy techniques and quality cannot be standardized.

[0040] The endoscopic examination evaluation system based on video understanding networks provided in this invention utilizes a video understanding network model based on dual-cascade spatiotemporal attention in the behavior and action recognition module to perform scene action recognition on endoscopic examination video frame images acquired by the video acquisition module, obtaining scene action recognition data. Then, a cleaning event recognition module associates and maps the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a cleaning event recognition data set. Based on this cleaning event recognition data set, the endoscopic examination recording module can determine the evaluation parameters of the endoscopic examination process. These evaluation parameters accurately reflect the time spent observing the intestines to find lesions. This invention effectively solves the aforementioned problems. This invention can perform semantic action classification and detection on endoscopic examination operation steps and corresponding image scenes, and form comparable data with the prescribed endoscopic examination action data for auxiliary prompts and indicator quantification, thereby promoting the standardization of endoscopic examinations and improving the lesion detection rate. This invention can effectively evaluate the endoscopic examination process and improve examination quality.

[0041] This invention utilizes a video understanding network model to effectively capture information about the length and distance of video, achieving good recognition results. This invention also employs a method for recognizing actions during colonoscopy based on a video understanding network, thereby eliminating time spent on clinical procedures such as treatment by the physician and improving examination quality.

[0042] In some embodiments, the video understanding network-based endoscopic examination evaluation system may further include an S3 auxiliary prompting module and an S6 client module;

[0043] The auxiliary prompting module is used to provide operation prompts corresponding to the scene action recognition data based on the scene action recognition data.

[0044] The client module is used to display the evaluation parameters and the operation prompts.

[0045] Specifically, the endoscopic examination evaluation system based on video understanding networks provided in this embodiment includes:

[0046] S1. Video acquisition module, which can read the video stream output by the endoscope device in real time and obtain the video frame image of the endoscope examination during the examination;

[0047] S2. The behavior and action recognition module utilizes a pre-constructed artificial intelligence video understanding neural network model based on a dual-cascade spatiotemporal attention module (i.e., a video understanding network model based on dual-cascade spatiotemporal attention) to recognize the endoscopic examination video frame images from the video acquisition module, and obtain the preset scene action recognition results (i.e., scene action recognition data) in the endoscopic examination; the scene actions in the endoscopic examination include: normal observation, biopsy, snare forceps removal, electrocautery resection, appearance of feces, immersion aspiration, hidden aspiration, saline irrigation, lens approaching the mucosa, and lens stillness;

[0048] S3. Assistive prompting module, which can provide auxiliary prompts based on scene action recognition results;

[0049] S4. The cleaning event recognition module can associate and map fecal scenes and cleaning actions (immersion water absorption, hidden water absorption, and saline flushing) based on continuous scene action recognition results to obtain cleaning event recognition results; and record the video frame segments corresponding to the cleaning events to obtain a cleaning event data set.

[0050] S5. Endoscopic examination recording module can assess intestinal cleanliness based on the action recognition results and cleaning event data set of the entire examination scene, record treatment records such as intraoperative cleaning efficiency, biopsy, and resection, and generate reports.

[0051] The S6 client module can automatically record the time and content of actions during surgical treatment on the upper left; display auxiliary prompts on the lower left; and display real-time video footage on the right.

[0052] Example 2

[0053] This invention provides a specific endoscopic examination evaluation system based on a video understanding network, the system comprising:

[0054] S1 Video Acquisition Module

[0055] This module uses a video capture card to transmit digital / analog endoscopic examination video frame images such as HDMI, DVI, SDI, and S-Video to the host computer. It then uses OpenCV to read the endoscopic examination video frame images and converts them frame by frame into RGB image format.

[0056] S2 Behavior and Action Recognition Module

[0057] This module is used to perform scene action recognition on the endoscopic examination video frame images acquired by the video acquisition module based on a pre-trained video understanding network model based on dual-cascade spatiotemporal attention, and to obtain scene action recognition data. It may include a model building unit and a recognition unit.

[0058] The model building unit is used to pre-build an AI video understanding network model based on dual-cascade spatiotemporal attention. It can train the pre-built dual-cascade spatiotemporal attention model using pre-acquired endoscopic examination video samples to obtain the video understanding network model based on dual-cascade spatiotemporal attention. This includes building the network model structure, labeling pre-collected endoscopic examination video segments, and training the network model structure to obtain the video understanding network model. Details:

[0059] 1. Construct a network model structure, which consists of one standardized module, four sets of double-cascaded unit blocks, and one prediction head module connected in series.

[0060] Optionally, the standardization module is used to convert the endoscopic examination video frame images acquired by the video acquisition module into two input data streams using downsampling, based on multiple preset dimensions. More specifically,

[0061] The input data of dimension T0×W0×H0×C0 is downsampled and converted into two inputs, with the two input dimensions being T×W×H×C and T′×W′×H′×C′, respectively.

[0062] Where T0 is the number of video frames, in this embodiment T0 = 2 (seconds) * 25 (fps) = 50; W0 is the original width of the video frame; H0 is the original height of the video frame; C0 is the number of channels per video frame, where the channels are RGB, so C0 = 3;

[0063] T is the number of video frames after dimensionality reduction. In this embodiment, T = 2 (seconds) * 10 = 20; W is the width of the video frame after dimensionality reduction, W = 448; H is the height of the video frame after dimensionality reduction, H = 448; C is the number of channels per video frame, where the channels are RGB, so C = 3.

[0064] T′×W′×H′×C′ corresponds to the other dimension after dimensionality reduction, which are the number of video frames after dimensionality reduction, the width of the video frame after dimensionality reduction, the height of the video frame after dimensionality reduction, and the number of channels per video frame, respectively.

[0065] Optionally, the multiple sets of dual-cascaded unit blocks are connected in series. For any set of dual-cascaded unit blocks, they are used to perform convolution operations on the two multi-dimensional input data using two sets of spatially separable convolutional blocks to obtain a first feature map and a second feature map, respectively; and are used to downsample the first feature map using a lateral fusion module to align the dimensions of the first feature map with the second feature map and concatenate them to obtain a third feature map; and are used to use the first feature map and the third feature map as input to a connection module; the connection module is the next set of dual-cascaded unit blocks or a prediction head module connected in series.

[0066] Specifically, the double-cascaded unit block is used for:

[0067] The first set of spatially separable convolutional blocks is used to perform convolution operations on the input data T×W×H×C to obtain the first feature map F1.

[0068] The second set of spatially separable convolutional blocks is used to perform convolution operations on the input data T′×W′×H′×C′ to obtain the second feature map F2.

[0069] The lateral fusion module is used to downsample the F1 feature map to align its dimensions with those of the F2 feature map, and the feature maps are then concatenated to obtain the third feature map.

[0070] First feature map F1 and third feature map As the input to the first spatially separable convolutional block and the second spatially separable convolutional block in the next set of double-cascaded unit blocks, and so on, two feature maps are obtained after 4 iterations.

[0071] Optionally, the prediction head module is used to perform SOFTMAX operation on the two feature maps output by the four sets of double-cascaded unit blocks, average the obtained scores, output the scene action recognition probability, and obtain scene action recognition data.

[0072] 2. Label the pre-collected endoscopic examination video samples. The labeling results should include the preset endoscopic examination scene actions, which may include normal observation, biopsy, snare removal, electrocautery resection, cleaning scene (feces appear), immersion aspiration, concealed aspiration, saline irrigation, lens approaching the mucosa, and lens stillness.

[0073] 3. Train the network model structure to obtain the video understanding network model. In detail: First, input the labeled video clips into the network model structure to obtain the output result y. pred Secondly, the cross-entropy loss function is used to calculate y. pred With the correct result y true The error loss is calculated, and the network model parameters are updated using backpropagation with the AdamW optimizer; finally, the model is iterated through round after round until it is reduced to the preset loss value (in this embodiment, the loss value is 1e). -5 Training can be stopped when the maximum number of iterations is reached (500 iterations in this embodiment), and the constructed video understanding network model is obtained.

[0074] The recognition unit is used to acquire and recognize segments of the video frame images in the S1 video acquisition module, specifically:

[0075] A set of consecutive video frame images within a 5-second timeframe is continuously saved to memory to obtain the set of video frame images to be identified. This set of video frame images is then input into a video understanding network model to obtain the output result y. pred ;

[0076] The probability of each category is predicted using a prediction function, specifically: P1, P2, ..., P i =SOFTMAX(y pred The system selects the category with the highest probability to obtain scene action recognition data for a set of video frame segments. The scene action recognition data includes normal observation, biopsy, snare removal, electrocautery excision, presence of feces, immersion aspiration, hidden aspiration, saline flushing, lens approaching mucosa, and lens stillness.

[0077] S3 Auxiliary Prompt Module

[0078] This module is used to provide operation prompts corresponding to the scene action recognition data. Optionally, when a scene with foreign objects to be cleaned (feces scene) is detected, the user is prompted to flush the intestines; when the lens is close to the mucosa, the user is prompted that there is a risk of intestinal perforation and the lens needs to be moved away; when a static scene occurs, the user is prompted that the static time has been too long. Real-time prompts can also be pushed to the client for display using long-connection WebSocket communication.

[0079] S4 Cleaning Event Recognition Module

[0080] This module is used to associate and map the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a cleaning event recognition data set. Specifically, it uses a dynamic time alignment kernel algorithm to associate and map the scenes to be cleaned (e.g., feces scenes) with foreign objects and the cleaning actions (immersion water absorption, hidden water absorption, saline rinsing) based on the scene action recognition data of continuous video frame segments to obtain cleaning event recognition data, and records the video frame segments contained in the cleaning event recognition data to obtain a cleaning event data set.

[0081] Specifically, firstly, the scene action recognition results of video frame segments of a specified length (500 in this invention, i.e., 20 (seconds) × 25 (fps)) are accumulated and combined to form sequence A; secondly, a dynamic event alignment kernel algorithm is called to calculate the cumulative similarity with a preset sequence B, obtaining the association mapping result of the current sequence A. A Gaussian kernel is selected as the kernel function to ensure that the cumulative similarity result is between 0 and 1; finally, it is determined whether the association mapping result of the current sequence A is greater than a preset clean event threshold. If it is greater, the current sequence A is recorded as a clean event; otherwise, it is not recorded.

[0082] Optionally, the method for determining the preset sequence B is to pre-set it. In this invention, it is set to a sequence of length 500, with the first 250 consecutive parameter values ​​being "scenario to be cleaned" and the last 250 consecutive values ​​being "cleaning action"; the cleaning event threshold is set to 0.65.

[0083] This invention establishes a correspondence between the scene to be cleaned (intestinal feces) and the cleaning action, and assesses only the proportion of uncleaned intestinal segments when evaluating cleanliness, which is more flexible and objective than traditional methods.

[0084] S5 Endoscopic Examination Recording Module

[0085] This module is used to determine the evaluation parameters of the endoscopic examination process based on the cleaning event identification data set. The evaluation parameters include intestinal cleanliness, "cleaning time", "uncleaned intestinal segment time", "operation time", "operational actions", and "precise withdrawal time".

[0086] In detail, this module can be used to assess intestinal cleanliness based on the cleaning event recognition data set of video frame segments and the cleaning time. Specifically, firstly, all fecal scenes are recorded; secondly, the scenes to be cleaned included in the cleaning event recognition data set are subtracted to obtain the uncleaned scenes; finally, the intestinal cleanliness is obtained by multiplying the number of uncleaned scenes by the number of sampling seconds and the total inspection time.

[0087] This module can also be used to record intraoperative cleaning efficiency, biopsy, resection and other surgical treatment records based on cleaning event recognition data of video frame segments, and generate reports, specifically including "cleaning time", "uncleaned intestinal segment time", "operation time", "surgical actions" and "precise withdrawal time".

[0088] S6 Client Module

[0089] The client can be a web interface using a B / S architecture for communication. The upper left corner automatically records and displays the time and content of actions in the treatment scene; the lower left corner displays warning information, pushed via a long-lived S3 websocket connection, with text displayed; and the right side of the interface uses a Canvas to display video frames in real time.

[0090] Example 3

[0091] This invention provides an endoscopic examination evaluation method based on video understanding networks, as shown in Figure 2. The endoscopic examination evaluation method based on video understanding networks includes:

[0092] S101, Based on the pre-trained video understanding network model based on dual-cascaded spatiotemporal attention, scene action recognition is performed on the acquired endoscopic examination video frame images to obtain scene action recognition data;

[0093] S102, associate and map the scene to be cleaned and the cleaning action in the scene action recognition data to obtain a set of cleaning event recognition data;

[0094] S103, Based on the cleaning event identification data set, determine the evaluation parameters for the endoscopic examination process.

[0095] Optionally, before S101, the process may further include: training a pre-constructed dual-cascade spatiotemporal attention model based on pre-acquired endoscopic examination video samples to obtain the video understanding network model based on dual-cascade spatiotemporal attention; the dual-cascade spatiotemporal attention model is obtained by concatenating a normalization module and multiple sets of dual-cascade unit blocks and a prediction head module.

[0096] The standardization module is used to convert the endoscopic examination video frame images acquired by the video acquisition module into two multi-dimensional input data by downsampling according to multiple preset dimensions.

[0097] In some embodiments, the preset multiple dimensions include the number of video frames, the original width of the video frame, the original height of the video frame, and the number of channels per video frame.

[0098] Optionally, the multiple sets of dual-cascaded unit blocks are connected in series. For any set of dual-cascaded unit blocks, they are used to perform convolution operations on the two multi-dimensional input data using two sets of spatially separable convolutional blocks to obtain a first feature map and a second feature map, respectively; and are used to downsample the first feature map using a lateral fusion module to align the dimensions of the first feature map with the second feature map and concatenate them to obtain a third feature map; and are used to use the first feature map and the third feature map as input to a connection module; the connection module is the next set of dual-cascaded unit blocks or a prediction head module connected in series.

[0099] The prediction head module is used to perform SOFTMAX operation on the first feature map and the third feature map of multiple sets of double concatenated unit blocks, average the obtained scores, output the scene action recognition probability, and obtain scene action recognition data.

[0100] In some embodiments, the endoscopic examination evaluation method based on video understanding networks may further include: providing operation prompts corresponding to the scene action recognition data based on the scene action recognition data; and displaying the evaluation parameters and the operation prompts.

[0101] The method provided in this invention can perform semantic action classification and detection on endoscopic examination operation steps and corresponding image scenes, and form comparable data with the prescribed action data of endoscopic examination to provide auxiliary prompts and quantify indicators, thereby promoting the standardization of endoscopic examination and improving the lesion detection rate.

[0102] Example 4

[0103] This invention provides a storage device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor;

[0104] When the computer program is executed by the processor, it implements the steps of the endoscopic examination evaluation method based on video understanding network as described in any one of Embodiments 3.

[0105] Example 5

[0106] This invention provides a computer-readable storage medium storing a solid-state drive (SSD) data management program. When the SSD data management program is executed by a controller, it implements the steps of the endoscopic examination and evaluation method based on a video understanding network as described in any one of Embodiment 3.

[0107] In the specific implementation process of Examples 2 to 5, Example 1 can be referred to, and the corresponding technical effects are achieved.

[0108] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An endoscopic examination evaluation system based on video understanding networks, characterized in that, The video understanding network-based endoscopic examination evaluation system includes: a video acquisition module, a behavior and action recognition module, a cleaning event recognition module, and an endoscopic examination recording module; The behavior and action recognition module is used to perform scene action recognition on the endoscopic examination video frame images acquired by the video acquisition module based on a pre-trained video understanding network model based on dual-cascaded spatiotemporal attention, and obtain scene action recognition data. The cleaning event recognition module is used to associate and map the scene to be cleaned and the cleaning action in the scene action recognition data to obtain a cleaning event recognition data set. The endoscopy examination record module is used to determine the evaluation parameters of the endoscopy examination process based on the cleaning event identification data set.

2. The endoscopic examination evaluation system based on video understanding network according to claim 1, characterized in that, The behavior recognition module is also used to train a pre-constructed dual-cascade spatiotemporal attention model based on pre-acquired endoscopic examination video samples to obtain the video understanding network model based on dual-cascade spatiotemporal attention; the dual-cascade spatiotemporal attention model is obtained by connecting a normalization module and multiple sets of dual-cascade unit blocks and a prediction head module.

3. The endoscopic examination evaluation system based on video understanding network according to claim 2, characterized in that, The standardization module is used to convert the endoscopic examination video frame images acquired by the video acquisition module into two multi-dimensional input data channels by downsampling according to multiple preset dimensions.

4. The endoscopic examination evaluation system based on video understanding network according to claim 3, characterized in that, The preset dimensions include the number of video frames, the original width of the video frame, the original height of the video frame, and the number of channels per video frame.

5. The endoscopic examination evaluation system based on video understanding network according to claim 3, characterized in that, The multiple sets of double-cascaded unit blocks are connected in series. For any set of double-cascaded unit blocks, it is used to perform convolution operations on the two multi-dimensional input data using two sets of spatially separable convolutional blocks to obtain a first feature map and a second feature map respectively; and it is used to downsample the first feature map using a lateral fusion module so that the dimensions of the first feature map are aligned with the second feature map and spliced ​​together to obtain a third feature map. And for using the first feature map and the third feature map as input to the connection module; the connection module is a series-connected next set of double-cascaded unit blocks or prediction head modules.

6. The endoscopic examination evaluation system based on video understanding network according to claim 5, characterized in that, The prediction head module is used to perform SOFTMAX operation on the first feature map and the third feature map of multiple sets of double concatenated unit blocks, average the obtained scores, output the scene action recognition probability, and obtain scene action recognition data.

7. The endoscopic examination evaluation system based on video understanding networks according to any one of claims 1-6, characterized in that, The video understanding network-based endoscopic examination evaluation system also includes an auxiliary prompting module and a client module; The auxiliary prompting module is used to provide operation prompts corresponding to the scene action recognition data based on the scene action recognition data. The client module is used to display the evaluation parameters and the operation prompts.

8. An endoscopic examination evaluation method based on video understanding networks, characterized in that, The endoscopic examination evaluation method based on video understanding networks includes: Based on a pre-trained video understanding network model with dual-cascaded spatiotemporal attention, scene action recognition is performed on the acquired endoscopic examination video frame images to obtain scene action recognition data. The scene to be cleaned and the cleaning action in the scene action recognition data are associated and mapped to obtain a set of cleaning event recognition data. Based on the cleaning event identification data set, evaluation parameters for the endoscopic examination process are determined.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor; When the computer program is executed by the processor, it implements the steps of the endoscopic examination evaluation method based on video understanding network as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an endoscopic examination evaluation program based on a video understanding network. When the endoscopic examination evaluation program based on a video understanding network is executed by the controller, it implements the steps of the endoscopic examination evaluation method based on a video understanding network as described in claim 8.

Images