Endoscopic examination evaluation system and method based on video understanding network

Abstract

The present invention relates to an endoscopic inspection evaluation system and method based on a video understanding network, the system comprising: a behavior action recognition module for performing scene action recognition on an endoscopic inspection video frame image acquired by a video acquisition module according to a pre-trained video understanding network model based on dual-cascade spatiotemporal attention, to obtain scene action recognition data; a cleaning event recognition module for associating and mapping the scene to be cleaned and the cleaning action in the scene action recognition data to obtain a cleaning event recognition data set; an endoscopic inspection recording module for determining evaluation parameters of the endoscopic inspection process according to the cleaning event recognition data set. The present invention performs semantic action classification detection on the endoscopic inspection operation and the corresponding video image, and forms comparable data between each semantic action data and the endoscopic inspection prescribed action data, performs auxiliary prompts and indicator quantification, thereby promoting the standardization of endoscopic inspection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an endoscopic examination evaluation system and method based on a video understanding network. Background Art

[0002] Colorectal cancer (CRC) is one of the most common malignant tumors. Many studies have reported that early CRC has a higher survival rate after treatment, so early screening plays a vital role in the diagnosis and treatment of CRC.

[0003] As the gold standard for diagnosing colorectal polyps (CP), colorectal endoscopy is the core link of the entire screening process. However, studies have shown that during the endoscopy process, the reasons for missed diagnosis of colorectal adenoma (CRA) are generally related to many factors such as the patient's intestinal preparation, endoscopy withdrawal time, physician's personal experience, and whether or what auxiliary observation imaging method is used. Among them, endoscopy withdrawal time is one of the important indicators for measuring the quality of colonoscopy. However, clinically, during the endoscopy withdrawal process, physicians have to perform non-observational operations such as intestinal flushing and surgical treatment. This shortcoming makes it impossible for the simple endoscopy withdrawal time to truly reflect the time used to observe the intestines and search for lesions. Therefore, the existing technology needs to be improved urgently. Summary of the invention

[0004] The embodiments of the present invention provide an endoscopic examination evaluation system and method based on a video understanding network to at least partially solve the above problems.

[0005] In a first aspect, the present invention provides an endoscope inspection evaluation system based on a video understanding network, wherein the endoscope inspection evaluation system based on a video understanding network comprises a video acquisition module, a behavior action recognition module, a cleaning event recognition module and an endoscope inspection recording module;

[0006] The behavior action recognition module is used to perform scene action recognition on the endoscopic inspection video frame images collected by the video acquisition module according to the pre-trained video understanding network model based on dual cascade spatiotemporal attention to obtain scene action recognition data;

[0007] The cleaning event recognition module is used to associate and map the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a cleaning event recognition data set;

[0008] The endoscopy recording module is used to determine evaluation parameters of the endoscopy process based on the cleaning event identification data set.

[0009] Optionally, the behavior action recognition module is also used to train a pre-constructed dual-cascade spatiotemporal attention model based on pre-acquired endoscopic inspection video samples to obtain the video understanding network model based on dual-cascade spatiotemporal attention; the dual-cascade spatiotemporal attention model is obtained by connecting a standardization module and multiple groups of dual-cascade unit blocks and a prediction head module in series.

[0010] Optionally, the standardization module is used to convert the endoscopic inspection video frame images collected by the video acquisition module into two-way multi-dimensional input data by downsampling according to preset multiple dimensions.

[0011] Optionally, the preset multiple dimensions include the number of video frames, the original width of the video frames, the original height of the video frames, and the number of channels of a single video frame.

[0012] Optionally, the multiple groups of dual-cascade unit blocks are connected in series. For any group of dual-cascade unit blocks, they are used to perform convolution operations on the two-way multi-dimensional input data using two groups of spatially separable convolution blocks to obtain a first feature map and a second feature map respectively; and to downsample the first feature map using a lateral fusion module so that the dimension of the first feature map is aligned and spliced ​​with the second feature map to obtain a third feature map; and to use the first feature map and the third feature map as inputs to a connection module; the connection module is the next group of dual-cascade unit blocks or prediction head module connected in series.

[0013] Optionally, the prediction head module is used to perform SOFTMAX operations on the first feature maps and the third feature maps input by multiple groups of dual-cascade unit blocks, average the obtained scores, output the scene action recognition probability, and obtain the scene action recognition data.

[0014] Optionally, the video understanding network-based endoscopy assessment system further includes an auxiliary prompt module and a client module;

[0015] The auxiliary prompt module is used to provide an operation prompt corresponding to the scene action recognition data according to the scene action recognition data;

[0016] The client module is used to display the evaluation parameters and the operation prompts.

[0017] In a second aspect, the present invention provides an endoscopy evaluation method based on a video understanding network, the endoscopy evaluation method based on a video understanding network comprising:

[0018] According to the pre-trained video understanding network model based on dual cascade spatiotemporal attention, scene action recognition is performed on the collected endoscopic inspection video frame images to obtain scene action recognition data;

[0019] Associatively mapping the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a cleaning event recognition data set;

[0020] Based on the cleaning event identification data set, an evaluation parameter of an endoscopy procedure is determined.

[0021] In a third aspect, the present invention provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor;

[0022] When the computer program is executed by the processor, the steps of the endoscopic examination evaluation method based on the video understanding network as described in any one of the above items are implemented.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium, on which is stored an endoscopic inspection evaluation program based on a video understanding network. When the endoscopic inspection evaluation program based on a video understanding network is executed by a controller, the steps of the endoscopic inspection evaluation method based on a video understanding network as described in any one of the above items are implemented.

[0024] Each embodiment of the present invention can truly reflect the time used to observe the intestines and find lesions through evaluation parameters. The present invention can perform semantic action classification detection on the endoscopic examination operation steps and the corresponding image scenes, and form comparable data between each semantic action data and the endoscopic examination prescribed action data, and perform auxiliary prompts and index quantification, thereby promoting the standardization of endoscopic examination and improving the lesion detection rate. The present invention can effectively evaluate the endoscopic examination process and improve the inspection quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a block diagram of an endoscopy evaluation system based on a video understanding network according to an embodiment of the present invention;

[0026] Figure 2 4 is a flowchart of an endoscopy evaluation method based on a video understanding network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0028] Embodiment 1

[0029] The embodiment of the present invention provides an endoscopy evaluation system based on a video understanding network, such as Figure 1As shown, the endoscope inspection evaluation system based on the video understanding network includes S1 video acquisition module, S2 behavior action recognition module, S4 cleaning event recognition module and S5 endoscope inspection recording module;

[0030] The behavior action recognition module is used to perform scene action recognition on the endoscopic inspection video frame images collected by the video acquisition module according to the pre-trained video understanding network model based on dual cascade spatiotemporal attention to obtain scene action recognition data;

[0031] The cleaning event recognition module is used to associate and map the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a cleaning event recognition data set;

[0032] The endoscopy recording module is used to determine evaluation parameters of the endoscopy process based on the cleaning event identification data set.

[0033] Among them, the scene action recognition data include recognition data of scene actions such as normal observation, biopsy, snare removal, electrosurgical resection, scenes to be cleaned (presence of feces), immersion water absorption, hidden water absorption, saline flushing, lens close to mucosa, and lens stillness.

[0034] Cleaning actions include immersion suction, hidden suction, saline flushing, etc.

[0035] The evaluation parameters include intestinal cleanliness, cleaning time, uncleaned intestinal segment time, operation time, surgical movements, precise scope withdrawal time and other parameters.

[0036] In order to reduce the missed diagnosis rate of microscopic examination, the withdrawal time should be guaranteed to be greater than 6 minutes, so as to ensure a more complete and clear observation of the intestinal mucosa and a more careful inspection of the possible location of lesions, thereby reducing the incidence of CRC. However, factors such as physician experience and individual differences in intestinal conditions may have a certain impact on the quality of microscopic examination. During the examination, if the patient's intestinal preparation is insufficient and the observation field is blurred, the endoscopist needs to perform intestinal flushing, and the flushing efficiency is related to the physician's operation technique. The operation scores of different physicians for cleaning the intestine during the withdrawal process are related to the detection of adenomas and polyps. Therefore, this time should be removed from the withdrawal time to avoid the extension of the withdrawal time caused by insufficient intestinal preparation, and at the same time remove the confounding factors caused by differences in physician operation. There are two concepts about the optimal withdrawal time: visual withdrawal time (VWT) and overall withdrawal time (OWT). VWT excludes the time for biopsy sampling and treatment, and only retains the time of diagnostic nature, which helps to analyze the withdrawal process more accurately. Therefore, the withdrawal time should accurately exclude the time for surgical treatment and biopsy. In addition, during the microscopic examination, doctors observe for a relatively long time to avoid missed diagnosis. However, staying too long at a location that is clearly free of lesions will not only increase the examination time, but will also have no substantial significance for improving ADR. Therefore, the lens static time should be accurately excluded.

[0037] Currently, there are no tools to accurately analyze the semantics of scenes such as normal observation, biopsy, snare removal, electrosurgical resection, the presence of feces, immersion water suction, hidden water suction, saline flushing, lens close to mucosa, and lens stillness. Therefore, it is impossible to accurately give the withdrawal time and intestinal cleanliness, resulting in the current situation where colonoscopy techniques and quality cannot be standardized.

[0038] The endoscope inspection evaluation system based on the video understanding network provided by the embodiment of the present invention performs scene action recognition on the endoscope inspection video frame image collected by the video acquisition module through the video understanding network model based on the dual cascade spatiotemporal attention of the behavior action recognition module, and obtains scene action recognition data, and then associates and maps the scene to be cleaned and the cleaning action in the scene action recognition data through the cleaning event recognition module to obtain a cleaning event recognition data set, so that the evaluation parameters of the endoscope inspection process can be determined according to the cleaning event recognition data set through the endoscope inspection recording module, and the time used to observe the intestine to find lesions can be truly reflected through the evaluation parameters. The embodiment of the present invention can effectively solve the above problems. The embodiment of the present invention can perform semantic action classification detection on the endoscope inspection operation steps and the corresponding image scene, and form comparable data between each semantic action data and the endoscope inspection prescribed action data, and perform auxiliary prompts and index quantification, thereby promoting the standardization of endoscope inspection and improving the lesion detection rate. The embodiment of the present invention can effectively evaluate the endoscope inspection process and improve the inspection quality.

[0039] The embodiment of the present invention can effectively capture the information of the long and short distance of the video through the video understanding network model, and achieve a good recognition effect. The embodiment of the present invention uses a method for recognizing colonoscopy behavior based on the video understanding network, based on which the operation time of the clinician's treatment is eliminated, thereby improving the quality of the examination.

[0040] In some embodiments, the video understanding network-based endoscopy assessment system may further include an S3 auxiliary prompt module and an S6 client module;

[0041] The auxiliary prompt module is used to provide an operation prompt corresponding to the scene action recognition data according to the scene action recognition data;

[0042] The client module is used to display the evaluation parameters and the operation prompts.

[0043] In detail, the endoscopy evaluation system based on the video understanding network provided by this embodiment includes:

[0044] S1. Video acquisition module, which can read the video stream output by the endoscope device in real time to obtain the video frame image of the endoscope inspection under inspection;

[0045] S2. A behavior action recognition module, using a pre-built artificial intelligence video understanding neural network model based on a dual-cascade spatiotemporal attention module (i.e., a video understanding network model based on dual-cascade spatiotemporal attention), identifies the endoscopic inspection video frame image of the video acquisition module, and obtains a preset scene action recognition result in the endoscopic inspection (i.e., scene action recognition data); the scene actions in the endoscopic inspection include: normal observation, biopsy, snare removal, electrosurgical resection, presence of feces, immersion water absorption, hidden water absorption, saline flushing, lens close to mucosa, and lens stillness;

[0046] S3. Auxiliary prompt module, which can provide auxiliary prompts based on the scene action recognition results;

[0047] S4. The cleaning event recognition module can associate and map the feces scene and the cleaning action (immersion water absorption, hidden water absorption, saline flushing) according to the continuous scene action recognition results to obtain the cleaning event recognition results; and record the video frame segments corresponding to the cleaning event to obtain the cleaning event data set;

[0048] S5. Endoscopic examination record module can evaluate intestinal cleanliness, record intraoperative cleaning efficiency, biopsy, resection and other treatment records, and generate reports based on the scene action recognition results and cleaning event data set of the entire examination.

[0049] S6. The client module can automatically record the time and content of the surgical treatment scene actions on the upper left side; display auxiliary prompt information on the lower left side; and display the video screen in real time on the right side.

[0050] Embodiment 2

[0051] An embodiment of the present invention provides a specific endoscopy evaluation system based on a video understanding network, the system comprising:

[0052] S1 video acquisition module

[0053] This module uses a video capture card to transmit digital / analog endoscopy video frame images such as HDMI, DVI, SDI, S-Video to the host, reads the endoscopy video frame images through opencv, and transcodes them into RGB image format frame by frame.

[0054] S2 Behavior Action Recognition Module

[0055] This module is used to perform scene action recognition on the endoscopic inspection video frame images collected by the video acquisition module according to a pre-trained video understanding network model based on dual cascade spatiotemporal attention to obtain scene action recognition data, and may include a model building unit and a recognition unit.

[0056] The model building unit is used to pre-build an artificial intelligence video understanding network model based on dual-cascade spatiotemporal attention, and can train the pre-built dual-cascade spatiotemporal attention model according to the pre-acquired endoscopic inspection video samples to obtain the video understanding network model based on dual-cascade spatiotemporal attention; including building a network model structure, annotating the pre-collected endoscopic inspection video clips, and training the network model structure to obtain a frequency understanding network model. In detail:

[0057] 1. Construct a network model structure, which is composed of 1 standardization module, 4 sets of double-cascade unit blocks and 1 prediction head module in series;

[0058] Optionally, the standardization module is used to convert the endoscopic inspection video frame image acquired by the video acquisition module into two-way input data by downsampling according to the preset multiple dimensions.

[0059] The input The data of dimension is converted into two inputs by downsampling, and the dimensions of the two inputs are .

[0060] in is the number of video frames. ; is the original width of the video frame; The original height of the video frame; is the number of channels of a single video frame, where the channels are RGB, so ;

[0061] is the number of video frames after dimensionality reduction. ; is the width of the video frame after dimensionality reduction, ; is the height of the video frame after dimensionality reduction, ; is the number of channels of a single video frame, where the channels are RGB, so ;

[0062] The corresponding dimensions of the other path after dimensionality reduction are the number of video frames after dimensionality reduction, the width of the video frame after dimensionality reduction, the height of the video frame after dimensionality reduction, and the number of channels of a single video frame;

[0063] Optionally, the multiple groups of dual-cascade unit blocks are connected in series. For any group of dual-cascade unit blocks, they are used to perform convolution operations on the two-way multi-dimensional input data using two groups of spatially separable convolution blocks to obtain a first feature map and a second feature map respectively; and to downsample the first feature map using a lateral fusion module so that the dimension of the first feature map is aligned and spliced ​​with the second feature map to obtain a third feature map; and to use the first feature map and the third feature map as inputs to a connection module; the connection module is the next group of dual-cascade unit blocks or prediction head module connected in series.

[0064] Specifically, the dual cascade unit block is used to:

[0065] The input data is processed using the first set of spatially separable convolutional blocks. Perform convolution operation to obtain the first feature map ;

[0066] The input data is processed using a second set of spatially separable convolutional blocks. Perform convolution operation to obtain the second feature map ;

[0067] Using the lateral fusion module, The feature map is downsampled to make its dimension equal to Align, and finally concatenate the feature maps to get the third feature map ;

[0068] First feature map With the third characteristic diagram As the input of the first spatially separable convolution block and the second spatially separable convolution block in the next set of double-cascaded unit blocks, and so on, two feature maps are obtained after 4 times.

[0069] Optionally, the prediction head module is used to perform SOFTMAX operations on the two feature maps output by the four groups of dual-cascade unit blocks, respectively, average the obtained scores, output the scene action recognition probability, and obtain the scene action recognition data.

[0070] 2. Label the pre-collected endoscopic video samples, and the labeling results should be preset endoscopic scene actions, which may include normal observation, biopsy, snare removal, electrosurgical resection, scenes to be cleaned (presence of feces), immersion suction, hidden suction, saline flushing, lens close to mucosa, and lens stillness.

[0071] 3. Train the network model structure to obtain the video understanding network model. In detail: First, input the labeled video clips into the network model structure and get the output results ; Secondly, use the cross entropy loss function to calculate With the correct result The error loss is 0.0000, and the network model parameters are updated by back propagation using the AdamW optimizer. Finally, the model is iterated until the loss value is reduced to the preset value (the loss value in this embodiment is 1e -5 ) or when the maximum number of iterations is reached (the maximum number of iterations in this embodiment is 500), the training is stopped to obtain a constructed video understanding network model.

[0072] The recognition unit is used to collect and recognize the video frame images in the S1 video acquisition module, specifically:

[0073] Continuously save continuous frame images within 5 seconds to the memory to obtain the video frame image set to be recognized; input the video frame image set into the video understanding network model to obtain the output result ;

[0074] The prediction function is used to predict the probability of each category. The specific formula is: , take the category with the highest probability, and obtain the scene action recognition data of the video frame segment set. The scene action recognition data includes normal observation, biopsy, snare removal, electrosurgical resection, presence of feces, immersion water absorption, hidden water absorption, saline flushing, lens close to mucosa, and lens stillness.

[0075] S3 auxiliary prompt module

[0076] This module is used to provide operation prompts corresponding to the scene action recognition data according to the scene action recognition data; optionally, when a scene to be cleaned (feces scene) with foreign objects is identified, the user is prompted to flush the intestines; when the lens is close to the mucosa, the user is prompted that there is a risk of intestinal perforation and the lens needs to be moved away; when a static scene appears, the user is prompted to stay still for too long. The real-time prompt results can also be pushed to the client for display using long-connection websocket communication.

[0077] S4 Cleaning Event Recognition Module

[0078] This module is used to associate and map the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a set of cleaning event recognition data. Specifically, it is used to associate and map the scenes to be cleaned (for example, feces scenes) with foreign objects and cleaning actions (immersion water absorption, hidden water absorption, saline flushing) using a dynamic time alignment kernel algorithm based on the scene action recognition data of continuous video frame fragments to obtain cleaning event recognition data, and record the video frame fragments contained in the cleaning event recognition data to obtain a set of cleaning event data.

[0079] Specifically, first, the scene action recognition results of video frame segments of a specified length (500 in this invention, i.e., 20 (seconds) × 25 (fps)) are accumulated to form a sequence A; secondly, the dynamic event alignment kernel algorithm is called to calculate the cumulative similarity with the preset sequence B to obtain the association mapping result of the current sequence A. Among them, the kernel function selects the Gaussian kernel to ensure that the cumulative similarity result is between 0 and 1; finally, it is determined whether the association mapping result of the current sequence A is greater than the preset cleaning event threshold. If so, the current sequence A is recorded as a cleaning event, otherwise, it will not be recorded.

[0080] Optionally, the method for determining the preset sequence B is pre-set, and the present invention is set to a sequence with a length of 500, the first 250 continuous parameter values ​​of the sequence are "scene to be cleaned", and the last 250 continuous values ​​are "cleaning actions"; the cleaning event threshold is set to 0.65.

[0081] The embodiment of the present invention establishes a correspondence between the scene to be cleaned (intestinal feces and water) and the cleaning action, and only evaluates the proportion of uncleaned intestinal segments when evaluating the cleanliness. Compared with the traditional method, the embodiment of the present invention is more flexible and objective.

[0082] S5 Endoscopy Recording Module

[0083] This module is used to identify the data set based on the cleaning event and determine the evaluation parameters of the endoscopic examination process, including intestinal cleanliness, "cleaning time", "uncleaned intestinal segment time", "operation time", "operation action" and "precise endoscope withdrawal time".

[0084] In detail, this module can be used to evaluate the intestinal cleanliness based on the cleaning event recognition data set and cleaning time of the video frame fragments; specifically, first, all fecal scenes are recorded; secondly, the scenes to be cleaned contained in the cleaning event recognition data set are deducted to obtain uncleaned scenes; finally, the intestinal cleanliness is obtained by the number of uncleaned scenes*sampling seconds / total inspection time.

[0085] The module can also be used to identify data based on cleaning events in video frame segments, record intraoperative cleaning efficiency, biopsy, resection and other surgical treatment records, and generate reports, specifically including "cleaning time", "uncleaned intestinal segment time", "operation time", "surgical action", and "precise scope withdrawal time".

[0086] S6 Client Module

[0087] The client can be a web interface, using B / S architecture communication. The time and content of the treatment scene actions are automatically recorded and displayed on the upper left side; the warning information is displayed on the lower left side, and the text is displayed according to the long connection websocket push information of S3; the video frame is displayed in real time using Canvas on the right side of the interface.

[0088] Embodiment 3

[0089] The embodiment of the present invention provides an endoscopic examination evaluation method based on a video understanding network, such as Figure 2 As shown, the endoscopy evaluation method based on the video understanding network includes:

[0090] S101, performing scene action recognition on the collected endoscopic inspection video frame images according to a pre-trained video understanding network model based on dual cascade spatiotemporal attention to obtain scene action recognition data;

[0091] S102, associating and mapping the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a cleaning event recognition data set;

[0092] S103: Determine evaluation parameters of an endoscopy process according to the cleaning event identification data set.

[0093] Optionally, before S101, it may also include: training a pre-constructed dual-cascade spatiotemporal attention model according to pre-acquired endoscopic inspection video samples to obtain the video understanding network model based on dual-cascade spatiotemporal attention; the dual-cascade spatiotemporal attention model is obtained by connecting a standardization module and multiple groups of dual-cascade unit blocks and a prediction head module in series.

[0094] Among them, the standardization module is used to convert the endoscopic inspection video frame images collected by the video acquisition module into two-way multi-dimensional input data by downsampling according to preset multiple dimensions.

[0095] In some embodiments, the preset multiple dimensions include the number of video frames, the original width of the video frame, the original height of the video frame, and the number of channels of a single video frame.

[0096] Optionally, the multiple groups of dual-cascade unit blocks are connected in series. For any group of dual-cascade unit blocks, they are used to perform convolution operations on the two-way multi-dimensional input data using two groups of spatially separable convolution blocks to obtain a first feature map and a second feature map respectively; and to downsample the first feature map using a lateral fusion module so that the dimension of the first feature map is aligned and spliced ​​with the second feature map to obtain a third feature map; and to use the first feature map and the third feature map as inputs to a connection module; the connection module is the next group of dual-cascade unit blocks or prediction head module connected in series.

[0097] The prediction head module is used to perform SOFTMAX operations on the first feature map and the third feature map input by multiple groups of dual-cascade unit blocks, average the obtained scores, output the scene action recognition probability, and obtain the scene action recognition data.

[0098] In some embodiments, the video understanding network-based endoscopic inspection evaluation method may further include: providing operation prompts corresponding to the scene action recognition data according to the scene action recognition data; and displaying the evaluation parameters and the operation prompts.

[0099] The method provided by the embodiment of the present invention can perform semantic action classification detection on endoscopic examination operation steps and corresponding image scenes, and form comparable data between each semantic action data and the prescribed action data of endoscopic examination, and perform auxiliary prompts and indicator quantification, thereby promoting the standardization of endoscopic examination and improving the detection rate of lesions.

[0100] Embodiment 4

[0101] An embodiment of the present invention provides a storage device, the storage device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor;

[0102] When the computer program is executed by the processor, the steps of the endoscopy evaluation method based on the video understanding network as described in any one of the third embodiments are implemented.

[0103] Embodiment 5

[0104] An embodiment of the present invention provides a computer-readable storage medium, on which a data management program for a solid-state hard disk is stored. When the data management program for the solid-state hard disk is executed by a controller, the steps of the endoscopic inspection and evaluation method based on a video understanding network as described in any one of the third embodiments are implemented.

[0105] In the specific implementation process of Embodiments 2 to 5, reference may be made to Embodiment 1, and corresponding technical effects are achieved.

[0106] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. An endoscopy evaluation system based on a video understanding network, characterized in that: The endoscope inspection evaluation system based on the video understanding network includes: a video acquisition module, a behavior action recognition module, a cleaning event recognition module and an endoscope inspection recording module; The behavior action recognition module is used to perform scene action recognition on the endoscopic inspection video frame images collected by the video acquisition module according to the pre-trained video understanding network model based on dual cascade spatiotemporal attention to obtain scene action recognition data; The cleaning event recognition module is used to associate and map the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a cleaning event recognition data set; The endoscopy recording module is used to determine evaluation parameters of the endoscopy process based on the cleaning event identification data set; The dual-cascade spatiotemporal attention model is obtained by connecting a normalization module and multiple groups of dual-cascade unit blocks and a prediction head module in series; the normalization module is used to convert the endoscopic inspection video frame images collected by the video acquisition module into two-way multi-dimensional input data by downsampling according to multiple preset dimensions; the multiple groups of dual-cascade unit blocks are connected in series, and for any group of dual-cascade unit blocks, two groups of spatially separable convolution blocks are used to perform convolution operations on the two-way multi-dimensional input data to obtain a first feature map and a second feature map respectively; and the first feature map is downsampled by a lateral fusion module so that the dimension of the first feature map is aligned and spliced ​​with the second feature map to obtain a third feature map; and the first feature map and the third feature map are used as inputs to a connection module; the connection module is the next group of dual-cascade unit blocks or prediction head modules connected in series.

2. The endoscopy evaluation system based on video understanding network according to claim 1, characterized in that: The behavior action recognition module is also used to train the pre-constructed dual-cascade spatiotemporal attention model according to the pre-acquired endoscopic examination video samples to obtain the video understanding network model based on the dual-cascade spatiotemporal attention.

3. The endoscopy evaluation system based on video understanding network according to claim 1, characterized in that: The preset dimensions include the number of video frames, the original width of the video frames, the original height of the video frames, and the number of channels per video frame.

4. The video understanding network-based endoscopy evaluation system according to claim 1, characterized in that: The prediction head module is used to perform SOFTMAX operations on the first feature map and the third feature map input by multiple groups of dual-cascade unit blocks, average the obtained scores, output the scene action recognition probability, and obtain the scene action recognition data.

5. The endoscopy evaluation system based on video understanding network according to any one of claims 1 to 4, characterized in that: The video understanding network-based endoscopy evaluation system further includes an auxiliary prompt module and a client module; The auxiliary prompt module is used to provide an operation prompt corresponding to the scene action recognition data according to the scene action recognition data; The client module is used to display the evaluation parameters and the operation prompts.

6. An endoscopy evaluation method based on a video understanding network, characterized in that: The endoscopy evaluation method based on the video understanding network includes: According to the pre-trained video understanding network model based on dual cascade spatiotemporal attention, scene action recognition is performed on the collected endoscopic inspection video frame images to obtain scene action recognition data; Associatively mapping the scenes to be cleaned and the cleaning actions in the scene action recognition data to obtain a cleaning event recognition data set; determining evaluation parameters of an endoscopy procedure based on the cleaning event identification data set; The dual-cascade spatiotemporal attention model is obtained by connecting a normalization module and multiple groups of dual-cascade unit blocks and a prediction head module in series; the normalization module is used to convert the endoscopic inspection video frame images collected by the video acquisition module into two-way multi-dimensional input data by downsampling according to multiple preset dimensions; the multiple groups of dual-cascade unit blocks are connected in series, and for any group of dual-cascade unit blocks, two groups of spatially separable convolution blocks are used to perform convolution operations on the two-way multi-dimensional input data to obtain a first feature map and a second feature map respectively; and the first feature map is downsampled by a lateral fusion module so that the dimension of the first feature map is aligned and spliced ​​with the second feature map to obtain a third feature map; and the first feature map and the third feature map are used as inputs to a connection module; the connection module is the next group of dual-cascade unit blocks or prediction head modules connected in series.

7. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor; When the computer program is executed by the processor, the steps of the endoscopy evaluation method based on the video understanding network as described in claim 6 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an endoscopic inspection evaluation program based on a video understanding network. When the endoscopic inspection evaluation program based on a video understanding network is executed by the controller, the steps of the endoscopic inspection evaluation method based on a video understanding network as described in claim 6 are implemented.

Images