Endoscopy-assisted examination system, method, device, and storage medium

Abstract

The present application relates to an endoscopy-assisted examination system, method, device, and storage medium. The system includes an anatomical region identification module that performs frame-by-frame identification of collected endoscopic examination images, identifies each observation point for each frame of the endoscopic examination image, and maps each observation point to a predetermined site. An attention recording module that identifies an observation sequence of each site based on the recorded observation time and observation completeness of each observation point identified by the anatomical region identification module. A guide module that identifies a target site to move to based on a current site corresponding to the current frame of the endoscopic examination image identified by the anatomical region identification module, the observation sequence of the site, and a predetermined physical depth of the current site, and guides the endoscopic examination based on the current site and the target site. The present invention guides an endoscopist to move the endoscope along the most efficient route.

Description

[Technical Field]

[0001] This application claims priority to a Chinese patent application, application number 202210828744.7, filed with the China Patent Office on July 15, 2022, entitled "Endoscopy-assisted examination system, method, device and storage medium," the entire contents of which are incorporated herein by reference. The present application relates to the medical technology field, and in particular to an endoscopy-assisted examination system, method, device, and storage medium. [Background technology]

[0002] Gastroscopy is an important tool for detecting abnormalities in the upper gastrointestinal tract and is the gold standard for diagnosing diseases of the upper gastrointestinal tract, including the esophagus, stomach, and duodenum. Traditional gastroscopy requires determining the location of the stomach at the time of capturing the video image based on the video image captured by the endoscope. A complete gastroscopy report typically includes at least 31 images covering 10 areas: the oropharynx, esophagus, cardia, fundus, body, angle of the stomach, antrum, pylorus, duodenal bulb, and descending limb. If a lesion or suspicious area is found, a more detailed image should be taken. Endoscopists must capture images in real time during the gastroscopy process and promptly conduct further examinations if a suspicious area is found.

[0003] Currently, gastrointestinal endoscopists require a long period of accumulated experience to complete a gastroscopy smoothly. Inexperienced gastrointestinal endoscopists often miss the examination site or fail to detect suspicious areas. Missing the examination site requires the patient to undergo a painful re-examination, which not only wastes the patient's time and money but also wastes the hospital's testing resources. Failing to detect suspicious areas puts the patient's life at risk. Furthermore, gastrointestinal endoscopists are essentially overloaded. Work overload reduces the quality of endoscopists' endoscopic examinations, making them prone to problems such as incomplete coverage of the examination site, incomplete lesion detection, and incomplete image collection.

[0004] Therefore, in order to improve the quality control level of clinical gastroscopy by ensuring that the examination area is completely covered and the lesion is completely examined during the upper gastrointestinal tract endoscopy process, and by ensuring that the endoscopist can perform a complete and detailed observation of the mucosa of each part of the upper gastrointestinal tract using accurate operating techniques, there is an urgent need for upper gastrointestinal tract endoscopic auxiliary examination techniques that can help endoscopists complete upper gastrointestinal tract endoscopic examination smoothly and ensure the accuracy of each upper gastrointestinal tract endoscopic examination operation and the completeness of mucosal observation. Summary of the Invention [Problem to be solved by the invention]

[0005] To solve or at least partially solve the above technical problems, the present application provides an endoscopy-assisted examination system, method, device, and storage medium. [Means for solving the problem]

[0006] In a first aspect, the present application provides an endoscopy-assisted examination system, the endoscopy-assisted examination system including an anatomical region identification module, an attention recording module, and a guide module; The anatomical region identification module is used to perform frame-by-frame identification on the collected endoscopic examination images, identify each observation point region corresponding to each frame of the endoscopic examination image, and map each observation point region to each pre-defined site region; The attention recording module is used to identify an observation sequence of the site sites based on the observation time of each observation point site and the observation completeness of each site site identified by the recorded anatomical site identification module; The guide module is used to identify a target site to transition to based on a current site corresponding to the endoscopic examination image of the current frame identified by the anatomical site identification module, an observation sequence of the site, and a preset physical depth of the current site, and to guide the endoscopic examination based on the current site and the target site.

[0007] In a second aspect, the present application provides an endoscopic assisted examination method, the endoscopic assisted examination method comprising: Identifying the collected endoscopy images frame by frame, identifying each observation point location for the endoscopy image of each frame, and mapping each observation point location to each pre-set site location; Identifying an observation sequence for the site site based on the recorded observation time for each observation point site and the observation completeness for each site site; Identifying a target site to move to based on a current site location corresponding to the endoscopic image of the current frame, an observation sequence of the site location, and a predetermined physical depth of the current site location; and guiding the endoscopy based on the current site location and the target site.

[0008] In a third aspect, the present application provides an endoscopy-assisted examination device, the endoscopy-assisted examination device including a memory, a processor, and a computer program stored in the memory and executable by the processor; When the computer program is executed by the processor, the steps of the endoscopy-assisted examination method are performed.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium having an endoscopic auxiliary examination program stored therein, and when the endoscopic auxiliary examination program is executed by a processor, the steps of the endoscopic auxiliary examination method described above are performed. [Effects of the Invention]

[0010] The above-mentioned solution provided by the embodiment of the present application has the following advantages over the prior art: Each embodiment of the present application identifies the mucosal regions of the upper gastrointestinal tract to be observed with an endoscope, mines the association relationships between each region of the gastrointestinal tract, presents an observation operation route for endoscopic movement, and guides the endoscopist to move the endoscope along the most efficient route, avoiding duplicate examinations, saving time, improving the fluency of gastroscopy, and ensuring the completeness of gastroscopy observation, without oversights or blind spots. [Brief explanation of the drawings]

[0011] The drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification, serve to explain the principles of the invention. In order to more clearly describe the embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for describing the embodiments or the prior art, and it is obvious that those skilled in the art can further obtain other drawings based on these drawings without any creative efforts. [Figure 1] 1 is a block diagram of an endoscopic assisted examination system provided by each embodiment of the present application. [Figure 2] FIG. 1 is a schematic diagram of a convolution layer provided by each embodiment of the present application. [Figure 3] 1 is a flowchart of an endoscopic assisted inspection method provided by each embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0012] It should be understood that the specific examples described herein are merely for purposes of illustrating the invention and are not intended to limit the invention.

[0013] In the following description, the suffixes used to indicate elements such as "module," "component," or "unit" are used only to facilitate the description of the present invention and do not have any specific meaning in themselves. Therefore, "module," "component," or "unit" may be used interchangeably.

[0014] Example 1 An embodiment of the present invention provides an endoscopy-assisted examination system, as shown in FIG. 1 , the endoscopy-assisted examination system includes the following modules: The S10 video acquisition module uses a video acquisition card to transmit digital / analog video signals such as HDMI, DVI, SDI, and S-Video into the host of the endoscopy auxiliary inspection system, reads the video signal through OpenCV, and converts it into RGB image format endoscopy images frame by frame. The S20 anatomical site identification module performs frame-by-frame identification on the collected endoscopic examination images, identifies each observation point site corresponding to each frame of the endoscopic examination image, and maps each observation point site to each pre-set site site; S30 attention recording module identifies an observation sequence of the site sites based on the observation time of each observation point site and the observation completeness of each site site identified by the recorded anatomical site identification module; The S40 guide module identifies a target site to move to based on the current site location corresponding to the endoscopic examination image of the current frame identified by the anatomical site identification module, the observation sequence of the site location, and the preset physical depth of the current site location, and guides the endoscopic examination based on the current site location and the target site.

[0015] Embodiments of the present invention are particularly applicable to gastroscopy, and identify the mucosal regions of the upper gastrointestinal tract to be observed with an endoscope, mine the correlation between each region of the gastrointestinal tract, present an observation operation route by moving the endoscope, and guide the endoscopist to move the endoscope along the most efficient route, avoiding duplicate examinations, saving time, improving the fluency of gastroscopy, and improving the completeness of gastric endoscopic observation, without oversights or blind spots.

[0016] The following describes in detail an endoscopic auxiliary examination system according to an embodiment of the present invention, using a specific embodiment. The endoscopic auxiliary examination system includes: an S10 video acquisition module, an S20 anatomical region identification module, an S30 attention recording module, an S40 guidance module, and an S50 monitoring module; The S10 video acquisition module uses a video acquisition card to transmit digital / analog video signals such as HDMI (registered trademark), DVI, SDI, and S-Video into the host of the endoscopy auxiliary inspection system, reads the video signal through OpenCV, and converts it into RGB image format frame by frame.

[0017] S20 Anatomical Identification Module The model training unit S201 is used to train an artificial intelligence neural network multi-label discrimination model, where the model can be realized by a neural network with classification function such as ResNet, VGG, or DenseNet. In this embodiment, to improve the classification effect, a network structure is constructed by selectively selecting a convolutional layer with a spatial grouping structure, a self-attention network layer, a fully connected layer, and a prediction layer. That is, a network structure is constructed based on a convolutional layer with a spatial grouping structure, a self-attention network layer, a fully connected layer, and a prediction layer, and pre-marked sample images are used as input information for training the network structure to obtain a neural network multi-label discrimination model. For example, the number of groups in the convolutional layer, the self-attention network layer, the fully connected layer, and the prediction layer are 4, 2, 1, and 1, respectively. The training process includes the following steps: In step 2011, a network structure is constructed. The network is mainly obtained by serially connecting a convolution layer including four groups of spatial grouping structures, two groups of self-attention network layers, one group of fully connected layers, and one group of prediction layers; Specifically, as shown in FIG. 2, the convolution layer including the spatial grouping structure has an internal group number (Group) of 64 and a bottleneck width of 32 (bottleneck-width), and is used to perform dimensional transformation and feature extraction on the input image and finally output a feature vector F0. The two groups of self-attention network layers first use the first group of self-attention network layers to perform self-attention weighting on the feature vector F0 to obtain the feature vector Q0, and then use the second group of self-attention network layers to perform joint self-attention weighting on the feature vector F0 and the feature vector Q0 to obtain the feature vector Q0.

number

number

number

number

[0018] In step 2014, after obtaining the loss value L(x), the model parameters are updated using backpropagation. Specifically, the training is preset to 500 rounds, and in each training round, the above steps 2011 and 2012 are repeated for all mark data. The process stops when the model reaches the preset number of rounds or the loss value becomes smaller than 1e-4, and the constructed neural network model is obtained.

[0019] The S202 identification unit is used to perform frame-by-frame identification on the collected endoscopic images using the neural network multi-label identification model trained by the model training unit, and identify each observation point location corresponding to each frame of the endoscopic image. That is, the trained model is used to identify the acquired frame-by-frame images, and finally, a prediction layer predicts and outputs the identification result of the subdivided anatomical location, i.e., the currently observed observation point location T {1,2…,j} Obtained, S203: The anatomical region mapping unit maps each observation point to each pre-set site based on the mapping relationship between the pre-set observation point and the site. In other words, the identification result of the subdivided anatomical region obtained as above is mapped to the corresponding site. The mapping relationship is as shown in the table below. The identification result of the site, that is, the currently observed site T i Here, a corresponding physical depth can be set for each site based on the order of endoscopy, and in this embodiment, the physical depth is 9 levels.

[0020] [Table 1]

[0021] In this embodiment, four sets of convolutional layers perform feature abstraction on the 2D endoscopic image to obtain an output feature vector. The first attention layer performs attention weighting on the interior of the one-dimensional feature vector to obtain a weighted feature Q0. The second attention layer then performs joint weighting on the features F0 and Q0 to obtain a feature Q0.

number

number

[0022] S30 Attention Recording Module The S301 time recording unit is used to record the observation time of the observation point site and the observation completeness of the site site for each frame identified by the anatomical site identification module, and to identify the observation sequence of the site site and the observation sequence of the observation point site based on the observation time of each observation point site and the observation completeness of each site site.

[0023] First, the observation completeness of the site and the observation time of the observation point recorded by the time recording unit are used. Specifically, the number of frames of the obtained identification result of the observation point are recorded, and the accumulated recording amount is converted into time in real time and stored, and the observation sequence M of the observation point is obtained. n =[m1,m2,…,m n ], Next, based on the observation time of the observation point site and the mapping relationship of the site, the completeness of each site site (completeness is obtained based on the following formula) is calculated, and the site site set observation sequence M k =[o1,o2,…,o k ],

number

[0024] S40 Guide Module It is used to guide the path during gastroscopy, and based on the calculation results, it displays the most rational examination route on the display interface, guiding the physician to perform the most efficient endoscope movement operation. It includes the following units: S401: The guide probability prediction unit determines a site transition probability matrix based on the current site, the observation sequence of the site, and the preset physical depth of the current site, and identifies a maximum value in the site transition probability matrix as the target site. Specifically, by using a first target function, the current observation site T in S20 i , the observed sequence of the site at S30 M k Based on the predetermined physical depth D, the site transition probability matrix S is calculated. k =[s1,s2,…,s k ](s kis the probability of transitioning from the current site to site k), and the maximum value in the matrix is calculated to find the new target site T i' or target observation point T n get.

[0025] The first target function is:

number

[0026] If the S402 guide presentation unit determines that the target site is different from the current site location, it draws a site guide line between the current site and the target site on a preset gastrointestinal UI interface; if it determines that the target site is the same as the current site location, it is used to present standard endoscope movement operations for endoscopic examination based on the preset observation direction of the target observation point location to be moved to, identified by the guide probability prediction unit.

[0027] In detail, when a new target site appears, a dynamic image guide is presented in the pre-defined upper gastrointestinal UI interface based on the new target site, and a site guide line is drawn using a Bezier curve within the UI model, allowing the user to easily identify the target site. i'When the user is prompted to advance or retreat the endoscope and continue observing at the current site, a new target observation point and the corresponding preset target observation point site orientation are obtained based on the second target function, and the user is prompted to rotate left, rotate right, advance, retreat, or continue observing, and a dynamic image of the rotation is drawn at the site.

[0028] In detail, T i' ≠ T i If so, the guide presentation unit is used to present a guide using dynamic images on a pre-defined upper gastrointestinal UI interface based on the target site that will be the new destination, and a site guide line is drawn using a Bezier curve within the UI model, and T i From T k A green dynamic flashing connecting line is drawn and displayed up to the target point, and at the same time, the text "Advance endoscope" or "Retract endoscope" is displayed based on the preset physical depth.

[0029] T i' = T i If so, it indicates that continuous observation should be performed at the current site. Using the second target function, the observation point site transition probability matrix S n =[s1,s2,…,s n ] (s n is the probability of transitioning from the current observation point location to the target observation point location n), calculate the maximum value in the matrix, and select the new target observation point location T n Obtained, The second target function is:

number

[0030] Using the guide presentation unit, characters such as left rotation, right rotation, forward movement, backward movement, or continuing observation are presented based on the direction of the pre-set observation point. i Renders dynamic images of rotation actions on the site.

[0031] In this embodiment, two target functions are applied to estimate and derive the AI identification results to obtain the transition area, and at the same time, a display of the current state of operation is provided based on the orientation and depth of the preset area in the real world, which can effectively improve the inspection effect compared to the fixed display in prior art.

[0032] S50 Monitoring Module The S501 endoscope movement operation identification unit is used to identify the current endoscope movement operation. Specifically, it models the observation matrix of S30 to obtain a pre-trained endoscope movement operation identification model, thereby realizing the identification of the doctor's actions such as endoscope rotation, endoscope advancement, endoscope retraction, etc., and obtains the current endoscope movement operation. If it detects that the doctor has performed an incorrect operation, it will be notified by the system, so that doctors at different levels can check according to the standard operation.

[0033] In detail, first, using the order queue matrix, the site identification result and observation point identification result in S20 are obtained once every 0.5 seconds and stored in the order queue matrix for a total of 10 seconds, resulting in a total of 20 pairs of identification results. When the result at the 11th second is stored, the original result at the 1st second is discarded and the result at the 11th second is inserted at the end of the order queue matrix.

[0034] Next, we build an endoscope movement action identification model. First, we collect the contents of the above order cue matrix and manually mark the actions of each order cue matrix. The above actions include seven common actions: "in-situ observation," "lens zoom in," "lens zoom out," "endoscope rotation to the left," "endoscope rotation to the right," "endoscope advance," and "endoscope retreat." This creates a dataset for training the model.

[0035] The model is modeled using a dynamic Bayesian network, and parameters of the dynamic Bayesian network are trained using the EM algorithm with each order cue matrix in the marked dataset as input to obtain a preset endoscope movement motion identification model.

[0036] Finally, during the inspection process, the order queue matrix is continuously collected according to the rules, and the movement of the endoscope is identified by a preset movement identification model of the endoscope.

[0037] The deviation early warning unit S502 is used to present a warning to the digestive tract UI interface when it determines that the current endoscope movement does not match the standard endoscope movement. That is, it displays a text message on the GUI interface based on the current endoscope movement identified as the guide movement presented by the S40 module. When the endoscope movement does not follow the transition site, it displays a clear mark on the GUI interface and provides a dynamic image guide. That is, it obtains the guide movement presented by S40 and determines whether the guide movement presented by S40 matches the endoscope movement performed by the operator. If they do not match, the GUI interface will present an early warning, for example, by displaying red in the upper right corner of the dynamic image area to notify the user.

[0038] The embodiments of the present invention employ an artificial intelligence neural network multi-label classification model to perform multi-label classification on gastroscopy images. During gastroscopy, multiple observation points, such as the anterior wall of the gastric antrum, the lesser curvature of the gastric antrum, the lesser curvature of the gastric fundus, and the posterior wall of the gastric fundus, are usually displayed in the same field of view captured by the lens. The embodiments of the present invention are closer to the actual clinical scene and have more accurate statistics.

[0039] The structure of four convolutional layers + two attention layers + one multi-label prediction layer adopted in each embodiment of the present invention constitutes an S2 neural network model, in which the four convolutional layers perform feature abstraction on the 2D image to obtain feature F0, the first attention layer performs attention weighting on the interior of feature F to obtain feature Q0, and the second attention layer jointly weights features F0 and Q0 to obtain feature Q0.

number

number

[0040] The gastroscopy guidance method disclosed in each embodiment of the present invention can provide the physician with detailed, full-coverage operation guidance, suggesting endoscope advancement or endoscope retraction when a site transition occurs, and suggesting left or right rotation when a transition of the target observation point occurs.

[0041] The gastroscopy physician operating behavior identification method proposed in each embodiment of the present invention uses multi-label observation sequences and machine learning methods to estimate the physician's current endoscope movement behavior, and combines with the S4 module to provide early warning information.

[0042] The display method and interface of the doctor attention level proposed in each embodiment of the present invention can record the multi-label classification results and display the recorded results as a heat map.

[0043] Each embodiment of the present invention presents the doctor with the progression path and observation angle of a gastroscopy examination in real time, corresponding to the doctor's advancement and retreat of the endoscope and rotation of the lens, providing a simple and easy-to-understand presentation and reducing overlap and omission of examination areas.

[0044] Each embodiment of the present invention provides real-time feedback on the consistency between the actual operation path and the preset path, helping the physician to immediately correct any deviations. Compared with feedback that can only provide information on previously covered areas and previously traveled paths, the intelligent tracking prediction effect of this navigation function is more significant.

[0045] In each embodiment of the present invention, the heat map clearly records the entire process of the doctor's observation of the area, and is an objective feedback of the doctor's subjective attention level, which is equivalent to smoothing the flow of gastric endoscopy and is a major innovation in testing feedback means.

[0046] Each embodiment of the present invention uses identification of the current gastric endoscopy site and site-related analysis to display the doctor's examination progress, provide an appropriate examination route, provide the operator with a more reasonable endoscope movement guide, improve the completeness and fluency of the examination, avoid duplicate examinations and omissions, and reduce patient discomfort. The present invention allows endoscopists of different levels to improve their skills to a level that meets basic standards.

[0047] Example 2 An embodiment of the present invention provides an endoscopy-assisted examination method, as shown in FIG. 3 , the endoscopy-assisted examination method includes the following steps: In S101, the collected endoscopic examination images are identified for each frame, and each observation point corresponding to the endoscopic examination image of each frame is identified, and each observation point is mapped to each predetermined site; In S102, an observation sequence of the site is identified based on the recorded observation time of each observation point site and the observation completeness of each site site; In S103, a target site to be moved to is identified based on a current site corresponding to the endoscopy image of the current frame, an observation sequence of the site, and a preset physical depth of the current site; In S104, an endoscopy is guided based on the current site location and the target site.

[0048] The embodiment of the present invention uses identification of the current gastric endoscopy site and analysis related to the site to display the doctor's examination progress, provide an appropriate examination route, provide the operator with a more reasonable guide for endoscope movement, improve the completeness and fluency of the examination, avoid duplicate examinations and omissions, and reduce the patient's discomfort. The present invention allows endoscopists of different levels to improve their level to meet basic standards.

[0049] In some embodiments, S101 includes recording the observation completeness of the site regions and the observation times of the observation point regions for each frame, and identifying an observation sequence of the site regions based on the observation times of each observation point region and the observation times of each site region; S102: Identifying a site transition probability matrix based on the current site, the observation sequence of the site, and a predetermined physical depth of the current site; and identifying a maximum value in the site transition probability matrix as the target site; If it is determined that the target site is different from the current site location, a site guide line between the current site and the target site is drawn on a preset digestive tract UI interface; if it is determined that the target site is the same as the current site location, a standard endoscope movement operation for endoscopic examination is presented based on the preset observation direction of the target observation point location to be moved to, identified by the guide probability prediction unit.

[0050] Optionally, a first target function is used to determine a site transition probability, and a second target function is used to determine the target observation point location according to an observation time of the current site location and a set of observation points of the current site; The first target function is:

number

number

[0051] Optionally, S102 further records the observation completeness of the site regions and the observation time of the observation point regions for each frame identified by the anatomical site identification module, and identifies an observation sequence of the observation point regions based on the observation time of each observation point region and the observation completeness of each site region; Displaying the observation sequence of the observation point site on the gastrointestinal UI interface in a heat map format.

[0052] In some embodiments, the endoscopy-assisted examination method further identifies the current endoscope movement motion, and if it is determined that the current endoscope movement motion does not match the standard endoscope movement motion, presents a notification on the gastrointestinal UI interface.

[0053] Example 3 An embodiment of the present invention provides an endoscopy-assisted examination device, the endoscopy-assisted examination device including a memory, a processor, and a computer program stored in the memory and executable by the processor, When the computer program is executed by the processor, the steps of the endoscopic assisted examination method described in any one of Example 2 are performed.

[0054] Example 4 An embodiment of the present invention provides a computer-readable storage medium, which stores an endoscopic assisted examination program. When the endoscopic assisted examination program is executed by a processor, the steps of the endoscopic assisted examination method described in any one of embodiment 2 are performed.

[0055] The specific implementation of the second to fourth embodiments can refer to the first embodiment, and have corresponding technical effects.

[0056] It should be noted that, in this specification, the terms "comprises," "including," or any other variations thereof, are intended to cover a non-exclusive inclusion, whereby a process, method, item, or apparatus comprising a set of elements not only includes those elements, but also other elements not expressly listed or inherent in such process, method, item, or apparatus. Absent more limitations, an element qualified by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, item, or apparatus that comprises that element.

[0057] The numbers of the above-mentioned embodiments of the present invention are merely for the purpose of explanation and do not represent the superiority or inferiority of the embodiments.

[0058] The numbers of the above embodiments of the present invention are merely for illustrative purposes and do not represent the superiority or inferiority of the embodiments.

[0059] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be realized by adding the necessary general hardware platform to software, and of course, they can also be realized by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solutions of the present invention can be essentially embodied or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM / RAM, magnetic disk, optical disk) and contains multiple commands to cause a terminal (which may be a mobile phone, computer, server, air conditioner, network device, etc.) to execute the methods described in each embodiment of the present invention.

[0060] Although the above describes the embodiments of the present invention with reference to the drawings, the present invention is not limited to the above specific embodiments, which are merely illustrative and not limiting. Those skilled in the art may implement many more forms under the guidance of the present invention without departing from the spirit of the present invention and the scope of protection of the claims, and all of these fall within the scope of protection of the present invention.

Claims

1. An endoscopy-assisted examination system, comprising: The endoscopy-assisted examination system includes an anatomical region identification module, an attention recording module, and a guide module; The anatomical region identification module is used to perform frame-by-frame identification on the collected endoscopic examination images, identify each observation point region corresponding to each frame of the endoscopic examination image, and map each observation point region to each pre-defined site region; The attention recording module is used to identify an observation sequence of the site sites based on the observation time of each observation point site and the observation completeness of each site site identified by the recorded anatomical site identification module; The endoscopic assisted examination system is characterized in that the guide module identifies a target site to transition to based on a current site corresponding to the endoscopic examination image of the current frame identified by the anatomical site identification module, an observation sequence of the site, and a predetermined physical depth of the current site, and is used to guide endoscopic examination based on the current site and the target site.

2. The attention recording module includes a time recording unit, and the guidance module includes a guidance probability prediction unit and a guidance presentation unit; The time recording unit is used to record the observation completeness of the site and the observation time of the observation point site for each frame identified by the anatomical site identification module, and identify the observation sequence of the site based on the observation time of each observation point site and the observation completeness of each site site; The guide probability prediction unit is used to identify a site transition probability matrix based on the current site, the observation sequence of the site, and the predetermined physical depth of the current site, and determine a maximum value in the site transition probability matrix as the target site; The endoscopic auxiliary examination system of claim 1, characterized in that, when the guide presentation unit determines that the target site is different from the current site location, it draws a site guide line between the current site and the target site on a preset digestive tract UI interface, and when it determines that the target site is the same as the current site location, it is used to present standard endoscopic movement operations for endoscopic examination based on the preset observation direction of the target observation point location that is the destination of movement identified by the guide probability prediction unit.

3. The guide probability prediction unit is further used to determine a site location transition probability according to a first target function, and to determine the target observation point location according to a second target function based on the observation time of the current site location and the observation location set of the current site location; The first target function is: [Equation 1] （１） However, s i' is the target site i ' indicates the probability of transitioning to M k (i ' ) is the target site i ' indicates the completeness of the observations, and D i' is the target site i ' indicates the physical depth of i denotes the physical depth of site i, λ is an arbitrary constant, The second target function is: [Equation 2] （２） However, s n indicates the probability of transitioning to the target observation point site n, and M n The endoscopic auxiliary inspection system of claim 2, characterized in that (n) indicates the observation time of target observation point site n, b(n) is the optimal observation time of target observation point site n, and O(i) indicates the target observation point site set corresponding to site site i.

4. The attention recording module further includes a heat map display unit; The time recording unit is used to record the observation completeness of the site portion and the observation time of the observation point portion for each frame identified by the anatomical site identification module, and identify the observation sequence of the observation point portion based on the observation time of each observation point portion and the observation completeness of each site portion; The endoscopic auxiliary examination system according to claim 2 , wherein the heat map display unit is used to display the observation sequence of the observation point region on the digestive tract UI interface in a heat map format.

5. The endoscope-assisted examination system further includes a monitoring module, the monitoring module including an endoscope movement operation identification unit and a deviation early warning unit; The endoscope movement operation identification unit is used to identify the current endoscope movement operation; The endoscopic auxiliary examination system of claim 2, wherein the deviation early warning unit is used to present on the gastrointestinal UI interface when it determines that the current endoscope movement behavior does not match the standard endoscope movement behavior.

6. the anatomical region identification module includes a model training unit, an identification unit, and an anatomical region mapping unit; The model training unit is used to construct a network structure based on a convolution layer of a spatial grouping structure, a self-attention network layer, a fully connected layer and a prediction layer, and to train the network structure using pre-marked sample images as input information to obtain a neural network multi-label discrimination model; The identification unit is used to perform frame-by-frame classification on the collected endoscopic examination images using the neural network multi-label identification model trained by the model training unit, and identify each observation point location corresponding to each frame of the endoscopic examination image; 5. The endoscopic auxiliary examination system according to claim 1, wherein the anatomical region mapping unit is used to map each observation point region to each predetermined site region based on a mapping relationship between the predetermined observation point regions and the predetermined site regions.

7. The model training unit specifically uses the spatial grouping structure convolution layer to perform dimension transformation and feature extraction on the input sample image, and obtains a feature vector F 0 is used to output The first group of self-attention network layers generates a feature vector F 0 Apply self-attention weighting to the feature vector Q 0 Then, the second group of self-attention network layers obtains the feature vector F 0 and the feature vector Q 0 We apply joint self-attention weighting to the feature vector [Equation 3] Obtained, The fully connected layer generates a feature vector [Equation 4] and feature vector F 0 Perform linear projection on to obtain a one-dimensional feature vector F, Based on the one-dimensional feature vector F, the prediction layer calculates a derivative for each preset category to obtain a multi-label classification probability; The endoscopic auxiliary inspection system of claim 6, wherein a loss function is used for calculation based on the pre-marked multi-label results to obtain a loss value, and then backpropagation is used to update the model parameters, and the process stops when a pre-set number of rounds is reached or the loss value is smaller than a pre-set value, thereby obtaining a neural network multi-label discrimination model.

8. An endoscopy-assisted examination method, comprising: The endoscopic assisted examination method includes: Identifying the collected endoscopy images frame by frame, identifying each observation point location for the endoscopy image of each frame, and mapping each observation point location to each pre-set site location; Identifying an observation sequence for the site site based on the recorded observation time for each observation point site and the observation completeness for each site site; Identifying a target site to move to based on a current site location corresponding to the endoscopic image of the current frame, an observation sequence of the site location, and a predetermined physical depth of the current site location; and guiding an endoscopy based on the current site location and the target site.

9. An endoscopic auxiliary examination device, The endoscopy-assisted examination device includes a memory, a processor, and a computer program stored in the memory and executable by the processor; An endoscope-assisted inspection device, characterized in that, when the computer program is executed by the processor, the steps of the endoscope-assisted inspection method according to claim 8 are performed.

10. 1. A computer-readable storage medium, comprising: A computer-readable storage medium, characterized in that an endoscope-assisted examination program is stored in the computer-readable storage medium, and when the endoscope-assisted examination program is executed by a processor, the steps of the endoscope-assisted examination method described in claim 8 are performed.

Images