Gastrointestinal endoscope diagnosis method based on AR glasses, controller, system and medium

Abstract

The invention relates to the technical field of medical treatment, and discloses a gastrointestinal endoscope diagnosis method based on AR glasses. The method comprises the steps that an initial detection image which is shot by a gastrointestinal endoscope and corresponds to a target object is obtained and sent to the AR glasses to be displayed; according to a preset planning path, utilizing a navigation algorithm to generate guidance indication information, and displaying the guidance indication information in the initial detection image displayed by the AR glasses so as to control the gastrointestinal endoscope to move according to a display picture of the AR glasses; performing focus recognition on all the real-time detection images through a preset focus recognition model to obtain a focus recognition result; according to the lesion identification result, determining a lesion marking area and a suggested biopsy position, and determining the lesion marking area as a lesion image; the focus image is sent to AR glasses for display, and a gastrointestinal endoscope diagnosis report is generated according to the focus image. According to the focus characteristics of the white light endoscopic image and the narrow-band wide image, determination of the diseased region, the marked focus region and the suggested biopsy region is achieved, and deep combination of gastrointestinal endoscope diagnosis, the AR technology and artificial intelligence is achieved.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular, to a gastroscopy and colonoscopy diagnosis method, a controller, a system and a medium based on an AR glasses. Background Art

[0002] In modern medical surgeries, especially in complex surgical operations, precise navigation is crucial for improving the success rate of surgeries and patient safety. With the progress of technology, augmented reality (AR) technology has been gradually introduced into neurosurgery or cardiac surgeries to display a detailed three-dimensional view of the surgical area and the position of surgical tools in real time. However, during the gastroscopy and colonoscopy detection process, it still relies on doctors' observation and judgment, and it is not easy to observe minor lesions, resulting in doctors easily overlooking minor lesions. Moreover, existing gastroscope devices lack a real-time visualization navigation function, and doctors need to repeatedly adjust the angle of the endoscope body to find the lesion. Therefore, there is an urgent need for a technology to solve the above problems. Summary of the Invention

[0003] Embodiments of the present invention provide a gastroscopy and colonoscopy diagnosis method, a controller, a system and a medium based on an AR glasses to solve the problems in the prior art that during the gastroscopy and colonoscopy detection process, it relies on manual observation, minor lesions are easily overlooked, and existing gastroscope devices lack a real-time visualization navigation function.

[0004] A gastroscopy and colonoscopy diagnosis method based on an AR glasses includes: Obtaining an initial detection image corresponding to a target object captured by a gastroscope and sending it to the AR glasses for display; the initial detection image includes a narrow-band detection image captured by the narrow-band imaging technology in the gastroscope, or / and a conventional detection image captured by the white light in the gastroscope in real time; According to a preset planned path, generating guiding indication information by using a navigation algorithm and displaying the guiding indication information in the initial detection image displayed by the AR glasses, so as to control the movement of the gastroscope and capture a real-time detection image according to the display screen of the AR glasses; the preset planned path is generated by performing path planning with a three-dimensional gastro model; the real-time detection image includes a narrow-band detection image captured by the narrow-band imaging technology in the gastroscope, or / and a conventional detection image captured by the white light in the gastroscope in real time; Performing lesion recognition on all the real-time detection images through a preset lesion recognition model to obtain a lesion recognition result; According to all the lesion recognition results, determining a lesion marking area corresponding to the target object and a recommended biopsy position corresponding to the lesion marking area, and determining the real-time detection image marked with the lesion marking area and the recommended biopsy position as a lesion image; Send the lesion image to the AR glasses for display, and generate a gastroscope diagnosis report corresponding to the target object according to the lesion image.

[0005] A controller includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The controller is used to execute the above-mentioned gastroscope diagnosis method based on AR glasses.

[0006] A gastroscope diagnosis system based on AR glasses includes a gastroscope with a multimodal sensor, AR glasses, and a controller as described above; the controller is communicatively connected to the gastroscope and the AR glasses; The gastroscope is used to capture narrowband detection images using narrowband imaging technology, or conventional detection images captured in real time using white light. The AR glasses are used to receive and display the narrowband detection images or the conventional detection images captured in real time by the gastroscope, and are used to prompt guiding indication information of the gastroscope.

[0007] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned gastroscope diagnosis method based on AR glasses.

[0008] For the above-mentioned gastroscope diagnosis method, controller, system, and medium based on AR glasses, the gastroscope diagnosis method based on AR glasses of the present invention realizes the deep combination of gastroscope diagnosis, AR technology, and artificial intelligence through a gastroscope, AR glasses, a preset lesion recognition model, and a navigation algorithm. By using the preset lesion recognition model to identify lesions in all real-time detection images, the identification of lesions of the target object is realized, and the acquisition of lesion recognition results is realized, so as to realize the identification of micro-lesions, reduce the dependence on artificial experience, and then realize the marking of the lesion area, and display the lesion marked area and the recommended biopsy position in the AR glasses. Through the lesion image, the generation of the gastroscope diagnosis report is realized, thereby reducing the examination time and improving the examination efficiency. Description of the Drawings

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the description of the embodiments of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0010] Figure 1 It is a flowchart of the gastroscope diagnosis method based on AR glasses in an embodiment of the present invention; Figure 2It is a flowchart of step S10 of the gastroscopy diagnosis method based on an AR glasses in an embodiment of the present invention; Figure 3 It is a flowchart of step S10 of the gastroscopy diagnosis method based on an AR glasses in another embodiment of the present invention; Figure 4 It is a flowchart of step S30 of the gastroscopy diagnosis method based on an AR glasses in an embodiment of the present invention; Figure 5 It is a flowchart of step S40 of the gastroscopy diagnosis method based on an AR glasses in an embodiment of the present invention; Figure 6 It is a flowchart of the gastroscopy diagnosis method based on an AR glasses in another embodiment of the present invention; Figure 7 It is a flowchart of the gastroscopy diagnosis method based on an AR glasses in yet another embodiment of the present invention. Detailed implementation manners

[0011] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts belong to the scope of protection of the present invention.

[0012] In one embodiment, as Figure 1 shown, a gastroscopy diagnosis method based on an AR glasses is provided, including the following steps: S10: Obtain an initial detection image corresponding to a target object captured by a gastroscope and send it to the AR glasses for display; the initial detection image includes a narrow-band detection image captured by the narrow-band imaging technology in the gastroscope, and / or a conventional detection image captured by the white light in the gastroscope in real time.

[0013] In this embodiment, the detection image of the target object is mainly captured by the gastroscope in real time and transmitted to the AR glasses for display.

[0014] It can be understood that the initial detection image includes a narrow-band detection image captured by the narrow-band imaging technology in the gastroscope, and / or a conventional detection image captured by the white light in the gastroscope in real time. An AR glasses refers to a device that uses augmented reality technology to fuse virtual information generated by a computer with the real-world environment. A gastroscope refers to a medical device used for examining gastrointestinal diseases. A narrow-band detection image refers to an image captured when the gastroscope uses narrow-band imaging technology to detect the stomach or intestine of a target object. A conventional detection image refers to an image captured when the gastroscope uses white light to detect the stomach or intestine of a target object. A target object refers to a patient who needs to be diagnosed.

[0015] Specifically, connect the AR glasses to the gastroscope in a communication manner, and connect the AR glasses and the gastroscope to a controller (the controller can be a cloud server, which is not limited here) in a communication manner. Then, detect the target object through the gastroscope, that is, use the multi-modal sensors in the gastroscope to take real-time pictures of the stomach or intestine of the target object, so as to obtain the initial detection image. Among them, the narrow-band imaging technology can be used to take real-time pictures of the stomach or intestine of the target object, so as to obtain the narrow-band detection image. The white light can also be used to take real-time pictures of the stomach or intestine of the target object, so as to obtain the conventional detection image. Switch the shooting mode of the gastroscope through the switching instruction. Furthermore, transmit the initially detected image taken in real time to the AR glasses. After receiving the initially detected image taken by the gastroscope in the AR glasses, the AR glasses display the initially detected image on the virtual screen. When the initially detected image is a narrow-band detection image, the narrow-band detection image is synchronously displayed in the AR glasses; when the initially detected image is a conventional detection image, the conventional detection image is synchronously displayed in the AR glasses. Among them, when it is necessary to retain the current initially detected image on the virtual screen, the current initially detected image can be retained in the virtual screen of the AR glasses through the retention instruction.

[0016] S20: According to the preset planned path, use the navigation algorithm to generate the guiding indication information, and display the guiding indication information in the initially detected image displayed by the AR glasses, so as to control the movement of the gastroscope and take the real-time detection image according to the display screen of the AR glasses; the preset planned path is generated by path planning with the three-dimensional gastrointestinal model; the real-time detection image includes the narrow-band detection image taken by the narrow-band imaging technology in the gastroscope, or / and the conventional detection image taken in real time by the white light in the gastroscope.

[0017] This embodiment is mainly used to generate the guiding indication information of the gastroscope and display the guiding indication information on the AR glasses.

[0018] It can be understood that the preset planned path refers to the moving path of the gastroscope when detecting the target object set in advance. The navigation algorithm refers to the path optimization algorithm set in advance. The guiding indication information refers to the direction to prompt the movement of the gastroscope, and can also include information such as the force and speed of the movement of the gastroscope. The real-time detection image includes the narrow-band detection image taken by the narrow-band imaging technology in the gastroscope, or / and the conventional detection image taken in real time by the white light in the gastroscope. The real-time detection image refers to the image taken by the gastroscope after adjusting the path according to the guiding indication information. The preset planned path is generated by path planning with the three-dimensional gastrointestinal model. The three-dimensional gastrointestinal model is a virtual model of the stomach and intestine constructed based on all images. The display screen refers to the initially detected image including the guiding indication information.

[0019] Specifically, obtain a preset planned path, that is, based on the basic information of the target object, select a preset planned path suitable for the target object. Then, according to the preset planned path, use a navigation algorithm to generate guiding indication information, that is, the navigation algorithm optimizes and adjusts the preset planned path based on the initial detection image, thereby generating guiding indication information, and sending the guiding indication information to the AR glasses, so that the guiding indication information is displayed in the initial detection image displayed by the AR glasses, and enabling the operator to control the movement of the gastroscope and capture real-time detection images according to the display screen of the AR glasses, that is, enabling the operator to control the movement direction and speed of the gastroscope according to the prompt information in the AR glasses. At the same time, the real-time detection images of the stomach or intestine of the target object are captured in real time. Among them, narrow-band imaging technology can be used to capture the stomach or intestine of the target object in real time, thereby obtaining narrow-band detection images. White light can also be used to capture the stomach or intestine of the target object in real time, thereby obtaining conventional detection images. The shooting mode of the gastroscope is switched through a switching instruction.

[0020] S30: Identify lesions in all the real-time detection images through a preset lesion identification model to obtain a lesion identification result.

[0021] In this embodiment, it is mainly used to identify whether the real-time detection image includes lesions through a preset lesion identification model, and the type of lesions when there are lesions.

[0022] Understandably, the preset lesion identification model refers to a model for lesion identification trained based on federated learning. The lesion identification result is used to characterize whether the narrow-band detection image or the conventional detection image contains a lesion area, and the type of lesion.

[0023] Specifically, obtain a preset lesion identification model, input the real-time detection images captured in real time into the preset lesion identification model, and identify lesions in all the real-time detection images through the preset lesion identification model, that is, first perform edge processing on the real-time detection images through an edge processing layer, then perform convolutional feature extraction on the real-time detection images through a convolutional layer, and reduce the dimension of the convolutional features through a pooling layer. Then, perform lesion identification on the pooled features through a fully connected layer, that is, perform lesion identification through the lesion identification ability learned during training, thereby obtaining a lesion identification result. For example, early gastric cancer (EGC) under endoscopy refers to cancer tissue limited to the gastric mucosa or submucosa, regardless of whether there is lymph node metastasis. Its endoscopic features vary depending on morphology, color, surface structure, etc. For example, typical endoscopic features: color change, redness or fading; uneven hue. Abnormal mucosal structure: rough or granular surface; disappearance or disorder of glandular openings. Edge features: unclear boundary or spiculated edge; interruption or fusion of surrounding mucosal folds. Abnormal blood vessels: irregular microvascular morphology.

[0024] In one embodiment, when the preset planned path is not adjusted by the navigation algorithm, the initial detection images are input into the preset lesion recognition model, and the preset lesion recognition model performs lesion recognition on all the initial detection images, so as to obtain the lesion recognition results corresponding to the initial detection images. When the preset planned path is adjusted by the navigation algorithm, the real-time detection images are input into the preset lesion recognition model, and the preset lesion recognition model performs lesion recognition on all the real-time detection images, so as to obtain the lesion recognition results corresponding to the real-time detection images.

[0025] S40: According to all the lesion recognition results, determine the lesion marking area corresponding to the target object and the recommended biopsy position corresponding to the lesion marking area, and determine the real-time detection image marked with the lesion marking area and the recommended biopsy position as the lesion image.

[0026] This embodiment is mainly used for marking the lesion area and recommending the biopsy position.

[0027] Understandably, the lesion marking area refers to the lesion area in the image recognized by the model. The recommended biopsy position refers to the position for biopsy sampling in the recommended lesion marking area.

[0028] Specifically, according to all the lesion recognition results, determine the lesion marking area corresponding to the target object, that is, analyze the lesion recognition results, determine the meaning represented by the lesion recognition results, and when the lesion recognition results indicate that the real-time detection image contains a lesion area, determine the lesion area recognized by the preset lesion recognition model as the lesion marking area. Then, obtain the historical biopsy positions corresponding to the lesion type of the lesion marking area, and then mark the biopsy positions on the real-time detection image of the lesion marking area through a large number of historical biopsy positions, so as to obtain the recommended biopsy position. And determine the real-time detection image marked with the lesion marking area and the recommended biopsy position as the lesion image.

[0029] S50: Send the lesion image to the AR glasses for display, and generate a gastroscopy diagnosis report corresponding to the target object according to the lesion image.

[0030] This embodiment is mainly used for displaying the lesion marking area and the recommended biopsy position in the AR glasses, and outputting a diagnosis report.

[0031] Understandably, the gastroscopy diagnosis report refers to the diagnosis report for the stomach and intestine of the target object.

[0032] Specifically, the lesion image is sent to the AR glasses, so that the AR glasses display the lesion marked area and the recommended biopsy location in the lesion image on the virtual screen. Among them, each lesion marked area and the recommended biopsy location can be displayed in sequence according to the time stamp order of the real-time detection images, or the lesion marked area and the recommended biopsy location viewed by the operator can be displayed through voice commands. Or, the eye movement direction is tracked by the eye tracking component on the AR glasses to display each lesion marked area and the recommended biopsy location in sequence according to the time stamp order of the real-time detection images. When the operator gazes at a certain real-time detection image, the lesion marked area and the recommended biopsy location in the real-time detection image are enlarged. Then, a gastroscopy diagnosis report corresponding to the target object is generated based on the lesion image, that is, first, the basic information is obtained to perform the first data filling on the preset report template, that is, the information such as the name, gender, and age of the target object is filled, and then the preset report template is filled with the second data through the lesion marked area in the lesion image, so as to obtain the gastroscopy diagnosis report corresponding to the target object.

[0033] In the gastroscopy diagnosis method based on AR glasses of the present invention, through the gastroscope, AR glasses, preset lesion recognition model and navigation algorithm, the deep combination of gastroscopy diagnosis, AR technology and artificial intelligence is realized. Through the preset lesion recognition model, the lesions in all real-time detection images are recognized, the lesions of the target object are recognized, and the recognition results of the lesions are obtained, so as to realize the recognition of micro-lesions, reduce the dependence on manual experience, and then realize the marking of the lesion area, and display the lesion marked area and the recommended biopsy location in the AR glasses. Through the lesion image, the generation of the gastroscopy diagnosis report is realized, thereby reducing the examination time and improving the examination efficiency.

[0034] In one embodiment, as Figure 2 shown, in step S10, the AR glasses receive and display the narrow-band detection image or the conventional detection image corresponding to the target object taken in real time by the gastroscope using narrow-band imaging technology or white light, including: S101, after the gastroscope uses white light to collect images of the target object to obtain a conventional detection image, the AR glasses receive all the conventional detection images transmitted in real time by the gastroscope and display the conventional detection images on the virtual screen.

[0035] S102, receive a blood vessel mode switching instruction. After the gastroscope uses narrow-band imaging technology to collect images of the target object to obtain a narrow-band detection image, the AR glasses receive all the narrow-band detection images transmitted in real time by the gastroscope and display the narrow-band detection images on the virtual screen.

[0036] Understandably, the blood vessel mode switching instruction refers to the instruction to switch the imaging mode to the blood vessel mode, that is, to use narrow-band imaging technology to photograph the stomach or intestine of the target object. For example, it can be triggered by a voice switching instruction, a button on the AR glasses, or a virtual button on the virtual screen.

[0037] Specifically, after connecting all devices through communication, the gastroscope is used to detect the target object according to the preset planned path. That is, the target object is detected through the multi-modal sensor in the gastroscope, and the stomach or intestine of the target object is photographed through the gastroscope host in the multi-modal sensor, that is, white light is used to collect images of the target object, so as to obtain a conventional detection image. And the collected conventional detection images are transmitted to the AR glasses and the controller in real time. After receiving all the conventional detection images transmitted by the gastroscope in real time, the AR glasses device displays the conventional detection images on the virtual screen. After receiving the conventional detection images, the controller performs lesion recognition on the conventional detection images through a preset lesion recognition model, so as to obtain a lesion recognition result.

[0038] Further, after receiving the blood vessel mode switching instruction, the imaging mode of the gastroscope is switched, that is, the gastroscope host in the multi-modal sensor uses narrow-band imaging technology to photograph the stomach or intestine of the target object, so as to obtain a narrow-band detection image. And the collected narrow-band detection images are transmitted to the AR glasses and the controller in real time. After receiving all the narrow-band detection images transmitted by the gastroscope in real time, the AR glasses device displays the narrow-band detection images on the virtual screen. Similarly, after receiving the narrow-band detection images, the controller performs lesion recognition on the narrow-band detection images through a preset lesion recognition model, so as to obtain a lesion recognition result.

[0039] In another embodiment, after receiving the normal mode switching instruction, the imaging mode of the gastroscope is switched again, that is, the gastroscope host uses white light to collect images of the target object, so as to obtain a conventional detection image. And the collected conventional detection images are transmitted to the AR glasses and the controller in real time. After receiving all the conventional detection images transmitted by the gastroscope in real time, the AR glasses device displays the conventional detection images on the virtual screen. After receiving the conventional detection images, the controller performs lesion recognition on the conventional detection images through a preset lesion recognition model, so as to obtain a lesion recognition result. Among them, during the movement of the gastroscope, the imaging mode can be converted, and the same area can be photographed by two shooting methods respectively, which is convenient for subsequent comprehensive analysis of the area. It can be understood that the shooting position after switching the mode according to the switching instruction each time can be changed or not, which is determined according to the actual situation.

[0040] In this embodiment, through white light imaging, the acquisition of conventional detection images is realized, thereby realizing the switching of the normal mode, and further realizing the display of conventional detection images on the AR glasses. Through narrowband imaging technology, the acquisition of narrowband detection images is realized, thereby realizing the switching of the blood vessel mode, and further realizing the display of narrowband detection images on the AR glasses.

[0041] In one embodiment, as Figure 3 shown, in step S10, according to the preset planned path, using a navigation algorithm to generate guiding indication information, and displaying the guiding indication information in the initial detection image displayed on the AR glasses, so as to control the movement of the gastroscope and capture real-time detection images according to the display screen of the AR glasses, including: S103, obtaining the historical data, physiological parameters of the target object, and the real-time images fed back by the gastroscope; the real-time images include narrowband detection images and conventional detection images.

[0042] S104, making the navigation algorithm optimize the preset planned path according to the historical data, the physiological parameters and the real-time images to obtain the guiding indication information of the gastroscope, and sending the guiding indication information to the AR glasses, so that the AR glasses use a virtual guiding line to display the guiding indication information of the gastroscope, so as to control the movement of the gastroscope and capture real-time detection images according to the display screen of the AR glasses.

[0043] In this embodiment, mainly through the real-time images fed back by the gastroscope in real time, as well as the historical data and physiological parameters of the target object, the movement path of the gastroscope is optimized and adjusted, and a virtual guiding line is used in the AR glasses to prompt the guiding indication information.

[0044] Understandably, physiological parameters refer to various quantitative indicators used to describe the physiological state of the target object. For example, respiratory rate, heart rate, blood pressure, etc. Historical data refers to the diagnostic information of the target object before the current time. Among them, when there is no historical data, only physiological parameters are obtained. Real-time images refer to the images fed back by the gastroscope in real time, which can be narrowband detection images, or conventional detection images, or narrowband detection images and conventional detection images, that is, the same area can be photographed in two ways respectively to obtain the narrowband detection image and the conventional detection image of this area.

[0045] Specifically, after obtaining the preset planned path, a detection operation is performed on the gastroscope according to the preset planned path. Then, historical data and physiological parameters of the target object, as well as real-time images feedback by the gastroscope, are obtained. The navigation algorithm optimizes the preset planned path based on the historical data, physiological parameters, and real-time images, that is, the navigation algorithm adjusts the preset planned path according to the historical data and physiological parameters of the feedback target object and the real-time images feedback by the gastroscope, so as to obtain the guiding indication information of the gastroscope, and send the guiding indication information to the AR glasses. After receiving the guiding indication information, the AR glasses display the guiding indication information of the gastroscope with a virtual guiding line in the AR glasses to prompt the operator, and an operation prompt pop-up window is added, so that the operator controls the movement of the gastroscope and takes real-time detection images according to the display screen of the AR glasses. For example, when the gastroscope moves along the preset planned path, when the navigation algorithm detects from the feedback real-time images that continuing to move forward will touch the stomach of the target object, the navigation algorithm adjusts the preset planned path to obtain the guiding indication information.

[0046] In another embodiment, the gastroscope simultaneously feedbacks the operation data of the operator (such as the angle of the endoscope body and the advancing speed). When the navigation algorithm adjusts the preset planned path, the guiding indication information also includes information such as the operation force of the operator, the specific angle of the endoscope body, and the advancing speed.

[0047] In this embodiment, through the historical data, physiological parameters, and the real-time images, the adjustment of the preset planned path is realized, thereby the acquisition of the target path is realized, and furthermore, the damage to the target object during the detection process is avoided, and the deep combination of gastroscope diagnosis, artificial intelligence, and AR technology is realized.

[0048] In one embodiment, as Figure 4 shown, in step S30, by using the preset lesion recognition model to perform lesion recognition on all the narrow-band detection images and all the conventional detection images to obtain a lesion recognition result, the preset lesion recognition model includes an edge processing layer, a convolutional layer, a pooling layer, and a fully connected layer; it includes: S301, perform edge processing on all the real-time detection images through the edge processing layer to obtain a target image.

[0049] S302, perform convolutional processing on the target image through the convolutional layer to obtain convolutional features.

[0050] S303, perform pooling processing on the convolutional features through the pooling layer to obtain pooling features.

[0051] S304, perform lesion recognition on all the pooling features through the fully connected layer to obtain a lesion recognition result.

[0052] In this embodiment, a preset lesion recognition model obtained through training is mainly used to recognize lesions in real-time detection images.

[0053] It can be understood that the target image refers to the image obtained after edge processing of the real-time detection image. The convolutional feature refers to the feature extracted through convolutional operations in a convolutional neural network. The pooling feature refers to the feature representation obtained by downsampling the convolutional feature through pooling operations.

[0054] Specifically, after synchronously obtaining the real-time detection image, a preset lesion recognition model is acquired, and the real-time detection image is input into the preset lesion recognition model. The preset lesion recognition model performs image segmentation on the real-time detection image, that is, first performs edge recognition on the real-time detection image through an edge recognition algorithm, and performs cutting processing on the recognized edges to obtain the target image. Then, the target image is subjected to convolutional processing through the convolutional layer in the preset lesion recognition model, that is, the convolutional kernels of the convolutional layer perform moving convolutions on the target image respectively to obtain the convolutional features corresponding to the target image. Next, all the convolutional features are subjected to pooling processing through the pooling layer in the preset lesion recognition model, that is, downsampling processing is performed on all the convolutional features through the pooling kernel to reduce the dimension, so as to obtain the pooling features corresponding to the target image. Finally, the preset lesion recognition model performs lesion recognition on all the pooling features through the fully connected layer, that is, prediction processing is performed on the pooling features through the positive and negative sample recognition capabilities learned during training, so as to obtain the lesion recognition result corresponding to the real-time detection image. Among them, the preset lesion recognition model may further include multiple convolutional layers and pooling layers to extract more detailed features. In this embodiment, the number of convolutional layers and pooling layers of the preset lesion recognition model is not limited.

[0055] In one embodiment, the preset lesion recognition model includes two recognition modules. The structures of the two recognition modules are the same, but the parameters are different. One narrowband recognition module is used to recognize lesions in narrowband detection images, and one conventional recognition module is used to recognize lesions in conventional detection images. When the image input into the preset lesion recognition model is a narrowband detection image, the narrowband recognition module is used to recognize whether there are lesions in the narrowband detection image and the type corresponding to the lesions. When the image input into the preset lesion recognition model is a conventional detection image, the conventional recognition module is used to recognize whether there are lesions in the conventional detection image and the type corresponding to the lesions.

[0056] In another embodiment, all historical lesion images are matched with the real-time detection images in terms of similarity, so as to obtain the similarity values corresponding to the real-time detection images. When the similarity value exceeds the preset similarity threshold, it is determined that the real-time detection image matches the historical lesion image, and based on the abnormal label corresponding to the historical lesion image (used to characterize the disease of the historical lesion image), the lesion recognition result corresponding to the real-time detection image is obtained.

[0057] In this embodiment, by performing edge processing on the real-time detection image, the cutting of the image edge and the acquisition of the target image are realized. Through the convolutional layer, pooling layer, and fully connected layer, the recognition of lesions in the target image is realized, thereby realizing the determination of the lesion recognition result, and further realizing the recognition of micro-lesions, reducing the dependence on personnel experience.

[0058] In one embodiment, as Figure 5 shown, in step S40, the determining the lesion marking area corresponding to the target object and the recommended biopsy position corresponding to the lesion marking area according to all the lesion recognition results includes: S401, according to all the lesion recognition results, mark the lesion areas with different colors to obtain the lesion marking area.

[0059] S402, obtain the historical biopsy positions corresponding to the lesion recognition results, and determine the recommended biopsy positions corresponding to each of the lesion marking areas according to the historical biopsy positions.

[0060] In this embodiment, different heat maps are mainly used to mark the lesion areas and determine the recommended biopsy positions of the lesion marking areas.

[0061] It can be understood that the lesion area refers to the area where lesions may exist identified by a preset lesion recognition model from narrow-band detection images or conventional detection images. The historical biopsy position refers to the biopsy position corresponding to the historical detection image. The historical detection image refers to the detection image before the current time, which can be the historical detection image of the target object or the historical detection image of a non-target object.

[0062] Specifically, after obtaining the lesion recognition results, according to all the lesion recognition results, determine all the lesion areas where lesions may exist and the lesion types corresponding to the lesion areas. Then, obtain the preset marking colors, and use different preset marking colors to mark the lesion areas according to different lesion grades, so as to obtain the lesion marking area. Next, obtain the historical biopsy positions corresponding to the lesion recognition results, and determine the recommended biopsy positions corresponding to each of the lesion marking areas according to the historical biopsy positions, that is, obtain at least one historical biopsy position through the lesion types in the lesion recognition results, and then mark the recommended biopsy positions of each lesion type according to the historical biopsy positions of a large number of the same lesion types, and determine them as the recommended biopsy positions corresponding to each of the lesion marking areas.

[0063] In this embodiment, by marking the lesion areas with different colors, the acquisition of the lesion marking area is realized. Through the historical biopsy positions, the recommendation of the biopsy positions of the lesion marking areas is realized, which facilitates the detection of the operator.

[0064] In one embodiment, as Figure 6 shown, after step S20, after determining the lesion marking area corresponding to the target object according to all the lesion recognition results, it further includes: S601, obtain a preset examination site, and detect whether the number of marks in the lesion marking area is the same as the preset number of the preset examination part.

[0065] S602, when the number of marks in the lesion marking area is less than the preset number of the preset examination site, determine the missed diagnosis areas in the preset examination site, and the AR glasses receive and display all the missed diagnosis areas.

[0066] In this embodiment, it is mainly used to confirm whether there are missed diagnosis areas. When there are missed diagnosis areas, the AR glasses prompt the missed diagnosis areas.

[0067] Understandably, the preset examination site refers to all positions that are prone to lesions obtained by analyzing a large amount of historical data. The missed diagnosis area refers to the area in the preset examination site that is different from the lesion marking area, that is, the area without marked lesions.

[0068] Specifically, after determining the lesion marking area corresponding to the target object, obtain the preset examination site corresponding to the target object, and detect whether the number of marks in the lesion marking area is the same as the preset number of the preset examination part, that is, detect whether there are missed diagnosis sites. By comparing the number of marks in the lesion marking area and the preset number of the preset examination part, it is determined whether there are missed diagnoses. When the number of marks in the lesion marking area is less than the preset number of the preset examination site, determine the missed diagnosis areas in the preset examination site, that is, by performing image matching on the image corresponding to the lesion marking area and the image corresponding to the preset examination site, it is determined whether all the images corresponding to the preset examination site correspond one by one to the image corresponding to the lesion marking area, and the image of the preset examination site that is not matched to the image corresponding to the lesion marking area is determined as the missed diagnosis area. And send all the missed diagnosis areas to the AR glasses to display all the missed diagnosis areas on the virtual screen of the AR glasses to prompt the operator of the missed diagnosis areas.

[0069] In this embodiment, through the comparison between the preset examination site and the lesion marking area, the prompt of the missed diagnosis area is realized, and then the lesion recognition of all areas is realized, ensuring the accuracy of the gastroscopy and colonoscopy diagnosis report.

[0070] In one embodiment, as Figure 7 shown, before step S30, before performing lesion recognition on all the narrowband detection images through the preset lesion recognition model to obtain the lesion recognition results corresponding to each narrowband detection image, it further includes: Train the preset lesion recognition model, where training the preset lesion recognition model includes: S701, sample and determine multiple first clients from all clients, send the initial model parameters to all the first clients, configure all the first clients as the first training model according to the initial model parameters, and after the first round of model training using the first training model, perform parameter aggregation on the first model parameters of all the first training models to obtain aggregated parameters.

[0071] S702, when the aggregated parameters do not meet the preset sampling condition, resample multiple second clients from all the clients, send the aggregated parameters to the second clients, and configure the second clients as the second training model through the aggregated parameters to perform the next round of model training through the second training model.

[0072] S703, when the aggregated parameters meet the preset sampling condition, send the aggregated parameters to all the clients, so that each client configures the local model according to the aggregated parameters and determines it as the preset lesion recognition model.

[0073] This embodiment mainly consists of two-stage processes, namely the server process and the client process. Set up a server and multiple participating clients. The server is mainly responsible for collecting the model parameter information sent by all participating clients, aggregating the received model parameter information, and then distributing it to the participating clients. The client is mainly responsible for training the model using the local dataset and sending the model parameters or partial statistical information to the server after completion.

[0074] Understandably, all clients refer to all parties participating in model training. In this embodiment, all clients refer to all hospitals participating in model training. The first client refers to the client participating in the first round of model training. The second client refers to the client participating in the second round of model training. Among them, the first client and the second client can be partially the same, completely different, or completely the same. The first training model refers to the model constructed based on the initial model parameters. The second training model refers to the model constructed based on the aggregated parameters. The local model refers to the model deployed on this client that meets the preset sampling condition. The sampling condition can be the sampling round threshold or the aggregated parameter deviation threshold.

[0075] Specifically, the server samples from all clients to determine multiple first clients. Then, each first client configures its first training model according to the initial model parameters, and then uses the sample data in each first client to train the first training model. After the training is completed, the server aggregates the first model parameters of all the first training models. That is, the average value of the first model parameters can be used as the aggregated parameter, or the server determines the parameter weights of the first model parameters corresponding to each first client according to the proportion of the training sample volume corresponding to each first client in the total number of training samples in this round, and performs a weighted sum of all the first model parameters according to the parameter weights to obtain the aggregated parameter. Among them, in another embodiment, the parameter weights are dynamically adjusted according to the data quality of the sample data corresponding to each client. When the client trains the model, the five-fold cross-validation evaluation index is used, and the model parameters are optimized through the feedback misdiagnosis cases.

[0076] Further, it is determined whether the aggregated parameter has reached the sampling condition of the server. When the aggregated parameter does not reach the preset sampling condition, multiple second clients are resampled from all clients, and the aggregated parameter is sent to the second clients to configure the second clients as the second training model through the aggregated parameter, so as to perform the next round of model training through the second training model. After the second training model is completed, the second model parameters of all the second training models are aggregated to obtain a new aggregated parameter. When the new aggregated parameter does not reach the preset sampling condition, the next round of model training continues. Until when the aggregated parameter reaches the preset sampling condition, the aggregated parameter is sent to all clients, so that each client configures the local model according to the aggregated parameter and determines the local model as the preset lesion recognition model. Among them, during the training process, the EfficientNet-B7 pre-trained model can be selected for the model, and the Dice Loss and Focal Loss are used as the loss function to solve the problem of class imbalance.

[0077] In this embodiment, the local model is trained through federated learning, which realizes that the data of the participants is retained locally, avoids leakage during the data transmission process, and further realizes the training of the preset lesion recognition model, improving the accuracy of the preset lesion recognition model.

[0078] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0079] In one embodiment, a controller includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The controller is used to execute the above-mentioned gastroscopy diagnosis method based on the AR glasses.

[0080] Specific limitations on the memory, processor, and their respective units and modules can be referred to the limitations on the gastroscope diagnosis method based on AR glasses in the above text, and will not be elaborated here. Each module in the above processor can be implemented in whole or in part by software, hardware, and their combination. Understandably, the processor includes a processor, a memory, a network interface, and a database connected by a device bus. Each module of the processor can be embedded in or independent of the processor in hardware form, or stored in the memory in software form, so that the processor can call and execute the operations corresponding to each of the above modules. Among them, the processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program, and a database. The internal memory provides an environment for the operation of the operating device and the computer program in the non-volatile storage medium. The database is used to store the data used in the gastroscope diagnosis method based on AR glasses in the above embodiments. The network interface is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a gastroscope diagnosis method based on AR glasses.

[0081] In one embodiment, a gastroscope diagnosis system based on AR glasses includes a gastroscope with multi-modal sensors, AR glasses, and a controller as described above; the controller is communicatively connected to the gastroscope and the AR glasses; The gastroscope is used to capture narrow-band detection images taken using narrow-band imaging technology, or conventional detection images taken in real time using white light; The AR glasses are used to receive and display the narrow-band detection images or the conventional detection images captured in real time by the gastroscope, and to prompt guiding indication information of the gastroscope.

[0082] In one embodiment, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned gastroscope diagnosis method based on AR glasses.

[0083] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention and should all be included in the protection scope of the present invention.

Claims

1. A gastroscope diagnosis method based on AR glasses, characterized in that, Including: Obtain the initial detection image corresponding to the target object captured by the gastroscope and send it to the AR glasses for display; the initial detection image includes a narrow-band detection image captured by the narrow-band imaging technology in the gastroscope, or / and a conventional detection image captured in real time by white light in the gastroscope; According to the preset planned path, use the navigation algorithm to generate guiding indication information and display the guiding indication information in the initial detection image displayed by the AR glasses, so as to control the movement of the gastroscope and capture real-time detection images according to the display screen of the AR glasses; the preset planned path is generated by path planning with the three-dimensional gastrointestinal model; the real-time detection image includes a narrow-band detection image captured by the narrow-band imaging technology in the gastroscope, or / and a conventional detection image captured in real time by white light in the gastroscope; Perform lesion recognition on all the real-time detection images through a preset lesion recognition model to obtain a lesion recognition result; According to all the lesion recognition results, determine the lesion marking area corresponding to the target object and the recommended biopsy position corresponding to the lesion marking area, and determine the real-time detection image marked with the lesion marking area and the recommended biopsy position as the lesion image; Send the lesion image to the AR glasses for display and generate a gastroscope diagnosis report corresponding to the target object according to the lesion image.

2. The gastroscope diagnosis method based on an AR glasses according to claim 1, characterized in that, The obtaining the initial detection image corresponding to the target object captured by the gastroscope and sending it to the AR glasses for display; the initial detection image includes a narrow-band detection image captured by the narrow-band imaging technology in the gastroscope, or / and a conventional detection image captured in real time by white light in the gastroscope, includes: After the gastroscope uses white light to collect an image of the target object to obtain a conventional detection image, send the conventional detection image to the AR glasses so that the conventional detection image is displayed on the virtual screen of the AR glasses; Receive a blood vessel mode switching instruction, after the gastroscope uses the narrow-band imaging technology to collect an image of the target object to obtain a narrow-band detection image, send the conventional detection image to the AR glasses so that the narrow-band detection image is displayed on the virtual screen of the AR glasses.

3. The gastrointestinal endoscopy diagnosis method based on an AR glasses according to claim 1, characterized in that The preset lesion recognition model includes an edge processing layer, a convolutional layer, a pooling layer, and a fully connected layer; The performing lesion recognition on all the real-time detection images through a preset lesion recognition model to obtain a lesion recognition result, includes: Perform edge processing on all the real-time detection images through the edge processing layer to obtain a target image; Perform convolutional processing on the target image through the convolutional layer to obtain convolutional features; Perform pooling processing on the convolutional features through the pooling layer to obtain pooling features; Perform lesion recognition on all the pooling features through the fully connected layer to obtain a lesion recognition result.

4. The gastrointestinal endoscopy diagnosis method based on an AR glasses according to claim 1, characterized in that The determining the lesion marking area corresponding to the target object and the recommended biopsy position corresponding to the lesion marking area according to all the lesion recognition results, includes: According to all the lesion recognition results, the lesion areas are marked with different colors to obtain the lesion marked areas; Obtain the historical biopsy positions corresponding to the lesion recognition results, and determine the recommended biopsy positions corresponding to each of the lesion marked areas according to the historical biopsy positions.

5. The gastroscope diagnosis method based on an AR glasses according to claim 1, characterized in that, After generating the guiding indication information by using the navigation algorithm according to the preset planned path and displaying the guiding indication information in the initial detection image displayed by the AR glasses, so as to control the movement of the gastroscope and capture real-time detection images according to the display screen of the AR glasses, it further includes: Obtain the historical data, physiological parameters of the target object, and the real-time images fed back by the gastroscope; the real-time images include narrow-band detection images and conventional detection images; Let the navigation algorithm optimize the preset planned path according to the historical data, the physiological parameters and the real-time images to obtain the guiding indication information of the gastroscope, and send the guiding indication information to the AR glasses, so that the AR glasses display the guiding indication information of the gastroscope by using a virtual guiding line, so as to control the movement of the gastroscope and capture real-time detection images according to the display screen of the AR glasses.

6. The gastroscope diagnosis method based on an AR glasses according to claim 1, wherein, After determining the lesion marked areas corresponding to the target object according to all the lesion recognition results, it further includes: Obtain the preset examination site, and detect whether the number of marks in the lesion marked area is the same as the preset number of the preset examination part; When the number of marks in the lesion marked area is less than the preset number of the preset examination site, determine the missed diagnosis areas in the preset examination site, and the AR glasses receive and display all the missed diagnosis areas.

7. The gastroscope diagnosis method based on an AR glasses according to claim 1, wherein Before obtaining the lesion recognition results by performing lesion recognition on all the real-time detection images through a preset lesion recognition model, it further includes: Training the preset lesion recognition model, wherein training the preset lesion recognition model includes: Sample and determine a plurality of first clients from all the clients, send the initial model parameters to all the first clients, and configure all the first clients as the first training model according to the initial model parameters. After the first round of model training is performed by using the first training model, parameter aggregation is performed on the first model parameters of all the first training models to obtain the aggregated parameters; When the aggregated parameters do not meet the preset sampling conditions, resample a plurality of second clients from all the clients, and send the aggregated parameters to the second clients, so as to configure the second clients as the second training model through the aggregated parameters, so as to perform the next round of model training through the second training model; When the aggregated parameters meet the preset sampling conditions, send the aggregated parameters to all the clients, so that each client configures the local model according to the aggregated parameters and determines it as the preset lesion recognition model.

8. A controller, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The controller is used to execute the gastroscope diagnosis method based on the AR glasses according to any one of claims 1 to 7.

9. A gastroscopy diagnosis system based on AR glasses, characterized in that, Comprising a gastroscope with multi-modal sensors, an AR glasses, and a controller as described in claim 8; the controller is communicatively connected to the gastroscope and the AR glasses; The gastroscope is used for narrow-band detection images captured by using narrow-band imaging technology, or conventional detection images captured in real time by using white light; The AR glasses are used for receiving and displaying the narrow-band detection images or the conventional detection images captured in real time by the gastroscope, and for prompting guiding indication information of the gastroscope.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the gastroscope diagnosis method based on AR glasses as described in any one of claims 1 to 7.

Images