Medical assistance operation method, device, equipment and computer storage medium

Abstract

Exemplary implementations of the present disclosure relate to a medical assistance operation method, apparatus, device, and computer storage medium. In one method, input data is obtained from an endoscope; and information related to the operation behavior of the endoscope is determined based on the input data. By using the above method, information related to the operation behavior of the endoscope can be provided in real time based on the current position of the endoscope, and thus medical assistance operations can be provided in real time during a doctor's endoscopy examination. Further, corresponding apparatuses, devices, and computer storage media are provided.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure relate to the technical field of medical assistance. Further, they relate to information processing methods, devices, equipment, and computer storage media for medical assistance, and more specifically, to medical assistance operation methods, devices, equipment, and computer storage media. Background Art

[0002] The medical examination process performed on patients usually involves complex manual operations. Currently, the development of computer technology has provided more and more support for medical assistance operations. For example, in endoscopic examinations, doctors need to move the endoscope inside the patient's body to obtain image data at multiple positions inside the patient's body. The operations of different doctors may vary. For example, experienced doctors can independently complete the full set of endoscopic examination processes, while inexperienced doctors may miss certain predetermined key point positions and / or cause discomfort to the patient due to improper movement of the endoscope. Therefore, it is desirable to provide an effective technical solution to provide medical assistance and thereby guide the operation of endoscopic examinations. Summary of the Invention

[0003] Exemplary implementations of the present disclosure provide technical solutions for medical assistance operations.

[0004] According to a first aspect of the present disclosure, a medical assistance operation method is proposed. In this method, input data is obtained from an endoscope; and information related to the operation behavior of the endoscope is determined based on the input data.

[0005] According to a second aspect of the present disclosure, a medical assistance operation device is proposed. The device includes: an input module configured to obtain input data from an endoscope; and an output module configured to output information related to the operation behavior of the endoscope determined based on the input data.

[0006] According to a third aspect of the present disclosure, a medical assistance operation device is proposed. The device includes: at least one processing unit; at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit causing the device to perform the method described in the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has computer-readable program instructions stored thereon for performing the method described in the first aspect.

[0008] The Summary of the Invention section is provided to introduce, in a simplified form, a selection of concepts that will be further described in the Detailed Description below. The Summary of the Invention section is not intended to identify key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. Brief Description of the Drawings

[0009] The above and other objects, features, and advantages of the present disclosure will become more apparent from the following more detailed description of exemplary implementations of the present disclosure in conjunction with the accompanying drawings, in which like reference numerals generally represent like components in the exemplary implementations of the present disclosure.

[0010] Figure 1 A block diagram schematically showing a human body environment in which endoscopic examination can be performed according to an exemplary implementation of the present disclosure;

[0011] Figure 2 A block diagram schematically showing a medical assistance operation according to an exemplary implementation of the present disclosure;

[0012] Figure 3 A flowchart schematically showing a method of medical assistance operation according to an exemplary implementation of the present disclosure;

[0013] Figure 4A A block diagram schematically showing a motion model according to an exemplary implementation of the present disclosure;

[0014] Figure 4B A block diagram schematically showing a process of obtaining a motion model according to an exemplary implementation of the present disclosure;

[0015] Figure 5 A block diagram schematically showing a process of mapping a set of image sequences in an image sequence to a set of key point positions according to an exemplary implementation of the present disclosure;

[0016] Figure 6 A block diagram schematically showing a process of selecting an image associated with a key point position for storage according to an exemplary implementation of the present disclosure;

[0017] Figure 7A A block diagram schematically showing a data structure of a motion trajectory according to an exemplary implementation of the present disclosure;

[0018] Figure 7B A block diagram schematically showing a process of providing a next destination position according to an exemplary implementation of the present disclosure;

[0019] Figure 8 A block diagram schematically showing a user interface for providing a medical assistance operation according to an exemplary implementation of the present disclosure;

[0020] Figure 9 A block diagram schematically showing another user interface for providing medical assistance operations according to an exemplary implementation of the present disclosure;

[0021] Figure 10 A block diagram schematically showing a medical assistance operation device according to an exemplary implementation of the present disclosure; and

[0022] Figure 11 A block diagram schematically showing a medical assistance operation device according to an exemplary implementation of the present disclosure. Detailed Implementation Modes

[0023] The preferred exemplary implementations of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred exemplary implementations of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the exemplary implementations set forth herein. On the contrary, these exemplary implementations are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0024] The term "including" and its variants used herein mean open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The term "an example exemplary implementation" and "an exemplary implementation" mean "at least one example exemplary implementation". The term "another exemplary implementation" means "at least one additional exemplary implementation". The terms "first", "second", etc. may refer to different or the same objects. There may also be other explicit and implicit definitions hereinafter.

[0025] Machine learning techniques have been applied to various application fields including medicine. Medical examination devices usually involve complex operation processes. In particular, for endoscopic examinations, an endoscope needs to be inserted into a patient's body to collect images of various human positions. The examination process needs to ensure that images at a set of key point positions are obtained. The endoscope can move along different movement trajectories according to the doctor's operation, and improper operation may result in missing some key point positions that should have been examined originally. Therefore, how to provide medical assistance operations in a more effective way has become a research hotspot.

[0026] Endoscopes can be applied to the examination of multiple human body parts. For example, according to the human body parts, they can be divided into various types such as esophagoscopes, gastroscopes, duodenoscopes, colonoscopes, etc. In the following, only gastroscopes will be used as an example to describe the details of the exemplary implementations of the present disclosure. First, refer to Figure 1 Describe the application environment of the exemplary implementations of the present disclosure. Figure 1FIG. 100 is a block diagram schematically showing a human body environment in which endoscopic examination can be performed according to an exemplary implementation of the present disclosure. According to the endoscopic operation specification, the endoscope should reach a set of predetermined key point positions during the examination and should capture images at these key point positions to determine whether an abnormality appears at that position. As Figure 1 shown, during the process of inserting the endoscope into the human stomach, it can pass through multiple key point positions 110, 112, 114, 116, 118, and 120, etc.

[0027] The endoscope can first pass through the pharynx and reach the key point position 110. As shown by the arrow 130, the endoscope can enter the stomach downward along the esophagus and can reach the key point position 112. Further, as shown by the arrow 132, the endoscope can reach the key point position 114. It will be understood that due to the relatively large space inside the human body and due to the different operation methods of the doctor, the endoscope can move in different directions within the stomach. For example, when the endoscope reaches the key point position 114, it can reach the key point position 118 along the direction shown by the arrow 134, or it can also reach the key point position 116 along the direction shown by the arrow 136. Although a set of key point positions has been defined in the operation specification, the doctor can only adjust the movement trajectory of the endoscope based on his own experience, and there may be a situation where the movement trajectory can only cover a part of the key point positions.

[0028] To at least partially solve the above-mentioned defects in endoscopic examination, according to an exemplary implementation of the present disclosure, a technical solution for medical assistant operation is provided. First, refer to Figure 2 to describe the outline of this technical solution. Figure 2 FIG. 200 is a block diagram schematically showing a medical assistant operation according to an exemplary implementation of the present disclosure. As Figure 2 shown, as the endoscope 210 is inserted into the human body and moves within the human body, the endoscope 210 can capture the video 220, and the doctor can observe the video 220 in real time.

[0029] It will be understood that the input data 230 (for example, including an image data sequence) can be obtained based on the video 220. For example, the input data 230 can include one or more video segments. One video segment can include images related to the endoscope 210 passing near the pharynx of the human body, and another video segment can include images related to the endoscope 210 passing near the esophagus of the human body. It will be understood that the format of the input data 230 is not limited in the context of the present disclosure. For example, the input data 230 can be video data, a set of image sequences arranged in chronological order in the video, or it can also be multiple image data with time information. According to an exemplary implementation of the present disclosure, the input data can be saved in the original video format or in a custom intermediate format.

[0030] It will be understood that unique identifiers can be used to identify the input data. For example, a doctor ID and the time of performing an examination can be used as identifiers, an endoscope device ID and the time of performing an examination can be used as identifiers, a patient ID and the time of performing an examination can be used as identifiers, or the above can also be combined to obtain a unique identifier. Subsequently, information related to the operation behavior 240 of the endoscope 210 can be determined based on the input data 230.

[0031] In this way, effective medical assistance can be provided to the doctor and the operation of the doctor (especially an inexperienced doctor) can be guided to avoid missing a certain / some key point positions. Further, by using the exemplary implementation of the present disclosure, the doctor can be guided to traverse all key point positions as soon as possible, which can improve the efficiency of the endoscope examination, shorten the time when the endoscope 210 is located in the patient's body, and thus reduce the patient's bad experience.

[0032] Specifically, medical assistance operations can be provided in real time during the doctor's performance of the endoscope examination. Information related to the operation behavior of the endoscope can be provided in real time based on the current position of the endoscope. For example, at least any one of the following can be provided in real time: the key point position where the endoscope is currently located, an image of the key point position, the movement trajectory that the endoscope has passed through, the next destination position of the endoscope, and statistical information of the endoscope operation, etc. For example, the above information can be displayed on a dedicated display device, alternatively and / or additionally, the above information can also be displayed on the display device of the endoscope device.

[0033] Hereinafter, Figure 3 more details of the medical assistance operation will be described. Figure 3 A flowchart of a medical assistance operation method 300 according to an exemplary implementation of the present disclosure is schematically shown. At block 310, input data 230 can be obtained from the endoscope 210. It will be understood that as the endoscope 210 moves in the human body, input data at different positions can be obtained.

[0034] The input data 230 can be used to determine the information or data required for endoscopic examination. Further, the input data 230 can also be used to determine the information or data related to the operation behavior of the endoscope. Exemplarily, the input data 230 can include image data collected at multiple positions during the movement of the endoscope 210. It will be understood that the image data here can be the originally collected data or the data after processing (e.g., noise reduction processing, brightness processing, etc.). Based on the acquisition frequency of the image acquisition device of the endoscope 210, the image data can include, for example, 30 frames per second (or other frame rates) of images. It will be understood that in the context of the present disclosure, the format of the input data 230 is not limited.

[0035] The input data 230 here can include at least any one of the following: video data, a set of image sequences arranged in chronological order, and multiple image data with time information. For example, the video data can include a video stream format and can support the standards of multiple video formats. Again, for example, the image sequence can also include a series of individual images. At this time, as the endoscopic examination progresses, the quantity of the obtained input data 230 can gradually increase. For example, when the endoscope reaches the pharynx, an image sequence of the pharynx can be acquired; when the endoscope reaches the esophagus, an image sequence of the esophagus can be further acquired.

[0036] In addition, an identifier corresponding to the input data 230 can be further acquired or determined to identify the input data 230. Different identifiers can distinguish one or more combinations of the following: different patients, different detection times, different detection sites, and different detection operators.

[0037] According to an exemplary implementation of the present disclosure, at block 320, information related to the operation behavior of the endoscope is determined based on the input data. This information can include various aspects, for example, the current position of the endoscope, the image data collected at the current position, the movement trajectory of the endoscope, the next destination position of the endoscope, the statistical information of the input data, and the statistical information of the operation behavior, and so on. In the following, reference will be made to Figure 4A and Figure 4B to describe more relevant details.

[0038] Exemplarily, information related to the operation behavior of the endoscope can be determined according to the timing relationship of the input data 230. In addition, according to an exemplary implementation of the present disclosure, various aspects of information related to the operation behavior 240 can be determined based on machine learning techniques and by using the input data 230. For example, the current position, movement trajectory of the endoscope 210, and whether the movement trajectory has reached a key point position expected to be examined can be determined, and so on. Further, the destination position to be reached next can be determined. Specifically, the sample data collected during historical operations can be used, and a motion model 410A can be obtained based on machine learning techniques. Figure 4A FIG. 400A schematically shows a block diagram of a motion model 410A according to an exemplary implementation of the present disclosure. The motion model 410A can include an association relationship between sample input data 412A and a sample movement trajectory 414A. Here, the sample input data 412A can be collected at multiple sample positions during an endoscopic examination, and the sample movement trajectory 414A can include the movement trajectory of the endoscope for collecting the sample input data 412A.

[0039] It will be understood that the sample input data 412A and the sample movement trajectory 414A here can be sample training data for training the motion model 410. According to an exemplary implementation of the present disclosure, one training can be performed using the sample input data 412A and the corresponding sample movement trajectory 414A. In the context of the present disclosure, one or more trainings can be performed by respectively using the sample training data from one or more endoscopic examinations.

[0040] It will be understood that only an example of the motion model 410A is schematically shown above. According to an exemplary implementation of the present disclosure, other models can also be provided. For example, another model can include an association relationship between sample input data collected at multiple sample positions during an endoscopic examination and the corresponding key point positions of the multiple positions where the sample input data is collected. Using this model, each image data in the input data 230 can be respectively mapped to the corresponding key point position. Thus, based on this model and the input data, the key point positions passed by the endoscope can be determined. Further, based on the acquisition time of the image data and the above key point positions, the movement trajectory of the endoscope can be determined.

[0041] Exemplarily, training can be performed based on technologies such as Recurrent Neural Network (RNN) and Long Short Term Memory (LSTM) to obtain the motion model 410A. According to an exemplary implementation of the present disclosure, the above training method can be utilized to obtain the motion model 410A based on the sample input data and the corresponding sample motion trajectories collected during the historical examination. According to an exemplary implementation of the present disclosure, an endoscopic examination operation can be performed by a doctor, and the collected data can be used as samples to train the above model.

[0042] For example, an experienced doctor can operate the movement of the endoscope in accordance with the endoscopic operation specifications. At this time, the sample motion trajectory of the endoscope will cover all the key point positions required for the medical examination. For the input data obtained during an endoscopic examination, the association relationship between each sample image in the input data and the position of the sample image in the motion trajectory can be identified based on the annotation method.

[0043] For example, an experienced doctor can perform endoscopic examinations multiple times to obtain a sequence of relevant sample images of multiple sample motion trajectories. Also, for example, multiple experienced doctors can each perform one or more endoscopic examinations to obtain more abundant training data. When sufficient training data has been obtained, the motion model 410A can be trained based on the sequence of sample images and the sample motion trajectories. Here, the endoscopic operation specifications define all the key point positions to be examined, and an experienced doctor can ensure that the examinations performed can meet the requirements in the specifications to the greatest extent. By using the training data obtained in this way to perform the training, it can be ensured that the obtained motion model 410A can accurately reflect the association relationship between the image and the motion trajectory. In addition, the motion model 410A can also be obtained by computer simulation.

[0044] For the sake of convenience in description, in the following, only the sequence of images will be used as an example of the input data 210 to describe the exemplary implementation according to the present disclosure. When the input data 210 is stored in other formats, the processing method is similar. For example, when the input data 210 is in video format, the sequence of images in the video can be obtained and processed for the sequence of images.

[0045] In the following, reference will be made to Figure 4B Describe the process of obtaining the motion model 410A. Figure 4BFIG. 400B schematically shows a block diagram of a process for obtaining a motion model 410A according to an exemplary implementation of the present disclosure. Training can be performed based on a sample image sequence and a sample motion trajectory collected during a historical inspection process. A plurality of sample image sequences can be divided into a plurality of groups, each group including N>3 images. Subsequently, a group of multi-frame sample images 410B (e.g., N consecutive frames starting from the T-Nth frame) can be input into a neural network layer 412B, a group of multi-frame sample images 420B (e.g., N consecutive frames starting from the Tth frame) can be input into a neural network layer 422B; a group of multi-frame sample images 430B (e.g., N consecutive frames starting from the T+Nth frame) can be input into a neural network layer 432B. In this way, the association relationship between the image sequence and the motion trajectory can be obtained.

[0046] It will be understood that the foregoing is only with reference to Figure 4B FIG. 400A schematically shows one implementation that can be used to obtain the motion model 410A. According to an exemplary implementation of the present disclosure, the motion model 410A can be obtained according to other machine learning techniques that are currently known and / or will be developed in the future.

[0047] Based on the motion model 410A and the input data, the motion trajectory of the endoscope 210 can be determined. According to an exemplary implementation of the present disclosure, the motion trajectory of the endoscope 210 includes a set of key point positions during the motion of the endoscope 210. Here, a set of key point positions includes at least a part of a set of predetermined human body positions of the endoscope 210 during endoscopy, and a plurality of positions passed during the motion of the endoscope can be within a predetermined range around the key point positions.

[0048] It will be understood that a set of key point positions herein can be positions defined according to endoscopy specifications. For example, it can include positions such as the pharynx, esophagus, cardia, pylorus, etc. Suppose the endoscope passes through the pharynx and 3 images have been collected at a plurality of positions near the pharynx during the motion (e.g., 0.5 cm before reaching the pharynx, the pharynx, 0.5 cm after leaving the pharynx). At this time, it can be determined that the motion trajectory includes the key point position "pharynx". As the endoscope 210 further moves, the motion trajectory can include more key point positions, such as the pharynx, esophagus, etc. The above positions can also be further divided into more positions. For example, for the esophagus, it can further include more positions such as the upper part, middle part, lower part of the esophagus, etc. In other words, the motion trajectory herein can include one or more key point positions passed by the motion of the endoscope 210.

[0049] According to an exemplary implementation of the present disclosure, the acquired input data 230 can be input into the motion model 410A in a manner similar to obtaining the motion model 410A. For example, the input data 230 can be divided into multiple groups (each group includes N frame images), and the multiple groups are sequentially input into the motion model 410A. At this time, at a certain layer in the motion model 410A, the features corresponding to the current N frame images can be continuously output (as latent variables), and the features are iteratively input to the position of the next layer. The motion model 410A can output the motion trajectory of the endoscope according to the input data.

[0050] According to an exemplary implementation of the present disclosure, using the motion model 410A, the input data can be mapped to a set of key point positions respectively. Continuing to refer to Figure 4B , as Figure 4B shown on the right side, CLSC(T) represents the prediction of the key point position to which the consecutive N frame images starting from the T-th frame belong, CLSN(T) represents the prediction of the key point position to which the subsequent N frame images belong, CLSP(T) represents the prediction of the key point position to which the previous N frame images belong, and Y(T) represents the prediction of the motion trajectory to which the current image sequence belongs. The prediction of the motion trajectory here can include multiple key point positions. For example, the prediction of a motion trajectory can include: key point position 110 -> key point position 112 -> key point position 114; the prediction of the motion trajectory can include: key point position 110 -> key point position 112 -> key point position 116. According to the currently input N frame images, the prediction of the motion trajectory can include different key points. The next destination position can be determined based on the prediction of the motion trajectory. Further, information associated with other frames can be determined in a similar manner.

[0051] Figure 5 Figure 500 schematically shows a block diagram of a process for mapping input data to a set of key point positions according to an exemplary implementation of the present disclosure. As Figure 5 shown, as the movement time of the endoscope 210 in the human body increases, the input data 210 will include more and more images. Figure 5 Only the situation at the initial stage of endoscopy is schematically shown. At this time, the endoscope 210 has acquired a large number of images near the key point positions 110, 112, and 114.

[0052] Using the method described above, these images can be mapped to the corresponding key point positions. For example, a set of image data 510 in an image sequence can be mapped to the key point position 110 to indicate that the set of image data 510 is an image collected near the key point position 110. Similarly, a set of image data 512 in an image sequence can be mapped to the key point position 112, and a set of image data 514 in an image sequence can be mapped to the key point position 114, and so on.

[0053] Using the exemplary implementation of the present disclosure, based on the input data 230 collected during the movement of the endoscope 210, the position where each image data is collected can be determined. Compared with the technical solution that completely relies on the personal experience of the doctor, the above technical solution can determine the key point position associated with the image data in a more accurate manner, thereby helping to select which images to store later.

[0054] According to the exemplary implementation of the present disclosure, based on the time sequence in which the image sequences associated with the key point positions are collected, the movement trajectory can be determined. Continuing to refer to Figure 5 , it has been determined that a set of image data 510 is associated with the key point position 110, a set of image data 512 is associated with the key point position 112, and a set of image data 514 is associated with the key point position 114. Assuming that the time sequence of the collection of each image is: a set of image data 510, a set of image data 512, and a set of image data 514. At this time, it can be determined that the movement trajectory 1 includes: key point position 110 -> key point position 112 -> key point position 114.

[0055] It will be understood that the movement trajectory includes the key point positions arranged in time sequence. Therefore, if the order of a set of key point positions is different, it represents a different movement trajectory. For example, the movement trajectory 2 can include: key point position 110 -> key point position 114 -> key point position 112. Then the movement trajectory 2 is different from the movement trajectory 1.

[0056] In addition, the movement trajectory can also be the actual movement trajectory of the endoscope in the human body part determined based on the input data. The actual movement trajectory includes both key point positions and non-key point positions, so as to reflect the operation behavior of the endoscope in real time, and thus can better analyze and assist in guiding the inspection operation of the endoscope.

[0057] Using the exemplary implementation of the present disclosure, based on the time sequence in which each image data is collected, the movement trajectory of the endoscope 210 can be recorded in a more accurate manner. Further, the determined movement trajectory can also be used for post-processing. For example, based on the key point positions that the endoscope 210 has reached, the key point positions that should be reached can be determined.

[0058] Generally speaking, during the process of performing an endoscopic examination, on the one hand, the doctor has to manipulate the endoscope to reach the desired key point position, and on the other hand, the doctor also needs to store the images for later diagnosis. Since the image sequence collected during the examination process will occupy a large amount of storage space, usually the doctor only selects an appropriate angle to collect and store images based on his own experience after reaching near the key point position. For example, a foot pedal can be set at the endoscopic examination device, and the doctor can step on the foot pedal to store the image. This may lead to the situation that the doctor misses some key point positions and / or the stored images have poor quality and cannot be used for diagnosis.

[0059] According to an exemplary implementation of the present disclosure, image analysis can also be performed on a determined set of images to select the image that best reflects the human body state at a certain key point position. Hereinafter, more details regarding the selection and storage of images will be described with reference to Figure 6 Describe more details about the selection and storage of images. Figure 6 FIG. 600 is a block diagram schematically showing a process of selecting an image associated with a key point position for storage according to an exemplary implementation of the present disclosure. Specifically, for a given key point position in a set of key point positions, a set of given images mapped to the given key point position in the input data can be determined.

[0060] As Figure 6 shown, the image quality evaluation of a set of given images can be determined respectively based on the image quality of the set of given image data. Subsequently, the image for storage can be selected based on the determined image quality evaluation. In Figure 6 , a set of image data 510 related to the key point position 110 has been determined. At this time, the image quality evaluation can be determined for the set of image data 510. Subsequently, the selected image data 610 can be obtained from the set of image data 510 and stored in the storage device 620. Similarly, the selected image data 612 can be obtained from the set of image data 512 and stored in the storage device 620; and the selected image data 614 can be obtained from the set of image data 514 and stored in the storage device 620. Subsequently, the relevant information of the stored images can be displayed to the doctor, for example, the number of stored images, the associated key point positions, etc.

[0061] It will be understood that the image quality here can have multiple meanings. For example: an image that can preferably reflect the position of the key points to be examined. For example, the image quality can include one or more of the following: the clarity of the human mucosa in the image collected by the endoscope, whether the mucosa is contaminated, whether the mucosa is covered by secretions, etc., the shooting angle of the endoscope, and so on. If the human mucosa is clearly visible, not contaminated, and not covered by secretions, it can be determined that the image has high quality. On the contrary, it can be determined that the image has low quality.

[0062] It will be understood that various methods can be adopted here to determine the image quality. For example, the clarity of the image can be determined based on an image processing method, and then an image quality evaluation can be obtained. For another example, based on machine learning, a quality prediction model can be established using pre-labeled sample data. According to the exemplary implementation manners of the present disclosure, other image processing technologies that have been developed currently and / or will be developed in the future can also be adopted to obtain the image quality evaluation.

[0063] Using the exemplary implementation manners of the present disclosure, one or more images with the best image quality can be selected from a large number of images obtained at a given key point position. Compared with the technical solution of manually selecting and storing images based on the doctor's personal experience, the efficiency of selecting images can be significantly improved, the time occupied by the doctor in selecting and storing images can be shortened, and thus the efficiency of endoscopy can be improved. On the other hand, since the mapping, selection, and storage of images are performed in an automatic manner, the situation of omission caused by doctor errors can be avoided as much as possible. In addition, further, the images with better image quality can be assisted in selection according to the temporal relationship of the acquired input data (such as the association relationship between the image sequence or the key point position images).

[0064] According to the exemplary implementation manners of the present disclosure, based on the movement trajectory of the endoscope 210 and the predetermined movement trajectory of the endoscopy, an evaluation of the movement trajectory can be determined. The predetermined movement trajectory here can be the sequence of a series of key point positions defined according to the endoscope operation specifications. For example, the predetermined movement trajectory can include pharynx -> esophagus -> cardia -> pylorus, etc. It is expected that the doctor can operate the movement of the endoscope according to the predetermined movement trajectory, and thus the evaluation can be determined based on the consistency between the actual movement trajectory of the endoscope 210 and the predetermined movement trajectory.

[0065] According to the exemplary implementation manners of the present disclosure, the evaluation can include various types. For example, the evaluation can be represented by a score within a certain range (such as a real number between 0 and 1); the evaluation can be represented in the form of a grade (such as high, medium, low); the evaluation can be represented in a text description manner; or the evaluation can also be represented in the form of an image or other ways.

[0066] In the following, reference will be made toFigure 7A Describe more details about the evaluation of determining the motion trajectory. Figure 7A FIG. 700A is a block diagram schematically showing a data structure of a motion trajectory according to an exemplary implementation of the present disclosure. In Figure 7A , the motion trajectory 710 of the endoscope 210 includes three key point positions: key point positions 110, 112, and 114. At this time, the endoscope 210 is located at the key point position 114, and the evaluation of the motion trajectory 710 can be determined based on comparing the motion trajectory 710 with a predetermined motion trajectory of the endoscopy. Further, the relevant evaluation can be displayed to the doctor.

[0067] It will be understood that various ways can be adopted herein to determine the evaluation. A numerical range of the evaluation can be specified. For example, the evaluation can be represented within a range of 0-1. Assume that the predetermined motion trajectory includes: key point position 110 -> key point position 112 -> key point position 114 -> key point position 118…, and the current motion trajectory 710 includes key point position 110 -> key point position 112 -> key point position 114. It can be determined that the motion trajectory 710 completely matches the start part of the predetermined motion trajectory, and thus a higher evaluation 712 can be given for the motion trajectory 710. For example, the evaluation 712 can be set to the highest score 1. For another example, assume that the predetermined motion trajectory deviates from the predetermined motion trajectory, then the value of the evaluation can be decreased at this time. For example, the evaluation can be set to 0.8.

[0068] It will be understood that the principle of determining the evaluation is only described in a schematic manner above. According to an exemplary implementation of the present disclosure, an evaluation prediction model can be established based on machine learning using pre-labeled sample data. According to an exemplary implementation of the present disclosure, other prediction techniques that have been developed currently and / or will be developed in the future can also be adopted to obtain the evaluation of the motion trajectory.

[0069] Hereinafter, more details about determining the next destination position will be referred to Figure 7B According to an exemplary implementation of the present disclosure, a set of candidate positions can be determined based on one or more key point positions near the last key point position in the motion trajectory. Figure 7B FIG. 700B is a block diagram schematically showing a process for providing the next destination position according to an exemplary implementation of the present disclosure. As Figure 7BAs shown, a set of candidate positions of the endoscope 210 at the next time point can be determined first. Continuing with the example above, the current position of the endoscope 210 is at the key point position 114, and there are key point positions 116 and 118 near the key point position 114. At this time, a set of candidate positions can include the key point positions 116 and 118. Subsequently, the evaluation of each candidate position in the set of candidate positions can be determined, and the next destination position can be selected from the set of candidate positions based on the determined evaluation.

[0070] Specifically, for a given candidate position in a set of candidate positions, a candidate movement trajectory of the endoscope 210 can be generated based on the movement trajectory and the candidate position. As Figure 7B shown, based on the movement trajectory 710 and the key point position 116, a candidate movement trajectory 720 can be generated; based on the movement trajectory 710 and the key point position 118, a candidate movement trajectory 730 can be generated. Subsequently, the methods described above can be used to determine the evaluations 722 and 732 of the two candidate movement trajectories 720 and 730 respectively based on the candidate movement trajectories 720 and 730 and the predetermined movement trajectory of the endoscopy. As Figure 7B shown, since the evaluation 732 is higher than the evaluation 722, a higher evaluation can be given to the key point position 118, and the key point position 118 can be used as the next destination position.

[0071] Using the exemplary implementation of the present disclosure, the key point position that is most matched to the predetermined movement trajectory of the endoscope 210 can be preferentially recommended to the doctor as the next destination position for moving the endoscope 210. In this way, guidance can be given for the doctor's movement operation, while improving the efficiency of the endoscopy, the potential risk of missing key point positions can also be reduced. Further, since the movement of the endoscope in the human body may cause discomfort to the patient, improving the inspection efficiency can shorten the time length of the endoscopy, and thus the pain of the patient can be reduced.

[0072] It will be understood that although specific examples of providing the next destination position have been described above with reference to the accompanying drawings. According to the exemplary implementation of the present disclosure, a subsequent recommended path can also be provided, and the recommended path can include one or more key point positions. The doctor can move the endoscope along the recommended path to cover all the key points required for the endoscopy.

[0073] According to an exemplary implementation of the present disclosure, a candidate motion trajectory of the endoscope can also be directly generated based on the motion model 410A and the input data. It will be understood that the motion model 410A can be established in an end-to-end manner during the training phase. At this time, the input of the motion model 410A can be specified as an image sequence, and the output of the motion model 410A can be specified as a candidate motion trajectory. Here, the candidate motion trajectory can include a set of key point positions corresponding to the input image sequence and the next candidate key point position. When using the motion model 410A, a set of image sequences currently acquired by the endoscope can be input into the motion model 410A to obtain a candidate motion trajectory. At this time, the doctor can operate the endoscope to move along the candidate motion trajectory to traverse all key point positions.

[0074] According to an exemplary implementation of the present disclosure, by using a historical sample image sequence with markers and a historical sample candidate motion trajectory, the motion model 410A including the association relationship between the image sequence and the candidate motion trajectory can be directly obtained. By using the exemplary implementation of the present disclosure, the training process can be directly performed based on the historical sample data and the corresponding model can be obtained. In this way, the operation process can be simplified and the efficiency of obtaining the candidate motion trajectory can be improved.

[0075] According to an exemplary implementation of the present disclosure, further, information related to the operation behavior of the endoscope can be transmitted and / or stored.

[0076] According to an exemplary implementation of the present disclosure, information related to the current doctor's operation can be output in real time, and corresponding statistical and analysis functions can be provided. For example, the method 300 described above can further provide the following functions: determining the duration of the endoscopy, determining the information of the key point positions that have been scanned, determining the information of the key point positions that have not been scanned, determining the information of the next destination position, determining the operation evaluation of the doctor during the endoscopy, determining whether the images of each key point position that have been acquired are qualified, and so on.

[0077] Hereinafter, reference will be made to Figure 8 and Figure 9 describe the functions of outputting information related to operation behavior. According to an exemplary implementation of the present disclosure, the function of outputting the above information can be combined with the existing endoscope display interface. Figure 8 A block diagram schematically showing a user interface 800 for providing medical assistance operations according to an exemplary implementation of the present disclosure is shown. As Figure 8As shown, the user interface 800 may include: an image display section 810 for displaying in real time the video 220 captured by the endoscope 210; a motion trajectory management section 820 for displaying the motion trajectory that the endoscope 210 has passed through and a prompt for the next destination position; and a statistical information section 830 for displaying relevant information about the images captured during the endoscopy.

[0078] As shown in the motion trajectory management section 820, the solid line indicates that the motion trajectory that the endoscope 210 has passed through is: key point position 110 -> key point position 112 -> key point position 114. The dashed line part indicates the trajectory from the current position of the endoscope 210 (i.e., key point position 114) to the next destination positions (i.e., key point positions 116 and 118). The next destination position can be set as the key point position 118 based on the method described above. Further, a star mark 822 can be used to indicate that the recommended next destination position is the key point position 118. At this time, the doctor can move the endoscope 210 to the key point position 118 at the next time point. Figure 7B As shown in the statistical information section 830, relevant information about the captured images can be displayed. For example, for the key point position 110, 10 images have been selected, and the comprehensive evaluation of the 10 images is 0.8. It will be understood that an upper limit on the number of images expected to be captured for each key point can be predefined. For example, the upper limit can be defined as 10. The 10 images shown here can be images with relatively high image quality selected according to the method described above, and the evaluation of 0.8 here can be a comprehensive evaluation obtained based on the individual image quality evaluations.

[0079] According to an exemplary implementation of the present disclosure, a lower limit on the image quality evaluation can also be set. For example, it can be set to select only images with an evaluation higher than 0.6. According to an exemplary implementation of the present disclosure, it is also possible to select which images are expected to be stored based on both the upper limit of the number of images and the lower limit of the image quality evaluation. The statistical information section 830 further shows the statistical information about other key point positions: for the key point position 112, 5 images have been selected, and the comprehensive evaluation of the 5 images is 0.6; and for the key point position 114, 7 images have been selected, and the comprehensive evaluation of the 7 images is 0.9. Figure 6 For the key point position 110, 10 images have been selected, and the comprehensive evaluation of the 10 images is 0.8. It will be understood that an upper limit on the number of images expected to be captured for each key point can be predefined. For example, the upper limit can be defined as 10. The 10 images shown here can be images with relatively high image quality selected according to the method described above, and the evaluation of 0.8 here can be a comprehensive evaluation obtained based on the individual image quality evaluations.

[0080] According to an exemplary implementation of the present disclosure, a lower limit on the image quality evaluation can also be set. For example, it can be set to select only images with an evaluation higher than 0.6. According to an exemplary implementation of the present disclosure, it is also possible to select which images are expected to be stored based on both the upper limit of the number of images and the lower limit of the image quality evaluation. The statistical information section 830 further shows the statistical information about other key point positions: for the key point position 112, 5 images have been selected, and the comprehensive evaluation of the 5 images is 0.6; and for the key point position 114, 7 images have been selected, and the comprehensive evaluation of the 7 images is 0.9.

[0081] According to an exemplary implementation of the present disclosure, the user interface for managing the motion of the endoscope 210 can be separated from the existing endoscope display interface. Figure 9 A block diagram schematically shows another user interface 900 for providing medical assistance operations according to an exemplary implementation of the present disclosure. AsFigure 9 As shown, relevant information about medical assistance operations can be displayed in a separate user interface 900. In the user interface 900, information related to operation behaviors can be output.

[0082] According to an exemplary implementation of the present disclosure, information about the selected images of key point positions can also be displayed in the area 910. For example, the area 910 may include thumbnails of the images. Suppose an endoscopic examination process requires collecting images of 6 key point positions, 4 key point position images have been collected, and the remaining 2 key point position images have not been collected. Legends 912, 914, and 916 can be used to represent images of different types of key point positions respectively. For example, legend 912 indicates that a qualified image has been collected at a certain key point position, legend 914 indicates that a qualified image has not been collected at a certain key point position, and legend 916 indicates that a certain key point position has not been scanned. Using the exemplary implementation of the present disclosure, the scanned, unscanned, and unqualified image key point positions can be visually displayed to the doctor, thus facilitating the doctor's subsequent operations.

[0083] According to an exemplary implementation of the present disclosure, after the images for storage have been selected, based on the selected images, image anomalies associated with a given key point position can be identified. Further, the identified image anomalies can be displayed. Specifically, the content of the images can be analyzed based on currently known and / or image recognition technologies that will be developed in the future to determine possible image anomalies at this key point position. For example, the image anomalies can indicate ulcers, tumors, etc. Using the exemplary implementation of the present disclosure, images that may have anomalies can be identified, thus assisting the doctor in diagnosis.

[0084] According to an exemplary implementation of the present disclosure, based on the input data, the working state of the endoscope 210 is identified. It will be understood that various working states may be involved during the operation of the endoscope 210. For example, during the process of starting the endoscope 210 and inserting the endoscope 210 into the patient's body, the image content collected by the endoscope 210 will be different. Based on the analysis of the images collected by the endoscope 210, the patient being examined can be determined, whether the endoscope is currently inside or outside the patient can be determined, and the current examination site (such as the stomach or intestine, etc.) can be determined. For example, if a part of the image sequence involves external images and a subsequent part of the images is converted to internal images, it can be determined that an external / internal switch has occurred. Further, the switch of the identified working state can be recognized.

[0085] For another example, between examinations for two patients, a patient switch can be determined based on an analysis of the input data collected by the endoscope 210. Specifically, when the image sequence includes in-vivo images, ex-vivo images, and then in-vivo images different from the previous examination, a patient switch can be determined. For another example, an endoscopic examination can involve different human body parts. At this time, a switch in the examination location can be determined based on an analysis of the images collected by the endoscope 210. Specifically, switches in examination locations such as esophagoscopy, gastroscopy, duodenoscopy, and colonoscopy can be determined. Using the exemplary implementation of the present disclosure, relevant configurations of medical assistance operations can be selected based on the detected switch. For example, corresponding motion models can be selected for gastroscopy and colonoscopy.

[0086] It will be understood that endoscopic examination requires inserting the endoscope 210 into the patient's body, and preparatory work needs to be performed before the examination. According to the exemplary implementation of the present disclosure, based on the input data, the preparation status of the person performing the endoscopic examination can be identified. This preparation status describes the qualification of the person's physical condition for performing endoscopic examination. For patients, preparatory work includes, for example, fasting, emptying the digestive tract, taking medications as prescribed by the doctor to empty and clean the digestive tract, etc. For doctors, preparatory work includes, for example, cleaning the stomach and blowing air into the stomach for the folded parts to be examined, etc.

[0087] Specifically, if the collected gastroscopy images include food residues, etc., it can be determined that the patient's preparation status is poor and does not meet the requirement of emptying the digestive tract. If the collected gastroscopy images include a large amount of secretions, etc., it can be determined that the doctor's cleaning operation is insufficient, and the doctor can be prompted to further perform the cleaning operation. Further, the identified degree of preparation can be output. The output can be in the form of display or other prompting methods. Using the exemplary implementation of the present disclosure, corresponding precautions can be respectively prompted to the patient and the doctor based on the preparation status.

[0088] It will be understood that although specific examples of determining the preparation status based on the images in the input data 230 are described above, according to the exemplary implementation of the present disclosure, the preparation status can also be determined based on dedicated sensors deployed at the endoscope (for example, sensors for monitoring in-vivo environmental parameters).

[0089] During the movement of the endoscope 210 within the human body, if the movement is too fast, key point positions may be missed, and it may also cause discomfort such as nausea and pain to the patient. Therefore, it is also desirable to monitor the movement state of the endoscope 210 based on the smoothness of the movement, so that the movement trajectory of the endoscope can cover all key point positions and reduce the discomfort of the patient. According to an exemplary implementation of the present disclosure, the smoothness of the movement of the endoscope 210 can be identified based on a set of time points when the endoscope 210 reaches a set of key point positions. The smoothness here can represent the smoothness of the movement of the endoscope 210 within the patient's body. Further, the identified smoothness can be displayed.

[0090] According to an exemplary implementation of the present disclosure, a speed evaluation of the movement speed of the endoscope 210 can be determined based on the smoothness. For example, if the endoscope 210 moves a large distance in a short time, it indicates that the movement of the endoscope 210 is relatively intense and should be avoided. At this time, a lower speed evaluation can be given, and the doctor can be prompted that the movement is too intense and the speed should be reduced to prevent the situation of missing key point positions.

[0091] For another example, if the movement of the endoscope 210 is moderate, a higher speed evaluation can be given. For another example, if the endoscope 210 only moves a small distance in a long time, although the movement is relatively smooth at this time, the overall time of the endoscopy will be increased. Therefore, the speed evaluation can be reduced and the doctor can be prompted to move the endoscope 210 to the next destination position as soon as possible. For another example, the speed distribution during the endoscopy can also be monitored. Assuming that the endoscope 210 stays near 5 key point positions in the first half of the entire examination time and quickly passes through the remaining 33 key point positions in the second half of the time, the second half of the examination is likely to be insufficient. At this time, a lower speed evaluation can be given.

[0092] Compared with the technical solution of determining whether the doctor's operation is sufficient based on whether the overall examination time reaches the expected time (for example, 10 minutes), using the exemplary implementation of the present disclosure, it is possible to determine whether the doctor's operation meets the predetermined standard based on the speed distribution of the endoscope 210. It will be understood that although specific examples of determining the smoothness based on the images in the input data 230 are described above, according to an exemplary implementation of the present disclosure, the smoothness can also be determined based on a speed sensor deployed at the endoscope.

[0093] The details of the medical assistance operation method have been described above with reference to 2 to Figure 9 will be described below. With reference to Figure 10 each module in the medical assistance operation device will be described. Figure 10FIG. 1000 schematically shows a block diagram of a medical assistance operation device 1010 (or a medical assistance information processing device 1010) according to an exemplary implementation of the present disclosure. As Figure 10 shown, there is provided a medical assistance operation device 1010, including: an input module 1012 configured to obtain input data from an endoscope; and an output module 1018 configured to output information related to the operation behavior of the endoscope determined based on the input data.

[0094] According to an exemplary implementation of the present disclosure, the input data includes image data collected at multiple positions during the movement of the endoscope.

[0095] According to an exemplary implementation of the present disclosure, the device 1010 further includes: a processing module 1014 configured to determine the information related to the operation behavior of the endoscope based on the input data.

[0096] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to determine the next destination position of the endoscope based on the input data.

[0097] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: determine the movement trajectory of the endoscope based on the input data.

[0098] According to an exemplary implementation of the present disclosure, the movement trajectory is represented by a predetermined set of key point positions.

[0099] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: determine the next destination position of the endoscope based on the movement trajectory.

[0100] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: determine a set of candidate positions of the endoscope at the next time point; determine the evaluation of each candidate position in the set of candidate positions; and select the next destination position from the set of candidate positions based on the determined evaluation.

[0101] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: for a given candidate position in the set of candidate positions, generate a candidate movement trajectory of the endoscope based on the movement trajectory and the given candidate position; and determine the evaluation of the candidate position based on the candidate movement trajectory and the predetermined movement trajectory of the endoscopy.

[0102] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: determine the evaluation of the movement trajectory.

[0103] According to an exemplary implementation of the present disclosure, the apparatus 1010 further includes an identification module 1016 configured to: identify the working state of the endoscope, where the working state includes at least any one of the following: patient identification, inside or outside the body, and examination site.

[0104] According to an exemplary implementation of the present disclosure, the identification module 1016 is further configured to: identify the switching of the working state.

[0105] According to an exemplary implementation of the present disclosure, the apparatus 1010 further includes an identification module 1016 configured to: based on the input data, identify the preparation state of the endoscope examination site, where the preparation state indicates the qualification degree of the examination site for performing the endoscope examination.

[0106] According to an exemplary implementation of the present disclosure, the apparatus 1010 further includes an identification module 1016 configured to: based on a set of time points when the endoscope reaches a set of key point positions, determine the smoothness of the movement of the endoscope.

[0107] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: obtain a set of key point positions based on the input data; and determine the movement trajectory based on the time sequence of the input data associated with the key point positions.

[0108] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: determine a set of image data mapped to the key point positions in the input data; respectively determine the image quality evaluation of the set of image data based on the image quality of the set of image data; and select the image data in the set of image data for storage based on the determined image quality evaluation.

[0109] According to an exemplary implementation of the present disclosure, the apparatus 1010 further includes an identification module 1016 configured to: based on the selected image data, identify the image abnormality at the key point position.

[0110] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: obtain a first model describing the endoscope examination, where the first model includes the association relationship between the sample input data collected at multiple sample positions during the execution of the endoscope examination and the sample movement trajectory of the endoscope for collecting the sample input data; and determine the movement trajectory based on the first model and the input data.

[0111] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: obtain the sample input data collected in the endoscope examination performed in accordance with the endoscope operation specifications; obtain the sample movement trajectory associated with the sample input data; and train the first model based on the sample input data and the sample movement trajectory.

[0112] According to an exemplary implementation of the present disclosure, the processing module 1014 is further configured to: obtain a second model describing the endoscopy, where the second model includes an association relationship between sample input data collected at a plurality of sample positions during the execution of the endoscopy and corresponding key point positions of the plurality of positions where the sample input data is collected; and determine a movement trajectory of the endoscope based on the second model, the input data, and the acquisition time of the image data.

[0113] According to an exemplary implementation of the present disclosure, determining information related to the operation behavior of the endoscope based on the input data includes determining at least any one of the following: the current position of the endoscope; the image data collected at the current position; the movement trajectory of the endoscope; the next destination position of the endoscope; the statistical information of the input data; and the statistical information of the operation behavior.

[0114] According to an exemplary implementation of the present disclosure, the input data includes at least any one of the following: video data; a set of image sequences arranged in chronological order; and a plurality of image data with time information.

[0115] According to an exemplary implementation of the present disclosure, the output module 1018 is further configured to: transmit information related to the operation behavior of the endoscope.

[0116] According to an exemplary implementation of the present disclosure, each module of the medical assistance operation device 1010 may be implemented by one or more processing circuits.

[0117] Figure 11 A schematic block diagram of an example device 1100 that can be used to implement an exemplary implementation of the present disclosure is shown. For example, as Figure 1 shown, the computing device 130 can be implemented by the device 1100. As shown in the figure, the device 1100 includes a central processing unit (CPU) 1101, which can execute various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1102 or computer program instructions loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0118] A plurality of components in device 1100 are connected to I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a magnetic disk, an optical disc, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0119] Each of the processes and treatments described above, such as method 300, may be executed by processing unit 1101. For example, in some exemplary implementations, method 300 may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as storage unit 1108. In some exemplary implementations, part or all of the computer program may be loaded and / or installed onto device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by CPU 1101, one or more actions of method 300 described above may be executed.

[0120] According to an exemplary implementation of the present disclosure, a medical assistance operation device is provided, including: at least one processing unit; at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit causing the device to execute method 300 as described above.

[0121] The present disclosure may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for performing various aspects of the present disclosure.

[0122] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0123] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or can be downloaded to an external computer or an external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0124] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some exemplary implementations, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0125] Aspects of the present disclosure are described herein with reference to the flowchart and / or block diagram of a method, apparatus (system), and computer program product according to exemplary implementations of the present disclosure. It should be understood that each block of the flowchart and / or block diagram, and the combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0126] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is produced that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause a computer, a programmable data processing device, and / or other devices to work in a specific manner. Thus, the computer-readable medium storing the instructions includes a manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0127] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0128] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various exemplary implementations of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0129] While various embodiments of the present disclosure have been described above, the description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A medical assistance operation method, comprising: Obtaining input data from an endoscope, the input data including image data collected at multiple positions during the movement of the endoscope; And Determining information related to the operation behavior of the endoscope based on the input data, including: Based on the input data, determining the movement trajectory of the endoscope, including: Obtaining a first model describing endoscopy, the first model being a machine learning model and including an association relationship between sample input data and a sample movement trajectory, the sample input data being input data collected at multiple sample positions during the execution of endoscopy, and the sample movement trajectory being the movement trajectory of the endoscope for collecting the sample input data; and Based on the first model and the input data, determining the movement trajectory; and Based on the movement trajectory, determining the next destination position of the endoscope.

2. The method according to claim 1, wherein the movement trajectory is represented by a predetermined set of key point positions.

3. The method according to claim 1, wherein based on the movement trajectory, determining the next destination position of the endoscope includes: Determining a set of candidate positions of the endoscope at the next time point; Determining the evaluation of each candidate position in the set of candidate positions; And Based on the determined evaluation, selecting the next destination position from the set of candidate positions.

4. The method according to claim 3, wherein determining an evaluation of each candidate position in the set of candidate positions comprises: For a given candidate position in the set of candidate positions, Based on the movement trajectory and the given candidate position, generating a candidate movement trajectory of the endoscope; And Based on the candidate movement trajectory and the predetermined movement trajectory of the endoscopy, determining the evaluation of the candidate position.

5. The method according to claim 1, further comprising: Determining the evaluation of the movement trajectory.

6. The method according to claim 1, further comprising: Identifying the working state of the endoscope, the working state including at least any one of the following: patient identification, inside / outside the body, and examination site.

7. The method according to claim 6, further comprising: Identifying the switching of the working state.

8. The method according to claim 1, further comprising: Based on the input data, identifying the preparation state of the endoscopy examination site, the preparation state indicating the qualification degree of the examination site for performing the endoscopy examination.

9. The method according to claim 2, further comprising: Based on a set of time points when the endoscope reaches the set of key point positions, determining the smoothness of the movement of the endoscope.

10. The method according to claim 2, wherein based on the input data, determining the movement trajectory of the endoscope includes: Based on the input data, obtaining the set of key point positions; And Based on the time sequence of the input data associated with the key point positions, determining the movement trajectory.

11. The method according to claim 10, further comprising: Determining a set of image data in the input data that is mapped to the key point positions; Based on the image quality of the set of image data, respectively determining the image quality evaluation of the set of image data; And Based on the determined image quality evaluation, selecting the image data in the set of image data for storage.

12. The method according to claim 11, further comprising: Based on the selected image data, identifying image abnormalities at the key point positions.

13. The method according to claim 1, wherein obtaining the first model describing the endoscopic examination comprises: obtaining the sample input data collected in the endoscopic examination performed according to the endoscopic operation specifications; obtaining the sample movement trajectory associated with the sample input data; and training the first model based on the sample input data and the sample movement trajectory.

14. The method according to claim 1, wherein determining the movement trajectory of the endoscope based on the input data comprises: obtaining a second model describing the endoscopic examination, the second model being a machine learning model and including an association relationship between the sample input data collected at a plurality of sample positions during the performance of the endoscopic examination and the corresponding key point positions of the plurality of positions at which the sample input data is collected; and determining the movement trajectory of the endoscope based on the second model, the input data, and the acquisition time of the image data.

15. The method according to claim 1, wherein determining information related to the operation behavior of the endoscope based on the input data comprises determining at least any one of the following: the current position of the endoscope; the image data collected at the current position; the movement trajectory of the endoscope; the next destination position of the endoscope; the statistical information of the input data; and the statistical information of the operation behavior.

16. The method according to claim 1, wherein the input data comprises at least any one of the following: video data; a set of image sequences arranged in chronological order; and a plurality of image data with time information.

17. The method according to claim 1, further comprising: Transmit information related to the operation behavior of the endoscope.

18. A medical assistance operation device, comprising: an input module configured to obtain input data from an endoscope, the input data including image data collected at a plurality of positions during the movement of the endoscope; and an output module configured to output information related to the operation behavior of the endoscope determined based on the input data, including: a processing module configured to: determine the movement trajectory of the endoscope based on the input data, including: obtaining a first model describing the endoscopic examination, the first model being a machine learning model and including an association relationship between sample input data and a sample movement trajectory, the sample input data being input data collected at a plurality of sample positions during the performance of the endoscopic examination, and the sample movement trajectory being the movement trajectory of the endoscope for collecting the sample input data; and determining the movement trajectory based on the first model and the input data; and determining the next destination position of the endoscope based on the movement trajectory.

19. The device according to claim 18, wherein the movement trajectory is represented by a predetermined set of key point positions.

20. The device according to claim 18, wherein the processing module is further configured to: determine a set of candidate positions of the endoscope at the next time point; determine the evaluation of each candidate position in the set of candidate positions; and Based on the determined evaluation, select the next destination location from the set of candidate locations.

21. The apparatus according to claim 20, wherein the processing module is further configured to: for a given candidate location in the set of candidate locations, generate a candidate motion trajectory of the endoscope based on the motion trajectory and the given candidate location; and determine the evaluation of the candidate location based on the candidate motion trajectory and a predetermined motion trajectory of the endoscopy.

22. The apparatus according to claim 21, wherein the processing module is further configured to: determine an evaluation of the motion trajectory.

23. The apparatus according to claim 18, further comprising an identification module configured to: identify an operating state of the endoscope, the operating state including at least any one of the following: patient identification, inside / outside the body, and examination site.

24. The apparatus according to claim 23, wherein the identification module is further configured to: identify a change in the operating state.

25. The apparatus according to claim 18, further comprising an identification module configured to: identify a preparation state of an endoscopy examination site based on the input data, the preparation state indicating the qualification degree of the examination site for performing the endoscopy examination.

26. The apparatus according to claim 19, further comprising an identification module configured to: determine the smoothness of the motion of the endoscope based on a set of time points at which the endoscope reaches the set of key point locations.

27. The apparatus according to claim 19, wherein the processing module is further configured to: acquire the set of key point locations based on the input data; and determine the motion trajectory based on the time sequence of the input data associated with the key point locations.

28. The apparatus according to claim 27, wherein the processing module is further configured to: determine a set of image data mapped to the key point locations in the input data; respectively determine an image quality evaluation of the set of image data based on the image quality of the set of image data; and select image data from the set of image data for storage based on the determined image quality evaluation.

29. The apparatus according to claim 28, further comprising an identification module configured to: identify an image anomaly at the key point location based on the selected image data.

30. The apparatus according to claim 18, wherein the processing module is further configured to: acquire the sample input data collected in an endoscopy examination performed in accordance with an endoscope operation specification; acquire the sample motion trajectory associated with the sample input data; and train the first model based on the sample input data and the sample motion trajectory.

31. The apparatus according to claim 18, wherein the processing module is further configured to: Obtain a second model for describing endoscopic examination, where the second model is a machine learning model and includes the association relationship between the sample input data collected at multiple sample positions during the execution of endoscopic examination and the corresponding key point positions of the multiple positions where the sample input data is collected; and Based on the second model, the input data, and the acquisition time of the image data, determine the movement trajectory of the endoscope.

32. The apparatus according to claim 18, wherein determining information related to the operation behavior of the endoscope based on the input data includes determining at least any one of the following: The current position of the endoscope; The image data acquired at the current position; The movement trajectory of the endoscope; The next destination position of the endoscope; The statistical information of the input data; and The statistical information of the operation behavior.

33. The apparatus according to claim 18, wherein the input data includes at least any one of the following: Video data; A set of image sequences arranged in chronological order; and Multiple image data with time information.

34. The apparatus according to claim 18, further comprising: Transmit information related to the operation behavior of the endoscope.

35. A medical assistance operation device, comprising: At least one processing unit; At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the device to perform the method according to any one of claims 1-17.

36. A computer-readable storage medium having computer-readable program instructions stored thereon for performing the method according to any one of claims 1-17.

Images