Video processing method, device, electronic device, medium, and computer program

Abstract

To provide a technical idea which evaluates stability of collecting operation by analyzing an endoscope video.SOLUTION: A method includes: acquisition of an endoscopic video obtained by endoscope collecting operation; and determination of local speed information related to the endoscope collecting operation on the basis of site estimation of a current frame of the endoscope video, site estimation of at least one prior frame, and site transfer information. The method further includes determination of stability of the endoscope collecting operation at least on the basis of the local speed information. It is possible to acquire stability information at a video frame level in real time, to appropriately assist adjustment of too-fast or too-slow collecting operation by an operator, and to obtain such stability information by considering a site acquisition order during an endoscope test, so that it is accurate and comprehensive.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present disclosure relate to the field of image processing technology, and more particularly, to video processing methods, apparatus, electronic devices, computer-readable storage media, and computer program products. [Background technology]

[0002] Endoscopy is a type of optical equipment inspection that is sent from outside the body through the natural lumen of the human body into the body to inspect for diseases inside the body. An endoscope may be composed of a bendable part, a light source, and a set of lenses. When in use, the operator introduces the endoscope into the organ to be inspected, manipulates and moves the endoscope, and can directly look into the relevant part and record images and videos.

[0003] During endoscopic examination, there are differences in the operation levels of different operators, and one of the indicators is the stability of operation. Poor operation stability refers to the forward or backward operation of the endoscope being too fast or too slow, or switching between different parts randomly. Good operation stability can perform stable operation within a limited operation time, observe each part of an organ and / or lesion more clearly and comprehensively, and at the same time avoid or reduce injury to the human body. Therefore, it is necessary to introduce an accurate stability detection mechanism into the endoscopic system to improve the operation level of endoscopic examination and improve the experience of the examinee. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of this, an embodiment of the present disclosure puts forward a technical solution for evaluating the stability of the sampling operation by analyzing endoscopic video. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a video processing method, the method including: acquiring an endoscopic video obtained by an endoscopic sampling operation, determining local velocity information associated with the endoscopic sampling operation based on a part estimation of a current frame of the endoscopic video, a part estimation of at least one previous frame, and part relocation information, and determining a stability of the endoscopic sampling operation based on at least the local velocity information.

[0006] According to a second aspect of the present disclosure, there is provided a video processing device, comprising: a video acquisition unit configured to acquire an endoscopic video obtained by an endoscopic sampling operation, a local speed determination unit configured to determine local speed information related to the endoscopic sampling operation based on a part estimation of a current frame of the endoscopic video, a part estimation of at least one previous frame, and part relocation information, and a stability determination unit configured to determine a stability of the endoscopic sampling operation based on at least the local speed information.

[0007] According to a third aspect of the present disclosure, there is provided an electronic device comprising at least one processing unit and a memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions being executed by the at least one processing unit to perform the first aspect of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium comprising machine-executable instructions which, when executed by a device, cause the device to perform the method according to the first aspect of the present disclosure.

[0009] According to a fifth aspect of the present disclosure, there is provided a computer program product comprising machine executable instructions which, when executed by a device, cause said device to perform a method according to the first aspect of the present disclosure.

[0010] The following sections are provided to introduce in a simplified form a selection of concepts that are further described in the specific embodiments below, and are not intended to indicate critical or required features of the disclosure, nor are they intended to limit the scope of the disclosure. [Brief description of the drawings]

[0011] The above and other objects, features, and advantages of the present disclosure will become more apparent by describing in more detail the exemplary embodiments of the present disclosure with reference to the accompanying drawings, in which like reference numerals generally represent like parts in the exemplary embodiments of the present disclosure. [Figure 1] 1 illustrates a schematic diagram of an exemplary environment in which several embodiments of the present disclosure may be implemented. [Diagram 2] 1 shows a schematic flowchart of a video processing method according to an embodiment of the present disclosure. [Diagram 3] 1 shows a schematic diagram of a stability estimation system according to an embodiment of the present disclosure; [Figure 4] 1 shows a schematic flow chart of a part estimation process according to an embodiment of the present disclosure. [Diagram 5] 1 illustrates a schematic flow chart of a process for determining regional velocity information based on regional transfer information according to an embodiment of the present disclosure. [Figure 6] 1 shows a schematic flow chart of a process for determining local stability according to an embodiment of the present disclosure. [Figure 7] 1 shows a schematic block diagram of a video processing device according to an embodiment of the present disclosure. [Figure 8] FIG. 1 shows a schematic block diagram of an example device that can be used to implement the teachings of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Preferred embodiments of the present disclosure are shown in the accompanying drawings, but it should be understood that the present disclosure should not be limited to the embodiments described herein and can be implemented in various forms. Rather, these embodiments are provided to make the present disclosure clearer and more complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0013] As used herein, the term "including" and variations thereof refer to an open inclusion, i.e., "including, but not limited to." Unless otherwise noted, the term "or" refers to "and / or." The term "based on" refers to "based at least in part on." The terms "one exemplary embodiment" and "one embodiment" refer to "at least one exemplary embodiment." The term "another embodiment" refers to "at least one other embodiment." Terms such as "first," "second," and the like can refer to different objects or the same object. Other explicit and implicit definitions may be included below.

[0014] In endoscopic examination, the device operator moves the lens of the endoscope to observe the internal parts of the body, for example, by using a gastroscope or intestine to examine the corresponding parts of the human body and generate the corresponding video or image. Since the lens and catheter of the endoscope enter the human body and cause discomfort to the examinee, the endoscopic examination requires the operator to have good operating stability, so that each part and lesion of the organ can be clearly and comprehensively observed while avoiding or reducing the damage to the human body caused by the examination.

[0015] Some conventional methods evaluate the stability of endoscopic operation in real time through endoscopic video and help improve the operation level. For example, one method evaluates the forward and backward speed of the lens by analyzing the stability of the image frames of the video. Since the degree of curvature is different for each part of the body, even if the operation is very stable, the difference between the frames of different parts may be large, so it is difficult to evaluate the stability at all parts with the same threshold. Some other methods calculate the movement time of the intestinal endoscope between each part, evaluate whether the actual movement speed meets the constraints, and determine the stability of the endoscope operation. However, such a method of estimating the stability between different parts is too granular. For example, a gastroscope can only estimate 30 stabilities if there are 30 locations, and the stability of the operation cannot be detected immediately.

[0016] In view of this, a stability estimation method with higher accuracy and good real-time performance is required. In the method according to the embodiment of the present disclosure, first, an endoscopic video obtained by an endoscopic collection operation is acquired, and the video may include a series of video frames. Then, the corresponding stability is analyzed at the video frame level. Specifically, a site estimation of each frame of the video is acquired, and based on the site estimation of the current frame, the site estimation of the previous frame, and predefined site transfer information, local speed information corresponding to the current frame is determined, and the stability of the endoscopic collection operation is determined using the local speed information. Note that the speed information mentioned in the present disclosure reflects the stability of the operation and is not limited to speed in the conventional physical sense. Hereinafter, the details of the implementation of the embodiment of the present disclosure will be described in detail with reference to Figs. 1 to 8.

[0017] FIG. 1 illustrates a schematic diagram of an exemplary environment 100 in which embodiments of the present disclosure may be implemented. As illustrated, the environment 100 relates to an endoscopic system including a control device 101 operable by an operator to control the movement of a lens 102 of the endoscope within the human body to capture images and videos. The captured images and videos may be transmitted and stored in a computing device 105. A display 103 is coupled to the computing device 105 and may display images and videos captured through the lens 102 in real time. Although FIG. 1 illustrates an exemplary gastroendoscopic system, it should be understood that embodiments of the present disclosure may be applicable to other endoscopic systems, including, but not limited to, otolaryngological endoscopes, oral endoscopes, dental endoscopes, neuroscopes, urethrocystoscopes, electrotomies, laparoscopes, arthroscopes, sinusoscopes, laryngoscopes, enteroscopes, and the like.

[0018] The computing device 105 may be a general-purpose computing device or a device dedicated to endoscopy, and is configured for image and video processing, particularly medical image processing. The computing device 105 may be a terminal or a server device. If the computing device 105 is a server, it may be an independent server, or a server network or server cluster of servers, including but not limited to a computer, a network host, a single network server, a set of multiple network servers, or a cloud server built with multiple servers. The computing device 105 may be a desktop terminal or a mobile terminal, such as but not limited to a desktop, a mobile phone, a tablet, a laptop, medical support equipment, etc.

[0019] As described above, the lens 102 needs to enter the body during use, which may cause discomfort to the subject, so the lens 102 needs to be moved as fast as possible, but in order to ensure the observation effect of the endoscope, the movement speed at a specific part needs to be slowed down as much as possible to observe for a longer period of time and maintain good stability. In other words, the endoscope operator needs to more precisely control the stability of the lens 102 to improve the examination efficiency and accuracy. According to an embodiment of the present disclosure, the computing device 105 can analyze the endoscopic video to evaluate the stability of the collection operation and help the operator improve his / her operation level.

[0020] An exemplary environment in which embodiments of the present disclosure may be implemented has been described above with reference to Fig. 1. It should be understood that Fig. 1 is merely schematic, and the environment may include more modules or systems, or may omit some modules or systems, or may recombine the illustrated modules or systems. Embodiments of the present disclosure may be implemented in environments different from the environment illustrated in Fig. 1, and the present disclosure is not limited thereto.

[0021] 2 shows a schematic flow chart of a video processing method 200 according to an embodiment of the present disclosure. The method 200 may be implemented, for example, by the computing device 105 shown in FIG. 1. It should be understood that the method 200 may also include additional operations not shown and / or omit operations shown, and the scope of the present disclosure is not limited in this respect. The method 200 will now be described in detail with reference to FIG. 1.

[0022] In block 210, an endoscopic video obtained by an endoscopic sampling operation is acquired. During an endoscopy, an operator steers the control device 101 to control the movement of the lens 102 within the human body and the orientation of the lens 102 to capture and generate a video. The video may be transmitted to the computing device 105. In some embodiments, the computing device 105 may acquire the video obtained by the endoscopic sampling operation in real time or near real time and analyze the video. Alternatively, the video may be stored in the computing device 105 and analyzed retrospectively after the endoscopy is completed.

[0023] At block 220, local velocity information associated with the endoscopic harvesting operation is determined based on the location estimate of the current frame of the endoscopic video, the location estimate of at least one previous frame, and the location transfer information. For each frame of the endoscopic video, local velocity information corresponding to that frame may be determined.

[0024] The computing device 105 may generate a region estimation result for each video frame in the endoscopic video. The region estimation result may include likelihood values ​​for multiple regions of the inspected organ. For example, the region estimation result may have the form of a vector, with each component of the vector indicating the likelihood or probability that the frame belongs to the corresponding region. Among the region estimation results, the region corresponding to the maximum likelihood or probability may be considered as the region in which the frame is located.

[0025] The site transfer information may include site transfer probability information (also referred to as a site transfer probability map). In some embodiments, the site transfer probability may have a matrix format, where the rows and columns of the matrix correspond to each site, and the elements of the matrix indicate the probability of transferring from one site to another site during endoscopy (e.g., the probability of transferring from the cardia to the stomach is 0.8). The site transfer information represents the examination sequence of the endoscope and is used to constrain whether the transition between each site is normative or not, and belongs to the global constraint. The site transfer information can be used to evaluate whether the transfer from one site indicated by a previous frame (e.g., a plurality of previous frames as a whole) to a site indicated by a current frame meets this constraint, and further evaluate the current stability. The site transfer information may be pre-stored in the computing device 105. Table 1 below shows an example of site transfer information. [Table 1] The rows in Table 1 represent the regions obtained from the results of region estimation in the previous frame, the columns represent the regions obtained from the results of region estimation in the current frame, and the values ​​in Table 1 represent the transfer probability between the two regions. For example, if the result of region estimation in the previous frame indicates "esophagus" and the result of region estimation in the current frame indicates "cardia," local velocity information may be determined based on the probability "0.8" shown in Table 1.

[0026] In some embodiments, the probability obtained from the region relocation information may be used as a part of the local velocity information corresponding to the current frame. This part may be referred to as the first velocity information. In some embodiments, the local velocity information may include information based on the inter-frame similarity information, also referred to as the second velocity information, in addition to the velocity information based on the region relocation information. The computing device 105 may combine the first velocity information and the second velocity information to form the local velocity information corresponding to the current frame. For example, the computing device 105 may determine a linear combination of the first velocity information and the second velocity information as the local velocity information corresponding to the current frame. This will be described in detail below with reference to FIG. 3 to FIG. 6.

[0027] At block 230, a stability of the endoscopic harvesting operation is determined based on at least the local velocity information. As described above, the computing device 105 determines the local velocity information for each frame. In response, the computing device 105 may determine a stability of the endoscopic harvesting operation based on the local velocity information corresponding to each frame. The stability for each frame may be referred to as a local stability. The local stability may take a percentage form and may be classified as "too fast", "stable", or "too slow".

[0028] In some embodiments, the computing device 105 may determine the local stability based on the frame-level local velocity information and a suitable reference local velocity. For example, when evaluating the stability in real time, the computing device 105 may determine the local stability as "stable" if the local velocity information is within a predetermined range compared to the reference local velocity, and may otherwise issue a warning of "too slow" or "too fast" to help the operator adjust the capture operation. In some embodiments, the computing device 105 may provide an overall stability of the capture operation after the video capture is completed, for example, the computing device 105 may determine the overall stability as an average value of the local stabilities of multiple frames or all frames. The computing device 105 may also calculate the stability near a particular location.

[0029] According to the method 200 shown in Fig. 2, the current stability information at the video frame level can be obtained in real time, so as to help the operator to timely understand the stability of the current operation and timely adjust the collection operation, and such stability information takes into account the region collection order information during the endoscopy, so that it is more accurate and comprehensive.

[0030] Fig. 3 shows a schematic diagram of a stability estimation system 300 according to an embodiment of the present disclosure. The stability estimation system 300 may be implemented in the computing device 105 in the form of software or hardware and is suitable for implementing the method 200 described with reference to Fig. 2. It should be understood that the stability estimation system 300 shown in Fig. 3 is only exemplary and may include more or less modules or units.

[0031] In the system 300, an endoscopic video 310 is provided to a stability module 320. The video 310 includes a sequence of frames F1 through Ft, where t is any integer greater than 1. Frame Ft may be referred to as a current frame, and frames F1 through Ft-1 may be referred to as previous frames. The stability module 320 may be configured to calculate a stability measure corresponding to each frame and an overall stability measure of the endoscopic video 310.

[0032] As shown in the figure, the stability module 320 includes a part estimation unit 322, a similarity estimation unit 324, and pre-stored part transfer information 302. The part estimation unit 322 is used to generate part estimation results for frames F1 to Ft. The similarity estimation unit 324 is used to generate a similarity between frame Ft and its preceding frame. The part transfer information 302 includes information on the possibility of transfer between parts according to the endoscopic examination order.

[0033] FIG. 4 shows a schematic flow chart of a region estimation process 400 according to an embodiment of the present disclosure. The region estimation unit 322 may be configured to implement the region estimation process 400 shown in FIG. 4. In block 401, the region estimation unit 322 receives an input frame. In block 402, the region estimation unit 322 performs quality evaluation on the input frame. For example, first, invalid images, including but not limited to blurred images, overexposed images, contaminated images, and non-standard regions, etc., in the video 310 are removed by image quality evaluation, and thus video frames F1-Ft with higher quality are obtained by quality screening.

[0034] In block 403, the region estimation unit 322 performs region identification on the frames that pass the quality evaluation. In some embodiments, the region estimation unit 322 may obtain a region estimation result for the current frame Ft and a region estimation result for the previous frames F1 to Ft-1 based on the prediction of the deep network. For example, the deep network may be trained by collecting labeled samples, and the trained deep network may be used to predict the region to which each frame belongs. Alternatively, a feature extraction classifier (e.g., a support vector machine) may be employed to perform training and prediction, or image features may be extracted, a weighting scheme may be employed to obtain category probabilities, and a threshold may be compared, etc.

[0035] In block 404, the region estimation unit 322 outputs a region estimation result. The region estimation result may have the form of a vector including a likelihood value or probability for each of a plurality of regions. Note that such likelihood values ​​or probabilities are only used to indicate the relative likelihood, not the absolute likelihood, that the frame belongs to each region. Thus, the sum of the likelihood values ​​in the region estimation results is not necessarily "1". In some implementations, the region in the region estimation results for a frame that corresponds to the maximum likelihood (i.e., the frame is most likely to belong to this region) may be determined as the region in which the frame is located.

[0036] Continuing to refer to FIG. 3, the similarity estimation unit 324 may be used to calculate the similarity between the current frame Ft and one or more previous frames. In some embodiments, the similarity estimation unit 324 calculates the individual similarity between the current frame and any previous frame, and combines these individual similarities to obtain the similarity between the current frame and the previous frame. Methods for calculating the individual similarity may include, but are not limited to, sparse optical flow field, dense optical flow field, image feature extraction and matching, perceptual hashing, dual channel deep network, etc. The similarity between the current frame and the previous frame may be calculated by the following equation:

number

[0037] The result of part estimation for each video frame output by the part estimation unit 322, the similarity output by the similarity estimation unit 324, and the part relocation information 302 may be provided to the local speed information estimation unit 326 to calculate local speed information for the current frame Ft. In some embodiments, the local speed information estimation unit 326 may determine first speed information V1 based on the result of part estimation for each frame F1-Ft from the part estimation unit 322 and the part relocation information, and may determine second speed information V2 based on the similarity from the similarity estimation unit 324. Then, the local speed information estimation unit 326 may combine the first speed information and the second speed information to determine local speed information V(x) corresponding to the current frame Ft.

[0038] 5 shows a schematic flow chart of a process 500 for determining local velocity information based on regional transfer information according to an embodiment of the present disclosure. The local velocity information estimation unit 326 may be configured to perform the process 500.

[0039] In block 501, it is determined whether the maximum likelihood value of the region estimation result of the current frame exceeds a threshold T1. In block 501, it may be determined that the current frame Ft is a higher quality video frame, for example, the quality exceeds a threshold T2. If this condition is not met, the method 500 proceeds to block 504 and uses the first velocity information of the previous frame as the first velocity information V1 of the current frame. If the condition regarding the likelihood threshold and / or the quality threshold is met, the current frame may be considered to be a video frame that can represent entering a specific region, and further determines the first velocity information V1 based on the region corresponding to the current frame and another region corresponding to the previous frame and the region transfer information. In such a case, the method 500 proceeds to block 502.

[0040] At block 502, a first region corresponding to the current frame and a second region corresponding to another frame, different from the first region, are determined based on the region estimation of the current frame and the region estimation of at least one previous frame. As described above, a result of the region estimation of either the region estimation of the current frame or the region estimation of the at least one previous frame may include a likelihood value for each of the multiple regions. In some embodiments, determining the first region corresponding to the current frame may include determining the first region based on the region estimation of the current frame. For example, a region corresponding to a maximum likelihood among the results of the region estimation of the current frame may be determined as the first region in which the current frame is located.

[0041] To determine a second region different from the first region, the results of region estimation of the previous frames F1 to Ft-1 are analyzed. Specifically, based on the region estimation of these previous frames, another frame is determined, and the region estimation of this another frame has the maximum likelihood value for the other region different from the first region. In other words, in the previous frame, the frame number of the non-current frame region with the highest probability is obtained. Then, according to the result of the region estimation of the determined other frame, the second region is determined, and the specific manner may be the same as that of the first region.

[0042] In block 503, a first velocity information is determined based on the first site, the second site, and the site relocation information. In some embodiments, the site relocation information may be, for example, a site relocation probability based on an order of endoscopy sites (e.g., Table 1), and the first velocity information V1 may be determined by looking up the table.

[0043] The local velocity information estimation unit 326 may determine second velocity information V2 corresponding to the current frame based on the similarity Sim(x) from the similarity estimation unit 324. The second velocity information V2 may be determined by the following equation: V2=exp(-Sim(x)) / sigma^2) (2) Here, sigma is a fixed parameter, which may be determined empirically.

[0044] Then, the local rate information estimation unit 326 may determine the local rate information V(x) corresponding to the current frame according to the first rate information V1 and the second rate information V2, as shown in the following equation: V(x)=w1*V1+w2*V2 (3) Here, w1 and w2 are fixed parameters and may be determined empirically.

[0045] Returning to Fig. 3, the local velocity information estimation unit 326 may output the region and local velocity information of the previous frame and the current frame Ft to the stability estimation unit 328. The stability estimation unit 328 may be configured to evaluate the local stability according to the local velocity information and the reference endoscopic video 304. The reference endoscopic video 304 may include pre-stored standard and stable adjacent region video segments. The stability estimation unit 328 determines the local stability by comparing the local velocity information from the local velocity information estimation unit 326 with the reference velocity information indicated by the reference endoscopic video 304.

[0046] 6 shows a schematic flow chart of a process 600 for determining a local stability according to an embodiment of the present disclosure. The stability estimation unit 328 may be configured to perform the process 600 to determine the local stability of a current frame.

[0047] In block 610, according to the first portion corresponding to the current frame and its adjacent region portion, a corresponding segment in the reference endoscopic video is determined. The stability estimation unit 328 may obtain a standard and stable video segment of the adjacent region portion according to the first portion and the preceding portion of the current frame, and obtain an average velocity SV per frame in the video segment. In some embodiments, the average velocity SV per frame may be calculated and stored in advance by the local velocity information estimation unit 326, and the calculation method may be the same as that for the video 310.

[0048] In block 620, the reference speed information is determined based on the corresponding segment. The reference speed information depends on the given video segment, so that the reference speed information is different for different parts. In some implementations, the stability estimation unit 328 may obtain the corresponding reference speed information according to the identification of the video segment.

[0049] In block 630, based on the local speed information and the reference speed information, the local stability corresponding to the current frame is determined. In some embodiments, the stability estimation unit 328 may compare the local speed information and the reference speed information and determine their difference. If the difference is within a predetermined range r1, it may be determined that the local conforms to the endoscopic operation requirements. If the difference is outside the predetermined range r1, a notice indicating that the endoscopic acquisition operation is too fast or too slow may be generated. In some illustrations, when |V(x) - SV| < r1, the local stability corresponding to the current frame may be determined by the following equation. [Number] Here, S(x) represents the local stability of the current frame, r1 represents a predetermined threshold value, SV represents the reference speed information, and V(x) represents the local speed information corresponding to the current frame. Also, when V(x) - SV > r1, it is determined that the current frame is too fast, and the local stability S(x) may be set to 20%. When V(x) - SV < r1, it is determined that the current frame is too slow, and the local stability S(x) may be set to 20%.

[0050] Referring to FIG. 3, in some embodiments, the overall stability of the video may be determined according to the local stabilities of a plurality of frames. In some embodiments, the overall stability may be determined based on the average value of the local stabilities of a plurality of video frames in the endoscopic video. The overall stability may be calculated after the completion of the video acquisition operation.

[0051] As described above, with reference to FIGS. 1 to 6, a method or process for processing an endoscopic video to evaluate the stability related to video acquisition according to an embodiment of the present disclosure has been described. Compared with the conventional solutions, the embodiments of the present disclosure can obtain the current stability information in real time for each video frame, assist the operator in timely adjusting the acquisition operation, and such stability information is obtained considering the site acquisition order during the endoscopic examination, so it is more accurate and comprehensive.

[0052] Compared with the conventional single cue stability estimation, some embodiments of the present disclosure can also obtain one endoscope stability estimation result by combining multiple cues. These cues include pre-stored video segments between adjacent regions, which are all normative and stable acquisition videos, and the cues are used to constrain whether the acquisition of adjacent regions is normative in the acquisition process, which belongs to the global constraint. The cues further include pre-stored region transfer information, which is used to constrain whether the transition between regions is normative, which belongs to the global constraint. The cues further include removing invalid video frames in the actual acquisition process and obtaining region probabilities of frames of high-quality videos, which belongs to the global constraint. The cues further include obtaining similarities of adjacent video frames in the actual acquisition process, which belongs to the local constraint. Therefore, the embodiments of the present disclosure consider both global and local elements, and at the same time, consider the region acquisition order, and can obtain more accurate and comprehensive evaluation results.

[0053] 7 shows a schematic block diagram of a video processing apparatus 700 according to an embodiment of the present disclosure. The apparatus 700 may be implemented by the computing device 105, and may be implemented as software, hardware, or a combination of software and hardware.

[0054] As shown, the device 700 includes a video acquisition unit 710, a local speed determination unit 720, and a stability determination unit 730. The video acquisition unit 710 is configured to acquire an endoscopic video obtained by an endoscopic acquisition operation. The local speed determination unit 720 is configured to determine local speed information related to the endoscopic acquisition operation based on a region estimation of a current frame of the endoscopic video, a region estimation of at least one previous frame, and region transfer information. The stability determination unit 730 is configured to determine a stability of the endoscopic acquisition operation based on at least the local speed information. In some implementations, the local speed determination unit 720 may be implemented by, for example, the local speed information estimation unit 326 described with reference to FIG. 3, and the stability determination unit 730 may be implemented by, for example, the stability estimation unit 328.

[0055] In some embodiments, the local speed information may include first speed information, and the local speed determination unit 720 may be configured to determine a first portion corresponding to the current frame and a second portion corresponding to another frame of the at least one previous frame and different from the first portion based on the portion estimation of the current frame and the portion estimation of the at least one previous frame, and to determine the first speed information based on the first portion, the second portion, and the portion relocation information.

[0056] In some embodiments, the local speed information may further include second speed information, and the local speed determination unit 720 may be further configured to determine a similarity between the current frame and one or more previous frames of the at least one previous frame, and determine the second speed information based on the similarity.

[0057] In some embodiments, the results of the region estimation of any of the region estimation of the current frame and the region estimation of the at least one previous frame may include a likelihood value for each of a plurality of regions, and the local speed determination unit 720 may be further configured to determine first speed information based on the first region, the second region, and the region relocation information in response to the maximum likelihood value among the results of the region estimation of the current frame exceeding a threshold.

[0058] In some embodiments, the local speed determination unit 720 may be further configured to determine a first region based on the region estimation of the current frame, and to determine another frame of the at least one previous frame, where in the region estimation of the at least one previous frame, the region estimation of the another frame has a maximum likelihood value for another region different from the first region, and to determine a second region based on the region estimation of the another frame.

[0059] In some embodiments, the apparatus 700 may further include a region estimation unit. The region evaluation unit may be configured to obtain a result of a region estimation of any one of a region estimation of a current frame and a region estimation of at least one previous frame based on a prediction of the deep network.

[0060] In some embodiments, the device 700 may further include a quality assessment unit. The quality assessment unit may be configured to perform quality assessment on video frames in the endoscopic video to determine a current frame and at least one video frame that satisfy a quality requirement.

[0061] In some embodiments, the stability may include a local stability of the current frame, and the stability determination unit 730 may be further configured to determine reference velocity information based on the reference endoscopic video, and determine the local stability based on the local velocity information and the reference velocity information.

[0062] In some embodiments, the stability determination unit 730 may be further configured to determine a corresponding segment in the reference endoscopic video based on the first portion corresponding to the current frame and an adjacent region portion of the first portion, and determine reference velocity information based on the corresponding segment.

[0063] In some embodiments, the stability determination unit 730 may be further configured to determine that the local stability meets an operational requirement of the endoscope in response to a difference between the local velocity information and the reference velocity information being within a predetermined range.

[0064] In some embodiments, the apparatus may further comprise a reminding unit configured to generate a notice in response to a difference between the local velocity information and the reference velocity information being outside a predefined range, the reminding unit indicating whether the endoscopic sampling operation is performed too fast or too slow.

[0065] In some embodiments, the stability may further include an overall stability, and the stability determination unit 730 may be further configured to determine the overall stability based on an average value of multiple local stability measures of multiple video frames in the endoscopic video.

[0066] FIG. 8 shows a schematic block diagram of an exemplary device 800 according to an embodiment that may be used to implement the teachings of the present disclosure. For example, the computing device 105 according to an embodiment of the present disclosure may be implemented by the device 800. As shown, the device 800 includes a central processing unit (CPU) 801 that can perform various appropriate operations and processes according to computer program instructions stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data necessary for the operation of the device 800. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0067] Several components of the device 800 are connected to the I / O interface 805, including an input unit 806 such as a keyboard and a mouse, an output unit 807 such as various displays and speakers, a storage unit 808 such as a magnetic disk and an optical disk, and a communication unit 809 such as a network interface card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information and data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0068] Each of the processes and operations described above, e.g., methods 200, 400, 500, and / or 600, may be executed by a processing device. For example, in some embodiments, methods 200, 400, 500, and / or 600 may be implemented as a computer software program tangibly embodied in a machine-readable medium, e.g., storage unit 808. In some embodiments, some or all of the computer program may be loaded and / or installed in device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more operations of methods 200, 400, 500, and / or 600 described above may be performed.

[0069] The present disclosure may be a method, an apparatus, a system, and / or a computer program product, which may include a computer-readable storage medium having computer-readable program instructions for carrying out aspects of the present disclosure.

[0070] A computer readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (not an exhaustive list) of computer readable storage media include portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, machine-encoded devices such as punch cards or slot-in-projection structures on which instructions are stored, and any suitable combination of the above. A computer readable storage medium as used herein is not to be construed as a momentary signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated through a wave guide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted through electrical wiring.

[0071] The computer readable program instructions described herein may be downloaded from the computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, e.g., the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in the computer readable storage medium in each computing / processing device.

[0072] The computer program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and traditional process-based programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user computer, partially on the user computer, as a separate software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. When a remote computer is involved, the remote computer may be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to implement aspects of the present disclosure by customizing electronic circuitry, e.g., programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), that can execute the computer-readable program instructions.

[0073] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented by computer-readable program instructions.

[0074] These computer-readable program instructions can be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that, when these instructions are executed by the processing unit of the computer or other programmable data processing apparatus, a machine is produced that implements the functions / operations specified in one or more blocks in the flowcharts and / or block diagrams. These computer-readable program instructions can be stored on a computer-readable storage medium, and these instructions cause a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable medium on which the instructions are stored comprises an article of manufacture containing instructions that implement each aspect of the functions / operations specified in one or more blocks in the flowcharts and / or block diagrams.

[0075] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to produce a computer implemented process, such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more blocks in the flowcharts and / or block diagrams.

[0076] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of the systems, methods, and computer program products according to the embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel or in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented in a hardware-based dedicated system that performs a given function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0077] Although each embodiment of the present disclosure has been described above, the above description is illustrative, not exhaustive, and not limited to each disclosed embodiment. Many modifications and changes will be obvious to those skilled in the art without departing from the scope and spirit of each described embodiment. The selection of terms used in the present specification is intended to best interpret the principles, practical applications, or improvements to the technology in the market of each embodiment, or to enable those skilled in the art to understand each embodiment disclosed in the present specification.

Claims

1. To acquire endoscopic videos obtained through endoscopic sampling procedures, Based on the location estimation of the current frame of the endoscopic video, the location estimation of at least one preceding frame, and location transfer information, local velocity information related to the endoscopic sampling operation is determined. This includes determining the stability of the endoscopic sampling operation based at least on the local velocity information, Video processing methods.

2. The local velocity information includes first velocity information, Determining the aforementioned local velocity information means Based on the estimation of the region of the current frame and the estimation of the region of at least one preceding frame, a first region corresponding to the current frame and a second region different from the first region, corresponding to another frame among the at least one preceding frame, are determined. This includes determining the first velocity information based on the first part, the second part, and the part transfer information, The video processing method according to claim 1.

3. The local velocity information includes second velocity information, and determining the local velocity information is The degree of similarity between the current frame and one or more of the at least one preceding frame is determined. The further includes determining the second velocity information based on the similarity, The video processing method according to claim 2.

4. The result of the part estimation of either the part estimation of the current frame or the part estimation of at least one preceding frame includes probability values ​​for each of the multiple parts, and the determination of the first velocity information is as follows: The process includes determining the first velocity information based on the first part, the second part, and the part transfer information in response to the highest probability value among the results of part estimation of the current frame exceeding a threshold, The video processing method according to claim 2.

5. Determining the first part corresponding to the current frame includes determining the first part based on the estimation of the part of the current frame, Determining a second portion corresponding to another frame among the at least one preceding frame is: Determining the other frame from the at least one preceding frame, wherein in the part estimation of the at least one preceding frame, the part estimation of the other frame has the highest probability value for a part other than the first part, This includes determining the second part based on the part estimation of the other frame, The video processing method according to claim 4.

6. The process further includes obtaining a result for a region estimation of either the region estimation of the current frame or the region estimation of at least one preceding frame, based on the predictions of the deep network. The video processing method according to claim 1.

7. The process further includes performing a quality evaluation on the video frames in the endoscopic video and determining the current frame and the at least one video frame that satisfy the quality requirements. The video processing method according to claim 1.

8. The stability includes the local stability of the current frame. Determining the stability of the aforementioned endoscopic sampling procedure is Determining reference velocity information based on reference endoscopic video, This includes determining the local stability based on the local velocity information and the reference velocity information, The video processing method according to claim 1.

9. Determining reference velocity information based on reference endoscopic video is Based on the first region corresponding to the current frame and the adjacent region of the first region, the corresponding segment in the reference endoscopic video is determined. This includes determining the reference speed information based on the corresponding segment, The video processing method according to claim 8.

10. The further includes determining that the local stability satisfies the endoscopic operation requirements in response to the difference between the local velocity information and the reference velocity information being within a predetermined range. The video processing method according to claim 8.

11. The system further includes generating a warning indicating whether the endoscopic sampling operation is too fast or too slow in response to the difference between the local velocity information and the reference velocity information being outside a predetermined range. The video processing method according to claim 8.

12. A video acquisition unit configured to acquire endoscopic videos obtained by endoscopic sampling procedures, A local velocity determination unit is configured to determine local velocity information related to the endoscopic sampling operation based on the location estimation of the current frame of the endoscopic video, the location estimation of at least one preceding frame, and location transfer information. The system includes a stability determination unit configured to determine the stability of the endoscopic sampling operation based at least on the local velocity information, Video processing device.

13. At least one processing unit, The device comprises a memory connected to the at least one processing unit and storing instructions to be executed by the at least one processing unit, wherein when an instruction is executed by the at least one processing unit, the method according to any one of claims 1 to 11 is performed. Electronic devices.

14. When executed by a device, the device includes a machine-executable instruction causing the device to perform the method according to any one of claims 1 to 11. A computer-readable storage medium.

15. When executed by a device, the device includes a machine-executable instruction causing the device to perform the method according to any one of claims 1 to 11. Computer program.

Images