Imaging method for endoscope imaging system and endoscope imaging system

Abstract

The invention discloses an imaging method for an endoscope imaging system and the endoscope imaging system, and the method comprises the steps: obtaining a plurality of frames of white light images and a plurality of frames of special light images of a to-be-observed part, and alternately or synchronously obtaining the white light images and the special light images; obtaining motion estimation information based on the at least two frames of white light images; mapping the special light image based on the motion estimation information to obtain a middle frame special light image; and displaying the special light image and the intermediate frame special light image. According to the invention, frame insertion is carried out on the special light image by multiplexing the motion estimation information of the white light image, so that the frame rate of the special light image can be improved, the problems of special light imaging smear and video lagging are relieved, and the impression and comfort of a user in an operation process are improved.

Description

Technical Field

[0001] The present invention relates to the field of medical equipment, and more particularly to an imaging method for an endoscopic imaging system and an endoscopic imaging system. Background Art

[0002] An endoscope is one of the important tools for diagnosis and treatment in modern medicine, as it can present the tissue morphology of a patient's internal organs and the pathological conditions in the body during minimally invasive surgery, facilitating diagnosis and implementation of surgery. The light source of an endoscopic imaging system can provide illumination of light in different wavelengths and generate different types of endoscopic images containing different information. For example, an endoscopic imaging system can perform visible light imaging and fluorescence imaging. Visible light imaging is imaging the tissue of the part to be observed under visible light (white light), and fluorescence imaging is collecting the fluorescence generated by the excitation of the fluorescent reagent injected into the tissue for imaging.

[0003] There is a time difference between the fluorescence image and the white light image acquired by the endoscopic imaging system, and their respective frame rates are relatively low. This will cause problems such as ghosting and stuttering in the fusion of fluorescence images and white light images during clinical use, affecting the user's perception and comfort during surgical operations. Summary of the invention

[0004] A series of simplified concepts are introduced in the Summary of the Invention, which will be further described in detail in the Detailed Description of the Invention. The Summary of the Invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0005] A first aspect of an embodiment of the present invention provides an imaging method for an endoscopic imaging system, the method comprising:

[0006] Acquire multiple frames of white light images and multiple frames of special light images of the part to be observed, wherein the white light images and the special light images are acquired alternately or synchronously;

[0007] Obtaining motion estimation information based on at least two frames of the white light image;

[0008] Mapping the special light image based on the motion estimation information to obtain an intermediate frame special light image;

[0009] The special light image and the intermediate frame special light image are displayed.

[0010] In one embodiment, the white light image includes a first white light image and a second white light image acquired sequentially, and the special light image includes a first special light image, the first special light image is acquired after the first white light image and before the second white light image, or the first special light image and the first white light image are acquired simultaneously;

[0011] The obtaining of motion estimation information based on the white light image comprises: obtaining the motion estimation information based on the first white light image and the second white light image;

[0012] The mapping the special light image based on the motion estimation information includes: mapping the first special light image based on the motion estimation information to obtain the intermediate frame special light image.

[0013] In one embodiment, the method further comprises:

[0014] Mapping the first white-light image and the second white-light image based on the motion estimation information to obtain an intermediate frame white-light image;

[0015] The first white-light image, the intermediate-frame white-light image, and the second white-light image are displayed in sequence.

[0016] In one embodiment, the motion estimation information includes first motion estimation information from the second white light image to the intermediate frame white light image;

[0017] When the time interval between the first special light image and the first white light image is equal to the time interval between the first special light image and the second white light image, mapping the first special light image based on the motion estimation information to obtain an intermediate frame special light image includes:

[0018] Determine inverse motion estimation information of the first motion estimation information;

[0019] The first special light image is mapped based on the inverse motion estimation information to obtain the intermediate frame special light image.

[0020] In one embodiment, the intermediate frame white light image is displayed by being superimposed on the first special light image, and the second white light image is displayed by being superimposed on the intermediate frame special light image.

[0021] In one embodiment, the motion estimation information includes first motion estimation information from the second white light image to the intermediate frame white light image, and second motion estimation information from the first white light image to the intermediate frame white light image;

[0022] The mapping the first white light image and the second white light image based on the motion estimation information to obtain an intermediate frame white light image includes:

[0023] Mapping the first white-light image based on the second motion estimation information to obtain a first intermediate frame white-light image;

[0024] Mapping the second white-light image based on the first motion estimation information to obtain a second intermediate frame white-light image;

[0025] The first white-light image and the second white-light image are weightedly fused to obtain the intermediate frame white-light image.

[0026] In one embodiment, when the time interval between the first special light image and the first white light image is not equal to the time interval between the first special light image and the second white light image, mapping the first special light image based on the motion estimation information to obtain an intermediate frame special light image includes:

[0027] adjusting the first motion estimation information and / or the second motion estimation information according to a time interval between the first special light image and the first white light image and / or a time interval between the first special light image and the second white light image to obtain third motion estimation information;

[0028] The first special light image is mapped based on the third motion estimation information to obtain the intermediate frame special light image.

[0029] In one embodiment, obtaining motion estimation information based on at least two frames of the white light image includes:

[0030] Inputting the white light image into a motion estimation network, and obtaining motion estimation information output by the motion estimation network;

[0031] Alternatively, the motion estimation information is obtained through a motion estimation algorithm;

[0032] Alternatively, the motion estimation information is obtained by performing image registration on the at least two frames of white light images.

[0033] In one embodiment, the motion estimation information comprises optical flow information.

[0034] In one embodiment, the motion estimation network includes an optical flow estimation network, which includes a feature extraction module and an optical flow extraction pyramid, wherein the feature extraction module is used to extract image features of the white light image, and the optical flow extraction pyramid is used to obtain the optical flow information based on the image features.

[0035] In one embodiment, the special light image comprises a fluorescent or narrow-band light image.

[0036] Another aspect of the present invention provides an endoscope imaging system, including a light source, an endoscope, a camera host and a display, wherein:

[0037] The light source is used to provide illumination to the part to be observed;

[0038] The endoscope comprises an insertion portion and an operation portion, wherein the insertion portion is used to be inserted into a part of a patient to be observed;

[0039] The endoscope is provided with at least one image sensor, and the at least one image sensor is used to collect image signals of the part to be observed;

[0040] The camera host is used to obtain the image signal to execute the method as described above;

[0041] The display is used to display the endoscopic image generated by the camera host.

[0042] The imaging method for an endoscopic imaging system and the endoscopic imaging system of the embodiments of the present invention reuse the motion estimation information of the white light image to perform interpolation processing on the special light image, which can improve the frame rate of the special light image, alleviate the problems of special light imaging ghosting and video freeze, and improve the viewing experience and comfort of the user during operation. Since the clarity of the white light image is higher than that of the special light image, more accurate motion estimation information can be obtained based on the white light image, and interpolating the special light image using the motion estimation information of the white light image can obtain a more accurate intermediate frame special light image. At the same time, interpolating using the motion estimation information of the white light image does not require recalculating the motion estimation information based on the special light image, thereby saving computing power and improving computing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0044] In the attached picture:

[0045] Figure 1 A schematic block diagram of an endoscopic imaging system according to an embodiment of the present invention is shown;

[0046] Figure 2 A schematic flow chart showing an imaging method for an endoscopic imaging system according to an embodiment of the present invention;

[0047] Figure 3 A timing diagram showing a special light image and a white light image according to an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of an endoscopic imaging system according to an embodiment of the present invention is shown;

[0049] Figure 5 A schematic diagram showing an optical flow estimation network of an endoscopic imaging method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0051] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0052] It should be understood that the present invention can be implemented in different forms and should not be interpreted as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0053] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present invention. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0054] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0055] Next, first refer to Figure 1 An endoscopic imaging system according to an embodiment of the present invention is described. Figure 1 FIG. 4 shows a schematic structural block diagram of an endoscopic imaging system 100 according to an embodiment of the present invention.

[0056] like Figure 1 As shown, the endoscopic imaging system 100 includes a light source host 110, an endoscope 120, a camera host 130 and a display 140. The light source host 110 is used to output white light and special light. Exemplarily, the light source host 110 includes a white light source and a special light source, the white light source is used to output white light, and the special light source is used to output special light, that is, light of a different spectrum from white light, such as the excitation light or narrow-band light of a fluorescent agent. Among them, the white light source can be an LED light source. The special light source can be a laser light source corresponding to the fluorescent agent, such as a near-infrared light source. Before the endoscope system is used for imaging, a fluorescent agent is injected into the part to be observed, and the fluorescent agent can generate fluorescence after absorbing the excitation light generated by the excitation light source. Under visible light, since all tissue structures are illuminated, some tumors may be difficult to detect and are not specific to malignant lesions. The fluorescent dye injected into the part to be observed will preferentially accumulate in the malignant lesions, and when the fluorescent dye is excited under the excitation light, it will generate fluorescence, thereby increasing the contrast between malignant tissue and healthy tissue, thereby helping users to more accurately identify malignant tissue. In addition, the special light source can be a narrow-band light source.

[0057] The light source host 110 is connected to the endoscope 120. The endoscope 120 can transmit the white light and / or special light output by the light source host 120 to the patient's part to be observed, receive the reflected light and / or special light from the part to be observed, and convert the reflected light and special light into electrical signals through the image sensor. The endoscope 120 includes an insertion portion and an operating portion. The insertion portion is used to insert into the patient's part to be observed, and the operating portion is used for user handheld operation. The insertion portion and the operating portion can be an integrated structure or a detachable structure. The endoscope 120 also includes at least one image sensor (not shown). Exemplarily, the image sensor can be arranged at the front end of the insertion portion of the endoscope 120.

[0058] The other end of the endoscope 120 is connected to the camera host 130 through a cable, and the electrical signal is transmitted to the camera host 130 through the cable for processing. In some embodiments, the endoscope can also send the electrical signal to the camera host 130 by wireless transmission. The camera host 130 is used to generate image data based on the electrical signal. Specifically, the camera host 130 is provided with a processor, and the processor obtains the electrical signal output by the endoscope 120 and generates image data.

[0059] The display 140 is connected to the camera host 130, and is used to display the endoscopic image based on the image data output by the camera host 130. Exemplarily, the camera host 130 is connected to the display 140 via a video connection line, and is used to send the image data to the display 140 for display.

[0060] It should be noted that Figure 1The endoscopic imaging system 100 is merely an example and does not constitute a limitation on the endoscopic imaging system 100. The endoscopic imaging system 100 may include Figure 1 More or fewer components, or a combination of certain components, or different components may be shown. For example, the endoscope system 100 may also include a dilator, a smoke control device, input and output devices, a network access device, etc.

[0061] Below, refer to Figure 2 Describe the endoscopic imaging method of the embodiment of the present invention. The method can be referred to as Figure 1 An endoscopic imaging system implementation is described. Figure 2 is a schematic flow chart of an endoscopic imaging method 200 in an embodiment of the present invention, which specifically includes the following steps:

[0062] In step S210, a plurality of frames of white light images and a plurality of frames of special light images of the part to be observed are acquired, wherein the white light images and the special light images are acquired alternately or synchronously;

[0063] In step S220, motion estimation information is obtained based on at least two frames of the white light image;

[0064] In step S230, the special light image is mapped based on the motion estimation information to obtain an intermediate frame special light image; that is, the intermediate frame image is obtained using the deformation information contained in the motion estimation information.

[0065] In step S240, the special light image and the intermediate frame special light image are displayed.

[0066] Among them, the special light image can be a fluorescence image or a narrow-band light image. Fluorescence images are images generated based on the fluorescence generated by the excitation of fluorescent dyes, and can specifically include near-infrared fluorescence images, blue light imaging (Blue Light Imaging, BLI) fluorescence images or photodynamic fluorescence diagnosis (Photodynamic diagnosis, PDD) fluorescence images, etc. Narrow-band light images utilize the penetration characteristics of light: shorter wavelengths can only penetrate the surface mucosa, while longer wavelengths can penetrate deeper mucosa, that is, narrow-band light images are images generated by using a filter device to filter out the broadband spectrum emitted by the light source, leaving only the narrow-band spectrum.

[0067] In one example, a multi-frame white light image and a multi-frame special light image of the part to be observed can be alternately acquired in a stroboscopic manner of a light source. For example, when the special light image is a fluorescent image, the light source can be controlled to alternately output white light and excitation light, wherein the excitation light is used to excite the fluorescent reagent of the part to be observed. In one embodiment, the light source host includes a white light source and an excitation light source, and the excitation light source can be a laser light source such as a near-infrared light source. The white light source and the excitation light source can be turned on alternately, so as to alternately output white light and excitation light, that is, in the first imaging cycle, the white light source is turned on and the excitation light source is turned off, and in the second imaging cycle, the excitation light source is turned on and the white light source is turned off, thereby realizing stroboscopic illumination of white light and excitation light. Alternatively, the white light source and / or the excitation light source can also be always on, and the alternation of white light and excitation light can be realized by a rotating wheel. Optionally, the white light source can also be kept on for a long time, and the excitation light source can be stroboscopically illuminated, thereby increasing the illumination brightness. The embodiment of the present invention does not limit the specific manner in which the light source host realizes the alternating output of white light and excitation light.

[0068] When the light source outputs white light, the image sensor provided in the endoscope collects the white light signal obtained by the white light reflected by the part to be observed; when the light source outputs fluorescence, the image sensor provided in the endoscope collects the fluorescence signal obtained by the excitation of the fluorescent reagent in the part to be observed by the excitation light. In order to realize the miniaturization of the endoscope, the white light signal and the fluorescence signal can be alternately collected by the same image sensor in a time-sharing exposure manner. The exposure process of the image sensor cooperates with the illumination process of the light source, and it collects white light signals under white light illumination and collects fluorescence signals under excitation light illumination. The camera host of the endoscope imaging system generates a white light image according to the white light signal and generates a fluorescence image according to the fluorescence signal, that is, the white light image and the fluorescence image are collected alternately in time sequence.

[0069] In one example, if Figure 3 As shown, the endoscope imaging system sequentially acquires a first white light image, a first special light image, a second white light image, and a second special light image, respectively. Indicates that the corresponding acquisition times are t1, t2, t3, and t4 respectively. The first white light image and the second white light image may be any two adjacent white light images; the first special light image and the second special light image may be any two adjacent special light images that are alternately acquired from the first white light image and the second white light image.

[0070] In another example, white light and special light illumination may be provided to the part to be observed at the same time, and a white light image and a special light image of the part to be observed may be generated at the same time. Specifically, white light signals and special light signals may be respectively collected based on different image sensors, and white light images may be generated based on white light signals, and special light images may be generated based on special light signals; or white light signals and special light signals may be simultaneously collected based on the same image sensor, and white light images and special light images may be generated respectively after signal separation.

[0071] After acquiring the white light image, motion estimation information is obtained based on at least two frames of white light images. The motion estimation information describes the movement of an object in a continuous frame image. Exemplarily, the motion estimation information may include optical flow information. Specifically, optical flow refers to the motion vector field of pixels in an image, which describes the displacement of each pixel in two adjacent frames of images. The calculation of optical flow relies on the assumption of constant brightness, that is, the brightness of an object will not change significantly in a short period of time, so the brightness value of a certain pixel in the image should remain unchanged between two adjacent frames. When obtaining the optical flow information, the direction and speed of movement of each pixel between two temporally adjacent frames of white light images can be estimated based on the above assumptions. In addition to optical flow, motion estimation information can also be obtained by methods such as motion field matching and target tracking.

[0072] After obtaining the motion estimation information, the white light image can be interpolated according to the motion estimation information to improve the frame rate of the white light image. For example, after obtaining the motion estimation information according to the first white light image and the second white light image, the first white light image and the second white light image can be mapped (warping) based on the motion estimation information to obtain an intermediate frame white light image. The intermediate frame white light image is equivalent to a white light image captured after the first white light image and before the second white light image. Figure 3 In the example, the middle frame white light image Equivalent to the white light image collected at time t2. When displaying the white light video image, the first white light image is displayed in sequence Intermediate frame white light image and the second white light image Can increase the smoothness of white light video images.

[0073] In some embodiments, the motion estimation information includes the second white light image To the middle frame white light image The first motion estimation information F1, and the first white light image To the middle frame white light image The first motion estimation information F1 is the pixel displacement information from the back to the front in time sequence, describing the pixel displacement from the second white light image The corresponding moment moves to the middle frame white light image The second motion estimation information F2 is the displacement information of the pixel points from the front to the back in time sequence, which describes the displacement of the pixel points from the first white light image The corresponding moment moves to the middle frame white light image The displacement at the corresponding moment.

[0074] On this basis, the first white light image is imaged based on the motion estimation information. and the second white light image Mapping is performed to obtain an intermediate frame white light image include:

[0075] Based on the second motion estimation information F2, the first white light image Mapping is performed to obtain the first intermediate frame white light image

[0076]

[0077] Based on the first motion estimation information F1, the second white light image Mapping is performed to obtain the second intermediate frame white light image

[0078]

[0079] First white light image and the second white light image Perform weighted fusion to obtain the intermediate frame white light image

[0080]

[0081] Among them, α and β are pre-set fusion weights.

[0082] Furthermore, since the white light image and the special light image are collected simultaneously or alternately, the motion information of the pixel points between the white light image and the special light image is continuous, and thus the embodiment of the present invention also performs interpolation processing on the special light image according to the motion estimation information obtained from the white light image, that is, the special light image is mapped based on the motion estimation information to obtain an intermediate frame special light image. When displaying the special light video image, the first special light image, the intermediate frame special light image, and the second special light image are displayed in sequence to improve the frame rate of the special light image.

[0083] Since the clarity of white light images is higher than that of special light images, more accurate motion estimation information can be obtained based on white light images, and interpolating special light images with the motion estimation information of white light images can obtain more accurate intermediate frame special light images. At the same time, interpolating with the motion estimation information of white light images does not require recalculating the motion estimation information based on the special light images, thereby saving computing power and improving computing speed.

[0084] In one example, as described above, the motion estimation information includes first motion estimation information from the second white light image to the intermediate frame white light image, and second motion estimation information from the first white light image to the intermediate frame white light image. Figure 3 As shown, the first special light image is a special light image actually collected at time t2, the intermediate frame white light image is a white light image simulated at time t2, and the time interval between the first special light image and the first white light image is equal to the time interval between the first special light image and the second white light image, that is, the timing of the first special light image is basically the same as the intermediate frame white light image located between the first white light image and the second white light image. The first motion estimation information F1 is the motion estimation information from time t3 to time t2, so the reverse motion estimation information F1′ of the first motion estimation information F1 can be determined, that is, the motion estimation information from time t2 to time t3; the first special light image is mapped based on the reverse motion estimation information F1′ to obtain the intermediate frame special light image. The intermediate frame special light image is a special light image simulated at time t3, and is located between the first special light image collected at time t2 and the second special light image collected at time t4 in terms of timing. Thus, the interpolation process of the special light image is completed according to the motion estimation information of the white light image, and the obtained intermediate frame special light image has high synchronization with the second white light image and good fusion effect.

[0085] Optionally, other methods may be used to multiplex the motion estimation information of the white light image to perform interpolation processing on the special light image. For example, the first special light image may be mapped according to the inverse motion estimation information of the second motion estimation information to obtain an intermediate frame special light image simulated at time t1. Optionally, a weighted summation may be performed on the intermediate frame special light images obtained by the two methods to further improve the accuracy of the intermediate frame special light image.

[0086] Exemplarily, when the time interval between the first special light image and the first white light image is not equal to the time interval between the first special light image and the second white light image, the first motion estimation information and / or the second motion estimation information may be adjusted according to the time interval between the first special light image and the first white light image and / or the time interval between the first special light image and the second white light image to obtain the third motion estimation information; the first special light image may be mapped based on the third motion estimation information to obtain the intermediate frame special light image. In this example, the acquisition time of the first special light image is different from that of the intermediate frame white light image between the first white light image and the second white light image, and the first special light image is acquired before or after the intermediate frame white light image. Therefore, in order to improve the accuracy of interpolation, the special light image is not interpolated directly using the inverse motion estimation information of the first motion estimation information, but the first motion estimation information and / or the second motion estimation information may be adjusted according to the timing of the first special light image, so as to obtain more accurate third motion estimation information for mapping the first special light image.

[0087] Exemplarily, obtaining motion estimation information based on at least two frames of white light images includes: inputting the white light images into a motion estimation network and obtaining the motion estimation information output by the motion estimation network; or, obtaining the motion estimation information through a traditional motion estimation algorithm; or, obtaining the motion estimation information by performing image registration on at least two frames of white light images.

[0088] When the motion estimation information is optical flow information, the motion estimation network includes an optical flow estimation network. Specifically, the white light image can be input into the optical flow estimation network, and the optical flow information output by the optical flow estimation network can be obtained. The optical flow estimation network can adopt any feasible deep learning model, such as an optical flow estimation model based on FlowNet. For example, the first white light image and the second white light image can be input into the optical flow estimation network to obtain the first optical flow information and the second optical flow information:

[0089]

[0090] For example, Figure 5 As shown, the optical flow estimation network includes a feature extraction module and an optical flow extraction pyramid. The feature extraction module is used to extract image features of the white light image, and the optical flow extraction pyramid is used to obtain optical flow information based on the image features.

[0091] Exemplarily, in the inference stage, that is, in the actual application process, the first white light image and the second white light image are input into the optical flow estimation network, and the feature extraction module extracts the image features of the first white light image and the second white light image. For example, the image feature points with high stability and representativeness can be extracted by feature extraction algorithms such as SIFT and Harris, and input into the optical flow extraction pyramid. The optical flow extraction pyramid obtains the first optical flow information and the second optical flow information based on the image features of the first white light image and the second white light image. Specifically, the optical flow extraction pyramid includes a plurality of optical flow extraction modules cascaded in sequence, which use a layer-by-layer iterative method to estimate the motion optical flow of the object from coarse to fine and from shallow to deep. In the process of continuous iteration of the pyramid, the accuracy of the optical flow is also continuously improved.

[0092] In some embodiments of the present invention, each optical flow extraction module of the optical flow extraction pyramid propagates the extracted optical flow information to the next optical flow extraction module, and the last optical flow extraction module outputs the optical flow information. In the reasoning stage, each optical flow extraction module does not transmit the intermediate frame white light image backward, but directly uses the optical flow information predicted by the last optical flow extraction module to perform backward mapping on the original white light image, thereby eliminating the computing resource consumption caused by multiple backward mappings, improving the model reasoning efficiency, and enabling the optical flow estimation network to be efficiently integrated into the frame rate optimization scheme of the entire fluorescent image and white light image.

[0093] In the model training phase, that is, the training process of the optical flow estimation network, the sample white light image is input into the optical flow estimation network, and the optical flow information is extracted by the optical flow extraction module. The optical flow information extracted by each optical flow extraction module will be propagated to the next optical flow extraction module. At the same time, the optical flow information extracted by each optical flow extraction module will be backward mapped with the original white light image to obtain the intermediate frame white light image and output the intermediate frame white light image; the loss function is calculated based on the intermediate frame white light image output by each optical flow extraction module and the real intermediate frame white light image, and the network parameters of the optical flow estimation network are optimized according to the loss function. Among them, the parameters of each layer of the optical flow estimation network can be gradually optimized through gradient back propagation.

[0094] The sample white light image and the real intermediate frame white light image can be continuous video frames extracted from the white light video image collected by the endoscopic imaging system. For example, a clinical surgery video collected by the endoscopic imaging system can be obtained, and the odd frames therein can be input into the optical flow estimation network as sample white light images, and the even frames can be used as real labels for model training.

[0095] After completing the interpolation processing of the white light image and the fluorescent image, the white light image and the special light image can be superimposed and displayed. When the white light image and the special light image are acquired alternately, the intermediate frame white light image and the first special light image can be superimposed and displayed, and the second white light image and the intermediate frame special light image can be superimposed and displayed. When the white light image and the special light image are acquired simultaneously, the intermediate frame white light image and the intermediate frame special light image can be superimposed and displayed. Figure 3 As shown, when the intermediate frame white light image and the first special light image both correspond to the moment t2, and the second white light image and the intermediate frame special light image both correspond to the moment t3, the special light image and the white light image that are close in time sequence can be superimposed and displayed, which can also alleviate the problem of ghosting after the special light image and the white light image are fused. Among them, the superimposed display of the white light image and the special light image can be achieved by superimposing the layers of the white light image and the special light image. For example, the special light image can be processed to have a certain degree of transparency, and then the special light image can be superimposed on the white light image to form a fused image. Alternatively, each pixel on the fused image can be synthesized according to the pixel value of each pixel in the white light image and the pixel value of the corresponding pixel in the special light image. In addition, a separate white light image, a special light image, and a fused image can also be displayed simultaneously in the display interface.

[0096] In some embodiments, the endoscopic imaging system is a stereoscopic endoscopic imaging system. The stereoscopic endoscopic system achieves a stereoscopic imaging effect based on the binocular parallax principle. The binocular parallax principle means that when the human eye observes an object, the object is projected onto the retinas of the left and right eyes respectively, forming two images with parallax. The human brain can automatically calculate the depth information of the object based on the parallax, thereby forming a stereoscopic visual effect. Based on the above principle, the display of the endoscopic imaging system alternately or simultaneously displays the left-path image and the right-path image with horizontal parallax, and uses the corresponding image separation technology to make the left-eye image and the right-eye image with horizontal parallax enter the left and right eyes of the person respectively, so that the human brain automatically fuses the stereoscopic effect and obtains the depth information. That is, a stereoscopic endoscopic image refers to an image that can produce a stereoscopic visual effect for the observer, which essentially includes a left-path image output to the left eye and a right-path image output to the right eye.

[0097] There are many ways to output stereoscopic endoscopic images, for example, they can be output through dual monitors or single monitors. Dual monitor output includes helmet-mounted displays, dual-screen displays, etc., so that the observer's left and right eyes can view the left and right images displayed on different monitors respectively. Single monitor output mainly includes active shutter stereo display mode, passive polarization stereo display mode, etc. The active shutter stereo display mode needs to be used in conjunction with liquid crystal light valve glasses, and the passive polarization stereo display mode needs to be used in conjunction with polarized glasses. Both of them realize left and right eye image separation through the cooperation of the display and glasses.

[0098] When the endoscopic imaging system is a stereoscopic endoscopic imaging system, the white light image includes a left white light image and a right white light image, and the special light image includes a left special light image and a right special light image. The left white light image and the left special light image are collected by the same first image sensor, and the right white light image and the right special light image are collected by the same second image sensor, thereby reducing the number of image sensors and reducing the size of the endoscope; but compared with the solution in which the same sensor only collects white light images or special light images, the frame rate of the white light images and special light images collected by the aforementioned solution will be reduced, resulting in blurring, ghosting and other problems in the white light images and special light images. In order to solve this technical problem, the present solution creatively proposes to use motion estimation information of two frames of white light images to map the special light image and the white light image to obtain an intermediate frame special light image and an intermediate frame white light image.

[0099] At the same time, due to the different observation angles of the left and right paths, the motion vectors of the objects are different. Therefore, during the interpolation processing, the left-path optical flow information can be obtained based on at least two frames of left-path white light images; the left-path special light image is mapped based on the left-path optical flow information to obtain the intermediate frame left-path special light image; the right-path optical flow information is obtained based on at least two frames of right-path white light images, and the right-path special light image is mapped based on the right-path optical flow information to obtain the intermediate frame right-path special light image.

[0100] In summary, the imaging method 200 for an endoscopic imaging system of an embodiment of the present invention reuses the motion estimation information of a white light image to perform interpolation processing on a special light image, which can improve the frame rate of the special light image, alleviate the problems of special light imaging ghosting and video freeze, and improve the viewing experience and comfort of the user during operation. Since the clarity of a white light image is higher than that of a special light image, more accurate motion estimation information can be obtained based on the white light image, and multiplexing the motion estimation information of the white light image to interpolate the special light image can obtain a more accurate intermediate frame special light image. At the same time, multiplexing the motion estimation information of the white light image to interpolate does not require the motion estimation information to be calculated again based on the special light image, thereby saving computing power and improving computing speed.

[0101] Re-reference Figure 1The embodiment of the present invention further provides an endoscope imaging system 100, which includes a light source host 110, an endoscope 120, a camera host 130 and a display 140, wherein the light source 110 is used to provide illumination to the part to be observed; the endoscope 120 includes an insertion portion and an operation portion, and the insertion portion is used to insert into the part to be observed of the patient; the endoscope 120 is provided with at least one image sensor, and the at least one image sensor is used to collect image signals of the part to be observed; the camera host 130 is used to obtain image signals to execute the imaging method 200 for the endoscope imaging system as described above, specifically including: obtaining multiple frames of white light images and multiple frames of special light images of the part to be observed; obtaining motion estimation information based on at least two frames of white light images; mapping the special light image based on the motion estimation information to obtain an intermediate frame special light image; and displaying the special light image and the intermediate frame special light image. The display 140 is used to display the endoscope image generated by the camera host 130.

[0102] The specific structure of the endoscopic imaging system 100 and the specific steps of the imaging method 200 for the endoscopic imaging system have been described above and will not be repeated here. The endoscopic imaging system 100 of the embodiment of the present invention reuses the motion estimation information of the white light image to perform interpolation processing on the special light image, which can improve the frame rate of the special light image, alleviate the problems of special light imaging ghosting and video freeze, and improve the user's perception and comfort during operation.

[0103] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0104] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0105] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0106] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0107] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0108] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0109] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0110] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules according to embodiments of the present invention. The present invention can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0111] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0112] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment, and the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An imaging method for an endoscopic imaging system, characterized in that: The method comprises: Acquire multiple frames of white light images and multiple frames of special light images of the part to be observed, wherein the white light images and the special light images are acquired alternately or synchronously; Obtaining motion estimation information based on at least two frames of the white light image; Mapping the special light image based on the motion estimation information to obtain an intermediate frame special light image; The special light image and the intermediate frame special light image are displayed.

2. The method according to claim 1, characterized in that The white light image includes a first white light image and a second white light image acquired sequentially, and the special light image includes a first special light image, wherein the first special light image is acquired after the first white light image and before the second white light image, or the first special light image and the first white light image are acquired simultaneously; The obtaining of motion estimation information based on the white light image comprises: obtaining the motion estimation information based on the first white light image and the second white light image; The mapping the special light image based on the motion estimation information includes: mapping the first special light image based on the motion estimation information to obtain the intermediate frame special light image.

3. The method according to claim 2, characterized in that Also includes: Mapping the first white-light image and the second white-light image based on the motion estimation information to obtain an intermediate frame white-light image; The first white-light image, the intermediate-frame white-light image, and the second white-light image are displayed in sequence.

4. The method according to claim 3, characterized in that The motion estimation information includes first motion estimation information from the second white light image to the intermediate frame white light image; When the time interval between the first special light image and the first white light image is equal to the time interval between the first special light image and the second white light image, mapping the first special light image based on the motion estimation information to obtain an intermediate frame special light image includes: Determine inverse motion estimation information of the first motion estimation information; The first special light image is mapped based on the inverse motion estimation information to obtain the intermediate frame special light image.

5. The method according to claim 3, characterized in that The intermediate frame white light image is superimposed on the first special light image and displayed, and the second white light image is superimposed on the intermediate frame special light image and displayed.

6. The method according to claim 3, characterized in that The motion estimation information includes first motion estimation information from the second white light image to the intermediate frame white light image, and second motion estimation information from the first white light image to the intermediate frame white light image; The mapping the first white light image and the second white light image based on the motion estimation information to obtain an intermediate frame white light image includes: Mapping the first white-light image based on the second motion estimation information to obtain a first intermediate frame white-light image; Mapping the second white-light image based on the first motion estimation information to obtain a second intermediate frame white-light image; The first white-light image and the second white-light image are weightedly fused to obtain the intermediate frame white-light image.

7. The method according to claim 6, characterized in that When the time interval between the first special light image and the first white light image is not equal to the time interval between the first special light image and the second white light image, mapping the first special light image based on the motion estimation information to obtain an intermediate frame special light image includes: adjusting the first motion estimation information and / or the second motion estimation information according to a time interval between the first special light image and the first white light image and / or a time interval between the first special light image and the second white light image to obtain third motion estimation information; The first special light image is mapped based on the third motion estimation information to obtain the intermediate frame special light image.

8. The method according to claim 1, characterized in that Obtaining motion estimation information based on at least two frames of white light images, comprising: Inputting the white light image into a motion estimation network, and obtaining motion estimation information output by the motion estimation network; Alternatively, the motion estimation information is obtained through a motion estimation algorithm; Alternatively, the motion estimation information is obtained by performing image registration on the at least two frames of white light images.

9. The method according to claim 8, characterized in that The motion estimation information includes optical flow information.

10. The method according to claim 9, characterized in that The motion estimation network includes an optical flow estimation network, and the optical flow estimation network includes a feature extraction module and an optical flow extraction pyramid. The feature extraction module is used to extract image features of the white light image, and the optical flow extraction pyramid is used to obtain the optical flow information based on the image features.

11. The method according to claim 1, characterized in that The special light image includes a fluorescent light or a narrow-band light image.

12. An endoscopic imaging system, characterized in that: It includes a light source, an endoscope, a camera host and a display, wherein: The light source is used to provide illumination to the part to be observed; The endoscope comprises an insertion portion and an operation portion, wherein the insertion portion is used to be inserted into a part of a patient to be observed; The endoscope is provided with at least one image sensor, and the at least one image sensor is used to collect image signals of the part to be observed; The camera host is used to obtain the image signal to perform the method according to any one of claims 1 to 11; The display is used to display the endoscopic image generated by the camera host.

Images