Three-dimensional reconstruction method and device, image processing equipment and endoscope system

Abstract

The invention is suitable for the technical field of medical treatment, and provides a three-dimensional reconstruction method and device, image processing equipment and an endoscope system. The three-dimensional reconstruction method comprises the steps that a plurality of target images shot for a target physiological area under the condition that the target physiological area containing a focus is irradiated through light of various different wavebands are obtained, and the penetration depths of the light of the different wavebands to the target physiological area are not completely the same; the number of the target images shot under each wave band of light is greater than or equal to 1; and performing three-dimensional reconstruction on the target physiological region according to the obtained target image. According to the embodiment of the invention, the focus tissue contour can be reflected more accurately.

Description

Technical Field

[0001] The present application belongs to the field of medical technology, and in particular relates to a three-dimensional reconstruction method, device, image processing equipment and endoscope system. Background Art

[0002] In the field of lesion screening, some doctors will extract the diseased tissue by taking a biopsy to determine the depth of invasion and the type of cancer. This method is difficult to operate, costly, and may stimulate the activity of the lesion. Therefore, doctors usually determine the scope of surgery by injecting dyes, marking with an electric knife, etc. when performing endoscopic submucosal dissection (ESD), and then perform the dissection operation. When dyes are injected during the operation, the dyes can only be used to observe the size of the lesion on a plane, and cannot reflect the depth of the lesion, which can easily affect the accuracy and efficiency of the operation.

[0003] Therefore, a method that can more accurately reflect the contour of lesion tissue is needed. Summary of the invention

[0004] The embodiments of the present application provide a three-dimensional reconstruction method, apparatus, image processing device and endoscope system, which can more accurately reflect the contour of lesion tissue.

[0005] A first aspect of an embodiment of the present application provides a three-dimensional reconstruction method, comprising: obtaining a plurality of target images taken of a target physiological region containing a lesion while irradiating the target physiological region with light of a plurality of different wavelength bands, wherein the penetration depths of light of different wavelength bands into the target physiological region are not exactly the same, and the number of the target images taken under each wavelength band is greater than or equal to 1; and performing three-dimensional reconstruction of the target physiological region based on the acquired target images.

[0006] In some embodiments of the first aspect, after performing three-dimensional reconstruction of the target physiological region based on the acquired target image and obtaining feedback signals of the target physiological region to light signals of different bands, the method further includes: determining the range of the first target object of the lesion at each penetration depth of the target physiological region based on the feedback signals corresponding to different bands of the acquired target image, so as to perform three-dimensional reconstruction of the lesion according to the range where the lesion is located.

[0007] In some embodiments of the first aspect, determining the range of the lesion at each penetration depth of the target physiological area based on the acquired target image includes: determining the physiological tissue to which each position belongs at each penetration depth of the target physiological area based on the acquired target image; determining the distribution of blood vessels at each penetration depth of the target physiological area based on the physiological tissue to which each position belongs; determining the range of the lesion at each penetration depth of the target physiological area based on the distribution of the blood vessels.

[0008] In some embodiments of the first aspect, the distribution of the blood vessels includes at least one of the following: the distance between the blood vessels, and the number of the blood vessels within a unit range.

[0009] In some embodiments of the first aspect, the three-dimensional reconstruction of the target physiological region based on the acquired target image includes: determining point cloud data at each penetration depth of the target physiological region based on the acquired target image; clustering different physiological tissues on the point cloud data, and three-dimensionally reconstructing the target physiological region based on the clustering result.

[0010] In some embodiments of the first aspect, the point cloud data includes characteristic values ​​of the point cloud at each position in the target physiological area, and the characteristic values ​​are related to signal strength, which is the signal strength of light reflected by physiological tissue at the corresponding position.

[0011] In some embodiments of the first aspect, the three-dimensional reconstruction method also includes: determining three-dimensional size data of the lesion based on the results of the three-dimensional reconstruction, the three-dimensional size data including length, width and height data of the lesion; sending the three-dimensional size data to a display device to be superimposed on the two-dimensional image displayed in real time by the display device.

[0012] In some embodiments of the first aspect, the two-dimensional image or the three-dimensional reconstruction result is used for surgical planning, wherein the surgical planning at least includes: determining the cutting range of the lesion.

[0013] In some embodiments of the first aspect, the three-dimensional reconstruction method further includes: determining, based on the acquired target image, a range of a target blood vessel at each penetration depth of the target physiological region, the target blood vessel being a blood vessel having a diameter greater than a preset diameter, or the target blood vessel being a blood vessel of a preset type; and generating prompt information based on the range of the target blood vessel and the range of the lesion, the prompt information being used to indicate the position of the target blood vessel relative to the lesion.

[0014] In some embodiments of the first aspect, the lesion includes cell canceration; before performing three-dimensional reconstruction of the target physiological area, the three-dimensional reconstruction method also includes: obtaining the target image corresponding to the longest wavelength light, the longest wavelength light being the light with the longest wavelength among the multiple different wavelength lights, and the longest wavelength light corresponding to the deepest penetration depth of the target physiological area; based on the target image corresponding to the longest wavelength light, performing cell canceration detection at the deepest penetration depth; if cell canceration is detected at the deepest penetration depth, stopping the three-dimensional reconstruction and providing a canceration degree prompt.

[0015] In some embodiments of the first aspect, the longest wavelength corresponding to the plurality of light beams of different wavelength bands does not exceed 950 nm.

[0016] In some embodiments of the first aspect, the acquiring of a plurality of target images captured of the target physiological region containing a lesion when the target physiological region is irradiated with a plurality of light beams of different wavelengths comprises: acquiring a plurality of target images captured of the target physiological region when the target physiological region is irradiated with the plurality of light beams of different wavelengths in sequence; or, the light irradiating the target physiological region comprises the plurality of light beams of different wavelengths; and performing three-dimensional reconstruction of the target physiological region based on the acquired target images comprises: performing image reconstruction on the target images to obtain images to be processed corresponding to each of the plurality of light beams of different wavelengths; and performing three-dimensional reconstruction of the target physiological region based on the images to be processed.

[0017] A three-dimensional reconstruction device provided in the second aspect of an embodiment of the present application includes: an image acquisition unit, used to acquire multiple target images of a target physiological region containing a lesion when the target physiological region is irradiated with multiple different wavelength bands of light, wherein the penetration depths of different wavelength bands of light to the target physiological region are not exactly the same, and the number of the target images taken under each wavelength band of light is greater than or equal to 1; and a three-dimensional reconstruction unit, used to perform three-dimensional reconstruction of the target physiological region based on the acquired target images.

[0018] A third aspect of an embodiment of the present application provides an image processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned three-dimensional reconstruction method when executing the computer program.

[0019] A fourth aspect of an embodiment of the present application provides an image processing system, comprising an image processing device and a camera, wherein the camera is used to capture a target image, and the image processing device is used to execute the steps of the above-mentioned three-dimensional reconstruction method.

[0020] A fifth aspect of an embodiment of the present application provides an endoscope system, comprising a light source, a mirror body, an image processing device and a display device; wherein the image processing device is used to execute the steps of the above-mentioned three-dimensional reconstruction method.

[0021] A sixth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned three-dimensional reconstruction method are implemented.

[0022] A seventh aspect of an embodiment of the present application provides a computer program product. When the computer program product runs on an image processing device, the image processing device executes the steps of the above-mentioned three-dimensional reconstruction method.

[0023] In an embodiment of the present application, by acquiring multiple target images taken of a target physiological region containing a lesion while irradiating the target physiological region using multiple different wavelength light rays, a three-dimensional reconstruction of the target physiological region is performed based on the acquired target images. Since the penetration depths of the target physiological region by different wavelength light rays are not exactly the same, and the number of the target images taken under each wavelength light ray is greater than or equal to 1, the situation of the target physiological region can be analyzed at different penetration depths based on the target images, thereby being able to better reflect the contours of the lesions in the target physiological region at different penetration depths. Using the three-dimensional reconstruction method provided by the present application, doctors can accurately remove lesions based on the contours of the lesions at different penetration depths, thereby helping to improve the accuracy and efficiency of the surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of the specific structure of the endoscope system provided in an embodiment of the present application;

[0027] Figure 3 It is a schematic diagram of the implementation process of the three-dimensional reconstruction method provided in the embodiment of the present application;

[0028] Figure 4 is a schematic diagram of light penetrating the wall of the digestive tract provided in an embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of a specific implementation process of determining the lesion range provided in an embodiment of the present application;

[0030] Figure 6 is a schematic structural diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0031] Figure 7 It is a structural schematic diagram of the image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are protected by the present application.

[0033] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0034] In the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0036] In the field of lesion screening, some doctors will extract diseased tissue by taking a biopsy to determine the depth of infiltration and the type of cancer. This method is difficult to operate, costly, and may stimulate the activity of the lesion. Therefore, at present, doctors usually determine the scope of surgery by injecting dyes, marking with electric knives, etc. when performing endoscopic submucosal dissection, and then perform the dissection operation. When dyes are injected during the operation, the dyes can only be used to observe the size of the lesion on a plane, and cannot reflect the depth of the lesion, which can easily affect the accuracy and efficiency of the operation.

[0037] In view of this, the present application proposes a three-dimensional reconstruction method, which performs three-dimensional reconstruction of the physiological area based on images obtained by light with different penetrating powers, so as to reproduce the contours of lesions at different depths in the physiological area, and can more accurately reflect the contours of lesion tissue, thereby helping doctors improve the accuracy and efficiency of surgery.

[0038] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0039] Please refer to Figure 1 , Figure 1 An image processing system provided by the present application is shown. The image processing system may include a camera and an image processing device.

[0040] The camera may be used to capture the target image. In the embodiments of the present application, the camera may be an optical camera, a video camera or other types of cameras, which are not limited in the present application.

[0041] The target image is an image obtained by photographing the target physiological region by a camera. The target physiological region may refer to any physiological region of any human tissue. In the embodiments of the present application, the target image may specifically refer to an image taken of the target physiological region when the target physiological region containing the lesion is irradiated with a plurality of light beams of different wavelengths. Moreover, the penetration depths of light beams of different wavelengths into the target physiological region are not exactly the same, and the number of target images taken under each light beam of different wavelengths is greater than or equal to 1.

[0042] The image processing device can obtain the target image captured by the camera and perform three-dimensional reconstruction of the target physiological area.

[0043] In some embodiments of the present application, the above-mentioned image processing system may further include a light source, which can emit a plurality of light beams of different wavelengths to the target physiological area, so as to meet the need for capturing the target image under light beams of different wavelengths. It should be noted that the light source may emit a plurality of light beams of different wavelengths in sequence, or may emit a plurality of light beams of different wavelengths at the same time. When the light source emits a plurality of light beams of different wavelengths at the same time, these light beams may form a mixed light, such as white light. This application does not impose any limitation on this.

[0044] In some scenarios, the above-mentioned image processing system may specifically refer to an endoscope system. Figure 2 As shown, the endoscope system may include an image processing device, a light source, a display device, and a mirror body. A camera may be configured on the mirror body, and the mirror body enters the human body to capture a target image. The captured target image may be displayed on the display device in real time.

[0045] In some implementations, a control component may also be provided on the mirror body, and the control component may be used to control the camera to capture images.

[0046] certainly, Figure 1 and Figure 2 It is only a schematic diagram of the image processing system and the endoscope system. In actual applications, the image processing system and the endoscope system may also include more or fewer devices, for example, a bus, a power supply device, etc., and this application does not impose any limitation on this.

[0047] Please refer to Figure 3 , Figure 3 A schematic diagram of the implementation process of a three-dimensional reconstruction method provided in an embodiment of the present application is shown. The method can be applied to an image processing device and is suitable for situations where the contour of lesion tissue needs to be reflected more accurately.

[0048] Among them, the above-mentioned image processing device can be a smart device such as a computer or a mobile phone, or it can be a device specifically used for image processing in a medical system, such as an image processing device in an endoscope system, and this application does not impose any restrictions on this.

[0049] Specifically, the above three-dimensional reconstruction method may include the following steps S301 to S302.

[0050] Step S301 , obtaining a plurality of target images of a target physiological region when the target physiological region including the lesion is irradiated with a plurality of light beams of different wavelength bands.

[0051] The target physiological region may refer to any physiological region of any human tissue containing a lesion. In some embodiments of the present application, a lesion screening method such as an endoscope may be used to determine the approximate region where the lesion is located, and the region may be used as the target physiological region for three-dimensional modeling, so that the lesion removal process may be performed based on the three-dimensional model within the target physiological region.

[0052] In order to obtain the general outline of the target physiological area, in the embodiment of the present application, a plurality of different wavelength bands of light can be used to irradiate the target physiological area containing the lesion. The penetration depth of different wavelength bands of light to the target physiological area is not exactly the same, and the longer the wavelength, the deeper the penetration depth.

[0053] For easier understanding, please refer to Figure 4 , Figure 4 The schematic diagram shows the penetration of light of different wavelengths into the target physiological area. The wavelengths of light A, B, and C increase in sequence. Figure 4 In the figure, the curve shown in 41 can represent the skin or the wall of the digestive tract, and the light rays A, B, and C can penetrate the wall of the digestive tract and reach the penetration depths A', B', and C', respectively. The depths corresponding to the penetration depths A', B', and C' increase in sequence. At this time, the physiological tissue at the penetration depth A' can absorb and reflect the light ray A, and the reflected signal can be photographed to obtain the target image under the light ray A. Similarly, the physiological tissue at the penetration depth B' can absorb and reflect the light ray B, and the reflected signal can be photographed to obtain the target image under the light ray B, and the physiological tissue at the penetration depth C' can absorb and reflect the light ray C, and the reflected signal can be photographed to obtain the target image under the light ray C.

[0054] In practical applications, the above-mentioned multiple different wavelength bands of light can be narrow-band light of different wavelength bands. For example, it can be narrow-band light of 400-480nm, narrow-band light of 540-580nm, and narrow-band light of 600-900nm. Among them, narrow-band light of 400-480nm can reflect the information of shallow mucosa (penetration depth of about 0-0.5mm), narrow-band light of 540-580nm can reflect the mucosal information of the location of venous blood vessels in the middle layer of mucosa (penetration depth of about 0.5-1.5mm), and narrow-band light of 600-900nm can reflect the information of deep mucosa to submucosal layer (penetration depth of about 1.5-3mm).

[0055] As an embodiment of the present application, in the case where the lesion is cell canceration, since the cancerous cells penetrate to a certain depth, they need to be treated by surgery and are not suitable for early cancer treatment. The longest wavelength corresponding to the above-mentioned multiple different bands of light may not exceed 950nm, and the band used (that is, the detection depth) can be adapted to early cancer treatment. In other words, if the wavelength of the light exceeds 950nm and there is still a lesion at the corresponding penetration depth, surgery is required.

[0056] In the implementation manner of the present application, the number of target images captured under each wavelength band of light is greater than or equal to 1. Figure 4 As can be seen from the description, the target image under a single wavelength band of light can reflect the physiological tissue conditions at the corresponding penetration depth. The target images under different wavelength bands of light can respectively reflect the physiological tissue conditions at different penetration depths, and further reflect the contours of objects (such as lesions, blood vessels, etc.) at each penetration depth.

[0057] Step S302: Perform three-dimensional reconstruction of the target physiological region according to the acquired target image.

[0058] In the implementation of the present application, the target image under a single wavelength band of light can reflect the two-dimensional plane information at a single penetration depth. Based on the target images under different wavelength bands of light, the two-dimensional plane information of different depths of the target physiological area can be obtained. Therefore, the image processing device can perform three-dimensional reconstruction of the target physiological area based on the acquired target image.

[0059] It is understandable that the three-dimensional reconstruction of the target physiological region may be the three-dimensional reconstruction of the entire target physiological region, or it may be the three-dimensional reconstruction of only some specific physiological tissues in the target physiological region, for example, the three-dimensional reconstruction of only the lesions and blood vessels in the target physiological region, and this application does not impose any limitation on this.

[0060] The 3D model obtained by 3D reconstruction can be applied in different scenarios, such as surgery planning, lesion degree analysis, lesion display, etc., which is also not limited in this application.

[0061] In an embodiment of the present application, by acquiring multiple target images taken of a target physiological region containing a lesion while irradiating the target physiological region using multiple different wavelength light rays, a three-dimensional reconstruction of the target physiological region is performed based on the acquired target images. Since the penetration depths of the target physiological region by different wavelength light rays are not exactly the same, and the number of the target images taken under each wavelength light ray is greater than or equal to 1, the situation of the target physiological region can be analyzed at different penetration depths based on the target images, thereby being able to better reflect the contours of the lesions in the target physiological region at different penetration depths. Using the three-dimensional reconstruction method provided by the present application, doctors can accurately remove lesions based on the contours of the lesions at different penetration depths, thereby helping to improve the accuracy and efficiency of the surgery.

[0062] In some embodiments of the present application, the above-mentioned step S302 may include: determining the point cloud data at each penetration depth of the target physiological area based on the acquired target image, clustering different physiological tissues on the point cloud data, and performing three-dimensional reconstruction of the target physiological area based on the clustering processing results.

[0063] Specifically, the image processing device can determine the point cloud data at the corresponding penetration depth based on the target image corresponding to the single-band light. Among them, the point cloud data may include the characteristic value of the point cloud at each position in the target physiological area. The characteristic value may refer to the grayscale value obtained after image processing, which is related to the signal strength. The signal strength refers to the signal intensity of the physiological tissue at the corresponding position reflecting the light. Since the components of the same physiological tissue are the same or similar, the signal intensity of the light reflected is the same or similar. When clustering the point cloud data, the point cloud of the same physiological tissue can be clustered into a set according to the characteristic value and position to obtain the clustering processing result. At this time, according to the clustering processing result, a triangular mesh processing can be performed to reconstruct the physiological tissue in the target physiological area in three dimensions.

[0064] To facilitate surgical planning or lesion analysis, in step S302, the image processing device may perform three-dimensional reconstruction of the lesion in the target physiological region.

[0065] Specifically, in some embodiments of the present application, step S302 may include: determining the range of the lesion at each penetration depth of the target physiological region according to the acquired target image, so as to perform three-dimensional reconstruction of the lesion according to the range of the lesion.

[0066] At a single penetration depth, based on the target image under the light corresponding to the penetration depth, the range of the lesion at the penetration depth can be screened out. At this time, according to the target images corresponding to the light of different bands obtained, the range of the lesion can be determined at each penetration depth of the target physiological area, and the two-dimensional contour of the lesion at each different penetration depth can be obtained. Then, according to the range of the lesion, the lesion can be reconstructed in three dimensions.

[0067] In some embodiments of the present application, the image processing device may determine the range of the lesion based on the distribution of physiological tissues.

[0068] For details, please refer to Figure 5 The above-mentioned determining the range of the lesion at each penetration depth of the target physiological area based on the acquired target image may include steps S501 to S503.

[0069] Step S501 : determining the physiological tissue to which each position belongs at each penetration depth of the target physiological region according to the acquired target image.

[0070] Specifically, the pixel information of the pixel point in the target image can reflect the physiological tissue condition at the corresponding position, for example, different pixel values ​​can correspond to different physiological tissues. Based on the target image under a single band of light, the physiological tissue to which each position belongs can be determined on a two-dimensional plane at the corresponding penetration depth.

[0071] The above physiological tissues may include but are not limited to mucosa and blood vessels.

[0072] Step S502: at each penetration depth of the target physiological region, the distribution of blood vessels is determined according to the physiological tissue to which each position belongs.

[0073] In the embodiments of the present application, for a single penetration depth, the distribution of blood vessels at the penetration depth may be determined according to the physiological tissue to which each position belongs.

[0074] The distribution of blood vessels may include at least one of the following: the distance between blood vessels, the number of blood vessels within a unit range (or density), etc.

[0075] Step S503, determining the range of the lesion at each penetration depth of the target physiological region according to the distribution of blood vessels.

[0076] Since lesions such as cell cancer require a large amount of oxygen supply, the blood vessels in the range of the lesion are denser than normal cells, and the blood vessels are often in an adhesion state. Therefore, based on the distribution of blood vessels such as the distance between blood vessels and the number of blood vessels in a unit range, it can be determined whether the blood vessels are adhesion. The area where the blood vessels are adhesion is the range where the lesion is located. For example, when the distance between blood vessels is less than the distance threshold, or the number of blood vessels in a unit range is greater than the number threshold, blood vessel adhesion can be confirmed.

[0077] It should be noted that, in the embodiments of the present application, the range of the lesion is determined at each penetration depth of the target physiological area mainly depending on the distribution of capillaries.

[0078] The range of the lesion at each penetration depth is obtained, that is, the two-dimensional contour of the lesion at each penetration depth is obtained. At this time, three-dimensional reconstruction can be performed according to the range of the lesion to obtain a three-dimensional model of the lesion.

[0079] In order to facilitate doctors in surgical planning, after obtaining the range of the lesion, the image processing device can also reflect the range of the lesion at different penetration depths in the two-dimensional image.

[0080] Specifically, in some embodiments of the present application, the image processing device can determine the three-dimensional size data of the lesion based on the results of the three-dimensional reconstruction, and send the three-dimensional size data to the display device to be superimposed on the two-dimensional image displayed in real time by the display device.

[0081] The three-dimensional size data may include the length, width and height data of the lesion. The two-dimensional image may be a white light image obtained by photographing the target physiological area. The three-dimensional size data reflected by the lesion in the range of different penetration depths is superimposed on the two-dimensional plane presented by the two-dimensional image, so that the deep contour of the lesion can be reflected on the two-dimensional image.

[0082] Accordingly, the results of the two-dimensional image or three-dimensional reconstruction can be used for surgical planning. Among them, surgical planning at least includes: determining the cutting range of the lesion. In other words, when the doctor views the white light image, he can comprehensively consider the area of ​​the lower layer beyond the surface range to finally confirm the cutting range of the lesion, so that the cutting range can cover the range of the lesion at any penetration depth. Of course, for the endoscope system, the cutting range of the lesion can also be determined based on the results of the two-dimensional image or three-dimensional reconstruction, and the cutting range can be marked on the two-dimensional image for the doctor's reference.

[0083] Considering that the surgical process needs to avoid larger blood vessels to avoid the risk of massive bleeding, in some embodiments of the present application, the image processing device can also determine the range of the target blood vessel at each penetration depth of the target physiological area based on the acquired target image. Then, prompt information is generated based on the range of the target blood vessel and the range of the lesion.

[0084] The target blood vessel may specifically refer to a blood vessel with a blood vessel diameter greater than a preset diameter. Alternatively, the target blood vessel is a blood vessel of a preset type, such as an arterial blood vessel.

[0085] In the embodiments of the present application, the prompt information can be used to indicate the position of the target blood vessel relative to the lesion. The prompt information can be marked in the aforementioned two-dimensional image, or can be output in the form of text information or voice information, which is not limited by the present application. Based on the prompt information, the doctor can avoid the target blood vessel during surgery to reduce the risk of massive bleeding.

[0086] As mentioned above, when cancerous cells penetrate to a certain depth, they need to be treated through surgery, which is not suitable for the treatment of early cancer. Therefore, in some embodiments of the present application, before performing three-dimensional reconstruction of the target physiological area, the three-dimensional reconstruction method may also include: obtaining a target image corresponding to the longest wavelength light, and performing cell cancer detection at the deepest penetration depth based on the target image corresponding to the longest wavelength light. If cell cancer is detected at the deepest penetration depth, the three-dimensional reconstruction is stopped, and the degree of cancer is prompted.

[0087] The longest wavelength light is the light with the longest wavelength among the multiple different wavelength light, and the longest wavelength light corresponds to the deepest penetration depth of the target physiological area.

[0088] The method of detecting cell canceration can refer to the method of determining the range of the lesion, which is not described in detail in this application. The canceration degree prompt is used to indicate that the current lesion does not belong to early cancer.

[0089] That is to say, if cancerous cells are detected at the deepest penetration depth, it means that the cancerous cells have penetrated to a certain depth and are not early-stage cancers, and need to be treated by surgery. Considering that 3D reconstruction mainly serves the treatment of early-stage cancers, the image processing device can stop 3D reconstruction and provide a cancer degree prompt. In this way, the processing load of the image processing device can be reduced, and at the same time, prompts for mid- and late-stage cancers can be quickly provided to prompt patients to undergo surgery as soon as possible.

[0090] In addition, in step S301 and step S302, when the light source adopts different irradiation methods, the present application can provide different processing methods.

[0091] Specifically, in some embodiments of the present application, step S301 may include acquiring a plurality of target images of the target physiological region taken by sequentially irradiating the target physiological region with light of a plurality of different wavelength bands.

[0092] It should be noted that the present application does not limit the order in which light is irradiated.

[0093] In some embodiments, the irradiation can be performed in order from low to high wavelengths. For example, the light source can emit narrow-band light from the 400nm band to the 900nm band in a manner of increasing the wavelength by 10nm every 50 milliseconds. In this way, if no cell canceration is detected at any penetration depth, it is possible to stop acquiring target images with deeper penetration depths, or stop image processing of target images with deeper penetration depths, which helps to improve the efficiency of three-dimensional reconstruction.

[0094] In other embodiments, the irradiation may be performed in a descending order of wavelength. For example, the light source may emit narrow-band light from the 900nm band to the 400nm band in a manner that the wavelength is reduced by 10nm every 50 milliseconds. In this way, if cell canceration is detected at the deepest penetration depth, the three-dimensional reconstruction may be stopped and the degree of canceration may be indicated.

[0095] When the light source is controlled to irradiate light of different wavelength bands in sequence, the obtained target images each correspond to a wavelength band. At this time, the three-dimensional reconstruction can be completed according to the method of performing three-dimensional reconstruction based on the target image as described above.

[0096] In other embodiments of the present application, the light irradiating the target physiological area may include light of multiple different wavelength bands. For example, the light source may output mixed light of 400nm-900nm.

[0097] At this time, in step S302, the processing device can reconstruct the target image to obtain the image to be processed corresponding to each of the multiple different wavelength bands of light, and then reconstruct the target physiological area in three dimensions according to the image to be processed.

[0098] Specifically, in this case, it is necessary to calculate the image information of a specific band based on the target image containing light of each band, according to the camera's reflection coefficient L, the filter transmittance F, the distribution of each band of the light source E, the reflectivity O of physiological tissue to light of different bands, and the camera's response coefficient S to each wavelength. The specific formula can be expressed as:

[0099] v i =E i ×F i ×L i ×S i ×o i ;

[0100] Where i represents the wavelength value of the image to be processed, v i That is the image to be processed in a specific band.

[0101] After obtaining the images to be processed corresponding to each wavelength band, the target physiological area can be reconstructed in three dimensions according to the images to be processed. The method of reconstructing in three dimensions according to the images to be processed can refer to the description of the method of reconstructing in three dimensions according to the target image in the previous text, which will not be described in detail in this application.

[0102] It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders.

[0103] like Figure 6 FIG. 6 is a schematic structural diagram of a three-dimensional reconstruction device 600 provided in an embodiment of the present application. The three-dimensional reconstruction device 600 is configured on an image processing device.

[0104] Specifically, the 3D reconstruction device 600 may include:

[0105] An image acquisition unit 601 is used to acquire a plurality of target images of a target physiological region including a lesion when the target physiological region is irradiated with a plurality of light beams of different wavelengths, wherein the penetration depths of the light beams of different wavelengths into the target physiological region are not completely the same, and the number of the target images captured under each light beam of different wavelengths is greater than or equal to 1;

[0106] The three-dimensional reconstruction unit 602 is used to perform three-dimensional reconstruction of the target physiological region according to the acquired target image.

[0107] In some embodiments of the present application, the three-dimensional reconstruction unit 602 can be specifically used to: determine the range of the lesion at each penetration depth of the target physiological area based on the acquired target image, so as to perform three-dimensional reconstruction of the lesion based on the range of the lesion.

[0108] In some embodiments of the present application, the three-dimensional reconstruction unit 602 can be specifically used to: determine the physiological tissue to which each position belongs at each penetration depth of the target physiological area based on the acquired target image; determine the distribution of the blood vessels at each penetration depth of the target physiological area based on the physiological tissue to which each position belongs; determine the range of the lesion at each penetration depth of the target physiological area based on the distribution of the blood vessels.

[0109] In some embodiments of the present application, the distribution of blood vessels includes at least one of the following: the distance between the blood vessels, and the number of the blood vessels within a unit range.

[0110] In some embodiments of the present application, the three-dimensional reconstruction unit 602 can be specifically used to: determine the point cloud data at each penetration depth of the target physiological area based on the acquired target image; cluster different physiological tissues on the point cloud data, and perform three-dimensional reconstruction of the target physiological area based on the clustering processing results.

[0111] In some embodiments of the present application, the point cloud data may include a characteristic value of the point cloud at each position in the target physiological area, where the characteristic value is related to the signal strength, which is the signal strength of the physiological tissue at the corresponding position reflecting the light.

[0112] In some embodiments of the present application, the three-dimensional reconstruction device 600 may also include an information processing unit for determining three-dimensional size data of the lesion based on the result of the three-dimensional reconstruction, the three-dimensional size data including length, width and height data of the lesion; and sending the three-dimensional size data to a display device to be superimposed on the two-dimensional image displayed in real time by the display device.

[0113] In some embodiments of the present application, the above-mentioned information processing unit can also be specifically used to: determine the range of the target blood vessel at each penetration depth of the target physiological area according to the acquired target image; generate prompt information according to the range of the target blood vessel and the range of the lesion, and the prompt information is used to indicate the position of the target blood vessel relative to the lesion.

[0114] In some embodiments of the present application, the above-mentioned information processing unit can also be specifically used for: before the three-dimensional reconstruction of the target physiological area, the three-dimensional reconstruction method also includes: obtaining the target image corresponding to the longest wavelength light, the longest wavelength light is the light with the longest wavelength among the multiple different wavelength lights, and the longest wavelength light corresponds to the deepest penetration depth of the target physiological area; according to the target image corresponding to the longest wavelength light, performing cell canceration detection at the deepest penetration depth; if cell canceration is detected at the deepest penetration depth, stopping the three-dimensional reconstruction and providing a prompt of the degree of canceration.

[0115] In some embodiments of the present application, the longest wavelength corresponding to the multiple different wavelength bands of light does not exceed 950 nm.

[0116] In some embodiments of the present application, the image acquisition unit 601 may be specifically used to: acquire a plurality of target images of the target physiological region obtained by photographing the target physiological region when the target physiological region is irradiated with the plurality of light beams of different wavelengths in sequence.

[0117] In some embodiments of the present application, the light irradiating the target physiological area includes the multiple different-band light; the three-dimensional reconstruction unit 602 can be specifically used to: reconstruct the target image to obtain a to-be-processed image corresponding to each of the multiple different-band light; and perform three-dimensional reconstruction of the target physiological area based on the to-be-processed image.

[0118] It should be noted that, for the convenience and simplicity of description, the specific working process of the above-mentioned 3D reconstruction device 600 can be referred to Figures 1 to 5 The corresponding process of the method will not be repeated here.

[0119] like Figure 7 FIG. 7 is a schematic diagram of an image processing device 7 provided in an embodiment of the present application. Specifically, the image processing device 7 may include: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a 3D reconstruction program. When the processor 70 executes the computer program 72, the steps in the above-mentioned 3D reconstruction method embodiments are implemented, such as Figure 3 Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 6 The functions of the image acquisition unit 601 and the 3D reconstruction unit 602 are shown.

[0120] The computer program may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the image processing device.

[0121] For example, the computer program can be divided into: an image acquisition unit and a three-dimensional reconstruction unit. The specific functions of each unit are as follows: an image acquisition unit, used to acquire multiple target images of the target physiological region containing the lesion when the target physiological region is irradiated with multiple different wavelength light, wherein the penetration depths of the different wavelength light to the target physiological region are not exactly the same, and the number of the target images taken under each wavelength light is greater than or equal to 1; a three-dimensional reconstruction unit, used to perform three-dimensional reconstruction of the target physiological region based on the acquired target images.

[0122] The image processing device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of an image processing device and does not constitute a limitation of the image processing device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the image processing device may also include input and output devices, network access devices, buses, etc.

[0123] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0124] The memory 71 may be an internal storage unit of the image processing device, such as a hard disk or memory of the image processing device. The memory 71 may also be an external storage device of the image processing device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the image processing device. Further, the memory 71 may also include both an internal storage unit and an external storage device of the image processing device. The memory 71 is used to store the computer program and other programs and data required by the image processing device. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0125] It should be noted that, for the convenience and brevity of description, the structure of the above-mentioned image processing device 7 can also refer to the specific description of the structure in the method embodiment, and will not be repeated here.

[0126] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0127] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0128] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0133] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire a plurality of target images of the target physiological region when the target physiological region including the lesion is irradiated with a plurality of light beams of different wavelengths, wherein the penetration depths of the light beams of different wavelengths into the target physiological region are not completely the same, and the number of the target images taken under each light beam of different wavelengths is greater than or equal to 1; The target physiological region is three-dimensionally reconstructed according to the acquired target image.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The three-dimensional reconstruction of the target physiological region according to the acquired target image includes: According to the acquired target image, the range of the lesion is determined at each penetration depth of the target physiological region, so as to perform three-dimensional reconstruction of the lesion according to the range of the lesion.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: Determining the range of the lesion at each penetration depth of the target physiological region according to the acquired target image includes: Determine the physiological tissue to which each position belongs at each penetration depth of the target physiological region according to the acquired target image; At each penetration depth of the target physiological region, determining the distribution of blood vessels according to the physiological tissues to which each position belongs; According to the distribution of the blood vessels, the range of the lesion is determined at each penetration depth of the target physiological area.

4. The three-dimensional reconstruction method according to claim 3, characterized in that: The distribution of the blood vessels includes at least one of the following: the distance between the blood vessels, and the number of the blood vessels within a unit range.

5. The three-dimensional reconstruction method according to claim 1, characterized in that: The three-dimensional reconstruction of the target physiological region according to the acquired target image includes: Determining point cloud data of the target physiological region at each penetration depth according to the acquired target image; The point cloud data is clustered according to different physiological tissues, and the target physiological region is three-dimensionally reconstructed according to the clustering processing result.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: The point cloud data includes characteristic values ​​of the point cloud at various positions within the target physiological area, and the characteristic values ​​are related to signal strength, which is the signal strength of the light reflected by the physiological tissue at the corresponding position.

7. The three-dimensional reconstruction method according to claim 1, characterized in that: The three-dimensional reconstruction method further comprises: Determine the three-dimensional size data of the lesion based on the result of the three-dimensional reconstruction, wherein the three-dimensional size data includes the length, width and height data of the lesion; The three-dimensional size data is sent to a display device to be superimposed on a two-dimensional image displayed in real time by the display device.

8. The three-dimensional reconstruction method according to claim 1, characterized in that: The three-dimensional reconstruction method further comprises: Determining the range of a target blood vessel at each penetration depth of the target physiological region according to the acquired target image, wherein the target blood vessel is a blood vessel having a blood vessel diameter greater than a preset diameter, or the target blood vessel is a blood vessel of a preset type; Prompt information is generated according to the range where the target blood vessel is located and the range where the lesion is located, and the prompt information is used to indicate the position of the target blood vessel relative to the lesion.

9. The three-dimensional reconstruction method according to any one of claims 1 to 8, characterized in that: The lesions include cell canceration; Before performing three-dimensional reconstruction on the target physiological region, the three-dimensional reconstruction method further includes: Acquire the target image corresponding to the longest wavelength light, wherein the longest wavelength light is the light with the longest wavelength among the multiple different wavelength light, and the longest wavelength light corresponds to the deepest penetration depth of the target physiological area; Performing cell canceration detection at the deepest penetration depth according to the target image corresponding to the light of the longest wavelength band; If cell canceration is detected at the deepest penetration depth, the three-dimensional reconstruction is stopped and the degree of canceration is indicated.

10. The three-dimensional reconstruction method according to any one of claims 1 to 8, characterized in that: The longest wavelength corresponding to the multiple light beams of different wavelength bands does not exceed 950nm.

11. The three-dimensional reconstruction method according to any one of claims 1 to 8, characterized in that: The step of acquiring a plurality of target images of the target physiological region taken when the target physiological region including the lesion is irradiated with a plurality of light beams of different wavelengths comprises: acquiring a plurality of target images of the target physiological region taken when the target physiological region is irradiated with the plurality of light beams of different wavelengths in sequence; Alternatively, the light irradiating the target physiological area includes the multiple different-band light; accordingly, the three-dimensional reconstruction of the target physiological area is performed based on the acquired target image, including: image reconstruction of the target image to obtain an image to be processed corresponding to each of the multiple different-band light; and three-dimensional reconstruction of the target physiological area based on the image to be processed.

12. A three-dimensional reconstruction device, characterized in that: include: An image acquisition unit, used to acquire a plurality of target images taken of a target physiological region containing a lesion when the target physiological region is irradiated with a plurality of light beams of different wavelengths, wherein the penetration depths of the light beams of different wavelengths into the target physiological region are not completely the same, and the number of the target images taken under each light beam of different wavelengths is greater than or equal to 1; A three-dimensional reconstruction unit is used to perform three-dimensional reconstruction of the target physiological area according to the acquired target image.

13. An image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 11 are implemented.

14. An endoscope system, characterized in that: include: Light source, mirror, image processing equipment and display equipment; Wherein, the image processing device is used to execute the steps of the three-dimensional reconstruction method as claimed in any one of claims 1 to 11.

Images