Three-dimensional reconstruction method, device, image processing equipment and endoscope system

Abstract

The present application is applicable to the field of medical technology and provides a three-dimensional reconstruction method, apparatus, image processing device and endoscope system. The three-dimensional reconstruction method includes: obtaining multiple target images of the target physiological region containing the lesion when the target physiological region is irradiated with a plurality of light beams of different wavelengths, wherein the penetration depths of the target physiological region by the light beams of different wavelengths are not exactly the same, and the number of the target images taken under each light beam of wavelength is greater than or equal to 1; and performing three-dimensional reconstruction of the target physiological region based on the acquired target images. The embodiments of the present application can more accurately reflect the contours of the lesion tissue.

Description

Technical Field

[0001] The present application belongs to the field of medical technology, and in particular relates to a three-dimensional reconstruction method, apparatus, image processing equipment and endoscope system. Background Art

[0002] In the field of lesion screening, some doctors will extract diseased tissue through biopsy to determine the depth of invasion and the type of cancer. This method is difficult to operate, costly, and may stimulate lesion activity. Therefore, doctors currently usually determine the surgical scope by injecting dyes and using electrocautery markings when performing endoscopic submucosal dissection (ESD) before performing the dissection operation. When dyes are injected during surgery, they can only be used to observe the size of the lesion on a flat surface and cannot reflect the depth of the lesion, which can easily affect the accuracy and efficiency of the surgery.

[0003] Therefore, a method that can more accurately reflect the contour of lesion tissue is needed. Summary of the Invention

[0004] The embodiments of the present application provide a three-dimensional reconstruction method, apparatus, image processing device, and endoscope system, which can more accurately reflect the contours of lesion tissue.

[0005] A first aspect of an embodiment of the present application provides a three-dimensional reconstruction method, comprising: obtaining multiple target images of a target physiological region containing a lesion when the target physiological region is irradiated with light of multiple different wavelengths, wherein the penetration depths of light of different wavelengths into the target physiological region are not exactly the same, and the number of target images taken under each wavelength is greater than or equal to 1; and performing three-dimensional reconstruction of the target physiological region based on the acquired target images.

[0006] In some embodiments of the first aspect, after performing three-dimensional reconstruction of the target physiological area based on the acquired target image and obtaining feedback signals of the target physiological area to light signals of different bands, the method further includes: determining the range of the first target object of the lesion at each penetration depth of the target physiological area based on the feedback signals corresponding to different bands of the acquired target image, so as to perform three-dimensional reconstruction of the lesion based on the range where the lesion is located.

[0007] In some embodiments of the first aspect, determining the range of the lesion at each penetration depth of the target physiological region based on the acquired target image includes: determining the physiological tissue to which each position belongs at each penetration depth of the target physiological region based on the acquired target image; determining the distribution of the blood vessels at each penetration depth of the target physiological region based on the physiological tissue to which each position belongs; and determining the range of the lesion at each penetration depth of the target physiological region based on the distribution of the blood vessels.

[0008] In some embodiments of the first aspect, the distribution of the blood vessels includes at least one of the following: the distance between the blood vessels, and the number of the blood vessels within a unit range.

[0009] In some embodiments of the first aspect, the three-dimensional reconstruction of the target physiological region based on the acquired target image includes: determining point cloud data at each penetration depth of the target physiological region based on the acquired target image; clustering different physiological tissues on the point cloud data, and three-dimensionally reconstructing the target physiological region based on the clustering processing results.

[0010] In some embodiments of the first aspect, the point cloud data includes characteristic values ​​of the point cloud at each position in the target physiological area, and the characteristic values ​​are related to signal intensity, which is the signal intensity of light reflected by the physiological tissue at the corresponding position.

[0011] In some embodiments of the first aspect, the three-dimensional reconstruction method further includes: determining three-dimensional size data of the lesion based on the results of the three-dimensional reconstruction, the three-dimensional size data including length, width and height data of the lesion; and sending the three-dimensional size data to a display device to be superimposed on the two-dimensional image displayed in real time by the display device.

[0012] In some embodiments of the first aspect, the two-dimensional image or the three-dimensional reconstruction result is used for surgical planning, wherein the surgical planning at least includes: determining the excision range of the lesion.

[0013] In some embodiments of the first aspect, the three-dimensional reconstruction method further includes: determining, based on the acquired target image, a range of a target blood vessel at each penetration depth of the target physiological region, the target blood vessel being a blood vessel having a diameter greater than a preset diameter, or the target blood vessel being a blood vessel of a preset type; and generating prompt information based on the range of the target blood vessel and the range of the lesion, the prompt information being used to indicate the orientation of the target blood vessel relative to the lesion.

[0014] In some embodiments of the first aspect, the lesion includes cell canceration; before performing three-dimensional reconstruction of the target physiological area, the three-dimensional reconstruction method further includes: obtaining the target image corresponding to the longest wavelength light, the longest wavelength light being the light with the longest wavelength among the multiple different wavelength lights, and the longest wavelength light corresponding to the deepest penetration depth of the target physiological area; based on the target image corresponding to the longest wavelength light, performing cell canceration detection at the deepest penetration depth; if cell canceration is detected at the deepest penetration depth, stopping the three-dimensional reconstruction and providing a canceration degree prompt.

[0015] In some embodiments of the first aspect, the longest wavelength corresponding to the multiple lights of different wavelength bands does not exceed 950 nm.

[0016] In some embodiments of the first aspect, the acquiring of a plurality of target images of the target physiological region containing the lesion when the target physiological region is irradiated with a plurality of light rays of different wavelengths comprises: acquiring a plurality of target images of the target physiological region when the target physiological region is sequentially irradiated with the plurality of light rays of different wavelengths; or, the light irradiating the target physiological region comprises the plurality of light rays of different wavelengths; and performing three-dimensional reconstruction of the target physiological region based on the acquired target images comprises: performing image reconstruction on the target images to obtain images to be processed corresponding to each of the plurality of light rays of different wavelengths; and performing three-dimensional reconstruction of the target physiological region based on the images to be processed.

[0017] A second aspect of an embodiment of the present application provides a three-dimensional reconstruction device, including: an image acquisition unit, used to acquire multiple target images of a target physiological region containing a lesion when the target physiological region is irradiated with light of multiple different wavelengths, wherein the penetration depths of light of different wavelengths into the target physiological region are not exactly the same, and the number of target images taken under each wavelength is greater than or equal to 1; and a three-dimensional reconstruction unit, used to perform three-dimensional reconstruction of the target physiological region based on the acquired target images.

[0018] A third aspect of an embodiment of the present application provides an image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned three-dimensional reconstruction method when executing the computer program.

[0019] A fourth aspect of an embodiment of the present application provides an image processing system, comprising an image processing device and a camera, wherein the camera is used to capture a target image, and the image processing device is used to execute the steps of the above-mentioned three-dimensional reconstruction method.

[0020] A fifth aspect of an embodiment of the present application provides an endoscope system, comprising a light source, a scope, an image processing device, and a display device; wherein the image processing device is used to execute the steps of the above-mentioned three-dimensional reconstruction method.

[0021] A sixth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional reconstruction method are implemented.

[0022] A seventh aspect of an embodiment of the present application provides a computer program product, which, when executed on an image processing device, enables the image processing device to execute the steps of the above-mentioned three-dimensional reconstruction method.

[0023] In an embodiment of the present application, by obtaining multiple target images of the target physiological region containing the lesion while irradiating the target physiological region with multiple different wavelength light, a three-dimensional reconstruction of the target physiological region is performed based on the obtained target images. Since the penetration depths of the target physiological region by light of different wavelengths are not exactly the same, and the number of target images taken under each wavelength is greater than or equal to 1, the situation of the target physiological region can be analyzed at different penetration depths based on the target images, thereby better reflecting the contours of the lesion in the target physiological region at different penetration depths. Using the three-dimensional reconstruction method provided by the present application, doctors can accurately remove the lesion based on the contours of the lesion at different penetration depths, thereby helping to improve the accuracy and efficiency of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 is a structural diagram of an image processing system provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of the specific structure of the endoscope system provided in an embodiment of the present application;

[0027] Figure 3 Schematic diagram of the implementation process of the three-dimensional reconstruction method provided in the embodiment of the present application;

[0028] Figure 4 is a schematic diagram of light penetrating the wall of the digestive tract provided by an embodiment of the present application;

[0029] Figure 5 This is a schematic diagram of a specific implementation process for determining the range of a lesion provided in an embodiment of the present application;

[0030] Figure 6 Schematic diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;

[0031] Figure 7 It is a structural diagram of the image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without making any creative work are protected by this application.

[0033] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0034] In the description of this application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0036] In the field of lesion screening, some doctors perform biopsies to extract diseased tissue to determine the depth of invasion and type of cancer. This method is difficult and costly to perform, and may stimulate lesion activity. Therefore, doctors currently typically use methods such as injecting dye and using electrocautery to mark the surgical area during endoscopic submucosal dissection before proceeding with the dissection. The intraoperative injection of dye can only be used to observe the size of the lesion on a flat surface and cannot reflect the depth of the lesion, which can easily affect the accuracy and efficiency of the procedure.

[0037] In view of this, the present application proposes a three-dimensional reconstruction method, which performs three-dimensional reconstruction of physiological areas based on images obtained with light rays of different penetrating powers to reproduce the contours of lesions at different depths within the physiological area, and can more accurately reflect the contours of lesion tissue, thereby helping doctors improve the accuracy and efficiency of surgery.

[0038] In order to illustrate the technical solution of the present application, specific embodiments are provided below.

[0039] Please refer to Figure 1 , Figure 1 An image processing system provided by the present application is shown. The image processing system may include a camera and an image processing device.

[0040] The camera may be used to capture the target image. In the embodiments of the present application, the camera may be an optical camera, a video camera, or other types of cameras, which are not limited in the present application.

[0041] A target image is an image captured by a camera of a target physiological region. The target physiological region may refer to any physiological region of any human tissue. In the embodiments of the present application, a target image may specifically refer to an image of the target physiological region, including a lesion, captured by irradiating the target physiological region with light of multiple wavelengths. Furthermore, the penetration depths of light of different wavelengths into the target physiological region vary, and the number of target images captured using each wavelength is greater than or equal to one.

[0042] The image processing device can obtain the target image captured by the camera and perform three-dimensional reconstruction of the target physiological area.

[0043] In some embodiments of the present application, the image processing system may further include a light source capable of emitting light of multiple different wavelengths toward the target physiological region, thereby satisfying the need to capture target images under light of different wavelengths. It should be noted that the light source may emit light of multiple different wavelengths sequentially or simultaneously. When the light source emits light of multiple different wavelengths simultaneously, these light rays may form a mixed light, such as white light. This application does not impose any limitations on this.

[0044] In some scenarios, the above-mentioned image processing system may specifically refer to an endoscope system. Figure 2 As shown, an endoscope system may include an image processing device, a light source, a display device, and a scope. A camera may be configured on the scope and enter the human body to capture a target image. The captured target image may be displayed in real time on the display device.

[0045] In some embodiments, a control component may be provided on the mirror body, and the control component may be used to control the camera to capture images.

[0046] certainly, Figure 1 and Figure 2 This is merely a schematic illustration of the image processing system and the endoscope system. In actual applications, the image processing system and the endoscope system may include more or fewer devices, for example, a bus, a power supply device, etc., and this application does not impose any restrictions on this.

[0047] Please refer to Figure 3 , Figure 3 A schematic diagram of the implementation process of a three-dimensional reconstruction method provided in an embodiment of the present application is shown. The method can be applied to image processing equipment and is suitable for situations where the contour of lesion tissue needs to be reflected more accurately.

[0048] Among them, the above-mentioned image processing device can be a smart device such as a computer or a mobile phone, or it can be a device specifically used for image processing in a medical system, such as an image processing device in an endoscope system, and this application does not impose any restrictions on this.

[0049] Specifically, the above three-dimensional reconstruction method may include the following steps S301 to S302.

[0050] Step S301 : acquiring a plurality of target images of a target physiological region including a lesion by irradiating the target physiological region with light of a plurality of different wavelength bands.

[0051] The target physiological region may refer to any physiological region of any human tissue containing a lesion. In some embodiments of the present application, a lesion screening method such as an endoscope may be used to determine the approximate region where the lesion is located, and a three-dimensional model of this region may be generated as the target physiological region, so that the lesion can be removed based on the three-dimensional model within the target physiological region.

[0052] In order to obtain the general outline of the target physiological area, in the embodiments of the present application, multiple different wavelengths of light can be used to irradiate the target physiological area containing the lesion. The penetration depth of light of different wavelengths into the target physiological area is not exactly the same, and the longer the wavelength, the deeper the penetration depth.

[0053] For easier understanding, please refer to Figure 4 , Figure 4 A schematic diagram showing how light of different wavelengths penetrates a target physiological area. The wavelengths of light A, B, and C increase in sequence. Figure 4 In the figure, the curve shown in 41 can represent the skin or the wall of the digestive tract. Light rays A, B, and C can penetrate the digestive tract wall and reach penetration depths A', B', and C', respectively. The depths corresponding to penetration depths A', B', and C' increase in sequence. At this time, physiological tissue at penetration depth A' can absorb and reflect light ray A, and the reflected signal can be captured to obtain the target image under light ray A. Similarly, physiological tissue at penetration depth B' can absorb and reflect light ray B, and the reflected signal can be captured to obtain the target image under light ray B. Physiological tissue at penetration depth C' can absorb and reflect light ray C, and the reflected signal can be captured to obtain the target image under light ray C.

[0054] In practical applications, the above-mentioned multiple different wavelength bands of light can be narrowband light of different wavelength bands. For example, it can be narrowband light of 400-480nm, narrowband light of 540-580nm, and narrowband light of 600-900nm. Among them, the narrowband light of 400-480nm can reflect the information of shallow mucosa (penetration depth of about 0-0.5mm), the narrowband light of 540-580nm can reflect the mucosal information of the location of veins in the middle layer of mucosa (penetration depth of about 0.5-1.5mm), and the narrowband light of 600-900nm can reflect the information of deep mucosa to submucosa (penetration depth of about 1.5-3mm).

[0055] As an embodiment of the present application, for cases where the lesion is a cancerous cell, since the cancerous cells penetrate to a certain depth and require surgical treatment, it is not suitable to treat it as an early-stage cancer. The longest wavelength corresponding to the above-mentioned multiple different wavelength bands of light can be no more than 950nm, and the wavelength band used (i.e., the detection depth) can be adapted to the treatment of early-stage cancer. In other words, if the wavelength of the light exceeds 950nm and the lesion is still present at the corresponding penetration depth, surgical treatment is required.

[0056] In the embodiment of the present application, the number of target images captured under each wavelength band is greater than or equal to 1. Figure 4 As can be seen from the description, a target image under a single wavelength band can reflect the physiological tissue conditions at the corresponding penetration depth. Target images under different wavelength bands can respectively reflect the physiological tissue conditions at different penetration depths, and thus reflect the contours of objects (such as lesions, blood vessels, etc.) at each penetration depth.

[0057] Step S302: Perform three-dimensional reconstruction of the target physiological region based on the acquired target image.

[0058] In the embodiments of the present application, a target image captured by a single wavelength band of light can reflect two-dimensional planar information at a single penetration depth. Based on target images captured by different wavelength bands of light, two-dimensional planar information at different depths of the target physiological region can be obtained. Therefore, the image processing device can perform three-dimensional reconstruction of the target physiological region based on the acquired target images.

[0059] It is understandable that the three-dimensional reconstruction of the target physiological area can be a three-dimensional reconstruction of the entire target physiological area, or it can be a three-dimensional reconstruction of only some specific physiological tissues in the target physiological area, for example, only three-dimensional reconstruction of lesions and blood vessels in the target physiological area, and this application does not impose any restrictions on this.

[0060] The 3D model obtained by 3D reconstruction can be applied in different scenarios, such as surgical planning, lesion severity analysis, lesion display, etc., which is also not limited by this application.

[0061] In an embodiment of the present application, by obtaining multiple target images of the target physiological region containing the lesion while irradiating the target physiological region with multiple different wavelength light, a three-dimensional reconstruction of the target physiological region is performed based on the obtained target images. Since the penetration depths of the target physiological region by light of different wavelengths are not exactly the same, and the number of target images taken under each wavelength is greater than or equal to 1, the situation of the target physiological region can be analyzed at different penetration depths based on the target images, thereby better reflecting the contours of the lesion in the target physiological region at different penetration depths. Using the three-dimensional reconstruction method provided by the present application, doctors can accurately remove the lesion based on the contours of the lesion at different penetration depths, thereby helping to improve the accuracy and efficiency of the operation.

[0062] In some embodiments of the present application, the above-mentioned step S302 may include: determining the point cloud data at each penetration depth of the target physiological area based on the acquired target image, clustering the point cloud data into different physiological tissues, and performing three-dimensional reconstruction of the target physiological area based on the clustering processing results.

[0063] Specifically, the image processing device can determine the point cloud data at the corresponding penetration depth based on the target image corresponding to the single-band light. The point cloud data may include the characteristic value of the point cloud at each position in the target physiological area. The characteristic value may refer to the grayscale value obtained after image processing, which is related to the signal intensity. The signal intensity refers to the signal intensity of the physiological tissue at the corresponding position reflecting the light. Since the components of the same physiological tissue are the same or similar, the signal intensity of the light reflected is the same or similar. When clustering the point cloud data, the point cloud of the same physiological tissue can be clustered into a set according to the characteristic value and position to obtain the clustering processing result. At this time, according to the clustering processing result, triangulation (Triangle Mesh) processing can be performed to perform three-dimensional reconstruction of the physiological tissue in the target physiological area.

[0064] To facilitate surgical planning or lesion analysis, in step S302 , the image processing device may perform three-dimensional reconstruction of the lesion in the target physiological region.

[0065] Specifically, in some embodiments of the present application, step S302 may include: determining the range of the lesion at each penetration depth of the target physiological area based on the acquired target image, so as to perform three-dimensional reconstruction of the lesion based on the range of the lesion.

[0066] At a single penetration depth, the target image captured by the light corresponding to that penetration depth can be used to identify the range of the lesion at that penetration depth. At this point, based on the acquired target images corresponding to light of different wavelengths, the range of the lesion can be determined at each penetration depth within the target physiological region, resulting in a 2D outline of the lesion at each different penetration depth. Furthermore, a 3D reconstruction of the lesion can be performed based on its range.

[0067] In some embodiments of the present application, the image processing device may determine the range of the lesion based on the distribution of physiological tissues.

[0068] For details, please refer to Figure 5 The above-mentioned process of determining the range of the lesion at each penetration depth of the target physiological area based on the acquired target image may include steps S501 to S503.

[0069] Step S501 : determining the physiological tissue to which each position belongs at each penetration depth of the target physiological region according to the acquired target image.

[0070] Specifically, the pixel information of a pixel in the target image can reflect the physiological tissue at the corresponding location. For example, different pixel values ​​can correspond to different physiological tissues. Based on the target image under a single wavelength band of light, the physiological tissue to which each location belongs can be determined on a two-dimensional plane at the corresponding penetration depth.

[0071] The above physiological tissues may include but are not limited to mucous membranes and blood vessels.

[0072] Step S502 : At each penetration depth of the target physiological region, the distribution of blood vessels is determined according to the physiological tissue to which each position belongs.

[0073] In an embodiment of the present application, for a single penetration depth, the distribution of blood vessels at the penetration depth can be determined based on the physiological tissue to which each position belongs.

[0074] The distribution of blood vessels may include at least one of the following: the distance between blood vessels, the number of blood vessels within a unit range (or density), etc.

[0075] Step S503: determining the range of the lesion at each penetration depth in the target physiological area according to the distribution of blood vessels.

[0076] Because lesions such as cancerous cells require a large amount of oxygen, blood vessels are denser within the lesion area than normal cells, often exhibiting adhesions. Therefore, vascular adhesions can be determined based on the distribution of blood vessels, such as the distance between them and the number of blood vessels per unit area. The area where adhesions occur is the area where the lesion is located. For example, vascular adhesions can be confirmed when the distance between blood vessels is less than a distance threshold, or when the number of blood vessels per unit area is greater than a number threshold.

[0077] It should be noted that, in the embodiments of the present application, the range of the lesion is determined at each penetration depth of the target physiological area mainly depending on the distribution of capillaries.

[0078] The range of the lesion at each penetration depth is obtained, that is, the two-dimensional contour of the lesion at each penetration depth is obtained. At this time, three-dimensional reconstruction can be performed according to the range of the lesion to obtain a three-dimensional model of the lesion.

[0079] To facilitate surgical planning by doctors, after obtaining the range of the lesion, the image processing device can also reflect the range of the lesion at different penetration depths in the two-dimensional image.

[0080] Specifically, in some embodiments of the present application, the image processing device can determine the three-dimensional size data of the lesion based on the results of the three-dimensional reconstruction, and send the three-dimensional size data to the display device to be superimposed on the two-dimensional image displayed in real time by the display device.

[0081] The three-dimensional dimensional data may include the length, width, and height of the lesion. The two-dimensional image may be a white light image obtained by photographing the target physiological area. By superimposing the three-dimensional dimensional data of the lesion at different penetration depths onto the two-dimensional plane presented by the two-dimensional image, the deep contours of the lesion can be reflected on the two-dimensional image.

[0082] Accordingly, the results of the two-dimensional image or three-dimensional reconstruction can be used for surgical planning. Surgical planning at least includes determining the incision range of the lesion. In other words, when viewing the white light image, the doctor can comprehensively consider the area of ​​the lower layer that exceeds the surface range to ultimately determine the incision range of the lesion, so that the incision range can cover the range of the lesion at any penetration depth. Of course, for the endoscope system, the incision range of the lesion can also be determined based on the results of the two-dimensional image or three-dimensional reconstruction, and the incision range can be marked on the two-dimensional image for the doctor's reference.

[0083] Considering that surgical procedures require avoiding larger blood vessels to minimize the risk of massive bleeding, in some embodiments of the present application, the image processing device can also determine the range of the target blood vessel at various penetration depths within the target physiological region based on the acquired target image. A prompt message is then generated based on the range of the target blood vessel and the range of the lesion.

[0084] The target blood vessel may specifically refer to a blood vessel having a diameter greater than a preset diameter. Alternatively, the target blood vessel may be a blood vessel of a preset type, such as an artery.

[0085] In embodiments of the present application, prompt information can be used to indicate the location of the target vessel relative to the lesion. This prompt information can be marked on the aforementioned two-dimensional image, or output in the form of text or voice information, which is not limited by this application. Based on this prompt information, the doctor can avoid the target vessel during surgery to reduce the risk of massive bleeding.

[0086] As mentioned above, when cancerous cells penetrate to a certain depth, they need to be treated through surgery, which is not suitable for the treatment of early-stage cancer. Therefore, in some embodiments of the present application, before performing three-dimensional reconstruction of the target physiological area, the three-dimensional reconstruction method may also include: obtaining a target image corresponding to the longest wavelength light, and performing cell cancer detection at the deepest penetration depth based on the target image corresponding to the longest wavelength light. If cell cancer is detected at the deepest penetration depth, the three-dimensional reconstruction is stopped and the degree of cancer is prompted.

[0087] The longest wavelength light is the light with the longest wavelength among the multiple different wavelength light, and the longest wavelength light corresponds to the deepest penetration depth of the target physiological area.

[0088] The method of detecting cell canceration can refer to the method of determining the range of the lesion, which is not described in detail in this application. The canceration degree prompt is used to indicate that the current lesion is not an early stage cancer.

[0089] In other words, if cancerous cells are detected at the deepest penetration depth, indicating that the cancer has penetrated deep enough to be considered early-stage cancer and requiring surgical treatment, the image processing device can stop 3D reconstruction and instead provide an indication of the cancerous stage. This reduces the image processing workload while quickly identifying advanced cancers and prompting patients to undergo surgery as soon as possible.

[0090] In addition, in step S301 and step S302, when the light source adopts different illumination methods, the present application can provide different processing methods.

[0091] Specifically, in some embodiments of the present application, step S301 may include acquiring a plurality of target images of the target physiological region obtained by sequentially irradiating the target physiological region with light of multiple different wavelength bands.

[0092] It should be noted that this application does not limit the order of light irradiation.

[0093] In some embodiments, irradiation can be performed in ascending wavelength order. For example, a light source can emit narrowband light from the 400nm to 900nm wavelength band, increasing the wavelength by 10nm every 50 milliseconds. In this way, if no cancerous cells are detected at any penetration depth, acquisition of target images at deeper penetration depths can be stopped, or image processing can be discontinued for target images at deeper penetration depths, thereby improving 3D reconstruction efficiency.

[0094] In other embodiments, irradiation can be performed in descending wavelength order. For example, the light source can emit narrowband light from the 900nm to 400nm band, decreasing the wavelength by 10nm every 50 milliseconds. In this way, if cell canceration is detected at the deepest penetration depth, 3D reconstruction can be stopped and the extent of canceration can be indicated.

[0095] When the light source is controlled to sequentially irradiate light of different wavelength bands, the obtained target images each correspond to a wavelength band. At this time, the three-dimensional reconstruction can be completed according to the method of performing three-dimensional reconstruction based on the target image as described above.

[0096] In other embodiments of the present application, the light irradiating the target physiological area may include light of multiple different wavelength bands. For example, the light source may output mixed light of 400nm-900nm.

[0097] At this time, in step S302, the processing device can reconstruct the target image to obtain an image to be processed corresponding to each wavelength of light in a plurality of different wavelength bands, and then perform three-dimensional reconstruction of the target physiological area based on the image to be processed.

[0098] Specifically, in this case, it is necessary to calculate the image information of a specific wavelength band based on the target image containing light of various wavelengths, the camera's reflection coefficient L, the filter transmittance F, the distribution of the light source in each wavelength band E, the reflectivity O of physiological tissue to light of different wavelengths, and the camera's response coefficient S to each wavelength. The specific formula can be expressed as:

[0099] v i =E i ×F i ×L i ×S i ×o i ;

[0100] Among them, i represents the wavelength value of the image to be processed, v i That is, the image to be processed in a specific band.

[0101] After obtaining the images to be processed corresponding to each wavelength band, the target physiological area can be reconstructed in three dimensions based on the images to be processed. The method of performing three-dimensional reconstruction based on the images to be processed can refer to the description of the method of performing three-dimensional reconstruction based on the target image above, which will not be described in detail in this application.

[0102] It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are all expressed as a series of action combinations. However, those skilled in the art should be aware that this application is not limited to the described order of actions, because according to this application, certain steps can be performed in other orders.

[0103] like Figure 6 FIG2 is a schematic structural diagram of a three-dimensional reconstruction apparatus 600 provided in an embodiment of the present application. The three-dimensional reconstruction apparatus 600 is configured on an image processing device.

[0104] Specifically, the 3D reconstruction device 600 may include:

[0105] An image acquisition unit 601 is configured to acquire a plurality of target images of a target physiological region containing a lesion, when the target physiological region is irradiated with light of multiple wavelengths, wherein the penetration depths of the light of different wavelengths into the target physiological region are different, and the number of the target images captured under each wavelength is greater than or equal to one;

[0106] The three-dimensional reconstruction unit 602 is configured to perform three-dimensional reconstruction of the target physiological region according to the acquired target image.

[0107] In some embodiments of the present application, the three-dimensional reconstruction unit 602 can be specifically used to: determine the range of the lesion at each penetration depth of the target physiological area based on the acquired target image, so as to perform three-dimensional reconstruction of the lesion based on the range of the lesion.

[0108] In some embodiments of the present application, the three-dimensional reconstruction unit 602 can be specifically used to: determine the physiological tissue to which each position belongs at each penetration depth of the target physiological area based on the acquired target image; determine the distribution of the blood vessels at each penetration depth of the target physiological area based on the physiological tissue to which each position belongs; and determine the range of the lesion at each penetration depth of the target physiological area based on the distribution of the blood vessels.

[0109] In some embodiments of the present application, the distribution of blood vessels includes at least one of the following: the distance between the blood vessels, and the number of the blood vessels within a unit range.

[0110] In some embodiments of the present application, the three-dimensional reconstruction unit 602 can be specifically used to: determine the point cloud data at each penetration depth of the target physiological area based on the acquired target image; cluster the point cloud data of different physiological tissues, and perform three-dimensional reconstruction of the target physiological area based on the clustering processing results.

[0111] In some embodiments of the present application, the point cloud data may include characteristic values ​​of the point cloud at each position within the target physiological area, where the characteristic values ​​are related to signal strength, which is the signal strength of light reflected by the physiological tissue at the corresponding position.

[0112] In some embodiments of the present application, the three-dimensional reconstruction device 600 may further include an information processing unit for determining the three-dimensional size data of the lesion based on the results of the three-dimensional reconstruction, the three-dimensional size data including the length, width and height data of the lesion; and sending the three-dimensional size data to a display device to be superimposed on the two-dimensional image displayed in real time by the display device.

[0113] In some embodiments of the present application, the above-mentioned information processing unit can also be specifically used to: determine the range of the target blood vessel at each penetration depth of the target physiological area based on the acquired target image; generate prompt information based on the range of the target blood vessel and the range of the lesion, and the prompt information is used to indicate the orientation of the target blood vessel relative to the lesion.

[0114] In some embodiments of the present application, the above-mentioned information processing unit can also be specifically used for: before performing three-dimensional reconstruction of the target physiological area, the three-dimensional reconstruction method also includes: obtaining the target image corresponding to the longest wavelength light, the longest wavelength light is the light with the longest wavelength among the multiple different wavelength lights, and the longest wavelength light corresponds to the deepest penetration depth of the target physiological area; according to the target image corresponding to the longest wavelength light, performing cell canceration detection at the deepest penetration depth; if cell canceration is detected at the deepest penetration depth, stopping the three-dimensional reconstruction and providing a prompt of the degree of canceration.

[0115] In some embodiments of the present application, the longest wavelength corresponding to the multiple different wavelength bands of light does not exceed 950 nm.

[0116] In some embodiments of the present application, the image acquisition unit 601 may be specifically configured to acquire a plurality of target images of the target physiological region obtained by sequentially irradiating the target physiological region with the plurality of light beams of different wavelengths.

[0117] In some embodiments of the present application, the light irradiating the target physiological area includes the multiple different wavelength bands of light; the three-dimensional reconstruction unit 602 can be specifically used to: reconstruct the target image to obtain the image to be processed corresponding to each of the multiple different wavelength bands of light; and perform three-dimensional reconstruction of the target physiological area based on the image to be processed.

[0118] It should be noted that for the convenience and simplicity of description, the specific working process of the above-mentioned 3D reconstruction device 600 can be referred to Figures 1 to 5 The corresponding process of the method will not be described in detail here.

[0119] like Figure 7 FIG. 7 is a schematic diagram of an image processing device 7 provided in an embodiment of the present application. Specifically, the image processing device 7 may include: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a 3D reconstruction program. When the processor 70 executes the computer program 72, the steps in the above-mentioned 3D reconstruction method embodiments are implemented, such as Figure 3 Alternatively, when the processor 70 executes the computer program 72, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 6 The functions of the image acquisition unit 601 and the 3D reconstruction unit 602 are shown.

[0120] The computer program may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the image processing device.

[0121] For example, the computer program can be divided into: an image acquisition unit and a three-dimensional reconstruction unit. The specific functions of each unit are as follows: the image acquisition unit is used to acquire multiple target images of the target physiological region containing the lesion when the target physiological region is irradiated with multiple light beams of different wavelengths, wherein the penetration depths of the light beams of different wavelengths into the target physiological region are different, and the number of target images captured under each light beam of different wavelengths is greater than or equal to 1; the three-dimensional reconstruction unit is used to perform three-dimensional reconstruction of the target physiological region based on the acquired target images.

[0122] The image processing device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that Figure 7 It is only an example of an image processing device and does not constitute a limitation of the image processing device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the image processing device may also include input and output devices, network access devices, buses, etc.

[0123] The processor 70 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0124] The memory 71 may be an internal storage unit of the image processing device, such as a hard disk or memory of the image processing device. The memory 71 may also be an external storage device of the image processing device, such as a plug-in hard disk equipped on the image processing device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 71 may include both an internal storage unit of the image processing device and an external storage device. The memory 71 is used to store the computer program and other programs and data required by the image processing device. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0125] It should be noted that, for the convenience and brevity of description, the structure of the above-mentioned image processing device 7 can also refer to the specific description of the structure in the method embodiment, and will not be repeated here.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0127] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0128] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] In the embodiments provided in this application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0132] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0133] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquiring a plurality of target images of a target physiological region containing a lesion when irradiating the target physiological region with light of multiple wavelengths, wherein the penetration depths of the light of different wavelengths into the target physiological region are different, and the number of the target images captured under each wavelength is greater than or equal to one; Performing three-dimensional reconstruction of the target physiological region according to the acquired target image; Determining, based on the acquired target image, a range of a target blood vessel at each penetration depth of the target physiological region, wherein the target blood vessel is a blood vessel having a diameter greater than a preset diameter, or the target blood vessel is a blood vessel of a preset type; Prompt information is generated according to the range where the target blood vessel is located and the range where the lesion is located, and the prompt information is used to indicate the position of the target blood vessel relative to the lesion.

2. The three-dimensional reconstruction method according to claim 1, wherein: The three-dimensional reconstruction of the target physiological region according to the acquired target image includes: According to the acquired target image, the range of the lesion is determined at each penetration depth of the target physiological region, so as to perform three-dimensional reconstruction of the lesion according to the range of the lesion.

3. The three-dimensional reconstruction method according to claim 2, wherein: Determining the range of the lesion at each penetration depth of the target physiological region based on the acquired target image includes: determining, based on the acquired target image, the physiological tissue to which each position belongs at each penetration depth of the target physiological region; At each penetration depth of the target physiological region, determining the distribution of blood vessels according to the physiological tissues to which the respective positions belong; According to the distribution of the blood vessels, the range of the lesion is determined at each penetration depth of the target physiological area.

4. The three-dimensional reconstruction method according to claim 3, wherein: The distribution of the blood vessels includes at least one of the following: the distance between the blood vessels, and the number of the blood vessels within a unit range.

5. The three-dimensional reconstruction method according to claim 1, wherein: The three-dimensional reconstruction of the target physiological region according to the acquired target image includes: determining point cloud data of the target physiological region at various penetration depths based on the acquired target image; The point cloud data is clustered according to different physiological tissues, and the target physiological region is three-dimensionally reconstructed based on the clustering processing result.

6. The three-dimensional reconstruction method according to claim 5, wherein: The point cloud data includes characteristic values ​​of the point cloud at various positions within the target physiological area. The characteristic values ​​are related to signal strength, which is the signal strength of light reflected by the physiological tissue at the corresponding position.

7. The three-dimensional reconstruction method according to claim 1, wherein: The three-dimensional reconstruction method further includes: Determining three-dimensional size data of the lesion based on a result of the three-dimensional reconstruction, wherein the three-dimensional size data includes length, width, and height data of the lesion; The three-dimensional size data is sent to a display device to be superimposed on a two-dimensional image displayed in real time by the display device.

8. The three-dimensional reconstruction method according to any one of claims 1 to 7, wherein: The lesions include cell canceration; Before performing three-dimensional reconstruction on the target physiological region, the three-dimensional reconstruction method further includes: Acquire the target image corresponding to the longest wavelength light, where the longest wavelength light is the light with the longest wavelength among the multiple different wavelength light, and the longest wavelength light corresponds to the deepest penetration depth of the target physiological area; Performing cell canceration detection at the deepest penetration depth according to the target image corresponding to the light of the longest wavelength; If cell canceration is detected at the deepest penetration depth, the three-dimensional reconstruction is stopped and the degree of canceration is indicated.

9. The three-dimensional reconstruction method according to any one of claims 1 to 7, wherein: The longest wavelength corresponding to the multiple light beams of different wavelength bands does not exceed 950 nm.

10. The three-dimensional reconstruction method according to any one of claims 1 to 7, wherein: The step of acquiring a plurality of target images of the target physiological region including the lesion, taken when the target physiological region is irradiated with a plurality of light beams of different wavelengths, comprises: acquiring a plurality of target images of the target physiological region, taken when the target physiological region is irradiated with the plurality of light beams of different wavelengths in sequence; Alternatively, the light irradiating the target physiological area includes the multiple light rays of different wavelengths; accordingly, performing three-dimensional reconstruction of the target physiological area based on the acquired target image includes: performing image reconstruction on the target image to obtain an image to be processed corresponding to each of the multiple light rays of different wavelengths; and performing three-dimensional reconstruction of the target physiological area based on the image to be processed.

11. A three-dimensional reconstruction device, characterized in that: include: an image acquisition unit, configured to acquire a plurality of target images of a target physiological region containing a lesion, taken when the target physiological region is irradiated with light of multiple wavelengths, wherein the penetration depths of the light of different wavelengths into the target physiological region are different, and the number of the target images taken under each wavelength is greater than or equal to one; a three-dimensional reconstruction unit, configured to perform three-dimensional reconstruction of the target physiological region based on the acquired target image; an information processing unit, configured to determine, based on the acquired target image, a range of a target blood vessel at each penetration depth in the target physiological region, the target blood vessel being a blood vessel having a diameter greater than a preset diameter, or the target blood vessel being a blood vessel of a preset type; and to generate prompt information based on the range of the target blood vessel and the range of the lesion, the prompt information being used to indicate the position of the target blood vessel relative to the lesion.

12. An image processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 10 are implemented.

13. An endoscope system, characterized in that: include: Light source, mirror, image processing equipment and display equipment; The image processing device is used to execute the steps of the three-dimensional reconstruction method according to any one of claims 1 to 10.

Images