Positioning method and positioning device of hemorrhagic spot, electronic equipment and medium

Abstract

The invention discloses a bleeding point positioning method and device, electronic equipment and a medium. The bleeding point positioning method comprises the steps that continuous multiple frames of first images of a target area are acquired; performing image segmentation on the continuous multiple frames of first images to obtain at least one to-be-judged area; determining at least one to-be-determined area as a bleeding point area or a diffusion bleeding area, and obtaining a depth map of the target area for three-dimensional reconstruction to form a three-dimensional point cloud image; and marking a bleeding point area and a diffusion bleeding area in the three-dimensional point cloud image. According to the bleeding point positioning method and device, the electronic equipment and the medium provided by the invention, high-quality data is selected based on a video preprocessing algorithm, identification and three-dimensional reconstruction of the initial bleeding point and the diffusion bleeding area are carried out, the coordinates of the initial bleeding point are mapped to the point cloud coordinate system space through a point cloud registration method, and the positioning accuracy of the bleeding point is improved. Therefore, even if the bleeding range diffuses in the subsequent operation process, the original bleeding point coordinates can be accurately positioned, and the bleeding stopping efficiency can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a positioning method, device, electronic equipment and medium in image data processing. Background Art

[0002] . Laparoscope is an instrument with a miniature camera. Laparoscopic surgery refers to surgery performed using a laparoscope and related instruments. It has the characteristics of less trauma, quick recovery, and good treatment effect. However, due to the complex structure of organs in the abdominal cavity, rich blood vessels, lack of experience of doctors and other reasons, heavy bleeding sometimes occurs during surgery. This situation is often sudden and rapid, causing the surgical field to be quickly covered by bleeding, which will not only affect the doctor's diagnostic field of view and the quality of the doctor's surgery, but in severe cases it may even endanger the patient's life and health. How to quickly locate bleeding points within a large area of ​​bleeding and take timely and effective hemostasis measures is of great clinical significance.

[0003] Although there are many methods for classifying and identifying bleeding images in the prior art, none of them can accurately and quickly locate the specific initial bleeding point. Therefore, it is urgent to propose an identification and positioning method that can mark and quickly identify the initial bleeding point. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the defects in the prior art and provide a positioning method, device, electronic equipment and medium that can quickly locate the bleeding point.

[0005] The present invention solves the above technical problems through the following technical solutions: A method for locating bleeding points, characterized in that it comprises the following steps:

[0006] Acquire a plurality of consecutive first image frames of the target area;

[0007] Performing image segmentation on the plurality of consecutive frames of first images to obtain at least one area to be determined;

[0008] Determining that the at least one area to be determined is a bleeding point area or a diffuse bleeding area, and recording a coordinate set of the bleeding area and the diffuse bleeding area in the target area;

[0009] Obtain the depth map of the target area for 3D reconstruction to form a 3D point cloud image;

[0010] Based on the coordinate sets of the bleeding area and the diffuse bleeding area in the target area, the bleeding point area and the diffuse bleeding area are marked in the three-dimensional point cloud image.

[0011] Preferably, the acquiring of the continuous multiple-frame first images of the target area includes acquiring a video image of the target area and performing preprocessing to acquire clear and effective continuous multiple-frame first images, wherein the video image is an image acquired by at least one of the binocular cameras of the laparoscope;

[0012] The obtaining of the depth map of the target area for three-dimensional reconstruction includes obtaining a second image based on the binocular camera.

[0013] Preferably, determining that the at least one area to be determined is a bleeding point area or a diffuse bleeding area includes calculating the area of ​​the at least one area to be determined, wherein an area less than a first preset threshold is a bleeding point area, and an area greater than or equal to the first preset threshold is a diffuse bleeding area.

[0014] Preferably, the determining that the at least one to-be-determined area is a bleeding point area or a diffuse bleeding area also includes performing an area overlap judgment on the at least one to-be-determined area obtained by the n-th frame of the first image and the at least one to-be-determined area at a corresponding position obtained by the n-1-th frame of the first image, and if the area overlap ratio is less than a preset ratio threshold, determining that the at least one to-be-determined area in the n-th frame of the first image is a non-bleeding area;

[0015] If the non-bleeding area appears in the to-be-determined area at a corresponding position in at least one subsequent frame of the first image and the area is enlarged, the non-bleeding area is determined to be a bleeding point area.

[0016] Preferably, the acquisition of the depth map of the target area for three-dimensional reconstruction to form a three-dimensional point cloud image is specifically as follows:

[0017] forming a disparity map based on a second image acquired by the binocular camera;

[0018] Converting the disparity map into a depth map;

[0019] The depth map is converted into a three-dimensional point cloud image, and each pixel information in the three-dimensional point cloud image includes its coordinate information and a corresponding color value.

[0020] Preferably, marking the bleeding point area and the diffuse bleeding area in the three-dimensional point cloud image based on the coordinate set of the bleeding area and the diffuse bleeding area in the target area includes:

[0021] When forming the three-dimensional point cloud image, based on the recorded coordinate sets of the bleeding point area and the diffuse bleeding area in the target area, a three-dimensional point cloud image coordinate set P1 of the bleeding point area and a three-dimensional point cloud image coordinate set P2 of the diffuse bleeding area are obtained;

[0022] The three-dimensional point cloud image coordinate set P1 of the bleeding point area and the three-dimensional point cloud image coordinate set P2 of the corresponding diffuse bleeding area are aligned, and the three-dimensional point cloud image coordinate set P1 of the bleeding point area is mapped to the three-dimensional point cloud image coordinate set P2 of the corresponding diffuse bleeding area and highlighted.

[0023] Another aspect of the present invention provides a bleeding point positioning device, characterized in that it comprises:

[0024] A first image acquisition unit, used to acquire a plurality of consecutive first images of a target area;

[0025] An image segmentation unit, used to segment the continuous plurality of frames of the first image respectively to obtain at least one area to be determined;

[0026] A determination unit, used to determine whether the area to be determined is a bleeding point area or a diffuse bleeding area;

[0027] A recording unit, used for recording a coordinate set of the bleeding point area and the diffuse bleeding area in the target area;

[0028] A second image acquisition unit, used to acquire a depth map of the target area;

[0029] A reconstruction unit, used for reconstructing the acquired depth map into a three-dimensional point cloud image;

[0030] The marking unit is used to mark the bleeding point area and the diffuse bleeding area in the three-dimensional point cloud image.

[0031] Preferably, the determining unit comprises:

[0032] An area calculation unit, used for calculating the area of ​​the area to be determined;

[0033] An area determination unit, used to determine the size of the area of ​​the to-be-determined area and the first preset threshold;

[0034] A ratio determination unit, used for determining the overlapping area ratio of at least one region to be determined obtained in two preceding and succeeding first image frames, and then determining a non-bleeding region;

[0035] The cycle determination unit is used to determine whether the area of ​​the non-bleeding region gradually expands in at least one subsequent frame of the first image.

[0036] Preferably, the second image acquisition unit further includes:

[0037] A disparity map conversion unit, used for converting the second image acquired by the binocular camera into a disparity map;

[0038] A depth map conversion unit, configured to convert the disparity map into a depth map;

[0039] The three-dimensional point cloud image converted by the reconstruction unit, wherein each pixel information includes its coordinate information and a corresponding color value.

[0040] Preferably, the marking unit comprises:

[0041] A conversion unit, used for converting a coordinate set of the bleeding point area and the diffuse bleeding area in the target area into a three-dimensional point cloud image coordinate set;

[0042] A registration unit, used for registering a three-dimensional point cloud image coordinate set P1 of a bleeding point area in a three-dimensional point cloud image with a three-dimensional point cloud image coordinate set P2 of a corresponding diffuse bleeding area;

[0043] And a highlight display unit is used to map the three-dimensional point cloud image coordinate set P1 of the bleeding point area to the three-dimensional point cloud image coordinate set P2 of its corresponding diffuse bleeding area and highlight it.

[0044] Another aspect of the present invention provides an electronic device, comprising:

[0045] at least one processor; and

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores instructions for execution by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform any of the above methods.

[0048] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any of the methods described above.

[0049] The positive and progressive effect of the present invention lies in: the bleeding point positioning method provided by the present invention selects high-quality data based on a video preprocessing algorithm, identifies the initial bleeding point and the diffuse bleeding area, and reconstructs the abdominal cavity in three dimensions, obtains the coordinate set of the initial bleeding point and the three-dimensional point cloud image of the abdominal cavity scene, and finally maps the coordinate set of the initial bleeding point to the point cloud coordinate system space through the point cloud registration method. Therefore, even if the bleeding range diffuses during the subsequent surgical process, the original bleeding point coordinate set can be accurately positioned, thereby effectively improving the hemostasis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of the flow of the method for locating bleeding points according to Example 1 of the present invention;

[0051] Figure 2This is a specific flow chart of step S3 in the method for locating bleeding points in Example 1 of the present invention;

[0052] Figure 3a is a schematic diagram of the nth frame of the first image in Embodiment 1 of the present invention, Figure 3b is a schematic diagram of the first image of the n-1th frame in Embodiment 1 of the present invention;

[0053] Figure 4 This is a specific flow chart of step S4 in the method for locating bleeding points in Example 1 of the present invention;

[0054] Figure 5 is a schematic block diagram of a bleeding point positioning device in Embodiment 2 of the present invention;

[0055] Figure 6 A schematic block diagram of an exemplary electronic device 500 provided in accordance with Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0057] Example 1

[0058] like Figure 1 As shown, a method for locating a bleeding point specifically comprises the following steps:

[0059] S1, acquiring a plurality of consecutive first images of a target area;

[0060] S2, performing image segmentation on the plurality of consecutive frames of the first image to obtain at least one region to be determined;

[0061] S3, determining that the at least one area to be determined is a bleeding point area or a diffuse bleeding area, and recording a coordinate set of the bleeding point area and the diffuse bleeding area in the target area;

[0062] S4, obtaining a depth map of the target area for three-dimensional reconstruction to form a three-dimensional point cloud image;

[0063] S5, marking the bleeding point area and the diffuse bleeding area in the three-dimensional point cloud image based on the coordinate set of the bleeding area or the diffuse bleeding area in the target area.

[0064] The target area is the surgical area explored by the laparoscope, and obtaining multiple continuous frames of first images of the target area includes obtaining a video image of the target area and preprocessing it to obtain clear and effective multiple continuous frames of first images, and the video image is an image obtained by at least one of the binocular cameras of the laparoscope.

[0065] The video image of the target area is obtained and preprocessed to obtain clear and effective continuous multi-frame first images. Conventional operations can be used, such as trimming the black borders of certain video frame images to retain only information, or removing non-information video frames, such as removing blurred or artifact video frames, to obtain clear and effective continuous multi-frame first images. Specifically, image classification algorithms, such as CNN convolutional neural network algorithms, can be used to preprocess the video image of the target area to ensure the accuracy of subsequent image segmentation or processing of the area to be determined.

[0066] If the laparoscope has binocular cameras, namely the first camera and the second camera, the first image can be obtained by any one of the binocular cameras, or the two cameras respectively obtain two sets of images with slightly different fields of view, and the two sets of images are fused frame by frame to obtain the first image of the entire complete field of view. Alternatively, the first camera and the second camera in the binocular camera are monitored and analyzed simultaneously in the early stage. If a possible bleeding area appears, the camera is selected for continuous monitoring and acquisition of the first image.

[0067] In step S2, the continuous multiple frames of the first image are segmented to obtain at least one area to be determined, specifically, at least one area to be determined that has obvious differences in features from surrounding tissues and organs is identified based on an image segmentation algorithm. More specifically, the image segmentation algorithm can be a threshold segmentation, a deep learning network such as UNet, a detection algorithm such as YOLO, Fast-RCNN, SSD and other series of deep learning detection algorithms, and can also include traditional target detection algorithms such as HOG (Histograms of Oriented Gradients) and other algorithms. There can be only one area to be determined in each frame of the first image, or there can be multiple areas to be determined.

[0068] In step S3, Figure 2 As shown, determining that the at least one area to be determined is a bleeding point area or a diffuse bleeding area specifically includes the following steps:

[0069] S31, calculating the area of ​​the at least one area to be determined; the area calculation of the area to be determined may be performed using a detection algorithm, such as calculating the area of ​​the minimum circumscribed rectangle, etc. Preferably, a mask method may also be used, after obtaining the area to be determined, the image values ​​of the pixels in the area to be determined are all set to 1 using an image mask, and the others are all set to 0, and only the area with the image value of 1 is calculated, so that the area of ​​the area to be determined can be obtained.

[0070] The area to be determined whose area is smaller than the first preset threshold is the bleeding point area, and the area to be determined whose area is greater than or equal to the first preset threshold is the bleeding spread area.

[0071] S32, performing an area overlap judgment on at least one to-be-determined region obtained in the nth frame of the first image and at least one to-be-determined region in a corresponding position obtained in the (n-1)th frame of the first image; if the area overlap ratio is less than a preset ratio threshold, determining that the region in the nth frame of the first image is a non-bleeding region.

[0072] S33, if the non-bleeding area appears in the same position as the area to be determined in at least one subsequent first image frame and the area is gradually enlarged, the non-bleeding area is determined to be a bleeding point area. The first preset threshold is a preset bleeding point area, which is usually small. If the area of ​​the area to be determined is smaller than the first preset threshold, it can usually be determined as an initial bleeding point, and if it is larger than the first preset threshold, it can be determined as a bleeding area that has spread. However, after determining the area to be determined in a single frame, misjudgment usually occurs.

[0073] For example, Figure 3a As shown in FIG. 1 , the to-be-determined regions M2, M3 and M4 are segmented from the first image of the nth frame. If the area of ​​M3 is smaller than the first preset threshold, it is determined as a bleeding point region. If the areas of M2 and M4 are larger than the first preset threshold, they are determined as diffuse bleeding regions. In the first image of the n-1th frame, as Figure 3b As shown, the area to be determined M1 appears at the position corresponding to the area to be determined M2, but the area to be determined does not appear at the position corresponding to M4, so M4 may be a misjudgment. Through step S33, it is determined that the area to be determined corresponding to the position of M4 does not appear in the first image of the n-1 frame. Therefore, the area overlap rate of the two is 0, which is necessarily less than the preset ratio threshold, indicating that M4 is a misjudgment and should be determined as a non-bleeding area. Between M1 and M2, the area overlap rate of the two is greater than the preset ratio threshold, and it should be determined as a bleeding area. Because its area is greater than the first preset threshold, the bleeding area is still diffused. The preset ratio threshold can be set as needed, such as 90%, 80%, etc., and the degree of bleeding diffusion needs to be considered.

[0074] Figure 3a In the first image of the n-1th frame, the area to be determined M3 appears, because the corresponding area to be determined does not appear in the first image of the n-1th frame. Therefore, through the judgment of the overlapping area, the area to be determined M3 should also be determined as a non-bleeding area. However, if in the subsequent first image of the n+1th frame or the n+2th frame, the area to be determined still appears at the corresponding position of the non-bleeding area M3, and its area is gradually expanding, it means that the non-bleeding area M3 should be a bleeding point area. Therefore, through the above-mentioned cyclic judgment of the first image of the n-1th frame, the nth frame and the subsequent n+1th frame, the bleeding point area and the diffuse bleeding area can be accurately determined.

[0075] Those skilled in the art can understand that the corresponding number of areas to be determined in the first image of each frame may be the same or different. For example, there is only one bleeding point area in the first image of the mth frame, and the bleeding point area in the first image of the m+1th frame can be determined to be a diffuse bleeding area, and in the first image of the m+2th frame, a larger diffuse bleeding area corresponding to the bleeding point area in the first image of the mth frame may appear, and a new bleeding point area may appear. Therefore, it is necessary to record the coordinate set of each bleeding point area and diffuse bleeding area in each first image. The coordinate set includes the coordinate set of the bleeding point area and the coordinate set of the diffuse bleeding area. Because the bleeding point area is small in area, it may be reflected in one pixel or several pixels in the image, while the diffuse bleeding area is usually larger in range, and its range is definitely not limited to one pixel, but a set of multiple pixels in an area, which is used to limit the range of the diffuse bleeding area. The coordinate set is the set of coordinates of the pixel points corresponding to the corresponding area in the image. And preferably, the coordinate set of the diffuse bleeding area can be a set of coordinates of all pixel points in the area, or it can be just a set of coordinates of its outermost circle of pixel points. Because when the area is large, as long as the coordinates of the outermost circle of pixel points in the area are obtained, the scope of the area is known.

[0076] The step S4 is as follows: Figure 4 As shown, the depth map of the target area is obtained for 3D reconstruction to form a 3D point cloud image. Specifically, it includes:

[0077] S41, forming a disparity map based on the second image acquired by the binocular camera;

[0078] S42, converting the disparity map into a depth map;

[0079] S43, converting the depth map into a three-dimensional point cloud image, wherein each pixel information in the three-dimensional point cloud image includes its coordinate information and a corresponding color value.

[0080] In step S41, the second image obtained by the binocular camera is specifically a first camera image and a second camera image with slightly different fields of view of the target area obtained by the binocular camera. The first camera image and the second camera image are processed, and the coordinate difference di between each pixel is calculated to obtain a disparity map D1, where di is the disparity of each pixel.

[0081] Step S42, converting the disparity map D1 into a depth map D2, specifically by using the principle of triangulation, through the formula Zi=f×B / di, Zi represents the depth value of each pixel, f is the focal length of the first camera and the second camera in the binocular camera in the laparoscope, and B is the distance between the centers of the first camera and the second camera in the binocular camera.

[0082] In another optional embodiment, those skilled in the art may also understand that the binocular camera may also be a color depth camera, which can directly obtain a depth map, thereby eliminating the steps of first obtaining a disparity map and then converting it into a depth map.

[0083] Step S43, converting the depth map into a three-dimensional point cloud image, wherein each pixel information in the three-dimensional point cloud image includes its coordinate information and a corresponding color value, and the color value is preferably an image RGB value, specifically converting the depth map coordinates into coordinates in the three-dimensional point cloud image by the following formula.

[0084]

[0085] Among them, (X, Y, Z) is the coordinate in the 3D point cloud image, (u, v) is the coordinate in the depth map, and f x 、f y are the horizontal and vertical focal lengths of the camera in the image coordinate system, corresponding to an ideal pinhole camera, f x =f y =f, f is the focal length of the camera. x With C y is the horizontal and vertical offset of the image origin relative to the camera optical center, which can be 1 / 2 of the image width and height respectively. For example, point (C x , C y ) is the center point of the imaging plane.

[0086] Furthermore, each pixel information in the reconstructed three-dimensional point cloud image includes not only its coordinate information but also the corresponding image color value, preferably, an RGB value, which is used to display the corresponding color during reconstruction.

[0087] Preferably, the above step S4, obtaining a depth map of the target area for three-dimensional reconstruction to form a three-dimensional point cloud image, can also be performed based on a monocular camera, for example, using a structure from motion (Structure from Motion, SfM) method, which only requires moving the endoscope lens to obtain a depth map of the target area.

[0088] The step S5, based on the coordinate set of the bleeding area or the diffuse bleeding area in the target area obtained in the step S3, is specifically:

[0089] In the process of reconstructing the three-dimensional point cloud image in the above step S4, a three-dimensional point cloud image coordinate set P1 of the bleeding point area and a three-dimensional point cloud image coordinate set P2 of the diffuse bleeding area are obtained.

[0090] Any bleeding point area is located in its corresponding diffuse bleeding area. The three-dimensional point cloud image coordinate set P1 of the bleeding point area is registered with the three-dimensional point cloud image coordinate set P2 of the corresponding diffuse bleeding area. Because in the first image of different consecutive frames, the coordinate set of the bleeding point area will not change once it appears, while the coordinate set of the diffuse bleeding area will continue to change with the change of different frames. In addition, multiple bleeding points and their corresponding multiple diffuse bleeding areas may appear. Therefore, it is necessary to match and register the bleeding point area and the diffuse bleeding area according to the position.

[0091] The 3D point cloud image coordinate set P1 of the bleeding point area is mapped to the 3D point cloud image coordinate set P2 of the corresponding diffuse bleeding area and highlighted. This step can use the commonly used iterative closest point registration algorithm (Iterative Closest Point, ICP) to accurately locate the position of the initial bleeding point after the bleeding area is diffused.

[0092] Preferably, the three-dimensional point cloud image coordinate set P1 of the bleeding point area and the corresponding three-dimensional point cloud image coordinate set P2 of the diffuse bleeding area are registered. Other algorithms may also be used, such as the random sampling consistency (RANSAC) point cloud registration algorithm, the point feature histogram (PFH)-based registration algorithm, the deep learning registration algorithm PCRNet, and the like.

[0093] Moreover, as another optional method of this embodiment, the three-dimensional point cloud image coordinate set P1 of the bleeding point area and the three-dimensional point cloud image coordinate set P2 of the corresponding diffuse bleeding area are registered. The three-dimensional point cloud image coordinate set P1 of the bleeding point area is converted from the coordinates of the bleeding point area in the target area in a certain frame of the first image when the bleeding point area just appeared, and the corresponding diffuse bleeding area will continue to change over time, so it is possible that the corresponding diffuse bleeding area will appear in multiple subsequent frames of the first image, and the diffuse bleeding area corresponding to the bleeding point area may correspond to multiple coordinate sets in different frames of the first image. Therefore, multiple different three-dimensional point cloud image coordinate sets P2, P2', P2" ... corresponding to the coordinate sets of multiple diffuse bleeding areas in different frames of the first image can also be obtained, and then the multiple diffuse bleeding areas corresponding to the bleeding point area can be selected to be dynamically presented in the reconstructed three-dimensional point cloud image at the same time, or only the three-dimensional point cloud image coordinate set of the diffuse bleeding area corresponding to the latest frame can be selected for display marking.

[0094] The step S1, acquiring a plurality of consecutive first image frames, and the step S4, acquiring a depth map of the target area, can be started and performed simultaneously, or can be performed sequentially. For example, when one of the binocular cameras of the laparoscope acquires a plurality of consecutive first image frames, the binocular camera is used to acquire a second image for three-dimensional image reconstruction. Alternatively, the first image may be acquired first, and then the second image may be acquired to obtain a depth map, and then the three-dimensional point cloud image may be reconstructed. Preferably, once a bleeding point area appears in the first image, the depth map is immediately acquired to perform the three-dimensional image reconstruction and marking steps.

[0095] The video frame selected when reconstructing the three-dimensional point cloud image is different from the video frame used when acquiring the aforementioned multiple frames of the first image, which will cause the difference in the final presentation of the image. In this embodiment, the inventor does not limit this, and the user can set it according to his needs, either at the same time or with differences.

[0096] The acquisition of the depth map of the target area and the three-dimensional reconstruction can also be performed automatically at an interval of n seconds according to a program setting, and the n seconds can be 2S, 3S or 5S, etc. It can be set as needed. Preferably, the depth map and the first image can be obtained by a binocular camera at the same time, or by different cameras, and it is only necessary to calibrate the target area for different cameras.

[0097] The bleeding point positioning method provided in this embodiment selects high-quality data based on a video preprocessing algorithm, identifies the initial bleeding point and the diffuse bleeding area, and reconstructs the abdominal cavity in three dimensions, thereby obtaining a coordinate set of the initial bleeding point and a three-dimensional point cloud image of the abdominal cavity scene. Finally, the coordinate set of the initial bleeding point is mapped to the point cloud coordinate system space through a point cloud registration method. Therefore, even if the bleeding range diffuses during the subsequent surgical process, the original bleeding point can be accurately positioned, thereby effectively improving the efficiency of hemostasis.

[0098] By registering the point cloud reconstructed at the time of initial bleeding with the point cloud after the bleeding spreads, the coordinate set of the initial bleeding point is highlighted in the point cloud model after the bleeding spreads. This achieves accurate mapping of the bleeding point. Moreover, this method can also be applied to various electronic endoscopes, such as laparoscopes, colonoscopes, hysteroscopes, bronchoscopes, brain endoscopes and other endoscopic devices.

[0099] Example 2

[0100] like Figure 5 As shown, this embodiment provides a bleeding point positioning device, comprising:

[0101] A first image acquisition unit 10 is used to acquire a plurality of consecutive first images of a target area;

[0102] An image segmentation unit 20 is used to segment the continuous first image frames to obtain at least one area to be determined;

[0103] A determination unit 30, configured to determine whether the area to be determined is a bleeding point area or a diffuse bleeding area;

[0104] A recording unit 40, used for recording a coordinate set of the bleeding point area and the diffuse bleeding area in the target area;

[0105] A second image acquisition unit 50, used to acquire a depth map of the target area;

[0106] A reconstruction unit 60, configured to reconstruct the acquired depth map into a three-dimensional point cloud image;

[0107] The marking unit 70 is used to mark the bleeding point area and its corresponding diffuse bleeding area in the three-dimensional point cloud image.

[0108] The first image acquisition unit 10 pre-processes the acquired video image of the target area to obtain a clear and effective continuous multi-frame first image, and the video image is an image acquired by at least one of the binocular cameras of the laparoscope. Specifically, the video image of the target area is acquired and pre-processed to obtain a clear and effective continuous multi-frame first image. Conventional operations can be used, such as cropping the black borders of certain video frame images to retain only information, or removing non-information video frames, such as removing blurred or artifact video frames, to obtain clear and effective continuous multi-frame first images. Specifically, an image classification algorithm, such as a CNN convolutional neural network algorithm, can be used to pre-process the video image of the target area to ensure the accuracy of subsequent image segmentation or processing of the area to be determined.

[0109] The image segmentation unit 20 is used to perform image segmentation on multiple consecutive frames of the first image to obtain at least one area to be determined; specifically, based on the image segmentation algorithm, at least one area to be determined that has obvious differences in features from surrounding tissues and organs is identified. More specifically, the image segmentation algorithm can be threshold segmentation, deep learning networks such as UNet, detection algorithms such as YOLO, Fast-RCNN, SSD and other series of deep learning detection algorithms, and can also include traditional target detection algorithms such as HOG (Histograms of Oriented Gradients) and other algorithms. There can be only one area to be determined in each frame of the first image, or there can be multiple areas to be determined.

[0110] The determination unit 30 is used to determine whether the area to be determined is a bleeding point area or a diffuse bleeding area; specifically,

[0111] The area calculation unit 31 is used to calculate the area of ​​the area to be determined; a detection algorithm may be used, for example, to calculate the area of ​​the minimum circumscribed rectangle, etc.

[0112] The area judgment unit 32 is used to judge the size of the area of ​​the to-be-judged area and the first preset threshold; the to-be-judged area whose area is smaller than the first preset threshold is the bleeding point area, and the to-be-judged area whose area is greater than or equal to the first preset threshold is the bleeding spread area.

[0113] The ratio judgment unit 33 is used to judge the overlapping area ratio of at least one to-be-determined area obtained in the two preceding and succeeding first image frames, and further determine the non-bleeding area; specifically, the at least one to-be-determined area obtained in the n-th frame of the first image is judged to overlap with at least one to-be-determined area at a corresponding position obtained in the (n-1)-th frame of the first image; if the area overlap ratio is less than a preset ratio threshold, the area in the n-th frame of the first image is determined to be the non-bleeding area.

[0114] The loop judgment unit 34 is used to judge whether the area of ​​the non-bleeding region gradually expands in at least one subsequent frame of the first image. If the non-bleeding region appears in the same position as the to-be-judged region in at least one subsequent frame of the first image and the area gradually expands, the non-bleeding region is judged to be a bleeding point region.

[0115] Preferably, the second image acquisition unit 50 further includes:

[0116] The disparity map conversion unit 51 is used to convert the second image acquired by the binocular camera into a disparity map; the second image acquired by the binocular camera is specifically a first camera image and a second camera image with slightly different fields of view of the target area acquired by the binocular camera. The first camera image and the second camera image are processed, and the coordinate difference di between each pixel is used to obtain a disparity map D1, where di is the disparity of each pixel.

[0117] The depth map conversion unit 52 is used to convert the disparity map into a depth map; the disparity map D1 is converted into a depth map D2, specifically by using the triangulation principle, through the formula Zi=f×B / di, Zi represents the depth value of each pixel, f is the focal length of the first camera and the second camera in the binocular camera in the laparoscope, and B is the distance between the centers of the first camera and the second camera in the binocular camera.

[0118] The reconstruction unit 60 is used to convert the acquired depth map into a three-dimensional point cloud image, wherein each pixel information in the three-dimensional point cloud image includes its coordinate information and image color value. The depth map is converted into a three-dimensional point cloud image, wherein each pixel information in the three-dimensional point cloud image includes its coordinate information and image color value, specifically, the depth map coordinates are converted into coordinates in the three-dimensional point cloud image by the following formula.

[0119]

[0120] Among them, (X, Y, Z) is the coordinate in the 3D point cloud image, (u, v) is the coordinate in the depth map, and f x 、f y are the horizontal and vertical focal lengths of the camera in the image coordinate system, corresponding to an ideal pinhole camera, f x =f y =f, f is the focal length of the camera. x With C y is the horizontal and vertical offset of the image origin relative to the camera optical center, which can be 1 / 2 of the image width and height respectively. For example, point (C x , C y ) is the center point of the imaging plane.

[0121] The marking unit 70 is used to mark the bleeding point area and its corresponding diffuse bleeding area in the three-dimensional point cloud image.

[0122] The marking unit further comprises a conversion unit 71 for converting a coordinate set of the bleeding point area and the diffuse bleeding area in the target area into a three-dimensional point cloud image coordinate set.

[0123] The registration unit 72 is used to register the three-dimensional point cloud image coordinate set P1 of the bleeding point area in the three-dimensional point cloud image with the three-dimensional point cloud image coordinate set P2 of the corresponding diffuse bleeding area.

[0124] And also includes a highlight display unit 73, which is used to map the three-dimensional point cloud image coordinate set P1 of the bleeding point area to the three-dimensional point cloud image coordinate set P2 of its corresponding diffuse bleeding area and highlight it.

[0125] The 3D point cloud image coordinate set P1 of the bleeding point area is mapped to the 3D point cloud image coordinate set P2 of the corresponding diffuse bleeding area and highlighted. A commonly used iterative closest point registration algorithm (ICP) can be used to accurately locate the initial bleeding point after the bleeding area diffuses.

[0126] Example 3

[0127] According to Embodiment 3 of the present invention, the present invention further provides an electronic device, a readable storage medium and a computer program product. Figure 6 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present invention is shown. The device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 502 or a computer program loaded from a storage unit 508 into a RAM (Random Access Memory) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0128] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as a method for locating a bleeding point. For example, in some embodiments, the method for locating a bleeding point may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the aforementioned bleeding point location method in any other appropriate manner (for example, by means of firmware).

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0133] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A method for locating a bleeding point, characterized in that: The following steps are included: Acquire a plurality of consecutive first image frames of the target area; Performing image segmentation on the plurality of consecutive frames of first images to obtain at least one area to be determined; Determining that the at least one area to be determined is a bleeding point area or a diffuse bleeding area, and recording a coordinate set of the bleeding area and the diffuse bleeding area in the target area; Obtain the depth map of the target area for 3D reconstruction to form a 3D point cloud image; Based on the coordinate sets of the bleeding area and the diffuse bleeding area in the target area, the bleeding point area and the diffuse bleeding area are marked in the three-dimensional point cloud image.

2. The method for locating bleeding points according to claim 1, characterized in that: The obtaining of the continuous multiple frames of first images of the target area includes obtaining a video image of the target area and performing preprocessing to obtain clear and effective continuous multiple frames of first images, wherein the video image is an image obtained by at least one of the binocular cameras of the laparoscope; The obtaining of the depth map of the target area for three-dimensional reconstruction includes obtaining a second image based on the binocular camera.

3. The method for locating bleeding points according to claim 1, characterized in that: The determining that the at least one area to be determined is a bleeding point area or a diffuse bleeding area includes calculating the area of ​​the at least one area to be determined. An area less than a first preset threshold is a bleeding point area, and an area greater than or equal to the first preset threshold is a diffuse bleeding area.

4. The method for locating bleeding points according to claim 3, characterized in that: The step of determining that the at least one area to be determined is a bleeding point area or a diffuse bleeding area also includes: Performing an area overlap judgment on at least one to-be-determined region obtained from the nth frame of the first image and at least one to-be-determined region at a corresponding position obtained from the n-1th frame of the first image, and if the area overlap ratio is less than a preset ratio threshold, determining that the at least one to-be-determined region in the nth frame of the first image is a non-bleeding region; If the non-bleeding area appears in the to-be-determined area at a corresponding position in at least one subsequent frame of the first image and the area is enlarged, the non-bleeding area is determined to be a bleeding point area.

5. The method for locating bleeding points according to claim 1, characterized in that: The depth map of the target area is obtained to perform three-dimensional reconstruction to form a three-dimensional point cloud image, specifically, forming a disparity map based on a second image acquired by the binocular camera; Converting the disparity map into a depth map; The depth map is converted into a three-dimensional point cloud image, and each pixel information in the three-dimensional point cloud image includes its coordinate information and a corresponding color value.

6. The method for locating bleeding points according to claim 1, characterized in that: Based on the coordinate sets of the bleeding area and the diffuse bleeding area in the target area, marking the bleeding point area and the diffuse bleeding area in the three-dimensional point cloud image, include, When forming a three-dimensional point cloud image, based on the recorded coordinate sets of the bleeding point area and the diffuse bleeding area in the target area, a three-dimensional point cloud image coordinate set (P1) of the bleeding point area and a three-dimensional point cloud image coordinate set (P2) of the diffuse bleeding area are obtained; The three-dimensional point cloud image coordinate set (P1) of the bleeding point area and the three-dimensional point cloud image coordinate set (P2) of the corresponding diffuse bleeding area are registered, and the three-dimensional point cloud image coordinate set (P1) of the bleeding point area is mapped to the three-dimensional point cloud image coordinate set (P2) of the corresponding diffuse bleeding area and highlighted.

7. A bleeding point positioning device, characterized in that: include, A first image acquisition unit, used to acquire a plurality of consecutive first images of a target area; An image segmentation unit, used to segment the continuous plurality of frames of the first image respectively to obtain at least one area to be determined; A determination unit, used to determine whether the area to be determined is a bleeding point area or a diffuse bleeding area; A recording unit, used for recording a coordinate set of the bleeding point area and the diffuse bleeding area in the target area; A second image acquisition unit, used to acquire a depth map of the target area; A reconstruction unit, used for reconstructing the acquired depth map into a three-dimensional point cloud image; The marking unit is used to mark the bleeding point area and the diffuse bleeding area in the three-dimensional point cloud image.

8. The bleeding point positioning device according to claim 7, characterized in that: The determining unit comprises: An area calculation unit, used for calculating the area of ​​the area to be determined; An area determination unit, used to determine the size of the area of ​​the to-be-determined area and a first preset threshold; A ratio determination unit, used for determining the overlapping area ratio of at least one region to be determined obtained in two preceding and succeeding first image frames, and then determining a non-bleeding region; The cycle determination unit is used to determine whether the area of ​​the non-bleeding region gradually expands in at least one subsequent frame of the first image.

9. The bleeding point positioning device according to claim 7, characterized in that: The second image acquisition unit further includes: A disparity map conversion unit, used for converting the second image acquired by the binocular camera into a disparity map; A depth map conversion unit, configured to convert the disparity map into a depth map; The three-dimensional point cloud image converted by the reconstruction unit, wherein each pixel information includes its coordinate information and a corresponding color value.

10. The bleeding point positioning device according to claim 9, characterized in that: The marking unit comprises: A conversion unit, used for converting a coordinate set of the bleeding point area and the diffuse bleeding area in the target area into a three-dimensional point cloud image coordinate set; A registration unit, used for registering a three-dimensional point cloud image coordinate set (P1) of a bleeding point area in a three-dimensional point cloud image with a three-dimensional point cloud image coordinate set (P2) of a corresponding diffuse bleeding area; And a highlight display unit is used to map the three-dimensional point cloud image coordinate set (P1) of the bleeding point area to the three-dimensional point cloud image coordinate set (P2) of the corresponding diffuse bleeding area and perform highlight display.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions for execution by the at least one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 6.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

Images