Method, device, equipment and storage medium for three-dimensional reconstruction of surgical video

Abstract

The present application relates to the field of video processing, and specifically to a method, device, equipment and storage medium for three-dimensional reconstruction of surgical videos, including: obtaining an image taken by a reference camera before the operation begins, the image taken includes a surgical site layer, and the image of the surgical site layer is used as a reference image; obtaining an image taken by a reference camera during the operation as a first reconstructed image; the first reconstructed image includes a surgeon's head layer; obtaining images taken by multiple acquisition cameras during the operation as a second reconstructed image; the second reconstructed image includes a surgeon's hand layer or a surgical site layer; based on the reference image and the surgeon's head layer in the first reconstructed image, the surgeon's hand layer or the surgical site layer in the second reconstructed image is three-dimensionally reconstructed to obtain a surgical video of the corresponding part. The entire surgical process is recorded effectively and completely, thereby achieving effective surgical demonstration teaching, postoperative retrospection and / or surgical guidance, etc.

Description

Technical Field

[0001] The present application relates to the field of video processing, and specifically to a method, device, equipment and storage medium for three-dimensional reconstruction of surgical videos. Background Art

[0002] There is a need to record video images during surgery for demonstration teaching, postoperative review and / or remote surgical guidance, etc. In order to record the surgery, an independent surgical camera / camera arm is generally installed in the operating room to record the surgical process by controlling the mechanical parts, or the nursing staff can also hold a camera to record the video.

[0003] Existing camera devices can only record with a single perspective. During surgery, most of the time, only the surgeon's head, hands or back can be captured, but the patient's surgical site and the surgeon's operation cannot be captured. This makes it impossible to achieve effective surgical demonstration teaching, postoperative review and / or surgical guidance. Summary of the invention

[0004] Therefore, the present application aims to solve the technical problem that the entire surgical process cannot be effectively and completely recorded by shooting from a single perspective, thereby providing a method, device, equipment and storage medium for three-dimensional reconstruction of surgical videos. The technical solution is as follows:

[0005] On the one hand, the present application provides a method for three-dimensional reconstruction of surgical videos, including a video acquisition matrix, wherein the video acquisition matrix includes a reference camera and multiple acquisition cameras, wherein the multiple acquisition cameras surround the reference camera, and the video acquisition matrix is ​​located above the surgical site of the patient; the method includes:

[0006] Acquire an image captured by the reference camera before the operation begins, wherein the captured image includes a surgical site layer, and use the image of the surgical site layer as a reference image;

[0007] Acquire an image captured by the reference camera during the operation of the operator as a first reconstructed image; wherein the first reconstructed image includes a layer of the operator's head;

[0008] Acquire images captured by the plurality of acquisition cameras during the operation as a second reconstructed image; wherein the second reconstructed image includes a layer of the operator's hand or a layer of the surgical site;

[0009] Based on the reference image and the operator's head layer in the first reconstructed image, the operator's hand layer or the surgical site layer in the second reconstructed image is three-dimensionally reconstructed to obtain a surgical video of the corresponding site.

[0010] Optionally, the three-dimensional reconstruction of the operator's hand layer in the second reconstructed image based on the reference image and the operator's head layer in the first reconstructed image to obtain a surgical video of the corresponding part includes:

[0011] Calculating the projection of the operator's head layer in the first reconstructed image in the reference image to obtain a projection plane;

[0012] Calculating the occlusion of the operator's hand layer in the second reconstructed image by the projection plane to obtain a first occlusion plane;

[0013] Deleting the first occlusion plane from the operator's hand layer in the second reconstructed image to obtain an unoccluded operator's hand layer;

[0014] The unobstructed surgeon's hand layer is used for three-dimensional reconstruction to obtain the surgical video of the surgeon's hand layer.

[0015] Optionally, the second reconstructed image includes a layer of the operator's head; the method further comprises: calculating the height of the layer of the operator's head in the second reconstructed image using the projection plane;

[0016] The height of the operator's head layer in the second reconstructed image is used to model the operator's head layer in the second reconstructed image using an optical flow method to correct the first occlusion plane.

[0017] Optionally, the three-dimensional reconstruction of the surgical site layer in the second reconstructed image based on the reference image and the surgeon head layer in the first reconstructed image to obtain a surgical video of the corresponding site includes:

[0018] Calculating the projection of the operator's head layer in the first reconstructed image in the reference image to obtain a projection plane;

[0019] calculating the occlusion of the surgical site layer in the second reconstructed image by the projection plane to obtain a second occlusion plane;

[0020] Deleting the second occlusion plane from the surgical site layer in the second reconstructed image to obtain an unoccluded surgical site layer;

[0021] The unobstructed surgical site layer is used for three-dimensional reconstruction to obtain the surgical video of the surgical site layer.

[0022] Optionally, the second reconstructed image includes a layer of the operator's head; the method further comprises: calculating the height of the layer of the operator's head in the second reconstructed image using the projection plane;

[0023] The height of the operator's head layer in the second reconstructed image is used to model the operator's head layer in the second reconstructed image using an optical flow method to correct the second occlusion plane.

[0024] Optionally, the operator head layer, the operator hand layer, and the surgical site layer are identified by the following steps:

[0025] Determining the focal planes of the two acquisition cameras;

[0026] Obtaining the focal length of the acquisition camera and a first distance between two acquisition cameras;

[0027] Obtain a second distance and a third distance of the light source point to be measured; wherein the second distance is the length of the intersection of the line connecting the first acquisition camera, the light source point to be measured and the focal plane from the intersection of the vertical focal plane of the light source point to be measured, and the second distance is the length of the intersection of the line connecting the second acquisition camera, the light source point to be measured and the focal plane from the intersection of the vertical focal plane of the light source point to be measured;

[0028] Based on the focal length, the first distance, the second distance and the third distance, the depth of the light source point to be measured is calculated, and the depth of the light source point to be measured is used to determine whether the light source point to be measured belongs to the operator's head layer, the operator's hand layer or the surgical site layer.

[0029] Optionally, the depth of the light source point to be measured is calculated by a first mathematical formula, and the first mathematical formula is:

[0030]

[0031] Wherein, Z is the depth of the light source point to be measured, L1 is the first distance, L2 is the second distance, and L3 is the third distance.

[0032] On the other hand, the present application provides a surgical video three-dimensional reconstruction device, including a video acquisition matrix, wherein the video acquisition matrix includes a reference camera and a plurality of acquisition cameras, wherein the plurality of acquisition cameras surround the reference camera, and the video acquisition matrix is ​​located above the surgical site of the patient; the device includes:

[0033] A first acquisition module is used to acquire an image captured by the reference camera before the operation begins, wherein the captured image includes a surgical site layer, and the image of the surgical site layer is used as a reference image;

[0034] A second acquisition module is used to acquire the image captured by the reference camera during the operation of the operator as a first reconstructed image; wherein the first reconstructed image includes a layer of the operator's head;

[0035] A third acquisition module is used to acquire images taken by the plurality of acquisition cameras during the operation as a second reconstructed image; wherein the second reconstructed image includes a layer of the operator's hand or a layer of the surgical site;

[0036] The reconstruction module is used to perform three-dimensional reconstruction on the operator's hand layer or the surgical site layer in the second reconstructed image based on the reference image and the operator's head layer in the first reconstructed image to obtain a surgical video of the corresponding site.

[0037] On the other hand, the present application provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned surgical video three-dimensional reconstruction method by executing the computer instructions.

[0038] On the other hand, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer device to execute the above-mentioned surgical video three-dimensional reconstruction method.

[0039] On the other hand, the present application provides a computer program product, which is used to implement the above-mentioned surgical video three-dimensional reconstruction method when the computer program product is run on a computer device.

[0040] The technical solution of this application has the following advantages:

[0041] The three-dimensional reconstruction method of surgical video provided by the present application uses a video acquisition matrix to record the surgical process. Multiple acquisition cameras of the video acquisition matrix shoot the operator's hand layer or surgical site layer from different angles to completely record the surgical process of the operator's hand layer or surgical site layer. The image captured by the reference camera is used as the reference image and the first reconstructed image, and the second reconstructed image captured by the multiple acquisition cameras is three-dimensionally reconstructed to obtain the surgical video of the corresponding part, so as to effectively and completely record the entire surgical process, thereby realizing effective surgical demonstration teaching, postoperative retrospection and / or surgical guidance, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is a flow chart of a method for three-dimensional reconstruction of surgical video in Example 1 of the present application;

[0044] Figure 2 This is a schematic diagram of the structure of the video acquisition matrix in Example 1 of the present application;

[0045] Figure 3This is a structural diagram of the reference image acquisition before the operation starts in Example 1 of the present application;

[0046] Figure 4 This is a schematic diagram of the structure of surgical video collection at the beginning of the surgery in Example 1 of the present application;

[0047] Figure 5 This is a schematic diagram of the structure for calculating the depth of the light source point to be measured in Example 1 of the present application;

[0048] Figure 6 This is a structural block diagram of a surgical video three-dimensional reconstruction device according to Example 2 of the present application;

[0049] Figure 7 This is a functional block diagram of the computer device of Example 3 of the present application. DETAILED DESCRIPTION

[0050] The technical solution of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0051] In the description of this application, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. In addition, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0052] In order to facilitate demonstration teaching, postoperative review and / or remote surgical guidance, it is usually necessary to record the operation process, record the operator's hand layer and the surgical site layer, and present them on the monitor in the form of a video. The surgical process is recorded using a surgical camera / camera arm, or by a nurse holding a camera, which can only be recorded from a single perspective. During the recording process, the single-perspective camera method can only capture the operator's head, hands or back, etc., and cannot capture the entire surgical site layer and the operator's hand layer (that is, the surgical instrument layer), which affects demonstration teaching, postoperative review and / or surgical guidance.

[0053] Example 1

[0054] This embodiment provides a method for three-dimensional reconstruction of surgical videos. Figure 13D reconstruction of a video captured by a video capture matrix according to some embodiments of the present application is described in detail. Although the process described below includes multiple operations that appear in a specific order, it should be clearly understood that these processes may also include more or fewer operations, which may be performed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment).

[0055] like Figure 2 As shown in FIG. 1 , it is a schematic diagram of a structure of a video acquisition matrix. In this embodiment, a video acquisition matrix is ​​used to record the surgical process. The video acquisition matrix includes a reference camera 101 and a plurality of acquisition cameras 102. The plurality of acquisition cameras 102 surround the reference camera 101. In the process of recording the surgical process, Figure 4 As shown in Figure 1, the video acquisition matrix is ​​located above the patient's surgical site. Figure 1 As shown, the surgical video 3D reconstruction method includes the following steps:

[0056] S101. Acquire an image captured by a reference camera before the operation begins, wherein the captured image includes a surgical site layer, and use the image of the surgical site layer as a reference image.

[0057] like Figure 3 As shown, before the surgeon starts the operation, the patient is located on the operating table 104, the video acquisition matrix is ​​located above the surgical site of the patient, the reference camera 101 acquires images of the patient, extracts the surgical site layer through the maximum boundary correlation algorithm, and uses the image of the surgical site layer as the reference image. It should be noted that the reference image is a static image (i.e., a single image) used as a background, and is the basis for the cutout analysis of the subsequent dynamic images (i.e., a video composed of multiple images). In order to ensure the quality of the subsequent three-dimensional reconstruction of the video, the reference image should include a complete surgical site layer, and the surgeon should be outside the shooting range of the reference camera 101 to reduce the impact on the reference image and the amount of data processing of the system.

[0058] During the subsequent surgery, the distance and position between the surgical site layer of the patient and the video acquisition matrix will not change, and the reference camera 101 is focused on the surgical site layer.

[0059] S102 , obtaining an image captured by a reference camera during the operation of the operator as a first reconstructed image; wherein the first reconstructed image includes a layer of the operator's head.

[0060] like Figure 4As shown, during the operation, the operator is located next to the operating table 104, and uses the surgical instrument 106 to process the surgical site layer 103 of the patient. The reference camera 101 shoots the operation process along the optical path d and collects the first reconstructed image, which includes the operator's head layer. It should be noted that the first reconstructed image is a plurality of images, and the plurality of first reconstructed images constitute the first reconstructed video.

[0061] Multiple acquisition cameras can be combined in pairs to form multiple sets of binocular cameras, such as Figure 4 As shown in the figure, the binocular camera focuses on the surgeon's head layer through optical path c to collect video information of the surgeon's head; the binocular camera focuses on the surgeon's hand layer through optical path a to collect video information of the surgical instrument's movements; the binocular camera focuses on the surgical site layer through optical path b to collect video information of the patient's surgical site. The surgical site video information is calculated and extracted through edge detection and surface polygon approximation; the surgical action video information is calculated and extracted through edge detection, gesture recognition, and surgical instrument recognition; the surgeon's head action video information is calculated and extracted through edge detection, human eye recognition and tracking.

[0062] S103, obtaining images captured by multiple acquisition cameras during the operation as a second reconstructed image; wherein the second reconstructed image includes a layer of the operator's hand or a layer of the surgical site.

[0063] like Figure 2 As shown, multiple acquisition cameras 102 are located around the reference camera 101 to form a video acquisition matrix. When the surgeon performs surgery on the patient, multiple acquisition cameras shoot from different angles, thereby recording the surgeon's hand layer (i.e., surgical instrument layer) and the surgical site layer without blind spots. Multiple acquisition cameras shoot the surgical process from different perspectives, and each acquisition camera will capture a series of images as the second reconstructed image, and each second reconstructed image includes the surgeon's hand layer or the surgical site layer. When shooting the surgical process, the acquisition camera 102 can focus on the surgeon's head layer, the surgeon's hand layer, or the surgical site layer.

[0064] For example, if Figure 4 As shown, the first acquisition camera 1021 and the second acquisition camera 1022 are located around the reference camera 101, and the first acquisition camera 1021 and the second acquisition camera 1022 form a binocular camera. For example, the first acquisition camera 1021 and the second acquisition camera 1022 focus on and shoot the surgical site layer 103 along the optical path b to obtain a second reconstructed image; for another example, the first acquisition camera 1021 and the second acquisition camera 1022 focus on and shoot the operator's hand layer (i.e., the surgical instrument layer) along the optical path a to obtain a second reconstructed image.

[0065] S104 , based on the reference image and the operator's head layer in the first reconstructed image, perform three-dimensional reconstruction on the operator's hand layer or the surgical site layer in the second reconstructed image to obtain a surgical video of the corresponding site.

[0066] The reference image is a single frame image as the basis for three-dimensional reconstruction. The first reconstructed image is a video formed by multiple images, and the second reconstructed image is a video formed by multiple images. The reference image and the surgeon's head layer in the first reconstructed image are used to perform three-dimensional reconstruction of the surgeon's hand layer or the surgical site layer in the second reconstructed image to obtain a surgical video of the corresponding part. It should be noted that the images taken by multiple acquisition cameras are used as the second reconstructed image, and the edge alignment and surface reconstruction algorithms are used to perform three-dimensional reconstruction of the surgeon's hand layer or the surgical site layer in the second reconstructed image. The obtained surgical video of the corresponding part is a three-dimensional image, which can more accurately display the surgical process.

[0067] For example, the operator's hand layer in the second reconstructed image is three-dimensionally reconstructed using the reference image and the operator's head layer in the first reconstructed image to obtain a surgical video of the operator's hand layer. For another example, the surgical site layer in the second reconstructed image is three-dimensionally reconstructed using the reference image and the operator's head layer in the first reconstructed image to obtain a three-dimensional reconstruction of the surgical site layer.

[0068] In summary, the video acquisition matrix is ​​used to record the surgical process. Multiple acquisition cameras of the video acquisition matrix shoot the operator's hand layer or the surgical site layer from different angles to completely record the surgical process of the operator's hand layer or the surgical site layer. The image captured by the reference camera is used as the reference image and the first reconstructed image, and the second reconstructed image captured by the multiple acquisition cameras is three-dimensionally reconstructed to obtain the surgical video of the corresponding part, so as to effectively and completely record the entire surgical process, thereby achieving effective surgical demonstration teaching, postoperative retrospection and / or surgical guidance, etc.

[0069] In one or more embodiments, based on the reference image and the operator's head layer in the first reconstructed image, the operator's hand layer in the second reconstructed image is three-dimensionally reconstructed to obtain a surgical video of the corresponding part, including the following steps (S201~S204).

[0070] S201 : Calculate the projection of the operator's head layer in the first reconstructed image in the reference image to obtain a projection plane.

[0071] like Figure 4As shown, the video during the operation can be divided into three layers of superposition, including the sequentially blocked surgeon head layer 105, the surgeon hand layer 106 and the surgical site layer 103. Based on the reference image taken by the reference camera 101 and the first reconstructed image, the projection of the surgeon head layer in the first reconstructed image in the reference image is calculated using the video edge extraction technology to obtain the projection plane.

[0072] S202 , calculating the occlusion of the operator's hand layer in the second reconstructed image by the projection plane to obtain a first occlusion plane.

[0073] S203 , deleting the first occlusion plane from the operator's hand layer in the second reconstructed image to obtain an unoccluded operator's hand layer.

[0074] The optical flow method or the pixel difference algorithm is used to delete the first occlusion plane from the operator's hand layer in the second reconstructed image to obtain an unoccluded operator's hand layer, thereby eliminating the influence of the operator's head layer on the operator's hand layer.

[0075] S204, using the unobstructed operator's hand layer to perform three-dimensional reconstruction to obtain a surgical video of the operator's hand layer. Using the two sets of videos to perform edge alignment, edge detection and surface reconstruction algorithms, the operator's hand layer is reconstructed in three dimensions.

[0076] In one or more embodiments, the second reconstructed image also includes a surgeon's head layer. The surgical video three-dimensional reconstruction method also includes: using the projection plane to calculate the height of the surgeon's head layer in the second reconstructed image; using the height of the surgeon's head layer in the second reconstructed image, using the optical flow method to model the surgeon's head layer in the second reconstructed image, so as to correct the first occlusion plane, and using the corrected first occlusion plane to participate in the three-dimensional reconstruction of the surgeon's hand layer, so as to obtain a surgical video of the surgeon's hand layer with higher accuracy.

[0077] In one or more embodiments, based on the reference image and the operator's head layer in the first reconstructed image, the surgical site layer in the second reconstructed image is three-dimensionally reconstructed to obtain a surgical video of the corresponding site, including the following steps (S301~S304).

[0078] S301, calculate the projection of the operator's head layer in the first reconstructed image in the reference image to obtain a projection plane. For details, please refer to the relevant description in step S201, which will not be repeated here.

[0079] S302 , calculating the occlusion of the projection plane on the surgical site layer in the second reconstructed image to obtain a second occlusion plane.

[0080] S303 : Delete the second occlusion plane from the surgical site layer in the second reconstructed image to obtain an unoccluded surgical site layer.

[0081] The first occlusion plane is deleted from the surgical site layer in the second reconstructed image by using an optical flow method or a pixel difference algorithm to obtain an unoccluded surgical site layer, thereby eliminating the influence of the surgeon's head layer on the surgical site layer.

[0082] S304, using the unobstructed surgical site layer to perform three-dimensional reconstruction to obtain a surgical video of the surgical site layer. Using the two sets of videos to perform edge alignment, edge detection and surface reconstruction algorithms, the surgical site layer is reconstructed in three dimensions.

[0083] In one or more embodiments, the second reconstructed image also includes a surgeon's head layer. The method for three-dimensional reconstruction of surgical videos also includes: using the projection plane to calculate the height of the surgeon's head layer in the second reconstructed image; using the height of the surgeon's head layer in the second reconstructed image, using the optical flow method to model the surgeon's head layer in the second reconstructed image, so as to correct the second occlusion plane, and using the corrected first occlusion plane to participate in the three-dimensional reconstruction of the surgical site layer, so as to obtain a surgical video of the surgical site layer with higher accuracy.

[0084] In one or more embodiments, identifying the operator's head layer, the operator's hand layer, and the surgical site layer includes the following steps (S401-S404).

[0085] S401: Determine the focal planes of two acquisition cameras.

[0086] For example, if Figure 5 As shown, the first acquisition camera 1021 and the second acquisition camera 1022 simultaneously acquire the light source point 202 to be measured, and the first acquisition camera 1021 and the second acquisition camera 1022 have a focal plane 201 .

[0087] S402: Obtain the focal length of the acquisition camera and the first distance between the two acquisition cameras. Figure 5 As shown, the first distance L1 between the first acquisition camera 1021 and the second acquisition camera 1022 , and the focal length f of the first acquisition camera 1021 and the second acquisition camera 1022 .

[0088] S403, obtain the second distance and the third distance of the light source point to be measured; wherein the second distance is the length of the intersection of the line connecting the first acquisition camera, the light source point to be measured and the focal plane from the intersection of the vertical focal plane of the light source point to be measured, and the third distance is the length of the intersection of the line connecting the second acquisition camera, the light source point to be measured and the focal plane from the intersection of the vertical focal plane of the light source point to be measured.

[0089] For example, Figure 5As shown, the distance between the first acquisition camera 1021 and the line 203 of the light source point 202 to be measured and the intersection point of the light source point 202 to be measured perpendicular to the focal plane 201 is L2 from the focal plane 201 and the line 203; the distance between the second acquisition camera 1022 and the line 203 of the light source point 202 to be measured and the intersection point of the light source point 202 to be measured perpendicular to the focal plane 201 is L3 from the focal plane 201 and the line 204.

[0090] S404: Based on the focal length, the first distance, the second distance, and the third distance, the depth of the light source point to be measured is calculated, and the depth of the light source point to be measured is used to determine whether the light source point to be measured belongs to the operator's head layer, the operator's hand layer, or the surgical site layer. Figure 5 As shown, based on the focal length, the first distance, the second distance and the third distance, the depth Z of the light source point to be measured is calculated. Since the distances of the operator's head layer, the operator's hand layer and the surgical site layer from the video acquisition matrix are inconsistent, the depth Z can be used to determine whether the light source point 202 to be measured is the operator's head layer, the operator's hand layer or the surgical site layer.

[0091] In one or more embodiments, the depth of the light source point to be measured is calculated by a first mathematical formula, and the first mathematical formula is:

[0092]

[0093] Wherein, Z is the depth of the light source to be measured, L1 is the first distance, L2 is the second distance, and L3 is the third distance. In some embodiments, constants may be added to the first mathematical formula.

[0094] The horizontal distance between the light source to be measured and the first acquisition camera is calculated by the second mathematical formula, and the second mathematical formula is:

[0095]

[0096] The lateral distance between the light source to be measured and the second acquisition camera is calculated by the third mathematical formula, and the third mathematical formula is:

[0097]

[0098] For example, if Figure 5 As shown, the lateral distance x1 between the light source point to be measured and the first acquisition camera is calculated by the second mathematical formula, and the lateral distance x2 between the light source point to be measured and the second acquisition camera is calculated by the third mathematical formula.

[0099] The video acquisition matrix expands the binocular stereo vision algorithm and realizes the reconstruction, splicing and segmentation of more optical planes. It meets the requirements of multi-directional, high-precision and fast three-dimensional reconstruction of scenes in the operating room with occlusion. The camera array three-dimensional scene reconstruction method is used to obtain a large-scale multi-view panoramic three-dimensional visual spherical layer model by splicing the point cloud reconstructed by the binocular stereo vision algorithm of the binocular camera at each viewing angle. A global splicing algorithm for N / 2 pairs of binocular vision systems based on the total number of acquisition camera arrays N is implemented. Based on the binocular stereo vision algorithm, a flexible camera array calibration method is designed to control and calibrate the parameters, positioning and phase difference of the acquisition cameras in the array. The binocular positioning accuracy reaches 1mm, and the comprehensive camera array calibration accuracy reaches 0.1mm. The binocular matching algorithm based on the Bayesian model is used to improve the operation speed. Finally, the position and parameters of the acquisition camera array calibration are used to align the 3D model dot matrix at each viewing angle to optimize the boundary alignment algorithm.

[0100] Example 2

[0101] This embodiment provides a surgical video 3D reconstruction device, which can be a computer device or can be set in a computer device. It includes a video acquisition matrix, which includes a reference camera and multiple acquisition cameras, and the multiple acquisition cameras surround the reference camera. The video acquisition matrix is ​​located above the surgical site of the patient. Figure 6 As shown, the device includes: a first acquisition module 301, a second acquisition module 302, a third acquisition module 303 and a reconstruction module 304.

[0102] The first acquisition module 301 is used to acquire the image taken by the reference camera before the operation starts. The image taken includes a surgical site layer, and the image of the surgical site layer is used as the reference image. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0103] The second acquisition module 302 is used to obtain the image taken by the reference camera during the operation of the operator as the first reconstructed image; wherein the first reconstructed image includes the operator's head layer; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0104] The third acquisition module 303 is used to acquire images taken by multiple acquisition cameras during the operation as the second reconstructed image; wherein the second reconstructed image includes the operator's hand layer or the surgical site layer; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0105] The reconstruction module 304 is used to perform three-dimensional reconstruction of the operator's hand layer or the surgical site layer in the second reconstructed image based on the reference image and the operator's head layer in the first reconstructed image to obtain a surgical video of the corresponding site. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0106] In one or more embodiments, the reconstruction module 304 includes a first computing unit ( Figure 6 Not shown), the second computing unit ( Figure 6 Not shown), the first deletion unit ( Figure 6 ) and the first reconstruction unit ( Figure 6 not shown).

[0107] The first calculation unit is used to calculate the projection of the operator's head layer in the first reconstructed image in the reference image to obtain a projection plane; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0108] The second calculation unit is used to calculate the occlusion of the operator's hand layer in the second reconstructed image by the projection plane to obtain a first occlusion plane; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0109] The first deleting unit is used to delete the first occlusion plane from the operator's hand layer in the second reconstructed image to obtain an unoccluded operator's hand layer; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0110] The first reconstruction unit is used to perform three-dimensional reconstruction using the unobstructed operator's hand layer to obtain a surgical video of the operator's hand layer. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0111] In one or more embodiments, the reconstruction module 304 includes a third computing unit ( Figure 6 Not shown), the fourth computing unit ( Figure 6 Not shown), the second deletion unit ( Figure 6 ) and a second reconstruction unit ( Figure 6 not shown).

[0112] The third calculation unit is used to calculate the projection of the operator's head layer in the first reconstructed image in the reference image to obtain a projection plane; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0113] The fourth calculation unit is used to calculate the occlusion of the surgical site layer in the second reconstructed image by the projection plane to obtain a second occlusion plane; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0114] The second deleting unit is used to delete the second occlusion plane from the surgical site layer in the second reconstructed image to obtain an unoccluded surgical site layer; for details, please refer to the relevant description in Example 1, which will not be repeated here.

[0115] The second reconstruction unit is used to perform three-dimensional reconstruction using the unobstructed surgical site layer to obtain a surgical video of the surgical site layer. For details, please refer to the relevant description in Example 1, which will not be repeated here.

[0116] The effects of the surgical video three-dimensional reconstruction device of this embodiment can be found in the relevant description in Embodiment 1, and will not be elaborated here.

[0117] Example 3

[0118] This embodiment provides a computer device, such as Figure 7 As shown, the computer device includes a processor 401 and a memory 402, wherein the processor 401 and the memory 402 can be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0119] The processor 401 may be a central processing unit (CPU). The processor 401 may also be other general-purpose processors, digital signal processors (DSP), graphics processors (GPU), embedded neural network processors (NPU), or other dedicated deep learning coprocessors, application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0120] The memory 402, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the three-dimensional reconstruction method of surgical videos using a neural network in the embodiment of the present invention (such as the first acquisition module 301, the second acquisition module 302, the third acquisition module 303 and the reconstruction module 304 in the above embodiment). The processor 401 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 402, that is, realizes the above-mentioned three-dimensional reconstruction method of surgical videos using a neural network.

[0121] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor 401, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 402 may optionally include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to the processor 401 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions can execute the method for three-dimensional reconstruction of surgical videos by neural networks in any of the above method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0123] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer device, the method for three-dimensional reconstruction of surgical videos in Embodiment 1 is implemented.

[0124] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A method for three-dimensional reconstruction of surgical videos, comprising a video acquisition matrix, wherein the video acquisition matrix comprises a reference camera and a plurality of acquisition cameras, wherein the plurality of acquisition cameras surround the reference camera, and the video acquisition matrix is ​​located above the surgical site of the patient; characterized in that: The method comprises: Acquire an image captured by the reference camera before the operation begins, wherein the captured image includes a surgical site layer, and use the image of the surgical site layer as a reference image; Acquire an image captured by the reference camera during the operation of the operator as a first reconstructed image; wherein the first reconstructed image includes a layer of the operator's head; Acquire images captured by the plurality of acquisition cameras during the operation as a second reconstructed image; wherein the second reconstructed image includes a layer of the operator's hand or a layer of the surgical site; Based on the reference image and the operator's head layer in the first reconstructed image, three-dimensionally reconstruct the operator's hand layer or the surgical site layer in the second reconstructed image to obtain a surgical video of the corresponding site; The three-dimensional reconstruction of the operator's hand layer in the second reconstructed image based on the reference image and the operator's head layer in the first reconstructed image to obtain a surgical video of the corresponding part includes: Calculating the projection of the operator's head layer in the first reconstructed image in the reference image to obtain a projection plane; Calculating the occlusion of the operator's hand layer in the second reconstructed image by the projection plane to obtain a first occlusion plane; Deleting the first occlusion plane from the operator's hand layer in the second reconstructed image to obtain an unoccluded operator's hand layer; The unobstructed surgeon's hand layer is used for three-dimensional reconstruction to obtain the surgical video of the surgeon's hand layer.

2. The method for three-dimensional reconstruction of surgical video according to claim 1, characterized in that: The second reconstructed image includes a layer of the operator's head; the method further includes: Calculating the height of the operator's head layer in the second reconstructed image using the projection plane; The height of the operator's head layer in the second reconstructed image is used to model the operator's head layer in the second reconstructed image using an optical flow method to correct the first occlusion plane.

3. The method for three-dimensional reconstruction of surgical video according to claim 1, characterized in that: The three-dimensional reconstruction of the surgical site layer in the second reconstructed image based on the reference image and the surgeon's head layer in the first reconstructed image to obtain a surgical video of the corresponding site includes: Calculating the projection of the operator's head layer in the first reconstructed image in the reference image to obtain a projection plane; Calculating the occlusion of the surgical site layer in the second reconstructed image by the projection plane to obtain a second occlusion plane; Deleting the second occlusion plane from the surgical site layer in the second reconstructed image to obtain an unoccluded surgical site layer; The unobstructed surgical site layer is used for three-dimensional reconstruction to obtain the surgical video of the surgical site layer.

4. The method for three-dimensional reconstruction of surgical video according to claim 3, characterized in that: The second reconstructed image includes a layer of the operator's head; the method further includes: Calculating the height of the operator's head layer in the second reconstructed image using the projection plane; The height of the operator's head layer in the second reconstructed image is used to model the operator's head layer in the second reconstructed image using an optical flow method to correct the second occlusion plane.

5. The method for three-dimensional reconstruction of surgical video according to any one of claims 1 to 4, characterized in that: Identify the surgeon's head layer, surgeon's hand layer, and surgical site layer by following these steps: Determining the focal planes of the two acquisition cameras; Obtaining the focal length of the acquisition camera and a first distance between two acquisition cameras; Obtain a second distance and a third distance of the light source point to be measured; wherein the second distance is the length of the intersection of the line connecting the first acquisition camera, the light source point to be measured and the focal plane from the intersection of the vertical focal plane of the light source point to be measured, and the second distance is the length of the intersection of the line connecting the second acquisition camera, the light source point to be measured and the focal plane from the intersection of the vertical focal plane of the light source point to be measured; Based on the focal length, the first distance, the second distance and the third distance, the depth of the light source point to be measured is calculated, and the depth of the light source point to be measured is used to determine whether the light source point to be measured belongs to the operator's head layer, the operator's hand layer or the surgical site layer.

6. The method for three-dimensional reconstruction of surgical video according to claim 5, characterized in that: The depth of the light source point to be measured is calculated by a first mathematical formula, wherein the first mathematical formula is: Wherein, Z is the depth of the light source to be measured, L1 is the first distance, L2 is the second distance, L3 is the third distance, and f is the focal length of the acquisition camera.

7. A surgical video 3D reconstruction device, using the surgical video 3D reconstruction method according to any one of claims 1 to 6, the surgical video 3D reconstruction device comprising a video acquisition matrix, the video acquisition matrix comprising a reference camera and a plurality of acquisition cameras, the plurality of acquisition cameras surround the reference camera, the video acquisition matrix is ​​located above the surgical site of the patient; characterized in that The device also includes: A first acquisition module is used to acquire an image captured by the reference camera before the operation begins, wherein the captured image includes a surgical site layer, and the image of the surgical site layer is used as a reference image; A second acquisition module is used to acquire the image captured by the reference camera during the operation of the operator as a first reconstructed image; wherein the first reconstructed image includes a layer of the operator's head; A third acquisition module is used to acquire images taken by the plurality of acquisition cameras during the operation as a second reconstructed image; wherein the second reconstructed image includes a layer of the operator's hand or a layer of the surgical site; The reconstruction module is used to perform three-dimensional reconstruction on the operator's hand layer or the surgical site layer in the second reconstructed image based on the reference image and the operator's head layer in the first reconstructed image to obtain a surgical video of the corresponding site.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the surgical video three-dimensional reconstruction method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the surgical video three-dimensional reconstruction method according to any one of claims 1 to 6.

Images