Three-dimensional fluorescence data generation method and system

Abstract

The present invention provides a method and system for generating three-dimensional fluorescence data, which can be applied to the field of surgical navigation technology. The method includes: obtaining a two-dimensional fluorescence image and three-dimensional point cloud data for the same lesion area; based on multiple pixel points and multiple initial data points, fusing the two-dimensional fluorescence image and the three-dimensional depth data to obtain initial three-dimensional fluorescence data, wherein the initial three-dimensional fluorescence data includes multiple target data points with fluorescence properties, and the multiple target data points are used to determine the three-dimensional shape representation of the lesion area; constructing an objective function based on the fluorescence properties of each of the multiple target data points, the neighboring data points of each of the multiple target data points, and preset parameters; solving the objective function to obtain target three-dimensional fluorescence data, wherein the target three-dimensional fluorescence data includes the updated three-dimensional shape representation of the lesion area.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical navigation, and in particular to a three-dimensional fluorescence data generation method and system. Background Art

[0002] Precise surgical resection is currently the main method for treating malignant tumors. The success of this treatment method depends on the doctor's determination of the location and morphology of the primary tumor and its local and distant metastases. In recent years, optical molecular imaging technology has been widely used in surgical navigation. By labeling specific molecules, it can achieve in vivo, real-time, and dynamic detection of molecular activities.

[0003] The application of traditional fluorescent molecular imaging technology in clinical practice usually involves direct navigation based on two-dimensional fluorescent images, or registering the three-dimensional anatomical structure obtained from preoperative structural images to the two-dimensional fluorescent images during surgery. However, the fluorescence information is still two-dimensional and does not reflect depth information. Moreover, in clinical surgical scenarios, because preoperative structural images may be deformed due to surgical operations, patient breathing, heartbeat, etc., they cannot be directly applied to intraoperative scenarios for registration and fusion with intraoperative fluorescent images. Therefore, the related technologies still have the problem of poor surgical guidance and inability to bring about better surgical assistance effects. Summary of the invention

[0004] In view of the above problems, the present invention provides a three-dimensional fluorescence data generation method and system.

[0005] According to one aspect of the present invention, a method for generating three-dimensional fluorescence data is provided, comprising: obtaining a two-dimensional fluorescence image and three-dimensional point cloud data for the same lesion area, wherein the two-dimensional fluorescence image includes a plurality of pixel points with fluorescence properties, the plurality of pixel points are used to determine an initial two-dimensional shape of the lesion area, and the three-dimensional point cloud data includes a plurality of initial data points for characterizing a three-dimensional contour of the lesion area; based on the plurality of pixel points and the plurality of initial data points, fusing the two-dimensional fluorescence image and the three-dimensional depth data to obtain initial three-dimensional fluorescence data, wherein the initial three-dimensional fluorescence data includes a plurality of target data points with fluorescence properties, the plurality of target data points are used to determine a three-dimensional shape representation of the lesion area; constructing an objective function based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points, and preset parameters; solving the objective function to obtain target three-dimensional fluorescence data, wherein the target three-dimensional fluorescence data includes an updated three-dimensional shape representation of the lesion area.

[0006] Another aspect of the present invention provides a three-dimensional fluorescence data generation system, comprising:

[0007] A fluorescence image acquisition module is used to acquire a two-dimensional fluorescence image of the lesion area; a three-dimensional data acquisition module is used to acquire three-dimensional point cloud data of the lesion area based on the same acquisition angle as the fluorescence image acquisition module; a processing module is used to obtain a two-dimensional fluorescence image from the fluorescence image acquisition module, and to obtain three-dimensional point cloud data from the three-dimensional data acquisition module, wherein the two-dimensional fluorescence image includes a plurality of pixel points corresponding to the lesion area, the pixel points have fluorescence properties, the fluorescence properties are used to determine the shape of the lesion area, and the three-dimensional point cloud data includes initial data points corresponding to each of the plurality of pixel points; a processing module is used to acquire a two-dimensional fluorescence image from the fluorescence image acquisition module, and to acquire three-dimensional point cloud data from the three-dimensional data acquisition module, wherein the two-dimensional fluorescence image includes a plurality of pixel points having fluorescence properties The three-dimensional point cloud data includes a plurality of initial data points for characterizing the three-dimensional contour of the lesion area; the processing module is further used to fuse the two-dimensional fluorescence image and the three-dimensional depth data based on the plurality of pixel points and the plurality of initial data points to obtain initial three-dimensional fluorescence data, wherein the initial three-dimensional fluorescence data includes a plurality of target data points with fluorescence properties, and the plurality of target data points are used to determine the three-dimensional shape representation of the lesion area; an objective function is constructed based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points, and preset parameters; and the objective function is solved to obtain target three-dimensional fluorescence data, wherein the target three-dimensional fluorescence data includes the updated three-dimensional shape representation of the lesion area.

[0008] According to the three-dimensional fluorescence data generation method of the present invention, by acquiring a two-dimensional fluorescence image and three-dimensional point cloud data for the same lesion area and fusing the two, the fluorescence properties in the two-dimensional fluorescence image are mapped to the three-dimensional point cloud data, and the objective function is constructed by the fluorescence properties of each of the multiple target data points, the neighboring data points of each of the multiple target data points, and the preset parameters, and the fluorescence properties of each target data point are reconstructed in the process of solving the objective function, so as to achieve the solution of the position and form of the fluorescent light source in the three-dimensional space, and then determine the three-dimensional shape representation of the lesion area. After the two-dimensional fluorescence image and the three-dimensional point cloud data are fused, the multiple target data points and the fluorescence properties of each of the multiple target data points are determined to construct the objective function, and the tomographic reconstruction of the three-dimensional shape representation of the lesion area is achieved by solving the objective function, so that at least part of the problem of poor surgical guidance in the related art is solved, and the technical effect of achieving more accurate surgical guidance by determining the three-dimensional shape representation of the lesion area is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0010] Figure 1 A flow chart of a method for generating three-dimensional fluorescence data according to an embodiment of the present invention is shown.

[0011] Figure 2 A flow chart of a method for generating three-dimensional fluorescence data according to another embodiment of the present invention is shown.

[0012] Figure 3 A schematic structural diagram of a three-dimensional fluorescence data generating system according to an embodiment of the present invention is shown.

[0013] Figure 4 A schematic structural diagram of a three-dimensional fluorescence data generating system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0014] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0015] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0016] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0017] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0018] It should be noted that the three-dimensional fluorescence data generation method and system of the present invention can be used in the field of surgical navigation technology, and can also be used in any field other than the field of surgical navigation technology, such as the field of artificial intelligence technology, the field of computer technology, etc. The present invention does not limit the application field of the three-dimensional fluorescence data generation method and system.

[0019] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, invention and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0020] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0021] During the research process, it was found that malignant tumors are the main cause of human death, and precise surgical resection is currently the main means of treating malignant tumors. The success of this treatment method depends on the doctor's determination of the location and morphology of the primary tumor and its local and distant metastases, and trying not to damage important tissues during the resection process. Existing intraoperative imaging methods include intraoperative ultrasound, intraoperative MRI, etc., which are difficult to meet the needs of precision surgery. Although optical molecular imaging technology brings opportunities for precision surgery, the application of traditional fluorescent molecular imaging technology in clinical practice is often based on direct navigation of two-dimensional fluorescent images, and it still has the problem of not being able to bring better surgical assistance effects.

[0022] An embodiment of the present invention provides a three-dimensional fluorescence data generation method, including: acquiring a two-dimensional fluorescence image and three-dimensional point cloud data for the same lesion area, wherein the two-dimensional fluorescence image includes a plurality of pixel points with fluorescence properties, and the plurality of pixel points are used to determine the initial two-dimensional shape of the lesion area, and the three-dimensional point cloud data includes a plurality of initial data points for characterizing the three-dimensional contour of the lesion area; based on the plurality of pixel points and the plurality of initial data points, fusing the two-dimensional fluorescence image and the three-dimensional depth data to obtain initial three-dimensional fluorescence data, wherein the initial three-dimensional fluorescence data includes a plurality of target data points with fluorescence properties, and the plurality of target data points are used to determine the three-dimensional shape representation of the lesion area; constructing an objective function based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points, and preset parameters; solving the objective function to obtain target three-dimensional fluorescence data, wherein the target three-dimensional fluorescence data includes the updated three-dimensional shape representation of the lesion area. Figure 1~Figure 2 The three-dimensional fluorescence data generating method according to the embodiment of the present invention is described in detail.

[0023] Figure 1 A flow chart of a method for generating three-dimensional fluorescence data according to an embodiment of the present invention is shown.

[0024] like Figure 1 As shown, the method includes operations S110 to S140.

[0025] In operation S110, a two-dimensional fluorescence image and three-dimensional point cloud data are obtained for the same lesion area, wherein the two-dimensional fluorescence image includes a plurality of pixel points having fluorescence properties, the plurality of pixel points are used to determine an initial two-dimensional shape of the lesion area, and the three-dimensional point cloud data includes a plurality of initial data points for characterizing a three-dimensional contour of the lesion area.

[0026] In operation S120, based on multiple pixel points and multiple initial data points, the two-dimensional fluorescence image and the three-dimensional depth data are fused to obtain initial three-dimensional fluorescence data, wherein the initial three-dimensional fluorescence data includes multiple target data points with fluorescence properties, and the multiple target data points are used to determine the three-dimensional shape representation of the lesion area.

[0027] In operation S130 , an objective function is constructed based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points, and preset parameters.

[0028] In operation S140 , the target function is solved to obtain target three-dimensional fluorescence data, wherein the target three-dimensional fluorescence data includes an updated three-dimensional shape representation of the lesion area.

[0029] According to an embodiment of the present invention, the lesion area may be an area in a human tissue or organ where an abnormality occurs, such as an area where a tumor exists.

[0030] According to an embodiment of the present invention, the two-dimensional fluorescence image may be an image having a larger fluorescence attribute in the lesion area, and the fluorescence attribute may include fluorescence intensity. In the two-dimensional fluorescence image, there is a large light intensity difference between the lesion area and the non-lesion area, so that the two-dimensional fluorescence image can be used to determine the two-dimensional shape and range of the lesion area.

[0031] According to an embodiment of the present invention, the initial data point may be a coordinate point in the three-dimensional point cloud data.

[0032] According to an embodiment of the present invention, after acquiring the two-dimensional fluorescence image and the three-dimensional point cloud data, pre-processing operations such as denoising, enhancement and rotation may be performed on the two respectively.

[0033] According to an embodiment of the present invention, the initial three-dimensional fluorescence data can be three-dimensional point cloud data with fluorescence properties, and the target data point is the coordinate point in the three-dimensional point cloud data. The initial three-dimensional fluorescence data can also be a tetrahedral mesh with fluorescence properties, and the target data point is the vertex of each mesh in the tetrahedral mesh.

[0034] According to an embodiment of the present invention, both the two-dimensional fluorescent image and the three-dimensional point cloud data may be data acquired in real time during surgery.

[0035] According to an embodiment of the present invention, the neighbor data point of the target data point may be a target data point whose distance from the target data point in multiple directions is less than a preset distance, or may be a target data point that is closest to the target data point in multiple directions. Specifically, in some embodiments, for each target data point, the data point closest to the target data point may be determined in multiple directions, and the respective nearest target data points in multiple directions may be used as the neighbor data point of the target data point.

[0036] According to the embodiments of the present invention, in some embodiments, the multiple directions can be multiple of the following directions: positive and negative directions of the X-axis, Y-axis, and Z-axis, and the positive and negative directions of the four straight lines x=y=z, x=-y=z, x=-y=-z, and x=y=-z.

[0037] According to an embodiment of the present invention, neighbor data points are determined for each target data point in the above-mentioned multiple directions, and 14 neighbor data points closest to the target data point can be determined for each target data point.

[0038] According to an embodiment of the present invention, the two-dimensional shape of the three-dimensional shape representation may be different from the initial two-dimensional shape.

[0039] According to an embodiment of the present invention, the process of fusing the two-dimensional fluorescence image and the three-dimensional point cloud data may be a process of mapping the fluorescence attribute of a pixel point to an initial data point corresponding to the pixel point.

[0040] According to an embodiment of the present invention, since the multiple target data points respectively have fluorescence properties, and the fluorescence properties may include fluorescence intensity, the three-dimensional shape representation of the lesion area may be determined by the multiple target data points and their respective fluorescence intensities.

[0041] According to the embodiment of the present invention, there is no limitation on the preset parameters, which may be parameters determined after multiple experiments during the experimental process of constructing the objective function.

[0042] According to an embodiment of the present invention, the objective function can be solved in an iterative manner. Specifically, the objective function can be solved to obtain the first three-dimensional fluorescence data; determine whether the first three-dimensional fluorescence data meets the preset conditions, and if so, determine that the first three-dimensional fluorescence data is the target fluorescence data; if the first three-dimensional fluorescence data does not meet the preset conditions, through operation S130, the first three-dimensional fluorescence data is used as the initial three-dimensional fluorescence data to reconstruct the objective function and obtain an updated objective function; and solve the updated objective function again to obtain the second three-dimensional fluorescence data, determine whether the second three-dimensional fluorescence data meets the preset conditions, and if it is determined that the second three-dimensional fluorescence data meets the preset conditions, determine that the second three-dimensional fluorescence data is the target three-dimensional fluorescence data; otherwise, repeat the above process until the three-dimensional fluorescence data that meets the preset conditions is determined, and the preset conditions can be that the three-dimensional shape representation of the lesion area meets expectations, etc.

[0043] According to an embodiment of the present invention, a limited-memory variant quasi-Newton (Limited-memory BFGS, L-BFGS) method can be used to solve the objective function. This method obtains the inverse of the Hessian matrix through historical information, which can reduce the storage amount, improve the calculation speed, and meet the real-time requirements during surgery.

[0044] According to an embodiment of the present invention, by constructing and solving the objective function, the reconstruction of the initial three-dimensional fluorescence data can be achieved, so that the final target three-dimensional fluorescence data can more accurately represent the three-dimensional shape of the lesion area, thereby better performing surgical navigation.

[0045] According to the embodiment of the present invention, the target three-dimensional fluorescence data can be visualized, so that the three-dimensional shape of the lesion area can be determined more intuitively.

[0046] According to the three-dimensional fluorescence data generation method of the present invention, by acquiring a two-dimensional fluorescence image and three-dimensional point cloud data for the same lesion area and fusing the two, the fluorescence properties in the two-dimensional fluorescence image are mapped to the three-dimensional point cloud data, and the objective function is constructed by the fluorescence properties of each of the multiple target data points, the neighboring data points of each of the multiple target data points and the preset parameters, and the fluorescence properties of each target data point are reconstructed in the process of solving the objective function, so as to achieve the solution of the position and form of the fluorescent light source in the three-dimensional space, and then determine the three-dimensional shape representation of the lesion area. Since the objective function is constructed by using the multiple target data points and the fluorescence properties of each of the multiple target data points determined after the fusion of the two-dimensional fluorescence image and the three-dimensional point cloud data, and the tomographic reconstruction of the three-dimensional shape representation of the lesion area is achieved by solving the objective function, the problem of poor surgical guidance in the related art is at least partially solved, and the technical effect of achieving more accurate surgical guidance by determining the three-dimensional shape representation of the lesion area is achieved.

[0047] According to an embodiment of the present invention, constructing an objective function based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points, and preset parameters may include the following operations.

[0048] Based on the fluorescence properties of each of the multiple target data points and the neighboring data points of each of the multiple target data points, the correlation values ​​between the multiple target data points are determined to obtain a correlation matrix; based on the correlation matrix, the fluorescence properties of each of the multiple target data points and preset parameters, an objective function is constructed.

[0049] According to an embodiment of the present invention, the correlation matrix may be determined by the fluorescence properties of each of the multiple target data points and the fluorescence properties of each of the multiple target data points' neighboring data points, and the correlation matrix may include correlation values ​​between each of the multiple target data points.

[0050] According to an embodiment of the present invention, in some embodiments, the objective function may be as shown in the following formula (1):

[0051] ; (1)

[0052] Wherein, x is the fluorescence intensity representation vector to be determined, which may include the fluorescence properties of each of multiple target data points in the target three-dimensional fluorescence data. is the correlation matrix, are preset parameters, which are used to characterize The weight parameters and The weight parameter, A is the system matrix containing the optical parameter information of near-infrared second-region fluorescence, It is a matrix composed of the fluorescence properties of multiple target data points in the initial three-dimensional fluorescence data.

[0053] According to an embodiment of the present invention, It is an adaptive aggregation regularization term. By adding it to the initial objective function, the aggregation of the light source distribution in the target three-dimensional fluorescence data can be guaranteed. is a regularization term used to ensure the sparsity of the light source distribution in the target three-dimensional fluorescence data. The light source distribution in the target three-dimensional fluorescence data can be determined by the fluorescence properties of each of the multiple target data points in the target three-dimensional fluorescence data.

[0054] According to an embodiment of the present invention, the objective function may be constructed by the present invention taking into account the following situations: abnormalities occurring in human tissues or organs, such as tumors, whose true morphological information is a diffuse tumor growth pattern, that is, the tumors are distributed in clusters rather than scattered in tissues and organs, and continue to diffuse and grow outward. Therefore, the adaptive aggregation regularization term included in the objective function of the present invention and the regularization term for ensuring the sparsity of the light source distribution in the target three-dimensional fluorescence data can fully consider the characteristics of lesions such as tumors, thereby obtaining more accurate target three-dimensional fluorescence data.

[0055] According to the embodiments of the present invention, considering that both regularization terms shown in formula (1) are non-smooth expressions, it is difficult to directly perform gradient calculations. The common non-smooth solution methods have a slow operation speed and may be difficult to meet the needs of real-time navigation during surgery. Therefore, in some embodiments of the present invention, a smooth strategy is introduced to replace the absolute value operation with a smooth function h(x), and the objective function of formula (1) is converted into a convex smooth objective function to accelerate the operation. The smooth function expression is shown in the following formula (2).

[0056] ; (2)

[0057] According to an embodiment of the present disclosure, u is a preset weight, and t is a value to be calculated.

[0058] According to an embodiment of the present invention, the smooth form objective function constructed in some embodiments can be determined by formula (1) and formula (2). Therefore, a smooth form objective function as shown in the following formula (3) can be constructed, and its gradient can be as shown in the following (4).

[0059] ; (3)

[0060] ; (4)

[0061] in, , All are preset parameters, which can represent preset weight values.

[0062] According to an embodiment of the present invention, the formula form of the objective function such as formula (1) can be obtained through the following derivation process: The time domain expression of photon transmission in complex biological tissue can be shown as formula (5).

[0063] (5)

[0064] Where c represents the propagation speed of light in biological tissues, Indicates location, represents the direction of the unit vector, and t represents the time; represents the radiance, that is, the energy of the photon flow per unit solid angle, is the absorption coefficient, represents the scattering coefficient, represents the spatial and angular distribution of the internal fluorescent light source, Represents the scattering phase coefficient, which is the single scattered photon from the incident direction Scattered The probability that The integral over the solid angle is 1.

[0065] According to an embodiment of the present invention, the diffusion equation may be used to approximate the formula (5), and the steady-state diffusion equation expression may be as shown in the formula (6).

[0066] (6)

[0067] in, is the absorption coefficient, is the diffusion coefficient, g represents the anisotropy coefficient, is the luminous flux intensity, is the light intensity.

[0068] According to the embodiment of the present invention, and based on the NIR-II light transmission model, it can be known that the excitation photons are transmitted from the biological surface through the biological tissue to the lesion area, the photons distributed in the lesion area are converted into emission photons by fluorescent dyes, and the emission photons are transmitted from the lesion area through the biological tissue to the biological surface in three stages. The transmission process of the excitation photons and the emission photons combined with the Robin boundary condition can obtain the following diffusion equations such as formula (7) to formula (9).

[0069]

[0070] Among them, formula (7) to formula (9) are the diffusion equations of the excitation process and the emission process, and (9) is the joint boundary condition. represents the entire area, e and c represent the excitation light and emission light respectively, are the fluorescence flux intensities of the excitation light and emission light, is the fluorescence flux intensity of the excitation light and the emission light, are the excitation light and emission light absorption coefficients, respectively, are the diffusion coefficients of the excitation light and the emission light, respectively, are the coordinates of the excitation point, is the light source intensity, Represents the fluorescence yield, v represents the unit vector normal to the body surface, and the continuous equation is discretized using the finite element method. The linear equation formula (10) can be obtained by integration.

[0071]

[0072] Where A is the system matrix containing the optical parameter information of near-infrared second-region fluorescence, is the surface fluorescence intensity vector, X is the fluorescence intensity representation vector to be determined, which may include the fluorescence intensity values ​​of multiple target data points in the target three-dimensional fluorescence data. Where m is the number of surface nodes of the initial three-dimensional fluorescence data, and the nodes may be target data points, and n is the number of internal nodes of the initial three-dimensional fluorescence data.

[0073] According to the embodiment of the present invention, since it is known that the number of internal nodes of the grid is much larger than the number of surface nodes, it can be seen that the above formula (10) is pathological and ill-posed, and it is difficult to obtain the correct X. Then, the present invention fully considers the characteristics of lesions such as tumors to determine the objective function such as formula (1).

[0074] According to an embodiment of the present invention, based on the fluorescence properties of each of the multiple target data points and the neighboring data points of each of the multiple target data points, the correlation values ​​between each of the multiple target data points are determined to obtain a correlation matrix, which may also include the following operations.

[0075] For each target data point, a data point to be calculated is determined from multiple target data points; when the data point to be calculated is determined to be a neighbor data point of the target data point, a correlation calculation result is determined based on the fluorescence attribute of the target data point, the fluorescence attribute of the data point to be calculated, the correlation weight determined by the minimum fluorescence attribute difference and the maximum fluorescence attribute difference, and the target distance between the target data point and the data point to be calculated, and the correlation calculation result is used as the target correlation value between the target data point and the data point to be calculated; when the data point to be calculated is determined to be a target data point, the target correlation value is determined to be a first preset value; when the data point to be calculated is determined to be a data point other than the target data point or the neighbor data point, the target correlation value is determined to be a second preset value; based on the target correlation values ​​between each of the multiple target data points, a correlation matrix is ​​obtained; wherein the target distance is determined by the position attribute of the data point to be calculated and the position attribute of the target data point, the minimum fluorescence attribute difference is the smallest of the fluorescence attribute differences between the target data point and at least one neighbor data point of the target data point, and the maximum fluorescence attribute difference is the largest of the fluorescence attribute differences between the target data point and at least one neighbor data point of the target data point.

[0076] According to an embodiment of the present invention, the location attribute may be information characterizing the location attribute such as a coordinate value.

[0077] According to an embodiment of the present invention, for each target data point, a target correlation value may be determined for the target data point and a plurality of target data points, thereby determining the correlation values ​​of the target data point and the plurality of target data points.

[0078] According to an embodiment of the present invention, the data point to be calculated is also a target data point, which is a target data point selected from a plurality of target data points for calculating a target correlation value with the current target data point.

[0079] According to an embodiment of the present invention, the target correlation values ​​between the target data point and each of the multiple target data points may be determined as shown in the following formulas (11) to (13).

[0080] ; (11)

[0081] ; (12)

[0082] ; (13)

[0083] Among them, l ij is the correlation value between the i-th target data point and the j-th data point to be calculated of the i-th target data point, is the Euclidean distance between the i-th target data point and the j-th neighbor data point of the i-th target data point when the node to be calculated is the neighbor data point of the i-th target data point, is a preset parameter, which represents the preset Gaussian kernel parameter, is the maximum fluorescence attribute difference determined from the fluorescence attribute differences between the ith target data point and multiple neighboring data points of the ith target data point in the kth iteration, is the minimum fluorescence attribute difference determined from the fluorescence attribute differences between the ith target data point and multiple neighboring data points of the ith target data point in the kth iteration, is the fluorescence property of the i-th target data point in the k-th iteration, is the fluorescence property of the jth neighbor data point of the i-th target data point in the k-th iteration, is the relevance weight, ( ) is the location attribute of the i-th target data point, ( ) is the location attribute of the jth neighbor data point of the i-th target data point, Characterize the entire area, is the point set of target data points, is the normalization parameter.

[0084] According to an embodiment of the present invention, The node to be calculated is the target data point itself, i≠j is the neighboring data point of the target data point, and others is other data points.

[0085] According to an embodiment of the present invention, the first preset value and the second preset value are not limited. Different first preset values ​​and second preset values ​​can be set according to actual conditions. The second preset value can be set to a smaller value. In some embodiments, the first preset value can be 1 and the second preset value can be 0.

[0086] According to an embodiment of the present invention, the fluorescence attribute of a target data point can be subtracted from the fluorescence attributes of multiple neighboring data points of the target data point, thereby obtaining the fluorescence attribute difference between the target data point and the multiple neighboring data points of the target data, and the largest one is determined as the maximum fluorescence attribute difference, and the smallest one is determined as the minimum fluorescence attribute difference.

[0087] According to an embodiment of the present invention, when the data point to be calculated is a data point other than the current target data point itself or a neighbor data point of the current target data point, the target correlation value between the target data point and the data point to be calculated is considered to be a second preset value.

[0088] According to an embodiment of the present invention, when determining the correlation between the target data point and the data point to be calculated, it is determined whether the data point to be calculated is the target data point itself, a neighbor data point of the target data point, or other data points, so that when the node to be calculated is other data points, a smaller target correlation value is obtained, and an adaptive target correlation value calculation method is adopted to calculate the target correlation value of the target data point and the neighbor data point, so that according to the prior information of the light intensity distribution in the neighborhood, the fluorescence distribution of the light source area and the background area is smoother, and the light intensity difference at the boundary of the light source is larger, so as to achieve edge enhancement, thereby making the final three-dimensional shape representation of the lesion area more accurate.

[0089] According to an embodiment of the present invention, when the node to be calculated is a neighbor node, the correlation value is obtained by replacing the fixed-size correlation weight with an adaptive correlation weight calculation method. In the iterative solution process, the light source area and the non-light source area will have a smaller light intensity variance, so that a larger correlation weight will be used to calculate the correlation value, and the edge area, that is, the separation area of ​​the light source area and the non-light source area, will have a larger light intensity variance, so that a smaller correlation weight will be used to calculate the correlation value, so that the objective function constructed by the correlation matrix obtained by the above-mentioned correlation value calculation method can obtain smoother reconstruction results in the light source area and the non-light source area, clearer reconstruction results in the edge area, and non-smooth target three-dimensional fluorescence data, that is, the three-dimensional shape representation of the lesion area is clearer and more accurate after the solution.

[0090] According to an embodiment of the present invention, the light source region may be a region of target data points having a fluorescence attribute greater than a preset value, and the background region may be a region of target data points having a fluorescence attribute less than or equal to a preset value, and the preset value is not limited and may be 0, etc. Usually, the light source region may be a lesion region, and the background region, which may also be called a non-light source region, may be a normal region in a tissue or organ other than the lesion region.

[0091] According to an embodiment of the present invention, each target data point has at least one neighbor data point; wherein, before determining the correlation values ​​between each of the multiple target data points based on the fluorescence properties of each of the multiple target data points and the neighbor data points of each of the multiple target data points to obtain a correlation matrix, the above method may further include the following operations.

[0092] For each target data point, the mean fluorescence attribute difference is determined based on the fluorescence attribute of at least one neighboring data point of the target data point and the fluorescence attribute of the target data point; the fluorescence area to which the target data point belongs is determined based on the mean fluorescence attribute difference, the minimum fluorescence attribute difference and the maximum fluorescence attribute difference.

[0093] According to an embodiment of the present invention, the fluorescent area may include: a high variance area and a low variance area.

[0094] According to an embodiment of the present invention, the fluorescent area to which the target data point belongs can be determined by the following formula (14) to formula (15).

[0095] ; (14)

[0096] ; (15)

[0097] in, Characterize the high variance area in the nth iteration result, represents the low variance area in the nth iteration result, 14 represents the neighbor data points determined from 14 directions, and its value changes according to the number of neighbor data points. represents the fluorescence property of the i-th target data point in the n-th iteration, represents the fluorescence property of the kth neighbor data point of the ith target data point at the nth iteration. represents the maximum fluorescence attribute difference determined from the fluorescence attribute differences between the ith target data point and multiple neighboring data points of the ith target data point in the nth iteration, represents the minimum fluorescence attribute difference determined from the fluorescence attribute differences between the ith target data point and multiple neighboring data points of the ith target data point in the nth iteration, It is a preset parameter, which represents the division threshold and takes a value between 0 and 1.

[0098] According to an embodiment of the present invention, since the difference in fluorescence properties between the light source area and the background area is relatively large, and the difference in fluorescence properties between the inside of the light source area and the inside of the background area is relatively small, the area to which the target data point belongs can be determined by the difference in fluorescence properties between the target data point and its adjacent data points, thereby facilitating the calculation method of the correlation value between the target data point and the remaining target data points.

[0099] According to an embodiment of the present invention, determining correlation values ​​between the multiple target data points based on the fluorescence properties of the multiple target data points and the neighboring data points of the multiple target data points to obtain a correlation matrix may include the following operations.

[0100] For each target data point, a data point to be calculated is determined from multiple target data points; when the data point to be calculated is determined to be a neighbor data point of the target data point, a correlation calculation result is determined based on the correlation weight corresponding to the fluorescent area and the target distance between the target data point and the data point to be calculated, and the correlation calculation result is used as the target correlation value between the target data point and the data point to be calculated; when the data point to be calculated is determined to be a target data point, the target correlation value is determined to be a first preset value; when the data point to be calculated is determined to be a data point other than the target data point or the neighbor data point, the target correlation value is determined to be a second preset value; based on the target correlation values ​​between each of the multiple target data points, a correlation matrix is ​​determined.

[0101] According to an embodiment of the present invention, the correlation matrix may be determined as shown in the following formula (16) to formula (17).

[0102] ; (16)

[0103] in,

[0104] ; (17)

[0105] in, is the Euclidean distance between the target data point and the data point to be calculated, are the correlation weights corresponding to the high variance region and the low variance region respectively.

[0106] According to an embodiment of the present invention, the correlation weights corresponding to the respective fluorescent regions are preset, and the light source region and the background region are low variance regions, having smaller light intensity variance and thus having larger correlation weights. Calculate the correlation value to make the reconstruction result in the corresponding area smoother; the high variance area has a larger light intensity variance as the edge area, so it has a smaller correlation weight. The correlation value is calculated, and the regularization weight calculated in this way is larger, which will make the edge reconstruction clearer and non-smooth.

[0107] According to an embodiment of the present invention, based on multiple pixel points and multiple initial data points, fusing the two-dimensional fluorescence image and the three-dimensional point cloud data to obtain initial three-dimensional fluorescence data may include the following operations.

[0108] An image group having the same shooting angle as the two-dimensional fluorescence image is obtained, wherein the image group includes an initial grayscale image and an initial color image; based on the initial grayscale image and the initial color image, the two-dimensional fluorescence image and the three-dimensional point cloud data are registered, and the matching relationship between multiple pixel points and each of the multiple initial data points is determined; based on the matching relationship and the fluorescence properties of each of the multiple pixel points, the multiple initial data points are assigned values ​​to obtain target data points with fluorescence properties; based on the multiple target data points and the fluorescence properties of each of the multiple target data points, the initial three-dimensional fluorescence data is determined. According to an embodiment of the present invention, the initial grayscale image and the two-dimensional fluorescence image have the same first image range; the initial color image includes a second image range that is larger than the first image range.

[0109] According to an embodiment of the present invention, after the initial grayscale image and the initial color image are acquired, pre-processing operations such as denoising, enhancement and rotation may be performed on the two images respectively.

[0110] According to an embodiment of the present invention, the three-dimensional point cloud data may be preprocessed before registering the two-dimensional fluorescence image with the three-dimensional point cloud data. The preprocessing process may be specifically as shown in the following manner.

[0111] The initial three-dimensional point cloud data can be read and a scatter plot can be drawn, and the three-dimensional coordinates of the multimodal visible registration points can be recorded; the area of ​​interest is cropped for the point cloud, and the area of ​​interest is the area including the lesion area. The target clustering method, such as k-means, is used for clustering sparseness to reduce the amount of calculation; the point cloud is translated to the first quadrant, the plane reference point is selected, the reference plane and its normal are drawn, and the rotation angle and direction required to rotate the normal to the positive direction of the z-axis are calculated. The point cloud is rotated according to this rotation angle and direction until the reference plane is parallel to the xy plane; independent empty points are removed, and the xy axis coordinate data of the point cloud is closed multiple times to remove holes, and the horizontal and vertical coordinates of the points that need to be filled are recorded. A certain depth is selected downward, and the original point cloud contour is filled with a solid three-dimensional matrix. Each layer of the matrix in the z direction is opened to remove independent points, thereby obtaining three-dimensional point cloud data. The recorded registration marker point coordinates are transformed according to the same steps as the point cloud processing;

[0112] According to an embodiment of the present invention, the matching relationship between the multiple pixel points and the multiple initial data points may include the initial data points that the multiple pixel points respectively match, and one pixel point may match one or more data points.

[0113] According to an embodiment of the present invention, the three-dimensional point cloud data can be first registered with the initial color image, and the registered initial color image can be used to register with the initial grayscale image, and finally the registered initial grayscale image can be used to register the two-dimensional fluorescence image, thereby achieving registration between the three-dimensional point cloud data and the two-dimensional fluorescence image.

[0114] According to an embodiment of the present invention, the multiple initial data points include multiple first registration points; the initial color image includes multiple second registration points; the grayscale image includes multiple third registration points; based on the initial grayscale image and the initial color image, the two-dimensional fluorescence image and the three-dimensional point cloud data are registered, and the matching relationship between the multiple pixel points and each of the multiple initial data points is determined, which may include the following operations.

[0115] The three-dimensional point cloud data is gridded to obtain three-dimensional grid data; based on multiple first registration points and multiple second registration points, the three-dimensional grid data and the initial color image are registered to obtain a registered target color image; based on multiple second registration points and multiple third registration points, the target color image and the grayscale image are registered to obtain a registered target grayscale image; the target grayscale image and the two-dimensional fluorescence image are registered to obtain a registered two-dimensional fluorescence image; based on the mapping relationship between multiple pixel points included in the registered two-dimensional fluorescence image and multiple grids included in the three-dimensional grid data, the grids corresponding to the multiple pixel points are determined, wherein the vertices of the grids are the initial data points; based on the vertices of the grids corresponding to the multiple pixel points, the matching relationship between the multiple pixel points and the multiple initial data points is determined.

[0116] According to the embodiment of the present invention, the means for gridding the three-dimensional point cloud data is not limited, and it can be achieved by using a gridding tool.

[0117] According to an embodiment of the present invention, the first registration point may be a point with obvious features in the three-dimensional point cloud data, such as a point on an obvious tissue fold, a point on a special instrument, a point on a circular retractor, and the like.

[0118] According to an embodiment of the present invention, the multiple second registration points include second registration points that match the multiple first registration points respectively, and the multiple third registration points are similar, including third registration points that match the multiple second registration points respectively. When the first registration point is a point on an obvious tissue fold, the second registration point that matches the first registration point is also a point on an obvious tissue fold, and the same applies to the third registration point.

[0119] According to an embodiment of the present invention, the range of tissues or organs included in the three-dimensional point cloud data may be the largest among the several images, such as including point cloud data within the range of the annular retractor. The initial color image may be the same as or slightly smaller than the image range of the three-dimensional point cloud data, and the initial grayscale image and the two-dimensional fluorescence image may include the same image range, which may be smaller than the initial color image and mainly include images of the lesion area, thereby improving the accuracy of the fluorescence.

[0120] According to an embodiment of the present invention, since the registered two-dimensional fluorescence image and the three-dimensional grid data are aligned, the mapping relationship between the multiple pixel points included in the registered two-dimensional fluorescence image and the multiple grids included in the three-dimensional grid data can be determined by the alignment relationship between the pixel points and the grids.

[0121] According to the embodiment of the present invention, the shape of the mesh is not limited, and meshes of different shapes may have different numbers of vertices.

[0122] According to an embodiment of the present invention, by determining the mapping relationship between the grid and the pixel points, and using the positioning of the grid as the initial data point to match the pixel points, the matching accuracy between the pixel points and the initial data points can be improved.

[0123] According to an embodiment of the present invention, based on a plurality of first registration points and a plurality of second registration points, the three-dimensional mesh data and the initial color image are registered to obtain a registered target color image, which may further include the following operations.

[0124] Determine a first angle registration point group and a first size registration point group from a plurality of first registration points, wherein the first angle registration point group includes at least two angle registration points, the at least two angle registration points are used to form a straight line with an angle greater than a preset angle, and the first size registration point group includes at least two size registration points, and the distance between the at least two size registration points is greater than a preset distance; determine a second angle registration point group matching the first angle registration point group and a second size registration point group matching the first size registration point group from a plurality of second registration points; based on a first angle value determined by the first angle registration point group and a second angle value determined by the second angle registration point group Angle value, determine the angle difference between the three-dimensional grid data and the initial color image; determine the size difference between the three-dimensional grid data and the initial color image based on the first distance value determined by the first size registration point group and the second distance value determined by the second size registration point group; adjust the initial color image based on the angle difference and the size difference to obtain an intermediate color image; determine the translation distance based on the position attributes of each of the multiple first registration points and the position attributes of the multiple fourth registration points in the intermediate color image, wherein the multiple fourth registration points correspond one-to-one to the multiple second registration points; translate the intermediate color image based on the translation distance to obtain a target color image.

[0125] According to the embodiment of the present invention, there is no limitation on the preset angle and the preset length, and they can be selected according to actual conditions.

[0126] According to an embodiment of the present invention, the first straight line registration point group and the second straight line registration point group match each other, and there is no limit on the number of the two groups, and multiple groups that match each other can be selected.

[0127] According to an embodiment of the present invention, the angle between the straight line formed by the first straight line registration point group and the xy plane can be calculated to obtain a first angle value, the angle between the straight line formed by the second straight line registration point group and the xy plane can be calculated to obtain a second angle value, the angle difference between the first angle value and the second angle value can be calculated, and the initial color image can be rotated by the angle of the angle difference to achieve the same orientation as the three-dimensional grid data.

[0128] According to an embodiment of the present invention, the first size registration point group and the second size registration point group match each other, and there is no limitation on the number of the first size registration point group and a plurality of groups that match each other can be selected.

[0129] According to an embodiment of the present invention, a first distance value between at least two size registration points in a first size registration point group can be calculated; a second distance value between at least two size registration points in a second size registration point group can be calculated; a distance difference between the first distance value and the second distance value can be calculated, and the distance difference can be used as the size difference between an initial color image and three-dimensional mesh data; the initial color image can be scaled proportionally using the size difference so that the initial color image is consistent in size with the three-dimensional mesh data.

[0130] According to an embodiment of the present invention, in the case where both the first size registration point group and the second size registration point group are multiple groups, the distance difference obtained for each group can be determined, and the distance differences of the multiple groups can be added and averaged to determine the size difference.

[0131] According to the embodiment of the present invention, since the intermediate color image is adjusted, the position attribute of the second registration point is changed, that is, the fourth registration point is the second registration point after the position attribute is changed.

[0132] According to an embodiment of the present invention, the translation distance between the three-dimensional mesh data and the intermediate color image can be determined based on the coordinate values ​​of each of the multiple first registration points and the coordinate values ​​of the fourth registration points corresponding to each of the multiple first registration points, and the intermediate color image can be moved based on the translation distance to achieve alignment of the intermediate color image with the three-dimensional mesh data.

[0133] According to an embodiment of the present invention, in the process of registering the initial grayscale image and the target color image, the linear registration point groups of the initial grayscale image and the target color image can be selected, and the angle difference of the straight line formed by the linear registration point groups can be calculated, so as to rotate the initial grayscale image and the two-dimensional fluorescent image, so that the initial grayscale image and the two-dimensional fluorescent image are in the same direction as the target color image. In addition, the size registration point groups of the initial grayscale image and the target color image can be selected, and the size difference determined by the size registration point groups can be used to scale the initial grayscale image and the two-dimensional fluorescent image to be consistent with the size of the target color image, and the translation distance between the initial grayscale image and the target color image can be calculated to align the initial grayscale image with the target color image, and the two-dimensional fluorescent image can be adjusted according to the same adjustment process as the initial grayscale image, so that the two-dimensional fluorescent image can be aligned with the three-dimensional grid data.

[0134] According to an embodiment of the present invention, since the initial grayscale image and the two-dimensional fluorescence image are images collected under the same optical path, their viewing angles and ranges are consistent, but the two-dimensional fluorescence image cannot determine the coordinates of the registration points. Therefore, through the third registration point of the initial grayscale image, the same steps as those for registering the initial grayscale image can be used to achieve registration of the two-dimensional fluorescence image with the three-dimensional point cloud data.

[0135] According to the embodiment of the present invention, by registering the two-dimensional fluorescence image with the three-dimensional point cloud data through the above-mentioned multiple adjustment angles, it is possible to consider the registration information from multiple aspects and achieve more accurate registration processing.

[0136] According to an embodiment of the present invention, assigning values ​​to a plurality of initial data points based on a matching relationship and respective fluorescence properties of a plurality of pixel points to obtain target data points with fluorescence properties may include the following operations.

[0137] For each pixel point, based on the matching relationship, determine the initial data point that matches the pixel point; when it is determined that there are M initial data points that match the pixel point, divide the fluorescence attribute of the pixel point by M to obtain the assigned fluorescence attribute assigned to each initial data point corresponding to the pixel point, where M is a positive integer greater than or equal to 1; for each initial data point, when the initial data point matches multiple pixel points, superimpose the assigned fluorescence attributes assigned to the multiple pixel points corresponding to the initial data point to obtain a target data point with fluorescence attributes. According to an embodiment of the present invention, the initial three-dimensional fluorescence data can be three-dimensional grid data with grid vertices having fluorescence attributes.

[0138] According to an embodiment of the present invention, before fusion, the fluorescence attributes of each of the multiple pixel points can be screened out. Specifically, the fluorescence attributes with fluorescence attributes less than the target fluorescence value can be deleted or replaced with a preset fluorescence value. The target fluorescence value is not limited and can be 5, and the preset fluorescence value can be 0.

[0139] According to an embodiment of the present invention, the initial data point corresponding to each pixel point may be determined, and the pixel point may be divided by M to be allocated to the initial data point corresponding to the pixel point.

[0140] According to an embodiment of the present invention, when the initial data point corresponds to a plurality of pixel points, the initial fluorescence properties respectively assigned to the plurality of pixel points may be superimposed to obtain the fluorescence property of the initial data point.

[0141] According to the embodiment of the present invention, there is no limitation on the superposition processing, and it may be processing methods such as accumulation and accumulation averaging.

[0142] Figure 2 A flow chart of a method for generating three-dimensional fluorescence data according to another embodiment of the present invention is shown.

[0143] like Figure 2 As shown, based on a preset instruction sequence, a two-dimensional fluorescence image 201, a three-dimensional point cloud data 202, an initial grayscale image 203 and an initial color image 204 are sequentially acquired, and the horizontal gyroscope readings on the respective acquisition devices of the two-dimensional fluorescence image, the three-dimensional point cloud data, the initial grayscale image and the initial color image are acquired.

[0144] According to the embodiment of the present invention, the two-dimensional fluorescence image 201 is subjected to operations such as flipping, rotation, denoising, and smoothing filtering to obtain a preprocessed two-dimensional fluorescence image 205. The three-dimensional point cloud data 202 is subjected to operations such as cropping, sparseness, flipping, hole filling, and selection of registration points to obtain preprocessed three-dimensional point cloud data 206, and the preprocessed three-dimensional point cloud data is subjected to structure completion and mesh construction to obtain three-dimensional mesh data 207. The initial grayscale image 203 and the initial color image 204 are subjected to operations such as flipping, cropping, and selection of registration points to obtain a preprocessed initial grayscale image 208 and a preprocessed initial color image 209.

[0145] According to an embodiment of the present invention, the preprocessed two-dimensional fluorescence image 205, the three-dimensional grid data 207, the preprocessed initial grayscale image 208 and the preprocessed initial color image 209 are registered, and the three-dimensional grid data 207 and the preprocessed two-dimensional fluorescence image 205 are fused to obtain the initial three-dimensional fluorescence data 210. The objective function is constructed by the initial three-dimensional fluorescence data 210, and the objective function is iteratively solved to obtain the reconstructed initial three-dimensional fluorescence data, i.e., the target three-dimensional fluorescence data 211.

[0146] According to an embodiment of the present invention, the following operations may be performed based on a preset instruction sequence to obtain the above-mentioned images and data.

[0147] The white light illumination device is turned on, and the white light camera in the same optical channel as the fluorescent camera is controlled to be aimed at the lesion area, the focal length is adjusted, an initial grayscale image 203 is captured, and the horizontal gyroscope reading on the white light camera is obtained.

[0148] The white light illumination device is turned off, the laser is controlled to be turned on, the laser is emitted, and the laser beam expander is aimed at the surgical area. The laser illumination area can be consistent with the expansion device of the tissue or organ such as the annular retractor, and the fluorescent camera is controlled to collect the two-dimensional fluorescent image 201, and the horizontal gyroscope reading on the fluorescent camera is obtained.

[0149] The laser is turned off, the lighting device is turned on, and the initial color image 204 in a larger range under the viewing angle is captured by using a white light camera at the same shooting angle, and the reading of the horizontal gyroscope on the white light camera is obtained.

[0150] The three-dimensional point cloud data 202 within the range of the annular retractor at the same shooting angle is collected by controlling the three-dimensional data collection module.

[0151] According to the embodiment of the present invention, the above-mentioned collection order is only illustrative, and the collection can be performed in different orders according to actual needs.

[0152] According to an embodiment of the present invention, the two-dimensional fluorescence image 201, the three-dimensional point cloud data 202, the initial grayscale image 203 and the initial color image 204 can be rotated to the same angle by using the horizontal gyroscope indications on their respective acquisition devices.

[0153] Figure 3 A schematic diagram of the structure of a three-dimensional fluorescence data generation system according to an embodiment of the present invention is shown; Figure 4 A schematic structural diagram of a three-dimensional fluorescence data generating system according to an embodiment of the present invention is shown.

[0154] like Figure 3 and Figure 4As shown, the three-dimensional fluorescence data generation system includes: a trolley packaging module 310, a fluorescence image acquisition module 320, a three-dimensional data acquisition module 330, an image group acquisition module 340, a processing module 350 and an image visualization module 360.

[0155] The fluorescence image acquisition module 320 is used to acquire a two-dimensional fluorescence image of the lesion area.

[0156] The three-dimensional data acquisition module 330 is used to acquire three-dimensional point cloud data of the lesion area based on the same acquisition angle as the fluorescence image acquisition module 320 .

[0157] The processing module 350 is used to obtain a two-dimensional fluorescence image from the fluorescence image acquisition module 320, and to obtain three-dimensional point cloud data from the three-dimensional data acquisition module 330, wherein the two-dimensional fluorescence image includes a plurality of pixel points with fluorescence properties, and the plurality of pixel points are used to determine the initial two-dimensional shape of the lesion area, and the three-dimensional point cloud data includes a plurality of initial data points for characterizing the three-dimensional contour of the lesion area; the processing module is also used to fuse the two-dimensional fluorescence image and the three-dimensional point cloud data based on the plurality of pixel points and the plurality of initial data points to obtain initial three-dimensional fluorescence data, wherein the initial three-dimensional fluorescence data includes a plurality of target data points with fluorescence properties, and the plurality of target data points are used to determine the three-dimensional shape representation of the lesion area; construct an objective function based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points, and preset parameters; solve the objective function to obtain target three-dimensional fluorescence data, wherein the target three-dimensional fluorescence data includes the updated three-dimensional shape representation of the lesion area.

[0158] According to the embodiment of the present invention, the image group acquisition module 340 includes: an acquisition submodule and a rotation angle determination submodule 343 .

[0159] According to the embodiment of the present invention, the processing module 350 may be used to execute the three-dimensional fluorescence data generating method according to the embodiment of the present invention.

[0160] The acquisition submodule is used to obtain an initial grayscale image and an initial color image having the same shooting angle as the two-dimensional fluorescence image.

[0161] The rotation angle determination submodule 343 is used to record the rotation angle of the acquisition submodule relative to the ground.

[0162] According to an embodiment of the present invention, the two-dimensional fluorescence image may be a near-infrared second-zone fluorescence image, and the fluorescence camera may be a near-infrared second-zone fluorescence camera.

[0163] According to an embodiment of the present invention, the fluorescence image acquisition module 320 can be used for the excitation and acquisition of near-infrared zone II fluorescence images, and can obtain a two-dimensional fluorescence image of the lesion area during surgery; the module may include a laser 321, a water cooler 322, a filter 323, a fluorescence camera 324, a laser optical fiber 325, a switching beam expander transposition, a zoom lens, etc.

[0164] According to an embodiment of the present invention, a laser 321 is used to excite excitation light with adjustable wavelength and power; a laser optical fiber 325 is used to conduct the excitation light; a switching beam expander transposition is used to connect the laser and the optical fiber to diffuse the excitation light everywhere in the optical fiber into a uniform light beam; a fluorescence camera 324 is used to collect near-two-dimensional fluorescence images; a zoom lens is used to adjust the imaging area and focus; and a filter 323 is used to filter stray light to obtain near-infrared zone II fluorescence in a specified spectral band.

[0165] According to an embodiment of the present invention, the image group acquisition module 340 can be used for illumination and acquisition of white light signals, and can obtain an initial grayscale image and an initial color image of the intraoperative lesion area, wherein the initial grayscale image is consistent with the viewing angle optical path of the two-dimensional fluorescence image, and the initial color image is used to obtain a complete feature image of the surgical area to facilitate alignment with the three-dimensional point cloud data; the module may include: a white light illumination device 341, a white light camera 342, a lighting camera, a rotation angle determination submodule 343, etc.

[0166] According to an embodiment of the present invention, the white light lighting device 341 is used to provide lighting for the white light camera to shoot the operating area; the white light camera 343 is used to collect an image group of the operating area; and the rotation angle determination submodule 343 is used to obtain the flip angle of the white light camera relative to the ground.

[0167] According to an embodiment of the present invention, a three-dimensional data acquisition module 330 is used to obtain a surface contour point cloud of the lesion area during surgery, construct a homogeneous three-dimensional structural model, replace the traditional preoperative image, and reversely reconstruct the light source with the two-dimensional fluorescence image registration, as the basis of the three-dimensional fluorescence surgery navigation model; the module may include a three-dimensional signal acquisition device 331, a rotation angle determination submodule 332, a projection device 333, a registration marker determination submodule 334, a main control unit, a structured light signal computer processing module, a packaging shell, etc.

[0168] According to an embodiment of the present invention, by Figure 4 A lesion area 370 and a registration marker 380 may also be determined. The registration marker 380 may be a marker used for registration, such as a ring retractor.

[0169] According to the embodiment of the present invention, the above-mentioned rotation angle determination submodules can all be implemented by a horizontal gyroscope.

[0170] According to an embodiment of the present invention, a projection device 333 is used to project a specific structured light pattern onto the surgical area to be measured; a three-dimensional signal acquisition device 331 is used to acquire an image of the surface of the surgical area to be measured after being irradiated with structured light; a registration marker determination submodule 334 is used to record the common visible marker points of multi-modal imaging of the surgical area; a main control unit is used to control the projection mode, light frequency and other related parameters of the projector to ensure the coordinated work of the system; a structured light signal computer processing module is used to receive and process the structured light data transmitted by the acquisition system to obtain point cloud data.

[0171] According to an embodiment of the present invention, the trolley packaging module 310 is used to support and package the entire system components, provide component connection interfaces and a mechanical control platform, and move operations during surgery; the module may include an external packaging submodule 311, a rotation angle determination submodule 312, a camera fixing device 313, a robotic arm control device 314, a trolley packaging shell 315, a trolley frame, etc.

[0172] According to an embodiment of the present invention, the trolley frame is used to support system components; the camera fixing device 313 is used to fix the fluorescence camera; the robotic arm control device 314 is used to control the position of the fluorescence acquisition camera and the signal acquisition direction; the rotation angle determination submodule 312 is used to obtain the rotation angle of the fluorescence camera relative to the ground to provide multimodal alignment information; the external packaging submodule 311 is used to package the entire system components to facilitate mobile operation.

[0173] According to an embodiment of the present invention, the processing module 350 may include a central control submodule 351 , a registration and fusion submodule 352 , and a reconstruction submodule 353 .

[0174] According to an embodiment of the present invention, the central control submodule 351 can be used to control the coordinated and orderly implementation of the fluorescence image acquisition module 320, the three-dimensional data acquisition module 330 and the image group acquisition module 340 in the system; the submodule may include a fluorescence image acquisition module control unit, a three-dimensional data acquisition module control unit, an image group acquisition module control unit, etc.

[0175] According to an embodiment of the present invention, the registration and fusion submodule 352 can be used to register and fuse the two-dimensional fluorescence signal, the initial color image, the initial grayscale image and the three-dimensional point cloud data under the same reference state, integrate the information, take the initial color image and the initial grayscale image as the reference, map the fluorescence spot of the two-dimensional fluorescence image to the three-dimensional grid constructed by the three-dimensional point cloud data, for reconstructing the three-dimensional shape of the lesion area; the submodule may include a fluorescence image processing unit, a three-dimensional point cloud data processing unit, a color image processing unit, and a tri-modal registration unit.

[0176] According to an embodiment of the present invention, a fluorescence image processing unit is used for basic filtering processing of two-dimensional fluorescence images, and flipping the image to be parallel to the ground according to the data provided by the rotation angle determination submodule placed near the fluorescence camera; a structured optical image processing unit is used for clustering and sparseness of structured light generated point cloud data, center area cropping, morphological closing operation hole filling, smoothing, depth filling, surgical area point cloud contour surface generation, finite element discretization grid generation, and three-dimensional coordinate reading of registration mark points; a color image processing unit is used for color image cropping processing, flipping the image to be parallel to the ground according to the data collected by the rotation angle determination submodule, and reading the pixel coordinates of the registration mark points; a tri-modal registration module is used to align the two-dimensional fluorescence image, the initial color image and the discretized grid, i.e., the three-dimensional grid data according to the relative coordinates of the registration points, and to map the fluorescence attribute information to the grid surface.

[0177] According to an embodiment of the present invention, the reconstruction submodule 353 can be used to reconstruct and solve the shape and position of the light source in the three-dimensional grid data, so that the discrete grid of the mapping fluorescence attribute obtained by the registration and fusion module can be used to accurately locate the three-dimensional position and shape of the tumor or other lesion area. This submodule includes a reconstruction calculation unit.

[0178] According to an embodiment of the present invention, the image visualization module 360 ​​is used to visualize the target three-dimensional data, and it may be a display.

[0179] According to the embodiment of the present invention, the specific number of the modules, sub-modules, units and specific devices mentioned above is not limited and can be one or more.

[0180] According to an embodiment of the present invention, a central control submodule is used to control a white light camera, a fluorescent camera and a signal acquisition device to perform signal acquisition in sequence, the three-modal information is aligned and fused in three-dimensional space through a registration and fusion module, and the position and shape of the fluorescent light source in the three-dimensional structure are solved through a reconstruction and visualization submodule, thereby comprehensively utilizing the information of the three modalities, breaking through the limitations of preoperative image structure changes, and realizing three-dimensional fluorescent surgical navigation in intraoperative scenarios.

[0181] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0183] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for generating three-dimensional fluorescence data, characterized in that: The method comprises: Acquire a two-dimensional fluorescence image and three-dimensional point cloud data for the same lesion area, wherein the two-dimensional fluorescence image includes a plurality of pixel points having fluorescence properties, the plurality of pixel points are used to determine an initial two-dimensional shape of the lesion area, and the three-dimensional point cloud data includes a plurality of initial data points used to characterize a three-dimensional contour of the lesion area; Based on the plurality of pixel points and the plurality of initial data points, the two-dimensional fluorescence image and the three-dimensional point cloud data are fused to obtain initial three-dimensional fluorescence data, wherein the initial three-dimensional fluorescence data includes a plurality of target data points having the fluorescence attribute, and the plurality of target data points are used to determine a three-dimensional shape representation of the lesion area; constructing an objective function based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points, and preset parameters; Solving the objective function to obtain target three-dimensional fluorescence data, wherein the target three-dimensional fluorescence data includes an updated three-dimensional shape representation of the lesion area; The fusing the two-dimensional fluorescence image and the three-dimensional point cloud data based on the plurality of pixel points and the plurality of initial data points to obtain initial three-dimensional fluorescence data comprises: Based on an initial grayscale image and an initial color image having the same shooting angle as the two-dimensional fluorescent image, registering the two-dimensional fluorescent image with the three-dimensional point cloud data, and determining a matching relationship between each of the plurality of pixel points and the plurality of initial data points; Based on the matching relationship and the fluorescence properties of the plurality of pixel points, assigning values ​​to the plurality of initial data points to obtain target data points having the fluorescence properties; The initial three-dimensional fluorescence data is determined based on the plurality of target data points and the fluorescence properties of each of the plurality of target data points.

2. The method according to claim 1, characterized in that The constructing of the objective function based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points and preset parameters comprises: Determine the correlation values ​​between the plurality of target data points based on the fluorescence properties of the plurality of target data points and the neighboring data points of the plurality of target data points to obtain a correlation matrix; An objective function is constructed based on the correlation matrix, the fluorescence properties of each of the plurality of target data points, and the preset parameters.

3. The method according to claim 2, characterized in that The step of determining the correlation values ​​between the plurality of target data points based on the fluorescence properties of the plurality of target data points and the neighboring data points of the plurality of target data points to obtain a correlation matrix further includes: For each of the target data points, Determining a data point to be calculated from the plurality of target data points; In the case where it is determined that the data point to be calculated is a neighbor data point of the target data point, based on the fluorescence property of the target data point, the fluorescence property of the data point to be calculated, the correlation weight determined by the minimum fluorescence property difference and the maximum fluorescence property difference, and the target distance between the target data point and the data point to be calculated, a correlation calculation result is determined, and the correlation calculation result is used as a target correlation value between the target data point and the data point to be calculated; In the case where it is determined that the data point to be calculated is the target data point, determining the target correlation value to be a first preset value; In the case where it is determined that the data point to be calculated is a data point other than the target data point or the neighbor data point, determining the target correlation value to be a second preset value; Based on the target correlation values ​​between each of the plurality of target data points, obtaining the correlation matrix; Among them, the target distance is determined by the position attribute of the data point to be calculated and the position attribute of the target data point, the minimum fluorescence attribute difference is the smallest of the fluorescence attribute differences between the target data point and at least one neighboring data point of the target data point, and the maximum fluorescence attribute difference is the largest of the fluorescence attribute differences between the target data point and at least one neighboring data point of the target data point.

4. The method according to claim 2, characterized in that: Each of the target data points has at least one neighbor data point; Wherein, before determining the correlation values ​​between the plurality of target data points based on the fluorescence properties of the plurality of target data points and the neighboring data points of the plurality of target data points to obtain the correlation matrix, the method further comprises: For each of the target data points, Determine a mean value of the fluorescence attribute difference based on the fluorescence attribute of each of at least one neighboring data point of the target data point and the fluorescence attribute of the target data point; Based on the fluorescence property difference mean, the minimum fluorescence property difference and the maximum fluorescence property difference, the fluorescence region to which the target data point belongs is determined.

5. The method according to claim 4, characterized in that Determining the correlation values ​​between the plurality of target data points based on the fluorescence properties of the plurality of target data points and the neighboring data points of the plurality of target data points to obtain the correlation matrix comprises: For each of the target data points, Determining a data point to be calculated from the plurality of target data points; In the case where it is determined that the data point to be calculated is a neighbor data point of the target data point, determining a correlation calculation result based on a correlation weight corresponding to the fluorescent area and a target distance between the target data point and the data point to be calculated, and using the correlation calculation result as a target correlation value between the target data point and the data point to be calculated; In the case where it is determined that the data point to be calculated is the target data point, determining the target correlation value to be a first preset value; In the case where it is determined that the data point to be calculated is a data point other than the target data point or the neighbor data point, determining the target correlation value to be a second preset value; The correlation matrix is ​​determined based on target correlation values ​​between respective ones of the plurality of target data points.

6. The method according to claim 1, characterized in that The plurality of initial data points include a plurality of first registration points; the initial color image includes a plurality of second registration points; and the grayscale image includes a plurality of third registration points; The registering process of the two-dimensional fluorescence image and the three-dimensional point cloud data based on the initial grayscale image and the initial color image, and determining the matching relationship between the plurality of pixel points and the plurality of initial data points, comprises: Meshing the three-dimensional point cloud data to obtain three-dimensional mesh data; Based on the plurality of the first registration points and the plurality of the second registration points, registering the three-dimensional grid data and the initial color image to obtain a registered target color image; Based on the plurality of second registration points and the plurality of third registration points, registering the target color image and the grayscale image to obtain a registered target grayscale image; Registering the target grayscale image and the two-dimensional fluorescence image to obtain a registered two-dimensional fluorescence image; Determine the grids corresponding to the plurality of pixel points based on the mapping relationship between the plurality of pixel points included in the registered two-dimensional fluorescence image and the plurality of grids included in the three-dimensional grid data, wherein the vertices of the grids are the initial data points; Based on the vertices of the grids corresponding to the plurality of pixel points, the matching relationships between the plurality of pixel points and the plurality of initial data points are determined.

7. The method according to claim 6, characterized in that The assigning values ​​to the plurality of initial data points based on the matching relationship and the fluorescence properties of the plurality of pixel points to obtain target data points having the fluorescence properties includes: For each of the pixel points, based on the matching relationship, determining an initial data point that matches the pixel point; When it is determined that there are M initial data points matching the pixel point, dividing the fluorescence attribute of the pixel point by M to obtain an assigned fluorescence attribute assigned to each of the initial data points corresponding to the pixel point, wherein M is a positive integer greater than or equal to 1; For each of the initial data points, when the initial data point matches a plurality of the pixel points, the assigned fluorescence attributes respectively assigned to the plurality of the pixel points corresponding to the initial data point are superimposed to obtain a target data point having the fluorescence attribute.

8. The method according to claim 6, characterized in that The step of registering the three-dimensional grid data and the initial color image based on the plurality of first registration points and the plurality of second registration points to obtain a registered target color image further includes: Determine a first angle registration point group and a first size registration point group from the plurality of first registration points, wherein the first angle registration point group includes at least two angle registration points, and the at least two angle registration points are used to form a straight line with an angle greater than a preset angle, and the first size registration point group includes at least two size registration points, and the distance between the at least two size registration points is greater than a preset distance; Determine, from the plurality of second registration points, a second angular registration point group that matches the first angular registration point group and a second size registration point group that matches the first size registration point group; Determine an angle difference between the three-dimensional mesh data and the initial color image based on a first angle value determined by the first angle registration point group and a second angle value determined by the second angle registration point group; Determine a size difference between the three-dimensional mesh data and the initial color image based on a first distance value determined by the first size registration point group and a second distance value determined by the second size registration point group; Adjusting the initial color image based on the angle difference and the size difference to obtain an intermediate color image; Determining a translation distance based on positional attributes of each of the plurality of first registration points and positional attributes of a plurality of fourth registration points in the intermediate color image, wherein the plurality of fourth registration points correspond one-to-one to the plurality of second registration points; The intermediate color image is translated based on the translation distance to obtain the target color image.

9. A three-dimensional fluorescence data generation system, characterized in that: The system comprises: A fluorescence image acquisition module, used for acquiring a two-dimensional fluorescence image of the lesion area; A three-dimensional data acquisition module, used to acquire three-dimensional point cloud data of the lesion area based on the same acquisition angle as the fluorescence image acquisition module; A processing module, used to obtain a two-dimensional fluorescence image from the fluorescence image acquisition module, and to obtain three-dimensional point cloud data from the three-dimensional data acquisition module, wherein the two-dimensional fluorescence image includes a plurality of pixel points with fluorescence properties, and the plurality of pixel points are used to determine the initial two-dimensional shape of the lesion area, and the three-dimensional point cloud data includes a plurality of initial data points for characterizing the three-dimensional contour of the lesion area; the processing module is also used to fuse the two-dimensional fluorescence image and the three-dimensional point cloud data based on the plurality of pixel points and the plurality of initial data points to obtain initial three-dimensional fluorescence data, wherein the initial three-dimensional fluorescence data includes a plurality of target data points with the fluorescence properties, and the plurality of target data points are used to determine the three-dimensional shape representation of the lesion area; construct an objective function based on the fluorescence properties of each of the plurality of target data points, the neighboring data points of each of the plurality of target data points, and preset parameters; solve the objective function to obtain target three-dimensional fluorescence data, wherein the target three-dimensional fluorescence data includes the updated three-dimensional shape representation of the lesion area; The method of fusing the two-dimensional fluorescence image and the three-dimensional point cloud data based on the plurality of pixel points and the plurality of initial data points to obtain the initial three-dimensional fluorescence data comprises: registering the two-dimensional fluorescence image and the three-dimensional point cloud data based on an initial grayscale image and an initial color image having the same shooting angle as the two-dimensional fluorescence image, and determining a matching relationship between each of the plurality of pixel points and each of the plurality of initial data points; assigning values ​​to the plurality of initial data points based on the matching relationship and the fluorescence properties of each of the plurality of pixel points to obtain target data points having the fluorescence properties; and determining the initial three-dimensional fluorescence data based on the plurality of target data points and the fluorescence properties of each of the plurality of target data points.

Images