Method for computing depth of objects in two-dimensional images using deep neural network cluster

Abstract

The application discloses a method for calculating the depth of a two-dimensional image object by using a deep neural network cluster, comprising the following steps: S1, inputting a two-dimensional image of a target object, identifying and labeling a plurality of feature points in the two-dimensional image, and taking each feature point as the center of a local image with a size of (m, n) pixels; S2, using a well-learned deep learning network model to perform height prediction on the center point of each local image, wherein the deep learning network model is fused with local image feature learning and height learning; and S3, planning a Luneburg chart with each height point as the vertex. According to the application, the three-dimensional structure of a target object in an image can be measured by inputting a two-dimensional image, and compared with the prior art, the application does not need to take multiple-direction photos of the target object, and the three-dimensional structure and volume of the target object can be calculated by analyzing a plane photo of the target object, so that the application can be more widely applied to the three-dimensional measurement of pathological sections and other objects with only a fixed perspective.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and in particular to a method for calculating the depth of objects in two-dimensional images using a deep neural network cluster. Background Technology

[0002] A two-dimensional image is an image represented on a plane, consisting of an array of pixels arranged in rows and columns. Each pixel has a specific position and color value; different images can be formed by changing the position and color of these pixels. Two-dimensional images only have length and width dimensions and lack depth information, therefore, they cannot inherently provide a correct three-dimensional structure of an object.

[0003] Currently, to measure the three-dimensional structure of an object using two-dimensional images, one or more digital cameras are needed to capture images of the target object from all directions. All the acquired two-dimensional images are then connected by calculating feature points, and combined with the angles and distances from which the two-dimensional images were captured, to obtain the three-dimensional structure of the target object. Methods for calculating feature points include SIFT and SURF. Furthermore, with the development of infrared imaging technology, stereoscopic imaging combining infrared and optical images is also widely used in the stereoscopic measurement of terrain and buildings.

[0004] The aforementioned existing technologies also have the following problems: because they require multi-directional imaging of the target object, they cannot be applied to stereoscopic measurements with only a fixed perspective, such as pathological slides, and especially cannot perform stereoscopic analysis on planar photographs of objects.

[0005] Therefore, it is necessary to provide a method for calculating the depth of objects in two-dimensional images using deep neural network clusters to solve the aforementioned technical problems. Summary of the Invention

[0006] This invention overcomes the shortcomings of the prior art and provides a method for calculating the depth of objects in two-dimensional images using a deep neural network cluster.

[0007] To achieve the above objectives, the technical solution adopted by this invention is as follows: a method for calculating the depth of objects in a two-dimensional image using a deep neural network cluster, comprising the following steps:

[0008] S1. Input a two-dimensional image of the target object, identify and label several feature points in the two-dimensional image, and create a local image of size (m,n) pixels centered on each feature point.

[0009] S2. Use the learned deep learning network model to predict the height of the center point of each local image. This deep learning network model integrates local image feature learning and height learning.

[0010] S3. Plan the Laure diagram with each elevation point as a vertex;

[0011] S4. Perform linear interpolation on the points within each triangle in the Laure plot to determine the height of the unlabeled feature points within each triangle region, thereby obtaining the three-dimensional structure and volume of the target object.

[0012] In a preferred embodiment of the present invention, in step S2, the learning step of the deep learning network model includes:

[0013] S21. Input learning data. The learning data consists of individual coordinates (x, y) and height data h of a two-dimensional image and its target object, i.e. (x, y, h) are feature points.

[0014] S22. Use a deep neural network model to extract image features from local images to obtain an image feature vector set;

[0015] S23. Connect the image feature vector and coordinates (x,y), use the height h as the teacher signal, and use a fully combined layer to learn the deep neural network model as a whole.

[0016] In a preferred embodiment of the present invention, in step S21, the marked feature points include at least the maximum height of the target object and cover the entire height.

[0017] In a preferred embodiment of the present invention, the object feature points in the identified two-dimensional image are located at the center of the local image, and all feature points are matched one-to-one with the local image and saved as a feature point local image set.

[0018] In a preferred embodiment of the present invention, the feature points include: color, corner points, edge points, spots, and texture features.

[0019] In a preferred embodiment of the present invention, in step S4, a three-dimensional integration method is used to calculate the volume of the target object. The continuous height distribution of the target object is integrated in three dimensions to calculate the volume of the target object.

[0020] In a preferred embodiment of the present invention, in step S4, after obtaining the continuous height distribution of the target object, all local images are recombined according to their original positions, and adjacent height points are connected and combined to form a three-dimensional structure of the target object.

[0021] In a preferred embodiment of the present invention, in S1, the two-dimensional image is captured by a camera device with a fixed viewing angle.

[0022] In a preferred embodiment of the present invention, the feature points in S1 are repeatedly stacked.

[0023] A system for calculating the depth of an object in a two-dimensional image, and a method for calculating the depth of an object in a two-dimensional image using a deep neural network cluster as described above, comprising a server, and a data receiving module, a recognition module, a segmentation module, a prediction module, an interpolation module, a data processing module, and a data output module mounted on the server;

[0024] The data receiving module is used to receive a two-dimensional image of the target object;

[0025] The recognition module is used to identify and label several feature points in a two-dimensional image;

[0026] The segmentation module is used to segment a two-dimensional image into several local images centered at each feature point and with a size of (m,n) pixels.

[0027] The prediction module is equipped with a deep learning network model, which is used to predict the height of the center point of each local image;

[0028] The interpolation module is used to plan the predicted height data into a Laure plot with each height point as a vertex, and to use a linear interpolation algorithm to determine the height of unlabeled feature points within each triangular region.

[0029] The data processing module calculates the volume of the target object based on all the obtained height points and the shape and area of ​​the target object in the two-dimensional image;

[0030] The data output module is used to output the three-dimensional structure and calculated volume of the target object.

[0031] This invention addresses the shortcomings of the prior art and has the following beneficial effects:

[0032] (1) This invention provides a method for calculating the depth of a two-dimensional image object using a deep neural network cluster. By inputting a two-dimensional image, the three-dimensional structure of the target object in the image can be measured. Compared with the prior art, this invention does not require multi-directional shooting of the target object. By analyzing the planar photograph of the target object, the three-dimensional structure and volume of the target object can be calculated. It can be more widely applied to three-dimensional measurements with only a fixed perspective, such as pathological slides.

[0033] (2) This invention utilizes a deep neural network model and a fully integrated layer to provide a two-dimensional image with labeled feature point data. The deep neural network model extracts image features from local images and connects them. The height data is used as the teacher signal. The fully integrated layer is used to learn the deep neural network model as a whole. Local image features and measured height are learned separately and finally merged into a whole deep learning network model. The measurement accuracy can be improved by simply stacking feature points without having to relearn the model.

[0034] (3) This invention uses Launet diagrams and linear interpolation to connect the image center height points predicted by the deep learning network model into a triangular network, and regards the height of each point in the local image as discrete data points. By interpolating these data points, the continuous height distribution of the object is obtained. The local image is recombined and the continuous height points are connected to obtain the three-dimensional structure of the target object. The volume data of the target object is calculated using the three-dimensional integral method, realizing the three-dimensional structure and volume of the object can be obtained by calculating only two-dimensional images. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of a preferred embodiment of the present invention;

[0037] Figure 2 This is a diagram illustrating the learning steps of a deep learning network according to a preferred embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, this invention provides a method for calculating the depth of objects in a two-dimensional image using a deep neural network cluster, comprising the following steps:

[0040] S1. Input a two-dimensional image of the target object, identify and label several feature points in the two-dimensional image, and create a local image of size (m,n) pixels centered on each feature point.

[0041] S2. Use the learned deep learning network model to predict the height of the center point of each local image. This deep learning network model integrates local image feature learning and height learning.

[0042] S3. Plan the Laue diagram with each height point as a vertex. The height point is the point at the height corresponding to the feature point. The height point is the local image center point plus its predicted height data, and the point coordinates in the three-dimensional space are obtained.

[0043] S4. Perform linear interpolation on the points within each triangle in the Laure plot to determine the height of the unlabeled feature points within each triangle region, thereby obtaining the three-dimensional structure and volume of the target object.

[0044] Among them, object feature points include: color, corner points, edge points, spots, and texture features.

[0045] It is worth noting that convolutional neural networks (CNNs) are preferred for identifying feature points of target objects in 2D images. A CNN is a deep learning model that uses a pre-trained CNN model, such as VGG, to identify object feature points in an image through a feature extraction layer. After identifying the object feature points, they are individually labeled, with the label location being the center of each individual feature point. Several labels are used to form a point map, which is then used for segmenting and locating objects in the 2D image.

[0046] Within the triangles formed in the Laure plot, linear interpolation is performed. Linear interpolation refers to an interpolation method using a first-order polynomial function. This fills in the missing information of unlabeled points between adjacent height points, resulting in a continuous height distribution. This makes the three-dimensional structure of the target object more complete and closer to the original object, improving the accuracy of the calculated three-dimensional structure and volume. The height of unlabeled feature points is calculated using the following formula: Where h1 and h2 are the height values ​​of the known vertices, (x1, y1) and (x2, y2) are the coordinates of the known vertices, (x, y) are the coordinates of the point to be interpolated, and h is the height value of the point to be interpolated. By applying linear interpolation to the triangle containing each unlabeled feature point, the height of the unlabeled feature points within each triangular region can be determined, thus obtaining a complete height map.

[0047] In this embodiment, in S1, the two-dimensional image is captured by a camera device with a fixed viewing angle.

[0048] This invention only requires photos with a fixed perspective or two-dimensional photos of the object to perform stereoscopic analysis, without the need to take photos of the target object from multiple directions.

[0049] This invention is applied to stereoscopic measurements of pathological slides and other materials with only a fixed perspective. It is used to accelerate the estimation of the volume and three-dimensional structure of pathological locations, and can more accurately assess the size and extent of lesions. This helps pathologists make pathological diagnoses and develop treatment plans, and can also provide clinicians with more information and evidence.

[0050] In this embodiment, the object feature points in the identified two-dimensional image are located at the center of the local image, and all feature points are matched one-to-one with the local image and saved as a feature point local image set.

[0051] For segmenting 2D images, meshing is preferred. The 2D image is divided into a series of equally sized grids. The size of each grid is determined by adjusting the number of rows and columns based on the point map, ensuring that each point is located near the center within the grid. Then, the pixels in the 2D image are assigned to the corresponding grids to form local images.

[0052] like Figure 2 As shown, in S2, the learning steps of the deep learning network model include:

[0053] S21. Input learning data. The learning data consists of individual coordinates (x, y) and height data h of a two-dimensional image and its target object, i.e. (x, y, h) are feature points.

[0054] S22. Use a deep neural network model to extract image features from local images to obtain an image feature vector set;

[0055] S23. Connect the image feature vector and coordinates (x,y), use the height h as the teacher signal, and use a fully combined layer to learn the deep neural network model as a whole.

[0056] This deep learning network model can learn local image features and measurement height separately, and then fuse them into a whole deep learning network model. It can improve measurement accuracy by simply stacking feature points, without having to relearn the model.

[0057] In S21, the labeled feature points include at least the maximum height of the target object and cover the entire height.

[0058] It is worth noting that feature points are the color, corner points, edge points, spots, and texture features of the target object. They are necessary points for volume calculation. These feature points combined together can display the three-dimensional structure of the target object and, combined with two-dimensional images, serve as learning data for deep learning network models.

[0059] In this embodiment, in S4, the volume of the target object is calculated using a three-dimensional integration method. The continuous height distribution of the target object is integrated in three dimensions to calculate the volume of the target object.

[0060] The three-dimensional integration method specifically refers to performing three-dimensional integration on discrete data points within a local image, that is, integrating the height distribution function in three-dimensional space. Numerical integration methods, such as the grid method, can be used to approximate the integral value, thereby obtaining the volume of the target object.

[0061] In this embodiment, after obtaining the continuous height distribution of the target object in S4, all local images are recombined according to their original positions, and adjacent height points are connected and combined to form the three-dimensional structure of the target object.

[0062] The specific process of combining the data to form the three-dimensional structure of the target object involves converting the distribution of all height points of the target object into point cloud identifiers, that is, saving the position and height information of each point as three-dimensional coordinates, and then using a point cloud processing algorithm to convert the point cloud data into a three-dimensional model, which is the three-dimensional structure of the target object.

[0063] In this embodiment, the feature points in S1 are repeatedly stacked and superimposed. The more feature points in the database, the higher the measurement accuracy.

[0064] This invention also provides a system for calculating the depth of a two-dimensional image object, using the aforementioned method for calculating the depth of a two-dimensional image object using a deep neural network cluster. The system includes a server and a data receiving module, a recognition module, a segmentation module, a prediction module, an interpolation module, a data processing module, and a data output module mounted within the server. The data receiving module receives a two-dimensional image of the target object. The recognition module identifies and labels several feature points in the two-dimensional image. The segmentation module segments the two-dimensional image into several local images centered at each feature point and with a size of (m,n) pixels. The prediction module carries a deep learning network model for predicting the height of the center point of each local image. The interpolation module projects the predicted height data into a Laureate diagram with each height point as a vertex, and uses a linear interpolation algorithm to determine the height of unlabeled feature points within each triangular region. The data processing module calculates the volume of the target object based on all the obtained height points and the shape and area of ​​the target object in the two-dimensional image. The data output module outputs the three-dimensional structure of the target object and the calculated volume.

[0065] It is worth noting that a communication protocol is set between the various modules. The data output module is connected to an external display screen to show the 3D structural point diagram of the target object and its calculated volume.

[0066] Based on the preferred embodiments of the present invention described above, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for calculating the depth of objects in a two-dimensional image using a deep neural network cluster, characterized in that, The method comprises the following steps: S1, inputting a two-dimensional image of a target object captured by a fixed-view camera device, identifying and labeling a plurality of feature points in the two-dimensional image, the feature points including color, corner points, edge points, spots and texture features, and dividing the two-dimensional image into a plurality of local images with each feature point as the center and a size of (m, n) pixels; S2, using a well-learned deep learning network model to predict the height of the center point of each local image, the deep learning network model integrating local image feature learning and height learning; wherein the learning steps of the deep learning network model comprise: S21, inputting learning data, the learning data being individual coordinate points (x, y) and height data h of the target object of the two-dimensional image, i.e. (x, y, h) being a feature point; S22, using a deep neural network model to extract image features from the local image to obtain an image feature vector set; S23, connecting the image feature vector and the coordinates (x, y), using the height h as a teacher signal, and using a full combination layer to learn the whole deep neural network model; S3, planning a Delaunay graph with each height point as a vertex; S4, performing linear interpolation on the points in each triangle in the Delaunay graph to determine the height of the unlabeled feature points in each triangular region, calculating the volume of the target object by three-dimensional integration based on the continuous height distribution obtained by interpolation, and connecting adjacent height points to form a point cloud to construct the three-dimensional structure of the target object.

2. The method of claim 1, wherein the method comprises: In the S21, at least the maximum height of the target object is included in the labeled feature points, and all heights are covered.

3. The method of claim 1, wherein the method comprises: The identified object feature points in the two-dimensional image are located at the center of the local image, and all the feature points are one-to-one corresponding to the local image, which are saved as a feature point local image set.

4. The method of claim 1, wherein the method comprises: In the S4, the three-dimensional integration method is used to calculate the volume of the target object, and the continuous height distribution of the target object is three-dimensionally integrated to calculate the volume of the target object.

5. The method of claim 1, wherein the method further comprises: In the S4, after obtaining the continuous height distribution of the target object, all the local images are recombined according to the original positions, and adjacent height points are connected and combined to form the three-dimensional structure of the target object.

6. The method of claim 1, wherein the method further comprises: The feature points in S1 are repeatedly superimposed and stacked.

7. A system for calculating the depth of a two-dimensional image object, using the method for calculating the depth of a two-dimensional image object using a depth neural network cluster according to any one of claims 1-6, characterized in that: The method comprises a server and a data receiving module, an identification module, a segmentation module, a prediction module, an interpolation module, a data processing module and a data output module mounted in the server; The data receiving module is used to receive a two-dimensional image of a target object; The identification module is used to identify and label a plurality of feature points in the two-dimensional image; The segmentation module is used to divide the two-dimensional image into a plurality of local images with each feature point as the center and a size of (m, n) pixels; The prediction module mounts a deep learning network model therein and is used to predict the height of the center point of each local image; The interpolation module is used to plan a Delaunay graph with each height point as a vertex based on the predicted height data, and determine the height of the unlabeled feature points in each triangular region by using a linear interpolation algorithm; The data processing module calculates the volume of the target object based on all the height points and the shape and area of the target object in the two-dimensional image; and The data output module is configured to output the three-dimensional structure and the calculated volume of the target object.

Images