Device and related products for evaluating a lesion area based on a neural network model

Abstract

The present invention relates to a device and related products for evaluating a lesion area based on a neural network model. The device includes one or more processors, a first neural network module, a second neural network module, and one or more computer-readable storage media, wherein the first neural network module receives and processes image data related to a lung lesion area image to obtain target vector data. The second neural network module receives and processes the target vector data to output an evaluation result for evaluating the lung lesion area. By using the solution of the present invention, high-order geometric features of the lung lesion area can be extracted and effective evaluation and prediction of including novel coronavirus infection can be carried out.

Description

Technical Field

[0001] The present invention generally relates to the field of image processing. More specifically, the present invention relates to an apparatus and a computer-readable storage medium for evaluating a pulmonary lesion area based on a neural network model. Background Art

[0002] With the continuous evolution of image processing technology, the current research on lesion area images including pulmonary lesion areas has developed rapidly. As is well known, lesion area images usually contain rich information helpful for clinical diagnosis. Therefore, it is particularly important to effectively extract and analyze the image features of the lesion area. Currently, the traditional processing method is to extract the imaging features of the lesion area and use the imaging features for subsequent analysis and research in order to evaluate the lesion area. However, how to effectively extract the features of the lesion area and effectively evaluate and predict the lesion area based on these features has become an urgent problem to be solved, especially when the pulmonary lesion area includes areas infected with the novel coronavirus. Summary of the Invention

[0003] To at least solve the above technical problems, the present invention provides an apparatus for evaluating a pulmonary lesion area based on a neural network model. Specifically, the present invention uses neural network-based technology to receive and process image data to output an evaluation result for evaluating the pulmonary lesion area. Using this evaluation result, the solution of the present invention can predict the development of the pulmonary lesion area over time. To this end, the present invention provides corresponding solutions in the following aspects.

[0004] In a first aspect, the present invention provides an apparatus for evaluating a pulmonary lesion area based on a neural network model, including: one or more processors; a first neural network module; a second neural network module; and one or more computer-readable storage media storing program instructions for implementing the first neural network module and the second neural network module, which, when executed by the one or more processors, cause: the first neural network module to receive and process image data related to a pulmonary lesion area image to obtain target vector data, where the image data includes raw data related to the pulmonary lesion area image and / or two-dimensional data related to the geometric features of the pulmonary lesion area image; and the second neural network module to receive and process the target vector data to output an evaluation result for evaluating the pulmonary lesion area.

[0005] In one embodiment, the image data related to the pulmonary lesion area image includes multiple sets of image sub-data related to the pulmonary lesion area acquired at multiple different times.

[0006] In one embodiment, the one or more computer-readable storage media further store program instructions for obtaining the two-dimensional data. When the program instructions are executed by the one or more processors, the following operations are performed: generating a two-dimensional grid based on the raw data; and using the two-dimensional grid to determine geometric features and representing the geometric features as a picture as the two-dimensional data.

[0007] In one embodiment, the geometric features include Gaussian curvature, mean curvature, or conformal factor obtained based on the image of the lung lesion area.

[0008] In one embodiment, the image of the lung lesion area is an image of the lung area infected with the novel coronavirus, and the first neural network module includes multiple encoders and a feature extractor, where: each of the multiple encoders includes multiple convolutional layers configured to perform multi-layer convolutional processing on the image data to obtain multiple feature vectors for different geometric features from the image data; and the feature extractor is configured to perform a feature fusion operation on the multiple feature vectors to obtain the target vector data.

[0009] In one embodiment, the multiple convolutional layers are connected in series, and the output end of the last convolutional layer connected in series is connected to the input end of the feature extractor.

[0010] In one embodiment, the feature fusion operation includes performing a data splicing operation on the multiple feature vectors to output the target vector data.

[0011] In one embodiment, the second neural network module includes a long short-term memory neural network configured to receive and process the target vector data to output an evaluation result for evaluating the lung lesion area.

[0012] In one embodiment, the evaluation result includes lesion quality information of the lung lesion area, and the lesion quality information is at least used to predict or judge the severity and / or development trend of the condition of a patient infected with the novel coronavirus.

[0013] In a second aspect, the present invention provides a computer-readable storage medium including a computer program for evaluating an image of a lung lesion area based on a neural network model. When the computer program is executed by the above device, the device outputs an evaluation result for evaluating the image of the lung lesion area.

[0014] Through the above description of the solution of the present invention in multiple aspects, those skilled in the art can understand that the solution of the present invention can efficiently utilize neural network technology to analyze and evaluate image data, so as to make a reasonable evaluation and prediction of the development of the lung lesion area included in the image. In an application scenario, when the lung lesion area includes a lesion area infected with the novel coronavirus, by using the device of the present invention to evaluate it, the severity and possible progression of the novel coronavirus infection can be predicted, so as to provide effective medical intervention for patients. Further, the two-dimensional data of the present invention includes data of geometric features extracted from the lung lesion area, making the obtained evaluation result more explanatory for the patient's condition, thus making the evaluation result more accurate and more referenceable. In addition, the neural network module of the present invention uses feature fusion operations to fuse data, so as to effectively extract and process features in the image data, thereby improving the accuracy of prediction and evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understandable. In the drawings, several embodiments of the present invention are shown in an exemplary rather than restrictive manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0016] Figure 1 is an architecture diagram showing an evaluation system for evaluating a lung lesion area based on a neural network model according to an embodiment of the present invention;

[0017] Figure 2 is a flowchart showing a method for extracting geometric features of a lung lesion area according to an embodiment of the present invention;

[0018] Figure 3 is the original image data of the lesion area that can be used in the present invention;

[0019] Figure 4a is a first topological disk obtained by cutting along a closed curve according to an embodiment of the present invention;

[0020] Figure 4b is a second topological disk obtained by cutting along a closed curve according to an embodiment of the present invention;

[0021] Figure 5 is a flowchart showing a method for obtaining a closed curve according to an embodiment of the present invention;

[0022] Figure 6 is an exemplary triangular mesh according to an embodiment of the present invention;

[0023] Figure 7It is a flowchart showing a method for mapping the interior of a topological disk into a unit rectangle to form harmonic image points of a harmonic mapping according to an embodiment of the present invention;

[0024] Figure 8 It is a detailed flowchart showing a method for mapping the interior of a topological disk into a unit rectangle to form harmonic image points of a harmonic mapping according to an embodiment of the present invention;

[0025] Figure 9 It is a simplified flowchart showing an operation for forming a geometric feature picture of a lesion area according to an embodiment of the present invention;

[0026] Figure 10a It is an exemplary schematic diagram showing an original unsegmented closed grid according to an embodiment of the present invention;

[0027] Figure 10b It is an exemplary schematic diagram showing the determination of pixel values according to an embodiment of the present invention;

[0028] Figure 11a It is a picture formed based on Gaussian curvature according to an embodiment of the present invention;

[0029] Figure 11b It is a picture formed based on mean curvature according to an embodiment of the present invention;

[0030] Figure 11c It is a picture formed based on conformal factor according to an embodiment of the present invention;

[0031] Figure 12 It is an operation block diagram showing a first neural network module according to an embodiment of the present invention;

[0032] Figure 13 It shows an operation block diagram of an encoder according to an embodiment of the present invention;

[0033] Figure 14 It is an operation block diagram showing a first neural network module and a second neural network module according to an embodiment of the present invention;

[0034] Figure 15 It is a schematic diagram of the operation principle of a second neural network module according to an embodiment of the present invention; and

[0035] Figure 16 It is a block diagram showing a device for evaluating a lung lesion area according to an embodiment of the present invention. Detailed implementation manner

[0036] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the embodiments described in this specification are only some embodiments provided by the present invention for the convenience of clearly understanding the solution and meeting the requirements of the law, rather than all the embodiments that can implement the present invention. All other embodiments obtained by those skilled in the art based on the embodiments disclosed in this specification without creative efforts belong to the scope of protection of the present invention.

[0037] Figure 1 It is an architecture diagram showing an evaluation system 100 for evaluating a pulmonary lesion area based on a neural network model according to an embodiment of the present invention.

[0038] As shown in the figure, the system 100 of the present invention includes a Computed Tomography (i.e., "CT") machine 102, which is used to scan the layers of the diseased part or suspected diseased part of a patient to obtain three-dimensional stereoscopic image data. In the context of the present invention, the diseased part here can be the lungs, especially the lung area that may or has been infected with the novel coronavirus. Thus, through the scanning of the CT machine, the three-dimensional image data of the present invention can be obtained.

[0039] After obtaining the above three-dimensional image data, the evaluation system of the present invention uses its database 104 to store the three-dimensional image data. Although not shown in the figure, in some scenarios, some preprocessing can also be performed on the three-dimensional image data before storage, such as triangulating the three-dimensional image data to obtain, for example, a two-dimensional network.

[0040] Further as Figure 1 shown in, the evaluation system 100 of the present invention further includes a device 106, which is exemplarily shown as a computer in the figure. It can be understood that the device of the present invention is not limited to the form of the computer shown in the figure, but can also be implemented as a mobile computing device or other forms of computing devices. Although not shown, the device 106 of the present invention may include one or more processors, and the one or more processors may include a general-purpose processor ("CPU") or a dedicated graphics processor ("GPU"). Further, the device of the present invention further includes a first neural network module 108 and a second neural network module 110.

[0041] As an example, the above-mentioned first neural network module and second neural network module of the present invention can be implemented as program instructions stored on a computer-readable storage medium of device 106. According to different application scenarios, the computer-readable storage medium here can be one or more, and can be various storage media capable of storing program instructions. During the execution of the evaluation task of the present invention, the processor can execute the program instructions stored on the computer-readable storage medium, so that the operation performed by the first neural network module and the second neural network module of the present invention is realized by the running of the program instructions.

[0042] Specifically, when the processor executes the foregoing one or more program instructions, the first neural network module of the present invention can be configured to receive and process image data related to the lung lesion area image (such as the image data 104 shown on the left side in the figure) to obtain target vector data. In one embodiment, the image data includes raw data related to the lung lesion area image and / or two-dimensional data related to the geometric features of the lung lesion area image. Correspondingly, the second neural network module of the present invention can be configured to receive and process the target vector data to output an evaluation result for evaluating the lung lesion area (for example, reflected in the form of a quality or volume ratio).

[0043] In one application scenario, the above-mentioned image data can include raw data related to the target image area and / or two-dimensional data related to the geometric features of the target image area. In one embodiment, the target image area can be, for example, an image area related to the lung lesion area of a patient, and its raw data can be CT image data obtained by, for example, electron computed tomography technology or equipment. In one implementation scenario, the image data related to the lung lesion area image includes multiple sets of image sub-data related to the lung lesion area acquired at multiple different times.

[0044] Based on the above description, the above-mentioned computer-readable storage medium also stores program instructions for obtaining the two-dimensional data. When the program instructions are executed by the one or more processors, a two-dimensional grid is generated according to the obtained raw data, and the geometric features of the target image area are determined through the two-dimensional grid. Further, through the device of the present invention, the geometric features (such as the Gaussian curvature, mean curvature or conformal factor obtained based on the lung lesion area image) can be represented in the form of a picture as two-dimensional data related to the geometric features of the target image area.

[0045] In one embodiment, the first neural network module in the device of the present invention can include multiple encoders and feature extractors (such as Figure 12(as shown in). In one implementation scenario, each of the foregoing multiple encoders may include a plurality of convolutional layers configured to perform multi-layer convolutional processing on the image data to obtain a plurality of feature vectors for different geometric features from the image data. In one embodiment, the plurality of convolutional layers in each encoder may be serially connected, and the output end of the last convolutional layer in the serial connection may be connected to the input end of the feature extractor. In one implementation scenario, the feature extractor may be configured to perform a feature fusion operation on the foregoing multiple feature vectors to obtain target vector data. As an example, the feature fusion operation may include performing a data splicing operation on the multiple feature vectors to output the target vector data.

[0046] In one embodiment, the second neural network module of the device of the present invention includes a Long Short-Term Memory (LSTM) neural network, which is configured to receive and process the above-mentioned target vector data to output an evaluation result for evaluating the pulmonary lesion area. In one application scenario, the evaluation result may include lesion quality information of the pulmonary lesion area, and the lesion quality information is at least used to predict or judge the severity and / or the development trend of the condition of a patient infected with the novel coronavirus.

[0047] As described above in connection with Figure 1 the evaluation system of the present invention and the devices therein. Next, in connection with Figures 2 - 1 1, a detailed description will be given of how the present invention extracts the foregoing geometric features.

[0048] Figure 2 FIG. is a flowchart of a method 200 for extracting geometric features of a lesion area image according to an embodiment of the present invention. It can be understood that the method 100 of the present invention can be implemented by various computing devices including, for example, a computer.

[0049] First, at step S202, the method 200 obtains three-dimensional image data of a pulmonary lesion area. In one embodiment, the foregoing three-dimensional image data may be three-dimensional image data obtained by, for example, CT technology or equipment. In another embodiment, the pulmonary lesion area here may be or include a pulmonary area infected with the novel coronavirus.

[0050] Next, at step S204, the method 200 generates a two-dimensional grid formed by connecting a plurality of vertices according to the three-dimensional image data of the lesion area. In one embodiment, those skilled in the art can understand the generation of the two-dimensional grid, and its essence is the generation of a grid on the outer surface of boolean variable (bool) data. In this embodiment, the three-dimensional image area of the foregoing lesion area may be marked with bool and set to be where is a smooth function, is the region where the three-dimensional image exists. Based on the foregoing definitions, it is possible to use to represent internal voxels, to represent external voxels, while (where 0 < < 1) represents the boundary region, and the isosurface mesh of the function is calculated by smooth interpolation of the function In one implementation scenario, it is possible to use, for example, the Computational Geometry Algorithms Library (abbreviated as "CGAL") to perform the foregoing interpolation to generate a two-dimensional mesh.

[0051] After generating the above two-dimensional mesh, the method flow then proceeds to step S206. At this step S206, method 100 uses a closed curve formed by some of the plurality of vertices to cut the two-dimensional mesh into two topological disks (such as Figure 4a the first topological disk shown in Figure 4b and the second topological disk shown in). In one embodiment, it is possible to calculate the function value of each vertex based on the two-dimensional mesh information, and select the vertex with the globally minimum function value from all vertices as the starting point of the foregoing closed curve and the vertex with the locally minimum function value as the intermediate point. Then, it is possible to start from the starting point, pass through N intermediate vertices in sequence, and return to the starting point to obtain the closed curve, and cut the two-dimensional mesh into two topological disks along this closed curve. Further, at step S208, method 200 maps the two topological disks to two unit rectangles respectively. According to the solution of the present invention, the mapping here can be a harmonic mapping.

[0052] After mapping to the above unit rectangle, the method flow then proceeds to step S210. At this step S210, method 200 uses the geometric eigenvalue of the above plurality of vertices to determine the pixel value of the corresponding point on the unit rectangle. According to different embodiments, the geometric eigenvalue of the present invention can be one of Gaussian curvature, mean curvature, or conformal factor. Finally, at step S212, method 200 forms a picture including the geometric features of the lung lesion region based on the foregoing pixel values. As an example, this picture can be Figure 11a the picture obtained by Gaussian curvature in Figure 11b the picture obtained by mean curvature in or is Figure 11c the picture obtained by conformal factor. As described above, when the above lesion region includes the lung region infected with the novel coronavirus, the picture generated at this time is a picture including the geometric features of the novel coronavirus infection region.

[0053] In one embodiment, in performing the operation of mapping a topological disk to a unit rectangle, method 200 may include mapping the boundary of the topological disk to the boundary of the unit rectangle and harmonically mapping the interior of the topological disk into the unit rectangle to form harmonic image points of the harmonic mapping.

[0054] Figure 3 The original three-dimensional data of the lesion area that can be used in the present invention is shown. In combination with the above Figure 2 description, Figure 3 the grid shown in Figure 2 may be the two-dimensional grid generated at step S204 of method 200 shown in Figure 2 Further, the black curve indicated by the arrow in Figure 2 may be the closed curve obtained at step S206 of method 200 shown in Figure 4a and Figure 4b respectively shown.

[0055] Figure 5 is a flowchart showing method 500 for obtaining a closed curve according to an embodiment of the present invention. It should be understood here that method 500 is Figure 2 a specific implementation of forming a closed curve in method 200 shown in

[0056] Therefore, the description made about method 200 also applies to method 500. Figure 1 As mentioned in combination with Figure 5 shown, the present invention uses the closed curve formed by some of the plurality of vertices to divide the two-dimensional grid generated by method 200 into two topological disks. Based on this, as Figure 6 shown, at step S502, method 500 may determine (e.g., by calculation) the Laplace-Beltrami matrix based on the grid according to the topology of the two-dimensional grid and the side lengths of the grid edges. The topology of the grid here can be understood as the connection relationship of the grid, specifically the connection relationship between the vertices on the triangular grid. When the total number of vertices is M, an M-order Laplace-Beltrami matrix can be formed. For the convenience of discussion, the Laplace-Beltrami matrix will be described below first in combination with

[0057] Figure 6 is an exemplary triangular grid showing an embodiment of the present invention. As can be seen from Figure 6 , two triangular grids are shown here, which include four vertices , , , and . Further, vertices and The edge formed between and the vertex and The included angle between the edges formed between is , and the vertex and The edge formed between and the vertex and The included angle between the edges formed between is . Additionally, it can be seen from the figure that the vertices share an edge. Based on the vertices, edge lengths, and included angles exemplarily shown here, each element value in the Laplace - Beltrami matrix can be determined by the following formula, that is, the weight of the (relationship) between the vertices :

[0058]

[0059] Where:

[0060]

[0061] The inner edge mentioned above indicates that this edge is shared by two triangular meshes, " " represents the cotangent value, and the boundary edge indicates that this edge is not shared by two triangular meshes but is only contained by one triangular mesh.

[0062] Returning to Figure 5 , after calculating the Laplace - Beltrami matrix based on the topology of the two - dimensional grid and the edge lengths of the grid edges as described above, the process of method 500 proceeds to step S504. At this step S504, method 500 can calculate the aforementioned matrix to obtain the non - zero eigenvalue with the smallest absolute value, thereby determining the eigenfunction corresponding to the non - zero eigenvalue with the smallest absolute value. Then, at step S506, method 500 can determine the function values at each grid vertex on the aforementioned two - dimensional grid according to the aforementioned eigenfunction, and obtain the starting point and intermediate points of the closed curve based on the function values.

[0063] After obtaining the starting point and intermediate points of the closed curve, the method 500 then selects, at step S508, the vertex with the globally minimum function value from all vertices as the starting point of the closed curve. Then, at step S510, the method 500 selects the vertex with the locally minimum function value from the vertices adjacent to the foregoing starting point as the first intermediate vertex. Next, at step S512, for each intermediate vertex from the second to the Nth intermediate vertex, the method 500 performs the following selection operation until returning to the starting point (i.e., the end point of the closed curve): selects the vertex with the locally minimum function value from the vertices adjacent to the (N - 1)th intermediate vertex as the Nth intermediate vertex, where N can be a positive integer greater than or equal to 2. After completing the above operations, the method 500 obtains a closed curve that starts from the starting point, passes through N intermediate nodes in sequence, and then returns to the starting point. For example Figure 3 the closed curve represented by the black line curve in the three-dimensional image of the lesion area shown

[0064] Figure 4a and Figure 4b are respectively diagrams showing the first topological disk and the second topological disk obtained by cutting along the closed curve in Figure 3 According to the embodiments of the present invention. As described above, by dividing the closed curve obtained by the method 500 of the present invention, the first topological disk as shown in Figure 4a and the second topological disk as shown in Figure 4b can be obtained

[0065] Figure 7 is a flowchart of a method 700 for forming a harmonic image point of a harmonic mapping by mapping the inside of a topological disk into the unit matrix according to an embodiment of the present invention. It should be understood here that the method 700 is Figure 2 a specific implementation manner of forming a harmonic image point of a harmonic mapping in the method 208 shown in

[0066] As Figure 7 shown, at step S702, the internal points of the topological disk are initially mapped into the unit rectangle to form image points after the initial mapping (abbreviated as "initial image points"). In one embodiment, it can be assumed that for the internal points of the unit rectangle, their coordinates are initially set to . Then, at step S704, the harmonic energy between the initial image points within the unit rectangle is determined (e.g., by the device of the present invention). In one embodiment, the harmonic energy can be defined as:

[0067]

[0068] where

[0069]

[0070] The initialized harmonic energy can be calculated from the above formulas (3) and (4). , and let .

[0071] As Figure 7 further shown, at step S706, the coordinates of the image points can be adjusted according to the harmonic energy and a preset energy gradient threshold to obtain the harmonic image points of the harmonic mapping. For example, in an implementation scenario, the preset energy gradient threshold can be , and the coordinates of the initial image points are adjusted according to the following formula (5), that is

[0072]

[0073] and the adjusted harmonic energy is calculated . Then, the above calculation results can be compared with the preset energy gradient threshold to obtain the harmonic image points of the harmonic mapping.

[0074] Figure 8 is a detailed flowchart showing a method 800 for forming harmonic image points of a harmonic mapping by mapping the interior of a topological disk to a unit matrix according to an embodiment of the present invention. It should be understood here that the method 800 is Figure 7 a specific implementation manner of the method 700 shown, so the descriptions made about the method 700 also apply to the method 800.

[0075] Specifically, at step S802, the coordinates of the initial image points of the initial mapping are adjusted. Then, at step S804, the aforementioned harmonic energy is updated according to the adjusted coordinates of the initial image points. At step S806, the harmonic energy can be compared with a preset energy gradient threshold. In one embodiment, the harmonic energy can be calculated using the above formula (3), and the preset energy gradient threshold is . When the harmonic energy (or the harmonic energy difference) is greater than the preset energy gradient threshold, that is , the coordinates of the initial image points are adjusted using the above formula (5), that is, the process returns to execute step S802. Then, at step S804, the harmonic energy is updated according to the adjusted coordinates of the initial image points for the next comparison with the preset energy gradient threshold.

[0076] When it is determined at step S806 that the harmonic energy is less than (or equal to) the preset energy gradient threshold, the image points at the time of stopping the adjustment are used as the harmonic image points of the harmonic mapping. For example, when the above harmonic energy is less than or equal to the preset energy gradient threshold, for example Then, the coordinate adjustment of the image point is stopped, and the coordinates of the image point at this time are used as the coordinates of the harmonic image point of the aforementioned harmonic mapping, that is, the harmonic image point is determined. It can be understood that in combination with methods 700 and 800, the solution of the present invention finally maps the internal points of the topological disk into the unit rectangle to form the harmonic image point of the harmonic mapping.

[0077] Figure 9 FIG. 800 is a simplified flowchart of an operation of forming a geometric feature picture of a lesion area according to an embodiment of the present invention. According to different implementation scenarios, the geometric feature of the present invention may be one of Gaussian curvature, mean curvature, or conformal factor. In one embodiment, the Gaussian curvature, mean curvature, or conformal factor of the grid vertices may be calculated according to the mesh information. In one implementation scenario, the Gaussian curvature is equal to 2π minus the angles corresponding to the adjacent meshes at the vertices of the original uncut closed mesh. To facilitate the understanding of the Gaussian curvature mentioned here, first, in combination with Figure 10a a simple description will be given. Figure 10a FIG. 9 is an exemplary schematic diagram of an original uncut closed mesh according to an embodiment of the present invention, where a vertex in the mesh is denoted as P, and the angles corresponding to the meshes adjacent to vertex P are respectively denoted as and , and the Gaussian curvature at vertex P is denoted as k, then k = . Based on this, the Gaussian curvature values of all vertices on the original mesh surface can be calculated.

[0078] Regarding the mean curvature involved in the above-mentioned geometric feature, in one implementation scenario, first, the normal vectors of each mesh surface are calculated on the original uncut mesh, and the normal vectors of its adjacent surfaces are respectively denoted as and , and arc = . When arc is less than zero, the mean curvature of this edge is side length * [π - acos(arc)]; when arc is greater than zero, the mean curvature of this edge is side length * acos(arc). Here, "acos" represents the inverse cosine value. For a point, the mean curvature of each point is the average value after summing the mean curvatures of all edges around this point.

[0079] Regarding the conformal factor also involved in the above-mentioned geometric feature, in another implementation scenario, first, the total area of the original uncut mesh surface and the area of each vertex are calculated, where the area of each vertex can be, for example, one-third of the area around the vertex. Then, the total area of the mesh after harmonic mapping is calculated, where the area of the vertex after harmonic mapping is one-third of the total area of the mesh after harmonic mapping, so the area ratio is the original total area / the total area after harmonic mapping. Thus, the conformal factor of each vertex is the area ratio * the area of the vertex after harmonic mapping / the total area of the mesh.

[0080] Based on the above exemplary operations, the Gaussian curvature, mean curvature, or conformal factor of each vertex on the mesh surface can be obtained, and the calculated result is rounded to be used as the pixel value of the corresponding harmonic image point on the unit rectangle. And a picture representing the geometric features of the lesion area is formed according to the pixel value. Specifically, as Figure 9 shown, at step S902, pixel points are evenly arranged on the unit rectangle. In an exemplary scenario, for example, 256*256 pixel points can be evenly arranged. Then, the following steps (i.e., steps S904 and S906) can be performed for each of the pixel points to obtain the pixel value.

[0081] First, at step S904, it is determined the position of the above pixel point on the unit rectangle. For example, the pixel point can be at the four vertices of the unit rectangle, can be on the four sides of the unit rectangle, or can be inside the unit rectangle. In one embodiment, the position of the pixel point inside the unit rectangle can be determined according to the coordinates of the aforementioned harmonic image point. Then, at step S906, the value of the Gaussian curvature, mean curvature, or conformal factor of the pixel point is determined according to the position, so as to finally determine the pixel value of the pixel point. In one embodiment, when the pixel point is at the four vertices of the unit rectangle, the Gaussian curvature, mean curvature, or conformal factor of the four vertices of the unit rectangle is used as the pixel value of the pixel point. In another embodiment, when the pixel point is on the four sides of the unit rectangle, linear interpolation is used to calculate the Gaussian curvature, mean curvature, or conformal factor to be used as the pixel value of the corresponding pixel point. In yet another embodiment, when the pixel point is inside the unit rectangle, barycentric coordinate interpolation is used to calculate the Gaussian curvature, mean curvature, or conformal factor to be used as the pixel value of the corresponding pixel point. The following will be combined with Figure 10b to describe the process of determining the pixel value of the pixel point at different positions.

[0082] Figure 10b is an exemplary schematic diagram showing the determination of the pixel value according to an embodiment of the present invention. As Figure 10b shown on the left side in the figure, the horizontal line and the vertical line intersect to form a rectangular grid. The size of the grid can be 256*256, and 256*256 pixel points are arranged on the grid, such as pixel point P1, pixel point P2, and pixel point P3 (the remaining pixel points are not shown in the figure). The multiple triangular grids shown on the left side in the figure are formed by connecting some harmonic image points of the harmonic mapping on the unit rectangle. In an exemplary scenario, it can be assumed that the vertices of the triangular grid are v1, v2, v3, v4, and v5 respectively, and the function values at points v1, v2, v3, v4, and v5 are respectively denoted as , and 。In this scenario, the , and can be any one of the geometric eigenvalue such as Gaussian curvature, mean curvature or conformal factor obtained based on the foregoing description, and the geometric eigenvalue is used to determine the pixel value.

[0083] Combined with the above Figure 10b description, in an implementation scenario, when the pixel point is located at the four vertices of the unit rectangle, for example Figure 10b the pixel point P1 shown in. In this scenario, the pixel value of the pixel point P1 is the function value at the grid vertex v1 . The can be any one of Gaussian curvature, mean curvature or conformal factor.

[0084] In another implementation scenario, when the pixel point is located on the boundary of the unit rectangle, for example Figure 10b the pixel point P2 shown in. In this scenario, the pixel value at the pixel point P2 is determined by the function values at v1 and v2. Specifically, assuming that the side length from v1 to the pixel point P2 is , the side length from v2 to the pixel point P2 is , and the side length from v1 to v2 is , then the pixel value at the pixel point P2 is calculated based on linear interpolation = , where are the function values at v1 and v2 respectively. Similarly, is any one of Gaussian curvature, mean curvature or conformal factor.

[0085] In yet another implementation scenario, when the pixel point is located inside the unit rectangle, for example Figure 10b the pixel point P3 shown in. In this scenario, the pixel value at the pixel point P3 is determined by the area ratio of the triangle formed by v3, v4 and v5 and through barycentric coordinate interpolation. Here, the area ratio can be understood as the weights at v3, v4 and v5. Specifically, as shown in the right figure of Figure 10b , assuming that the area corresponding to v3 is denoted as S3, the area corresponding to v4 is denoted as S4, and the area corresponding to v5 is denoted as S5, and assuming that the total area of the triangle is S, then the pixel value at the pixel point P3 = , where are the function values at v3, v4 and v5 respectively. Similarly, It can also be any one of the Gaussian curvature, the mean curvature, or the conformal factor. Based on the foregoing description, the area of each part can be determined by the vertex coordinates of the triangle and the coordinates of the pixel points. For example, in an exemplary scenario, assume that the coordinates at v3 are (x1, y1), the coordinates at v4 are (x2, y2), and the coordinates at v5 are (x3, y3), and the coordinates at pixel point P3 are ( ), interpolation is performed using barycentric coordinates, for example:

[0086]

[0087] where , , respectively represent the weights at vertices v3, v4, and v5, and the weights are the areas corresponding to vertices v3, v4, and v5. Express the above formulas (7)-(9) in terms of weights as:

[0088] (10)

[0089] The area S3 corresponding to v3, the area S4 corresponding to v4, and the area S5 corresponding to v5 are obtained from the above formulas, and finally the pixel value at pixel point P3 is obtained.

[0090] Based on the foregoing description, the pixel values of 256*256 pixel points are determined. The solution of the present invention can finally form, for example, Figure 11a the picture formed based on the Gaussian curvature as shown, for example, Figure 11b the picture formed based on the mean curvature as shown; or for example, Figure 11c the picture formed based on the conformal factor as shown. By performing deep learning in the field of artificial intelligence, for example, on the pictures representing the foregoing three geometric features, a lesion development prediction model can be obtained and corresponding predictions can be made.

[0091] After obtaining the image data including the geometric features based on the extraction method described in combination with the above Figures 2 - 1 1 (for example, two-dimensional data) or the raw data collected by CT technology, since the image data is usually represented by gray values in the range of 0 to 255, it is usually necessary to preprocess the obtained image data. In one embodiment, the present invention proposes to normalize the gray value of the image data to a floating point number between 0 and 1 using the maximum-minimum (max-min) criterion. Then, the first neural network module of the present invention receives the preprocessed image data and processes it to obtain target vector data. In the implementation scenario of applying the geometric feature extraction solution of the present invention, the foregoing image data can also be one-dimensional data and / or three-dimensional data related to the geometric features of the target image region.

[0092] Figure 12 It is an operational block diagram showing the first neural network module 108 according to an embodiment of the present invention. It should be understood that Figure 12 the first neural network module shown is Figure 1 a specific implementation of the first neural network module in the evaluation system 100 shown. Thus, regarding Figure 1 the relevant details and features of the evaluation system 100 described also apply to Figure 12 the description of

[0093] As shown in the figure, different types of image data are represented in the left dotted box in the figure. From top to bottom, they represent raw data 1201, three-dimensional data 1202, two-dimensional data 1203, and one-dimensional data 1204 in sequence. In one embodiment, the one-dimensional data can be stored in TXT format, and its size can be 1*400 (i.e., 400 data bits in one row); the two-dimensional data can be stored in picture (e.g., png) format, and its pixel size can be, for example, 256*256. This picture can exemplarily be Figure 11a , Figure 11b or Figure 11c the picture containing geometric features shown in

[0094] After receiving the above-mentioned image data (e.g., the image data preprocessed by normalization), the first neural network module 108 of the present invention first extracts the feature vectors corresponding to different types of image data by passing the image data through different encoders.

[0095] Specifically, encoder 1 processes the raw data 1201 to output the feature vector 1201-1. Similarly, the three-dimensional data 1202, two-dimensional data 1203, and one-dimensional data 1204 can respectively extract the corresponding feature vectors 1202-1, feature vector 1203-1, and feature vector 1204-1 through encoder 2, encoder 3, and encoder 4. It should be understood that Figure 12The number of dimensions of the image data dimensions and the number of encoders shown in the figure are merely exemplary and not restrictive. Those skilled in the art can select other image data formats or types according to requirements. For example, in some application scenarios, any one of the raw data, one-dimensional to three-dimensional data can be used for evaluation. In other application scenarios, any two or more of the aforementioned raw data and one-dimensional to three-dimensional data can be combined for evaluation. Therefore, the present invention does not impose any restrictions on data formats and data usage methods, etc. Similarly, the present invention also does not impose any restrictions on the number and type of encoders corresponding to the aforementioned data formats.

[0096] In one embodiment, the above encoder of the present invention can be implemented through a convolutional layer (or convolutional operator) in a neural network. In one implementation scenario, it can be implemented through a layer structure including two convolutional layers and an adaptive convolutional layer as shown in Figure 13 to perform the encoding operation on the data to obtain the feature vector data as described above, which is specifically described as follows.

[0097] Figure 13 Shows an operation block diagram of the encoder 1300 according to an embodiment of the present invention. It can be understood that the encoder 1300 can be any one of the encoders 1 - 4 in Figure 12 . As shown in the figure, the encoder 1300 can include a convolutional layer 1301, a convolutional layer 1302, and an adaptive convolutional layer 1303. Assuming that the data on the left side of the figure is the two-dimensional data 1203 in the above figure (such as a picture represented by the extracted Gaussian curvature, mean curvature, or conformal factor), the encoder 1300 is set as the encoder 3 in Figure 12 . Thus, the two-dimensional data 1203 undergoes the first convolution through the convolutional layer 1301 in the encoder 1300. Then, it undergoes the second convolution through the convolutional layer 1302, and optionally undergoes the third convolution through the adaptive convolution 1303 to obtain the feature vector 1203 - 1. Similarly, through the processing of the encoder as described above, the feature vectors 1202 - 1, 1203 - 1, and 1204 - 1 in Figure 12 can be obtained.

[0098] According to the actual application scenario, the first two convolutional layers in the above-mentioned encoder of the present invention can respectively use 128 and 64 convolutional kernels to perform convolutional operations. In this case, the inputs can be feature maps of sizes 256*256 and 128*128 respectively, and the outputs can be feature maps of sizes 128*128 and 64*64 respectively. For the third convolutional layer, it can be an adaptive convolutional layer using 32 convolutional kernels, and its output is a feature map of size 32*32. Here, the purpose of adding the adaptive convolutional layer is only to fix the output size of the encoder, that is, to make the encoder of the present disclosure always output a feature map of a fixed size, such as the aforementioned 32*32 feature map. Based on this, those skilled in the art can understand that the adaptive convolutional layer of the present invention is an optional setting, and in some other application scenarios, it may not be used or replaced by another convolutional layer. Further, the size of the convolutional kernels in the neural network of the present invention can all be 3*3 arrays, and can be initialized using a uniform distribution.

[0099] Combined Figure 12 with Figure 13 the content shown, those skilled in the art can understand that the multiple convolutional layers in the multiple encoders in the first neural network module of the present invention are serially connected, and the output end of the last convolutional layer in the serial connection is connected to the input end of the feature extractor of the first neural network module (i.e., Figure 12 the feature extractor 1205 in Figure 13 ). For the convolutional layer structure shown in

[0100] Referring to Figure 12The shown feature extractor 1205 performs a feature fusion operation on, for example, the above four types of feature vectors (i.e., the operation in the middle dotted box in the figure). Specifically, first, a convolution is performed on the feature vector 1201-1, the feature vector 1202-1, the feature vector 1203-1, and the feature vector 1204-1 respectively to obtain their respective convolution results, and then the convolution results of each feature vector are fused (such as concatenated) to obtain the target vector data 1206. For example, the convolution results of the feature vector 1202-1, the feature vector 1203-1, and the feature vector 1204-1 are concatenated with the convolution result of the feature vector 1201-1 to obtain the feature vector 1201-2. Similarly, the convolution results of the feature vector 1201-1, the feature vector 1203-1, and the feature vector 1204-1 are concatenated with the convolution result of the feature vector 1202-1 to obtain the feature vector 1202-2. Thus, the feature vector 1203-2 and the feature vector 1204-2 can also be obtained. Then, for the feature vector 1201-2, the feature vector 1202-2, the feature vector 1203-2, and the feature vector 1204-2, the above operation is performed multiple times (such as twice) to obtain the feature vector 1201-10, the feature vector 1202-10, the feature vector 1203-10, and the feature vector 1204-10, and they are concatenated to form the target vector data 1206.

[0101] In an implementation scenario, the size of the convolution kernel used for the above convolution can be 3*3, 1*1 (when convolving with itself), and the number of convolution times can be three times. The present invention does not limit this. In addition, the dimension of the target vector data can also be set according to requirements, and the present invention does not limit this either. For example, the target vector data obtained in the present invention is 1024-dimensional, and this target vector data relates to the image data obtained from a patient's one-time CT. In the scenario of applying to the analysis of the lesion area image, usually, multiple CT images of the patient at different times can be collected respectively, and based on the operations described in the above first neural network module, multiple target vector data at different times can be obtained to be used as the input end of the second neural network module, such as Figure 14 shown.

[0102] Figure 14It is an operation block diagram showing a first neural network module and a second neural network module according to an embodiment of the present invention. The leftmost part of the figure represents the acquisition of image data of a patient at Tn time periods (such as CT1 image data, CT2 image data, and CTn image data). The first neural network module 108 respectively receives the foregoing image data and processes it, thereby respectively obtaining the target vector data 1401 at time T1, the target vector data 1402 at time T2, and the target vector data 1410 at the T-th time period. Then, the second neural network module 108 receives the obtained multiple target vector data and processes it to finally obtain the evaluation result of the lesion area. In one embodiment, the second neural network module may be a Long Short-Term Memory neural network (Long Short-Term Memory, "LSTM"), for example Figure 15 as shown in

[0103] Figure 15 It is a schematic diagram showing the operation principle of the second neural network module 110 according to an embodiment of the present invention. As described above, the second neural network module of the present invention can be implemented as an LSTM neural network in one implementation scenario, and the LSTM neural network may include an input layer, one or more hidden layers, and an output layer.

[0104] As Figure 15 shown in, representing the image data of the patient's CT t period, CT t-1 period, and CT t+1 period, after feature fusion by the first neural network module, respectively obtain the target vector data X t , the target vector data X t-1 , and the target vector data X t+1 . The LSTM neural network takes the target vector data X t , the target vector data X t-1 , and the target vector data X t+1 as inputs, and uses the target vector data X t and the memory S t-1 of the patient's t-1-th CT at the previous moment to calculate the memory S t of the patient's t-th CT at the current moment. Similarly, the memory S t of the patient's t-th CT can be used to calculate the memory S t+1. In an implementation scenario, the foregoing operations (such as adjusting the weights U from the input layer to the hidden layer, the weights V from the hidden layer to the output layer, and the weights W from the hidden layer at the previous moment to the hidden layer at the current moment) can be repeated multiple times and the calculations for memories at all moments can be completed, and finally the output Ot is obtained, and this Ot is the evaluation result. Generally, the foregoing evaluation result can be expressed as quality (such as the quality of the lesion area) and volume ratio (the volume ratio of the lesion area to the entire image). By analyzing this evaluation result, the situation of the target features in the image data evolving over time can be determined. For example, when the image data is image data containing the lesion area of a novel coronavirus infection, the current state and development of the novel coronavirus infection can be evaluated and predicted.

[0105] For example, through the foregoing high-order geometric features extracted from the lesion area containing the novel coronavirus infection area of the present invention, the user of the device of the present invention (such as a medical professional) can determine the severity of the current user's condition by analyzing the quality or volume ratio of the lung infection area reflected by the evaluation result. Further, since the neural network module of the present invention processes data in the time dimension, the user can also evaluate the development of the patient's condition based on the evaluation result. For example, when the quality or volume ratio shows a gradually decreasing or declining trend, it can be judged that the patient is expected to recover within a certain period of time. In contrast, when the quality or volume ratio shows an increasing or rising trend, it can be judged that the patient's condition may deteriorate further to a certain extent. In this situation, medical staff can provide necessary treatment to the patient in a timely manner to control the development of the condition and prevent further deterioration of the condition.

[0106] Although the training process of the neural network module of the present invention is not mentioned above, based on the content disclosed in the present invention, those skilled in the art can understand that the neural network module of the present invention can be trained with training data to obtain a neural network module with high precision. For example, in the forward propagation process of neural network training, the present invention can use the image data including geometric features obtained by combining Figures 2 - 1 1 to train the neural network module of the present invention, and compare the training result with the expected result (or called the true value) to obtain the corresponding loss function. Further, in the backpropagation process of neural network training, the present invention uses the obtained loss function and updates the weights (such as Figure 15 the weights U, V, and W therein) based on, for example, the gradient descent algorithm to reduce the error between the output Ot and the true value.

[0107] In combination with the above description, using the image evaluation system according to the embodiments of the present invention, the first neural network module can perform feature fusion on the image data to obtain target vector data, and the second neural network module can process the target vector data to obtain the evaluation result of the image. For example, the CT image of a patient can be input into the image evaluation system of the present invention, and the evaluation result (such as the quality and volume ratio) of the lesion area of the patient can be directly obtained. By predicting the condition of the patient and the development trend of the lesion area through the quality or volume ratio, manual intervention can be facilitated.

[0108] Figure 16 FIG. 4 is a block diagram showing a device 1600 for evaluating a lung lesion area based on a neural network model according to an embodiment of the present invention. As Figure 16 shown, the device 1600 may include a central processing unit ("CPU") 1611, which may be a general-purpose CPU, a dedicated CPU, or other information processing and program execution units. Further, the device 1600 may further include a mass storage 1612 and a read-only memory ("ROM") 1613, where the mass storage 1612 may be configured to store various types of data, such as various image data related to the lesion area, algorithm data, intermediate results, and various programs required to operate the device 1600. The read-only memory ("ROM") 1613 may be configured to store data required for power-on self-test of the device 1600, initialization of each functional module in the system, driver programs for basic input / output of the system, and data for booting the operating system.

[0109] Optionally, the device 1600 may further include other hardware platforms or components, such as the shown tensor processing unit ("TPU") 1614, graphics processing unit ("GPU") 1615, field programmable gate array ("FPGA") 1616, and machine learning unit ("MLU") 1617. It can be understood that although various hardware platforms or components are shown in the device 1600 of the present invention, this is only exemplary and not restrictive, and those skilled in the art can add or remove the corresponding hardware according to actual needs. For example, the device 1600 may only include a CPU to implement the evaluation operation of the lung lesion area of the present invention.

[0110] In some embodiments, to facilitate the transfer and interaction of data with an external network, the device 1600 of the present invention further includes a communication interface 1618, so that it can be connected to a local area network / wireless local area network ("LAN / WLAN") 1605 through this communication interface 1618, and then can be connected to a local server 1606 or connected to the Internet ("Internet") 1607 through the LAN / WLAN. Alternatively or additionally, the device 1600 of the present invention can also be directly connected to the Internet or a cellular network based on wireless communication technology through the communication interface 1618, such as wireless communication technology based on the third generation ("3G"), the fourth generation ("4G") or the fifth generation ("5G"). In some application scenarios, the device 1600 of the present invention can also access the server 1608 and database 1609 of the external network as needed, in order to obtain various known image models, data and modules, and can remotely store various data, such as various types of data for presenting or evaluating the lesion area image.

[0111] The peripheral devices of the device 1600 of the present invention may include a display device 1602, an input device 1603, and a data transmission interface 1604. In one embodiment, the display device 1602 may include, for example, one or more speakers and / or one or more visual displays, which are configured to provide voice prompts and / or image video displays for the operation process or final result of displaying the lesion area image of the present invention. The input device 1603 may include, for example, a keyboard, a mouse, a microphone, a gesture capture camera, or other input buttons or controls, which are configured to receive the input of lesion area image data and / or user instructions. The data transmission interface 1604 may include, for example, a serial interface, a parallel interface, or a universal serial bus interface ("USB"), a small computer system interface ("SCSI"), serial ATA, FireWire, PCI Express, and a high-definition multimedia interface ("HDMI"), etc., which are configured for data transmission and interaction with other devices or systems. According to the solution of the present invention, the data transmission interface 1604 can receive the lesion area image or lesion area image data from a CT device (such as Figure 1 the CT device 102 shown therein), and transmit to the device 1600 the image data including the lesion area or various other types of data or results.

[0112] The above-mentioned CPU 1611, mass storage 1612, read-only memory ROM 1613, TPU 1614, GPU 1615, FPGA 1616, MLU 1617, and communication interface 1618 of the device 1600 according to the present invention can be interconnected via a bus 1619, and data interaction can be achieved with peripheral devices via this bus. In one embodiment, via this bus 1619, the CPU 1611 can control other hardware components and their peripheral devices in the device 1600.

[0113] The above combination Figure 16 has described a device that can be used to perform the evaluation of lung lesion areas based on a neural network model of the present invention. It should be understood that the device structure or architecture here is merely exemplary, and the implementation manner and implementation entity of the present invention are not limited by it, but can be changed without departing from the spirit of the present invention.

[0114] It should also be understood that any module, unit, component, server, computer, terminal, or device that executes instructions in the examples of the present invention may include or otherwise access a computer-readable medium, such as a storage medium, a computer storage medium, or a data storage device (removable) and / or non-removable, such as a magnetic disk, an optical disk, or a magnetic tape. A computer storage medium may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0115] It should be understood that the terms "first", "second", "third", and "fourth", etc. in the claims, the description, and the drawings of the present invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the description and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0116] It should also be understood that the terms used in the description of the present invention herein are merely for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the description and claims of the present invention, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms. It should be further understood that the term "and / or" used in the description and claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0117] As used in this specification and the claims, the term "if" can be construed contextually as "when" or "once" or "in response to determining" or "in response to detecting". Similarly, the phrase "if determined" or "if [described condition or event] is detected" can be construed contextually to mean "once determined" or "in response to determining" or "once [described condition or event] is detected" or "in response to detecting [described condition or event]".

[0118] Although the embodiments of the present invention are as described above, the above is only an example for the convenience of understanding the present invention, and is not intended to limit the scope and application scenarios of the present invention. Any person skilled in the art within the technical field of the present invention may make any modifications and changes in the form of implementation and details without departing from the spirit and scope disclosed by the present invention. However, the scope of patent protection of the present invention shall still be subject to the scope defined by the appended claims.

Claims

1. An apparatus for evaluating a pulmonary lesion area based on a neural network model, comprising: One or more processors; A first neural network module; A second neural network module; And One or more computer-readable storage media storing program instructions for implementing the first neural network module and the second neural network module, which, when executed by the one or more processors, cause: The first neural network module receives and processes image data related to a pulmonary lesion area image to obtain target vector data, where the image data includes original data related to the pulmonary lesion area image and / or two-dimensional data related to the geometric features of the pulmonary lesion area image; and The second neural network module receives and processes the target vector data to output an evaluation result for evaluating the pulmonary lesion area; Wherein, the method for obtaining image data related to the pulmonary lesion area image includes: Obtaining three-dimensional image data of the pulmonary lesion area; Generating a two-dimensional grid formed by connecting multiple vertices according to the three-dimensional image data of the pulmonary lesion area; Determining a grid-based Laplace-Beltrami matrix according to the topology of the two-dimensional grid and the side lengths of the grid edges; Calculating the Laplace-Beltrami matrix to obtain a non-zero eigenvalue with the smallest absolute value of the Laplace-Beltrami matrix, thereby determining the eigenfunction corresponding to the non-zero eigenvalue with the smallest absolute value; Determining the function values at each grid vertex on the two-dimensional grid according to the eigenfunction; Selecting the vertex with the globally minimum function value from all vertices as the starting point of the closed curve; Selecting the vertex with the locally minimum function value from multiple vertices adjacent to the starting point as the first intermediate vertex of the closed curve; For each intermediate vertex from the second to the Nth intermediate vertex of the closed curve, perform the following selection operation until returning to the starting point: Selecting the vertex with the locally minimum function value from multiple vertices adjacent to the (N - 1)th intermediate vertex as the Nth intermediate vertex of the closed curve, where N is a positive integer greater than or equal to 2; Cutting the two-dimensional grid into two topological disks along the closed curve; Mapping the two topological disks to two unit rectangles respectively; Using the geometric feature values of multiple vertices to determine the pixel values of corresponding points on the unit rectangle; and Forming a picture containing the geometric features of the pulmonary lesion area based on the pixel values.

2. The apparatus according to claim 1, wherein the image data related to the pulmonary lesion area image includes multiple sets of image sub-data related to the pulmonary lesion area obtained at multiple different times.

3. The apparatus according to claim 2, wherein the one or more computer-readable storage media further store program instructions for obtaining the two-dimensional data, which, when executed by the one or more processors, cause: Generating a two-dimensional grid based on the original data; and Using the two-dimensional grid to determine geometric features and representing the geometric features as a picture as the two-dimensional data.

4. The device according to claim 3, wherein the geometric feature includes Gaussian curvature, mean curvature or conformal factor obtained based on the image of the lung lesion area.

5. The device according to claim 1, wherein the image of the lung lesion area is an image of the lung area infected with novel coronavirus, and the first neural network module includes a plurality of encoders and feature extractors, wherein: Each of the plurality of encoders includes a plurality of convolutional layers configured to perform multi-layer convolutional processing on the image data to obtain a plurality of feature vectors for different geometric features from the image data; and The feature extractor is configured to perform a feature fusion operation on the plurality of feature vectors to obtain the target vector data.

6. The device according to claim 5, wherein the plurality of convolutional layers are connected in series, and the output end of the last convolutional layer connected in series is connected to the input end of the feature extractor.

7. The device according to claim 5, wherein the feature fusion operation includes performing a data splicing operation on the plurality of feature vectors so as to output the target vector data.

8. The device according to claim 7, wherein the second neural network module includes a long short-term memory neural network configured to receive and process the target vector data to output an evaluation result for evaluating the lung lesion area.

9. The device according to any one of claims 2-8, wherein the evaluation result includes lesion quality information of the lung lesion area, and the lesion quality information is at least used to predict or judge the severity of the condition and / or the trend of disease development of a patient infected with novel coronavirus.

Images