An evaluation system and related products for evaluating pulmonary lesion areas based on neural networks

Abstract

The present invention relates to an evaluation system and related products for evaluating a pulmonary lesion area based on a neural network. The evaluation system includes a processing subsystem and a neural network subsystem, wherein the processing subsystem includes one or more processors, and the neural network subsystem includes a first neural network unit and a second neural network unit, wherein the first neural network unit receives and processes tensor data related to the geometric features of the pulmonary lesion area image to obtain target vector data. The second neural network unit receives and processes the target vector data to output an evaluation result for evaluating the pulmonary lesion area. By using the solution of the present invention, high-order geometric features of the pulmonary lesion area can be extracted, and pulmonary diseases including novel coronavirus infection can be effectively evaluated and predicted.

Description

Technical Field

[0001] The present invention generally relates to the field of image processing. More specifically, the present invention relates to an evaluation system, a computing device, and a computer-readable storage medium for evaluating a pulmonary lesion area based on a neural network. Background Art

[0002] As is well known, the image of the pulmonary lesion area contains rich information helpful for the clinical diagnosis of lung diseases. Therefore, it is particularly important to effectively extract and analyze the image features of the pulmonary lesion area. Currently, the traditional processing method is to extract the imaging features of the lesion area and use the imaging features for subsequent analysis and research in order to evaluate the lesion area. However, how to effectively extract the features of the lesion area and effectively evaluate and predict the lesion area based on these features has become an urgent problem to be solved, especially when the pulmonary lesion area includes the area infected with the novel coronavirus. Summary of the Invention

[0003] To at least solve the above technical problems, the present invention provides a device for evaluating a pulmonary lesion area based on a neural network model. Specifically, the present invention uses neural network-based technology to receive and process image data to output an evaluation result for evaluating the pulmonary lesion area. Using this evaluation result, the solution of the present invention can predict the development of the pulmonary lesion area over time. To this end, the present invention provides corresponding solutions in the following aspects.

[0004] In a first aspect, an evaluation system for evaluating a pulmonary lesion area based on a neural network model according to the present invention includes: one or more processors; a first neural network unit; a second neural network unit; and one or more computer-readable storage media storing program instructions for implementing the first neural network unit and the second neural network unit. When the program instructions are executed by the one or more processors, it is caused that: the first neural network unit receives and processes image data related to the image of the pulmonary lesion area to obtain target vector data, where the image data includes raw data related to the image of the pulmonary lesion area and / or tensor data related to the geometric features of the image of the pulmonary lesion area; and the second neural network unit receives and processes the target vector data to output an evaluation result for evaluating the pulmonary lesion area.

[0005] In one embodiment, the lung lesion area image is an image of a lung area infected with the new coronavirus, and the first neural network unit includes multiple encoders and feature extractors, wherein: each of the multiple encoders includes multiple convolution layers, which are configured to perform multi-layer convolution processing on the image data to obtain multiple feature vectors for different geometric features from the image data; and the feature extractor is configured to perform a feature fusion operation on the multiple feature vectors to obtain the target vector data.

[0006] In one embodiment, the multiple convolutional layers are connected in series, and the output of the last convolutional layer connected in series is connected to the input of the feature extractor.

[0007] In one embodiment, the feature fusion operation includes performing a data concatenation operation on the multiple feature vectors to output the target vector data.

[0008] In one embodiment, the second neural network unit includes a long short-term memory neural network configured to receive and process the target vector data to output an evaluation result for evaluating the lung lesion area.

[0009] In one embodiment, the image data related to the lung lesion area image includes a plurality of groups of image sub-data related to the lung lesion area acquired at a plurality of different time points.

[0010] In one embodiment, the tensor data includes three-dimensional tensor data, and the one or more computer-readable storage media also store program instructions for obtaining the three-dimensional tensor data. When the program instructions are executed by the one or more processors, they enable: generating a tetrahedral mesh based on the original data; and determining geometric features using the tetrahedral mesh, and representing the geometric features as three-dimensional tensor data.

[0011] In one embodiment, the geometric features include Ricci curvature, gradient or mean curvature, and the evaluation results include lesion quality information of the lung lesion area, which is at least used to predict or judge the severity and / or disease development trend of patients infected with the new coronavirus.

[0012] In a second aspect, the invention provides a computing device comprising an evaluation system as described above.

[0013] In a third aspect, the present invention provides a computer-readable storage medium comprising a computer program for evaluating a lung lesion area based on a neural network model, wherein when the computer program is executed by one or more processors of a device, the device performs the operations of the aforementioned evaluation system.

[0014] Through the description of the solution of the present invention in multiple aspects above, those skilled in the art can understand that the solution of the present invention can efficiently utilize neural network technology to analyze and evaluate image data, so as to make a reasonable evaluation and prediction of the development of the lung lesion area included in the image. In an application scenario, when the lung lesion area includes a lesion area infected with the novel coronavirus, by using the device of the present invention to evaluate it, the severity and possible progression of the novel coronavirus infection can be predicted, so as to provide effective medical intervention for patients. Further, the tensor data of the present invention includes data extracted from the geometric features of the lung lesion area, making the obtained evaluation result more interpretable for the patient's condition, thus making the evaluation result more accurate and more referenceable. In addition, the neural network unit of the present invention uses feature fusion operations to fuse data, so as to effectively extract and process the features in the image data, thereby improving the accuracy of prediction and evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the drawings, several embodiments of the present invention are shown in an exemplary rather than restrictive manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0016] Figure 1 is a system architecture diagram showing the evaluation of the lung lesion area based on a neural network according to an embodiment of the present invention;

[0017] Figure 2 is a flowchart showing the method for processing the lung lesion area image according to an embodiment of the present invention;

[0018] Figure 3 is an exemplary flowchart showing the method for generating a two-dimensional grid according to an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram showing a tetrahedral mesh according to an embodiment of the present invention;

[0020] Figure 5 is a flowchart showing the method for replacing voxel values with geometric feature values according to an embodiment of the present invention;

[0021] Figure 6 is an exemplary schematic diagram showing some mesh vertices and their adjacent edges according to an embodiment of the present invention;

[0022] Figure 7 is an operation block diagram showing a first neural network unit according to an embodiment of the present invention;

[0023] Figure 8Shows an operational block diagram of an encoder according to an embodiment of the present invention;

[0024] Figure 9 Is an operational block diagram showing a first neural network unit and a second neural network unit according to an embodiment of the present invention;

[0025] Figure 10 Is a schematic diagram of the operation principle of the second neural network unit according to an embodiment of the present invention; and

[0026] Figure 11 Is a block diagram of a computing device for evaluating a lung lesion area according to an embodiment of the present invention. Detailed implementation manners

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the embodiments described in this specification are only some embodiments provided by the present invention for the convenience of clear understanding of the solution and compliance with legal requirements, rather than all embodiments that can implement the present invention. All other embodiments obtained by those skilled in the art based on the embodiments disclosed in this specification without creative efforts fall within the scope of protection of the present invention.

[0028] Figure 1 Is a system architecture diagram for evaluating a lung lesion area based on a neural network according to an embodiment of the present invention.

[0029] As shown in the figure, the system includes a computed tomography scanner for a computer (i.e., a "CT" scanner) 102, which is used to scan the layers of the diseased part or suspected diseased part of a patient to obtain three-dimensional stereoscopic image data. In the context of the present invention, the diseased part here can be the lungs, especially the lung area that may or has been infected with the novel coronavirus. Thus, through the scanning of the CT scanner, the three-dimensional image data 104 of the present invention can be obtained, as shown in the figure, the lung area infected with the novel coronavirus.

[0030] After obtaining the above-mentioned three-dimensional image data, the evaluation system 106 of the present invention (which is arranged on the computer shown in the figure) uses a memory to save the three-dimensional image data. Although not shown in the figure, in some scenarios, some preprocessing can also be performed on the three-dimensional image data before saving, for example, including triangulating the three-dimensional image data to obtain, for example, a two-dimensional network.

[0031] Furthermore, as shown in Figure 1As shown, the evaluation system 106 of the present invention may include a processing subsystem 112 and a neural subsystem 114. In one embodiment, the processing subsystem includes one or more processors, which may include a general-purpose processor ("CPU") or a dedicated graphics processing unit ("GPU"). Further, the neural network subsystem 114 of the present invention may include a first neural network unit 112 and a second neural network unit 114.

[0032] As an example, the above-mentioned first neural network unit and second neural network unit of the present invention may be implemented as program instructions stored on a computer-readable storage medium (not shown in the figure). According to different application scenarios, the computer-readable storage medium here may be one or more, and may be various storage media capable of storing program instructions. During the execution of the evaluation task of the present invention, the processor may execute the program instructions stored on the computer-readable storage medium, so that the operation performed by the first neural network unit and the second neural network unit of the present invention is realized by the running of the program instructions.

[0033] Specifically, when the processor executes the foregoing one or more program instructions, the first neural network unit of the present invention may be configured to receive and process image data related to the lung lesion area image to obtain target vector data. In one embodiment, the image data includes tensor data related to the geometric features of the lung lesion area image (such as Figure 1 the lung lesion area image 104 shown on the left in the figure). Correspondingly, the second neural network unit of the present invention may be configured to receive and process the target vector data to output an evaluation result for evaluating the lung lesion area (for example, reflected in the form of a quality or volume ratio).

[0034] In one application scenario, the above-mentioned image data may also be raw data related to the lung lesion area and / or two-dimensional data related to the geometric features of the lung lesion area. In one embodiment, the raw data of the image area related to the lung lesion area of the patient may be CT image data obtained by, for example, computed tomography technology or equipment. In one implementation scenario, the image data related to the lung lesion area image includes multiple sets of image sub-data related to the lung lesion area acquired at multiple different times (such as Figure 1 shown as 104 in the figure).

[0035] Based on the above description, the above-mentioned computer-readable storage medium also stores program instructions for obtaining the tensor data. When the program instructions are executed by the one or more processors, a tetrahedral mesh is generated based on the raw data; and the geometric features are determined using the tetrahedral mesh and the geometric features are represented as three-dimensional tensor data.

[0036] In one embodiment, the first neural network unit in the device of the present invention may include a plurality of encoders and feature extractors (such as Figure 7 shown in). In one implementation scenario, each of the foregoing plurality of encoders may include a plurality of convolutional layers configured to perform multi-layer convolutional processing on the image data to obtain a plurality of feature vectors for different geometric features from the image data. In one embodiment, the plurality of convolutional layers in each encoder may be serially connected, and the output end of the last convolutional layer in the serial connection may be connected to the input end of the feature extractor. In one implementation scenario, the feature extractor may be configured to perform a feature fusion operation on the foregoing plurality of feature vectors to obtain target vector data. As an example, the feature fusion operation may include performing a data splicing operation on the plurality of feature vectors so as to output the target vector data.

[0037] In one embodiment, the second neural network unit of the device of the present invention includes a long short-term memory neural network (“Long Short-Term Memory”, abbreviated as “LSTM”), which is configured to receive and process the above-mentioned target vector data to output an evaluation result for evaluating the lung lesion area. In one application scenario, the evaluation result may include lesion quality information of the lung lesion area, and the lesion quality information is at least used to predict or judge the severity and / or disease development trend of a patient infected with the novel coronavirus.

[0038] As described above in connection with Figure 1 the evaluation system of the present invention and the devices therein. Next, a detailed description will be given of how the present invention extracts the foregoing geometric features in connection with Figures 2 - 6 FIG.

[0039] Figure 2 is a flowchart of a method 200 for processing an image of a lung lesion area according to multiple embodiments of the present invention. It should be noted that the method 200 of the present invention can be implemented by various computing devices including, for example, a computer, and the three-dimensional image data of the lung lesion area involved therein can be three-dimensional image data obtained by, for example, computed tomography (abbreviated as “CT”) technology or equipment. Further, the three-dimensional image data of the lung lesion area of the present invention contains a cube structure, such as volume elements (abbreviated as “voxels”).

[0040] As is known to those skilled in the art, voxels are mainly used in the fields of three-dimensional imaging, scientific data, and medical imaging. A voxel is the smallest unit that digital data can be segmented and recognized in three-dimensional space. Further, the value of a voxel (abbreviated as "voxel value") can represent different characteristics. For example, in a "CT" image, the aforementioned voxel value is the Hounsfield Unit (abbreviated as "HU").

[0041] As Figure 2 As shown in [Figure], at step 202, method 200 can obtain three-dimensional image data of a pulmonary lesion region. In one embodiment, the three-dimensional image data can be obtained by a device supporting CT technology and the voxel values of the present invention can be obtained through calculation. In this case, the voxel value is the gray value of the image (i.e., the gray value discussed in the embodiments of the present invention hereinafter). Additionally, the aforementioned gray value can be converted to obtain a CT value, whose unit is the aforementioned Hounsfield Unit.

[0042] After obtaining the three-dimensional image data based on, for example, the CT technology discussed above, at step 204, method 200 can generate a tetrahedral mesh formed by connecting multiple vertices according to the three-dimensional image data of the pulmonary lesion region. In one embodiment, the operation of generating a tetrahedral mesh in the present invention can include generating the boundary of the tetrahedral mesh and then generating the internal vertices of the tetrahedral mesh. In this case, the boundary of the tetrahedral mesh can be a two-dimensional mesh generated according to the boundary of the three-dimensional image data, and the internal vertices of the tetrahedral mesh can be the vertices of voxels. Further, by using the generated two-dimensional mesh as the boundary of the tetrahedral mesh, prior information on the outer surface of the tetrahedral mesh can be obtained by using this boundary, thereby accelerating the generation speed of the tetrahedral mesh. Further, through constructing (or reconstructing) a tetrahedral mesh in the embodiments of the present invention, the shape of the pulmonary lesion region can be accurately described. Further, the present invention can determine the positions of higher-order geometric parameters (such as gradients) through this tetrahedral mesh, so as to provide more accurate data for subsequent analysis of the pulmonary lesion region. In one embodiment, the operation of generating a tetrahedral mesh can also be automatically implemented through a software package. For example, a software package including the function of Constrained Delaunay Tetrahedralization (abbreviated as "CDT") can be selected to directly generate a tetrahedral mesh.

[0043] It should be noted that there is a missing figure reference in the original text at "[Figure]" in the translation of item . Please check and correct it according to the actual situation.After generating the tetrahedral mesh through step 204, the process proceeds to step 206. At this step 206, method 200 uses the voxel values at the vertices to determine the geometric feature values at the vertices. As previously mentioned, the voxel values at the vertices can be directly obtained through a device or equipment that supports, for example, CT technology, and the obtained voxel values are usually the gray values of a CT image (i.e., the lung lesion area image in the embodiments of the present invention), and this gray value can be any corresponding value between 0 and 255. According to one or more embodiments of the present invention, the aforementioned geometric features may include, but are not limited to, Ricci curvature, gradient, or mean curvature. Then, at step 208, method 200 uses the geometric feature values to replace the voxel values to achieve the extraction of the geometric features of the lung lesion area.

[0044] According to different implementation scenarios, the geometric feature of the present invention can be one of the aforementioned Ricci curvature, gradient, or mean curvature, and accordingly, the Ricci curvature value, gradient value, or mean curvature value at the vertex is calculated. Based on this, the Ricci curvature value, gradient value, or mean curvature value at the vertex of the tetrahedral mesh obtained in the previous step 206 is used as the gray value at the vertex to replace the voxel value. In one embodiment, the obtained Ricci curvature value, gradient value, or mean curvature value can be tensor data with multiple dimensions, such as three-dimensional tensor data, for operations such as feature extraction of a deep convolutional network.

[0045] As described above in conjunction with Figure 2 the geometric feature extraction of the lung lesion area of the present invention has been described. Based on the above description, those skilled in the art can understand that the present invention reconstructs a tetrahedral mesh for the lesion area and calculates geometric feature values, such as Ricci curvature values, gradient values, or mean curvature values, based on the voxel values at the vertices of the tetrahedral mesh. Further, the obtained geometric feature values are used to replace the voxel values at the vertices, thereby extracting geometric features. In one embodiment, the geometric feature can be represented as three-dimensional tensor data for subsequent research and analysis, including, for example, training and prediction evaluation of a neural network model.

[0046] Combined with the above description, the present invention extracts the high-order geometric features of the lung lesion area (including the images of novel coronavirus infection), so that the obtained geometric feature data contains richer feature information, and thus can reflect the essential geometric attributes of the lung lesion image area. At the same time, compared with traditional feature extraction, the high-order geometric features extracted by the present invention based on the lung lesion area images are more interpretable and can be used to evaluate lung diseases from multiple aspects and angles. Further, by using the high-order geometric features obtained by the present invention and representing them as three-dimensional tensors as the training data of machine learning algorithms such as deep neural networks, a prediction model for the development trend of the lesion can be trained, so as to accurately predict the development of the lesion area for effective human intervention.

[0047] Figure 3 is an exemplary flowchart showing a method 300 for generating a two-dimensional grid according to multiple embodiments of the present invention. Combined with the above description of Figure 2 , those skilled in the art can understand that the boundary of the tetrahedral grid can be a two-dimensional grid generated according to the boundary of the three-dimensional image data. Thus, the present invention proposes to use Figure 3 the method 300 shown to obtain the aforementioned two-dimensional grid. It should be noted here that the method 300 is Figure 2 a specific implementation manner of some steps in the method 200 shown, so the corresponding description of the method 200 also applies to the discussion of the method 300 below.

[0048] As Figure 3 shown, at step 302, the method 300 uses a boolean variable to mark the three-dimensional image area, such as the three-dimensional image area including novel coronavirus infection. In one implementation scenario, those skilled in the art can understand that the generation of the two-dimensional grid is essentially the generation of the grid on the outer surface of the boolean variable (bool) data. Specifically, a three-dimensional image area of the lesion area can be marked with bool and set to , where is a smooth function, is the area where the three-dimensional image exists. Then, at step 304, the method 200 generates the two-dimensional grid according to the marked three-dimensional image area. For example, based on the aforementioned marked three-dimensional image area , the internal voxels can be represented by , the external voxels can be represented by , and (where 0 < < 1) represents the boundary of the three-dimensional image data, and the function is calculated by smooth interpolation Isosurface mesh. In one implementation scenario, the foregoing interpolation can be performed using, for example, the Computational Geometry Algorithms Library ("CGAL") to generate a two-dimensional mesh.

[0049] After generating the above two-dimensional mesh, those skilled in the art can perform the foregoing "CDT" processing on the two-dimensional mesh to ensure the boundary consistency between the two-dimensional mesh and the tetrahedral mesh. That is, the generated two-dimensional mesh is exactly the boundary of the tetrahedral mesh. Further, those skilled in the art can impose stronger restrictions on the tetrahedral mesh, i.e., using the voxel vertices of the three-dimensional data as the internal vertices of the tetrahedral mesh. Thus, based on the obtained two-dimensional mesh and voxel vertices, a tetrahedral mesh formed by connecting multiple vertices is finally generated. For ease of understanding, Figure 4 An exemplary schematic diagram showing a part of the tetrahedral mesh generated according to an embodiment of the present invention is shown. According to the obtained tetrahedral mesh, the gray value at each vertex of the mesh can be used to determine the geometric feature value at the vertex of the mesh. In one embodiment, the geometric feature value can be a Ricci curvature value, a gradient value, or an average curvature value. Further, the foregoing geometric feature value is used to replace the voxel value at the vertex of the tetrahedral mesh.

[0050] Figure 5 is a flowchart of a method 500 for replacing voxel values with geometric feature values according to multiple embodiments of the present invention. It should be understood here that the method 500 is Figure 2 a specific implementation of some steps in the method 200 shown, so the corresponding descriptions made about the method 200 also apply to the method 500.

[0051] According to the foregoing description, after generating a tetrahedral mesh formed by connecting multiple vertices, at step 502, the method 500 can calculate the Ricci curvature value, gradient value, or average curvature value at the vertex of the tetrahedral mesh according to the voxel value at the vertex of the generated tetrahedral mesh. In one implementation scenario, the Ricci curvature value can be calculated through the following described mathematical operations. First, the weights of the edges adjacent to the vertices in the tetrahedral mesh can be defined , and it is expressed as:

[0052]

[0053] where, represents the weight of the edge , and represent the weights at the vertices and respectively, represents all the weights related to the vertex Adjacent edges (excluding edge ), represent all the edges adjacent to vertex (excluding edge ). For ease of understanding, Figure 6 FIG. shows an exemplary schematic diagram of some mesh vertices and their adjacent edges according to multiple embodiments of the present invention.

[0054] As Figure 6 shown, and can represent two vertices sharing an edge in the above-generated tetrahedral mesh, where is the connecting edge between vertex and vertex also includes its adjacent edges , and . Similarly, vertex also includes its adjacent edges , and . In one embodiment, the weight at vertex is defined as , and the weight at vertex is defined as . The aforementioned weights and can be the voxel values (i.e., gray values) at vertex and vertex . Thus, based on the weights and at vertex and , the weight and of the edge shared by can be obtained:

[0055]

[0056] Combining the above formula (1) and formula (2), the weights of the edges adjacent to the vertices in the tetrahedral mesh can be obtained. Based on the aforementioned obtained weights , further, the Ricci curvature Ric at each vertex can be obtained according to the following formula:

[0057]

[0058] In the above formula (3), represents the edge adjacent to vertex , Denote all the edges adjacent to the vertex, which can represent the number of, that is, the number of edges adjacent to the point. In this case, the calculated result of the Ricci curvature value is a numerical value.

[0059] In another embodiment, the weights of the edges adjacent to the vertices in the tetrahedral mesh can be calculated only based on the above formulas (1) and (2). For example, the weights on three mutually orthogonal axes (i.e., the x-axis, y-axis, and z-axis) at the vertex can be calculated respectively, and the weights of these three axes are used as the Ricci curvature value. The aforementioned three-axis weights can represent the tensor data of a three-dimensional tensor. Thus, the Ricci curvature value can be represented as a three-dimensional tensor.

[0060] The above has given an exemplary description of how to calculate the Ricci curvature value. Regarding the gradient value involved in the above-mentioned geometric eigenvalue, in one implementation scenario, the tetrahedral mesh can be first convolved with a Gaussian function, and its gradient can be calculated based on the convolved tetrahedral mesh. Further, the modulus of the obtained gradient is calculated. The gradient value of the tetrahedral mesh is calculated and represented mathematically as . Specifically, represents a Gaussian distribution with variance , represents convolution, represents the voxel value (i.e., grayscale value) in the tetrahedral mesh. For the Gaussian convolution operation, those skilled in the art can directly call the Gaussian filtering function through image processing software (such as MATLAB) for calculation. It should be understood that in this case, the aforementioned obtained gradient value at the vertex is a real number. Those skilled in the art can also calculate the partial derivatives of the three axes (i.e., the x-axis, y-axis, and z-axis) at the vertex respectively, and use the partial derivatives on these three axes as the tensor data on the three dimensions of a three-dimensional tensor. Thus, the gradient value can also be represented as a three-dimensional tensor.

[0061] Furthermore, regarding the mean curvature also involved in the above-mentioned geometric eigenvalue, in yet another implementation scenario, assume that the lesion area image is a function , and the normal vector of the isosurface where the vertex is located is , then the mean curvature K at the vertex can be defined as:

[0062]

[0063] The above isosurface can be understood as a surface composed of a set of points with the same grayscale value. For three-dimensional data, it can be regarded as a set of multiple isosurfaces. It should be understood that the average curvature obtained based on the above definition is a real number. Therefore, the average curvature can be directly expressed as a real number calculated based on formula (4).

[0064] Back to Figure 5 , based on the above-mentioned Ricci curvature value, gradient value or average curvature value, method 500 then proceeds to step 504. At step 504, method 400 replaces the voxel value at the vertex with the Ricci curvature value, gradient value or average curvature value. Specifically, the voxel value at each vertex can be replaced with the Ricci curvature value, gradient value or average curvature value obtained above, where the Ricci curvature value and gradient value can be expressed as a three-dimensional tensor. Thus, the present invention realizes the extraction of geometric features of lung lesion areas, especially new coronavirus infection lesions, by determining the Ricci curvature value, gradient value or average curvature value.

[0065] In combination with the above description, the embodiment of the present invention extracts high-order geometric eigenvalues, such as Ricci curvature values, gradient values ​​or mean curvature values, from the lung lesion area, and represents the Ricci curvature values ​​and gradient values ​​as a three-dimensional tensor. Furthermore, the voxel values ​​of the lung lesion area in the embodiment of the present invention are replaced with the aforementioned geometric eigenvalues, which facilitates subsequent analysis and research. In an implementation scenario, those skilled in the art can use the obtained geometric eigenvalues ​​including the new coronavirus infection images as a data source, apply them to an artificial intelligence architecture such as a neural network, and after training or deep learning, a predictive model for the development trend of the lung lesion area can be obtained. Thus, the high-order geometric features of the present invention can be used to make accurate predictions about the development of the lung lesion area, so that medical staff can provide timely and effective treatment.

[0066] Based on the above Figures 2 - 6 After the described extraction method is used to obtain image data including the geometric features (such as three-dimensional tensor data) or the original data collected by CT technology, since the image data is usually represented by grayscale values ​​in the range of 0 to 255, it is usually necessary to preprocess the acquired image data. In one embodiment, the present invention proposes to normalize the grayscale values ​​of the image data to a floating point number between 0 and 1 using the maximum-minimum (max-min) criterion. Then, the first neural network unit of the present invention receives the preprocessed image data and processes it to obtain target vector data. In the implementation scenario of the geometric feature extraction scheme of the present invention, the aforementioned image data may also be one-dimensional data and / or three-dimensional data related to the geometric features of the target image area.

[0067] Figure 7is an operational block diagram showing a first neural network unit 112 according to an embodiment of the present invention. It should be understood that Figure 7 the first neural network unit shown is Figure 1 a specific implementation of the first neural network unit in the shown evaluation system 106. Thus, regarding Figure 1 the relevant details and features of the described evaluation system 106 also apply to Figure 7 the description of.

[0068] As shown in the figure, different types of image data are represented in the left dotted box in the figure. From top to bottom, they represent raw data 701, three-dimensional data 702, two-dimensional data 703, and one-dimensional data 704 respectively. In one embodiment, the one-dimensional data can be stored in TXT format, and its size can be 1*400 (i.e., 400 data bits per row); the two-dimensional data can be stored in picture (e.g., png) format, and its pixel size can be 256*256, for example. The three-dimensional data can be stored in nii format, and its size can be 512 bits * 512 bits * 512 bits. As mentioned above, the solution of the present invention proposes to normalize the above-mentioned image data using the max-min criterion, so as to keep the processed data format and size unchanged.

[0069] After receiving the above-mentioned image data (e.g., pre-normalized image data), the first neural network unit 112 of the present invention first extracts feature vectors corresponding to different types of image data through different encoders.

[0070] Specifically, encoder 1 processes the raw data 701 to output a feature vector 701-1. Similarly, the three-dimensional data 702, two-dimensional data 703, and one-dimensional data 704 can respectively extract corresponding feature vectors 702-1, feature vector 703-1, and feature vector 704-1 through encoder 2, encoder 3, and encoder 4. It should be understood that Figure 7 the number of dimensions of the image data shown and the number of encoders are merely exemplary and not restrictive. Those skilled in the art can select other image data formats or types according to needs. For example, in some application scenarios, any one of the raw data, one-dimensional to three-dimensional data can be used for evaluation. In other application scenarios, any two or more of the foregoing raw data and one-dimensional to three-dimensional data can be combined for evaluation. Therefore, the present invention does not impose any restrictions on the data format and data usage method, etc. Similarly, the present invention also does not impose any restrictions on the number and type of encoders corresponding to the foregoing data formats.

[0071] In one embodiment, the above encoder of the present invention can be implemented by a convolutional layer (or a convolutional operator) in a neural network. In one implementation scenario, it can be implemented by a layer structure including two convolutional layers and an adaptive convolutional layer as shown in Figure 8 to perform an encoding operation on the data to obtain the feature vector data as described above, which is specifically described as follows.

[0072] Figure 8 Fig. 6 shows an operation block diagram of an encoder 800 according to an embodiment of the present invention. It can be understood that the encoder 800 can be any one of the encoders 1-4 in Figure 7 . As shown in the figure, the encoder 800 can include a convolutional layer 801, a convolutional layer 802, and an adaptive convolutional layer 803. Assume that the data on the left side of the figure is the two-dimensional data 703 in the above figure (for example, a picture represented by the extracted Gaussian curvature, mean curvature, or conformal factor). The encoder 800 is set as the encoder 3 in the above Figure 7 . Thus, the two-dimensional data 703 undergoes a first convolution through the convolutional layer 801 in the encoder 800. Then, it undergoes a second convolution through the convolutional layer 802 and optionally a third convolution through the adaptive convolution 803 to obtain the feature vector 703-1. Similarly, through the encoder processing as described above, feature vectors 702-1, 703-1, and 704-1 as shown in Figure 7 can be obtained.

[0073] According to the actual application scenario, the first two convolutional layers in the above encoder of the present invention can respectively use 128 and 64 convolutional kernels to perform convolution operations. In this case, the inputs can be feature maps of sizes 256*256 and 128*128 respectively, and the outputs can be feature maps of sizes 128*128 and 64*64 respectively. For the third convolutional layer, it can be an adaptive convolutional layer using 32 convolutional kernels, and its output is a feature map of size 32*32. Here, the purpose of adding the adaptive convolutional layer is only to fix the output size of the encoder, that is, to make the encoder of the present disclosure always output a feature map of a fixed size, such as the aforementioned 32*32 feature map. Based on this, those skilled in the art can understand that the adaptive convolutional layer of the present invention is an optional setting, and in some other application scenarios, it can be not used or replaced by another convolutional layer. Further, the size of the convolutional kernels in the neural network of the present invention can all be 3*3 arrays and can be initialized with a uniform distribution.

[0074] Combined with Figure 7 and Figure 8As shown, those skilled in the art can understand that multiple convolutional layers in multiple encoders in the first neural network unit of the present invention are serially connected, and the output end of the last convolutional layer in the serial connection is connected to the input end of the feature extractor of the first neural network unit (i.e., Figure 7 the feature extractor 705 in Figure 8 ). Regarding the convolutional layer structure shown in

[0075] , the output end of the adaptive convolutional layer 803 in the encoder is connected to the input end of the feature extractor, so that the feature extractor performs a data splicing operation on multiple vector data to obtain target vector data. Figure 7 Referring to the feature extractor 705 shown in

[0076] , it performs a feature fusion operation on, for example, the above four feature vectors (i.e., the operation in the middle dotted box in the figure). Specifically, first, a convolution is performed on the feature vector 701-1, the feature vector 702-1, the feature vector 703-1, and the feature vector 704-1 respectively to obtain their respective convolution results, and then the convolution results of each feature vector are fused (e.g., spliced) to obtain the target vector data 706. For example, the convolution results of the feature vector 702-1, the feature vector 703-1, and the feature vector 704-1 are spliced with the convolution result of the feature vector 701-1 to obtain the feature vector 701-2. Similarly, the convolution results of the feature vector 701-1, the feature vector 703-1, and the feature vector 704-1 are spliced with the convolution result of the feature vector 702-1 to obtain the feature vector 702-2. Thus, the feature vector 703-2 and the feature vector 704-2 can also be obtained. Then, for the feature vector 701-2, the feature vector 702-2, the feature vector 703-2, and the feature vector 704-2, the above-mentioned operation is performed multiple times (e.g., twice) to obtain the feature vector 701-10, the feature vector 702-10, the feature vector 703-10, and the feature vector 704-10, and they are spliced to form the target vector data 706. Figure 9 As shown in

[0077] Figure 9 is an operational block diagram showing a first neural network unit and a second neural network unit according to an embodiment of the present invention. The leftmost part in the figure represents the image data of a patient collected at Tn time periods (such as CT1 image data, CT2 image data, and CTn image data). The first neural network unit 112 respectively receives the foregoing image data and processes it, thereby respectively obtaining the target vector data 901 at time T1, the target vector data 902 at time T2, and the target vector data 910 at the T-th time period. Then, the second neural network unit 114 receives the obtained multiple target vector data and processes it to finally obtain the evaluation result of the lesion area. In one embodiment, the second neural network unit may be a Long Short-Term Memory neural network (Long Short-Term Memory, "LSTM"), for example Figure 10 as shown in

[0078] Figure 10 is a schematic diagram showing the operation principle of the second neural network unit 114 according to an embodiment of the present invention. As described above, the second neural network unit of the present invention can be implemented as an LSTM neural network in one implementation scenario, and the LSTM neural network may include an input layer, one or more hidden layers, and an output layer.

[0079] As Figure 10 shown in the time period, the time period, and the time period of the image data, after feature fusion by the first neural network unit, respectively obtain the target vector data , the target vector data , and the target vector data . The LSTM neural network uses the target vector data , the target vector data , and the target vector data as inputs, and uses the target vector data and the memory of the patient's t-1-th CT at the previous moment to calculate the memory of the patient's t-th CT at the current moment . Similarly, the memory of the patient's t+1-th CT can be calculated based on the memory of the patient's t-th CT 。In an implementation scenario, the foregoing operations (such as adjusting the weights U from the input layer to the hidden layer, the weights V from the hidden layer to the output layer, and the weights W from the hidden layer at the previous moment to the hidden layer at the current moment) can be repeated multiple times and the calculations for all moments of the memory can be completed, and finally the output Ot is obtained, which is the evaluation result. Usually, the foregoing evaluation result can be expressed as quality (such as the quality of the lesion area) and volume ratio (the volume ratio of the lesion area to the entire image). By analyzing the evaluation result, the situation of the target features in the image data over time can be determined. For example, when the image data is image data containing a lesion area of a novel coronavirus infection, the current state and development of the novel coronavirus infection can be evaluated and predicted.

[0080] For example, through the foregoing high-order geometric features extracted from the lesion area containing the novel coronavirus infection area of the present invention, the user of the device of the present invention (such as a medical professional) can determine the severity of the current user's condition by analyzing the quality or volume ratio of the lung infection area reflected by the evaluation result. Further, since the neural network unit of the present invention processes data in the time dimension, the user can also evaluate the development of the patient's condition based on the evaluation result. For example, when the quality or volume ratio shows a gradually decreasing or declining trend, it can be judged that the patient is expected to recover within a certain period of time. In contrast, when the quality or volume ratio shows an increasing or rising trend, it can be judged that the patient's condition may deteriorate further to a certain extent. In this case, medical staff can provide necessary treatment to the patient in a timely manner to control the development of the condition and prevent further deterioration of the condition.

[0081] Although the training process of the neural network unit of the present invention is not mentioned above, based on the content disclosed in the present invention, those skilled in the art can understand that the neural network unit of the present invention can be trained with training data to obtain a neural network unit with high precision. For example, in the forward propagation process of neural network training, the present invention can use the image data (such as three-dimensional tensor data) including geometric features obtained by combining Figures 2 - 6 to train the neural network unit of the present invention, and compare the training result with the expected result (or true value) to obtain the corresponding loss function. Further, in the backpropagation process of neural network training, the present invention uses the obtained loss function and updates the weights (such as Figure 10 the weights U, V, and W therein) based on, for example, the gradient descent algorithm to reduce the error between the output Ot and the true value.

[0082] In combination with the above description, by using the image evaluation system according to the embodiments of the present invention, the first neural network unit can perform feature fusion on the image data to obtain target vector data, and the second neural network unit can process the target vector data to obtain the evaluation result of the image. For example, the CT image of a patient can be input into the image evaluation system of the present invention, and thus the evaluation result (such as the quality and volume ratio) of the lesion area of the patient can be directly obtained. By predicting the condition of the patient and the development trend of the lesion area based on this quality or volume ratio, artificial intervention can be facilitated.

[0083] Figure 11 FIG. 4 is a block diagram showing a device 1100 for evaluating a lung lesion area based on a neural network model according to an embodiment of the present invention. As Figure 11 shown, the device 1100 may include a central processing unit ("CPU") 1111, which may be a general-purpose CPU, a dedicated CPU, or other information processing and program execution units. Further, the device 1100 may also include a mass storage 1112 and a read-only memory ("ROM") 1113, where the mass storage 1112 may be configured to store various types of data, such as various image data related to the lesion area, algorithm data, intermediate results, and various programs required to run the device 1100. The read-only memory ("ROM") 1113 may be configured to store data required for power-on self-test of the device 1100, initialization of each functional module in the system, driver programs for basic input / output of the system, and data for booting the operating system.

[0084] Optionally, the device 1100 may further include other hardware platforms or components, such as the shown tensor processing unit ("TPU") 1114, graphics processing unit ("GPU") 1115, field programmable gate array ("FPGA") 1116, and machine learning unit ("MLU") 1117. It can be understood that although various hardware platforms or components are shown in the device 1100 of the present invention, they are merely exemplary rather than restrictive, and those skilled in the art can add or remove corresponding hardware according to actual needs. For example, the device 1100 may include only the CPU to implement the evaluation operation of the lung lesion area of the present invention.

[0085] In some embodiments, for the convenience of data transfer and interaction with an external network, the device 1100 of the present invention further includes a communication interface 1118, so that it can be connected to a local area network / wireless local area network ("LAN / WLAN") 1105 through this communication interface 1118, and then can be connected to a local server 1106 or connected to the Internet ("Internet") 1107 through the LAN / WLAN. Alternatively or additionally, the device 1100 of the present invention can also be directly connected to the Internet or a cellular network based on wireless communication technology through the communication interface 1118, such as based on the wireless communication technology of the third generation ("3G"), the fourth generation ("4G") or the fifth generation ("5G"). In some application scenarios, the device 1100 of the present invention can also access the server 1108 and possibly the database 1109 of the external network as needed, so as to obtain various known image models, data and modules, and can remotely store various data, such as various types of data for presenting or evaluating the image of the lesion area.

[0086] The peripheral devices of the device 1100 of the present invention may include a display device 1102, an input device 1103 and a data transmission interface 1104. In one embodiment, the display device 1102 may include, for example, one or more speakers and / or one or more visual displays, which are configured to perform voice prompts and / or image and video displays on the operation process or the final result of displaying the lesion area image of the present invention. The input device 1103 may include, for example, a keyboard, a mouse, a microphone, a gesture capture camera, or other input buttons or controls, which are configured to receive the input of lesion area image data and / or user instructions. The data transmission interface 1104 may include, for example, a serial interface, a parallel interface or a universal serial bus interface ("USB"), a small computer system interface ("SCSI"), serial ATA, FireWire, PCI Express and a high-definition multimedia interface ("HDMI"), etc., which are configured for data transmission and interaction with other devices or systems. According to the solution of the present invention, the data transmission interface 1104 can receive the lesion area image or lesion area image data from a CT device (such as Figure 1 the CT device 102 shown therein), and transmit to the device 1100 the image data including the lesion area or various other types of data and results.

[0087] The above-mentioned CPU 1111, mass storage 1112, read-only memory ROM 1113, TPU 1114, GPU 1115, FPGA 1116, MLU 1117, and communication interface 1118 of the device 1100 of the present invention can be interconnected through a bus 1119 and achieve data interaction with peripheral devices through this bus. In one embodiment, through this bus 1119, the CPU 1111 can control other hardware components and their peripheral devices in the device 1100.

[0088] The above combination Figure 11 has described a device that can be used to execute the evaluation of lung lesion areas based on a neural network according to the present invention. It should be understood that the device structure or architecture here is only exemplary, and the implementation manner and implementation entity of the present invention are not limited by it, but can be changed without departing from the spirit of the present invention.

[0089] It should also be understood that any module, unit, component, server, computer, terminal, or device that executes instructions in the examples of the present invention can include or otherwise access a computer-readable medium, such as a storage medium, a computer storage medium, or a data storage device (removable) and / or non-removable, such as a magnetic disk, an optical disk, or a magnetic tape. The computer storage medium can include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0090] It should be understood that the terms "first", "second", "third", and "fourth", etc. in the claims, the description, and the drawings of the present invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the description and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0091] It should also be understood that the terms used in the description of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the description and claims of the present invention, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms. It should be further understood that the term "and / or" used in the description and claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0092] As used in this specification and the claims, the term "if" can be construed contextually as "when" or "once" or "in response to determining" or "in response to detecting". Similarly, the phrase "if determined" or "if [described condition or event] is detected" can be construed contextually to mean "once determined" or "in response to determining" or "once [described condition or event] is detected" or "in response to detecting [described condition or event]".

[0093] Although the embodiments of the present invention are as described above, the above content is only an example adopted for the convenience of understanding the present invention, and is not intended to limit the scope and application scenarios of the present invention. Any person skilled in the art within the technical field of the present invention may make any modifications and changes in the form of implementation and details without departing from the spirit and scope disclosed by the present invention. However, the scope of patent protection of the present invention shall still be subject to the scope defined by the appended claims.

Claims

1. An evaluation system for evaluating a lung lesion area based on a neural network, comprising: A processing subsystem, which includes one or more processors; A neural network subsystem, which includes a first neural network unit and a second neural network unit; And One or more computer-readable storage media, which store program instructions for implementing the neural network subsystem. When the program instructions are executed by the one or more processors, it is caused that: The first neural network unit receives and processes image data related to a lung lesion area image to obtain target vector data, wherein the image data includes tensor data related to the geometric features of the lung lesion area image. The first neural network unit includes a plurality of encoders and a feature extractor. Each encoder in the plurality of encoders includes a plurality of convolutional layers, which are configured to perform multi-layer convolutional processing on the image data to obtain a plurality of feature vectors for different geometric features from the image data; and the feature extractor, which is configured to perform a feature fusion operation on the plurality of feature vectors to obtain the target vector data; And The second neural network unit receives and processes the target vector data to output an evaluation result for evaluating the lung lesion area. The second neural network unit includes a long short-term memory neural network. The evaluation result includes the lesion quality information of the lung lesion area, and the lesion quality information is at least used to predict or judge the severity and / or the development trend of the condition of a patient infected with the novel coronavirus; The tensor data includes three-dimensional tensor data, and the one or more computer-readable storage media further store program instructions for obtaining the three-dimensional tensor data. When the program instructions are executed by the one or more processors, it is caused that: Generate a tetrahedral mesh formed by connecting a plurality of vertices according to the three-dimensional image data of the lung lesion area, wherein the operation of generating the tetrahedral mesh includes generating the boundary of the tetrahedral mesh and generating the internal vertices of the tetrahedral mesh; the boundary of the tetrahedral mesh is a two-dimensional mesh generated according to the boundary of the three-dimensional image data, and the internal vertices of the tetrahedral mesh are voxel vertices; Use the voxel values at the vertices to determine the geometric feature values at the vertices; And Use the geometric feature values to replace the voxel values to achieve the extraction of the geometric features of the lung lesion area; Represent the geometric features as three-dimensional tensor data; the three-dimensional tensor data is used for feature extraction.

2. The evaluation system according to claim 1, wherein the lung lesion area image is an image of a lung area infected with the novel coronavirus.

3. The evaluation system according to claim 1, wherein the plurality of convolutional layers are serially connected, and the output end of the last convolutional layer in the serial connection is connected to the input end of the feature extractor.

4. The evaluation system according to claim 1, wherein the feature fusion operation includes performing a data splicing operation on the plurality of feature vectors so as to output the target vector data.

5. The evaluation system according to claim 1, wherein the image data related to the image of the lung lesion area includes multiple sets of image sub-data related to the lung lesion area acquired at multiple different times.

6. The evaluation system according to claim 1, wherein the geometric features include Ricci curvature, gradient or mean curvature.

7. A computing device, comprising the evaluation system according to any one of claims 1-6.

Images