Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an image processing method and device, an electronic device and a storage medium. The image processing method comprises: extracting a center line of a target object in a to-be-processed image; generating a target image of the target object based on the center line; performing convolution processing on the target image to obtain classification results of a plurality of first pixel points in the target image; and dividing the target object into a plurality of first regions according to the classification results of the plurality of first pixel points. The present disclosure can automatically realize naming of the target object by means of convolution processing on the target image, thereby reducing labor costs. Moreover, the classification results of the plurality of first pixel points in the target image are obtained by performing convolution processing on the target image of the target object generated based on the center line of the target object, so that the classification results are more in line with actual distribution characteristics of the target object, and the plurality of first regions divided based on the classification results have higher accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to an image processing method and apparatus, electronic device and storage medium. Background Technology

[0002] The bronchi are the passageways that carry air from the respiratory tract to the lungs and play a vital role in immune function. The bronchi are associated with many diseases, such as bronchitis, asthma, and pulmonary nodules. Doctors typically simulate the distribution of the trachea based on CT scans. This process usually requires manually naming the bronchial segments, which is both inefficient and inaccurate. Summary of the Invention

[0003] This disclosure proposes an image processing technology solution.

[0004] According to one aspect of this disclosure, an image processing method is provided, comprising: extracting the centerline of a target object in an image to be processed; generating a target image of the target object based on the centerline; performing convolution processing on the target image to obtain a classification result of a plurality of first pixels in the target image, wherein the classification result includes the name of the region to which the first pixel belongs; and dividing the target object into a plurality of first regions according to the classification result of the plurality of first pixels.

[0005] In one possible implementation, the convolution processing of the target image to obtain classification results for a plurality of first pixels in the target image includes: performing global graph convolution processing on the target image to obtain global features and a global adjacency matrix of the target image, wherein the global adjacency matrix includes feature distances between a plurality of first pixels in the target image; performing local graph convolution processing on the target image based on the global adjacency matrix to obtain local features of the target image; and performing classification processing on the plurality of first pixels in the target image based on the global features and the local features to obtain classification results for the plurality of first pixels.

[0006] In one possible implementation, the step of performing global graph convolution processing on the target image to obtain the global features and global adjacency matrix of the target image includes: performing K1 times of global graph convolution processing on the target image to obtain the global features and global adjacency matrix, wherein the global features are obtained through the K1th global graph convolution processing, and the global adjacency matrix is ​​obtained through the first global graph convolution processing, where K1 is a positive integer.

[0007] In one possible implementation, when i is less than or equal to K1 and greater than 1, the i-th global graph convolution process includes: performing edge graph convolution processing on the intermediate global features output after the (i-1)-th global graph convolution process to obtain a first edge graph convolution processing result and an intermediate adjacency matrix; and performing fitting processing on the intermediate global features output after the (i-1)-th global graph convolution process based on the intermediate adjacency matrix and the first edge graph convolution processing result to obtain the intermediate global features output after the i-th global graph convolution process, wherein the fitting processing includes convolution processing and / or regularization processing.

[0008] In one possible implementation, the step of performing local graph convolution processing on the target image based on the global adjacency matrix to obtain local features of the target image includes: performing K2-fold local graph convolution processing on the target image based on the global adjacency matrix to obtain local features, wherein the local features are obtained through the K2-fold local graph convolution processing, and K2 is a positive integer.

[0009] In one possible implementation, when j is less than or equal to K2 and greater than 1, the j-th local graph convolution process includes: performing edge graph convolution processing on the intermediate local features output after the (j-1)-th local graph convolution process to obtain a second edge graph convolution processing result; and performing fitting processing on the intermediate local features output after the (j-1)-th local graph convolution process according to the global adjacency matrix and the second edge graph convolution processing result to obtain the intermediate local features output after the j-th local graph convolution process, wherein the fitting processing includes convolution processing and / or regularization processing.

[0010] In one possible implementation, the target image includes a tree topology graph; the convolution processing of the target image to obtain classification results for multiple first pixels in the target image includes: dividing the tree topology graph into multiple sampling regions according to the hierarchical structure of the tree topology graph; sampling pixels in the multiple sampling regions respectively; generating a sampling image based on the obtained multiple sampling points; performing graph convolution processing on the sampling image to obtain classification results for multiple sampling points in the sampling image; and determining the classification results for multiple first pixels in the target image based on the classification results of the multiple sampling points.

[0011] In one possible implementation, the target image includes a tree topology graph; the convolution processing of the target image to obtain classification results for multiple first pixels in the target image includes: dividing the target object in the image to be processed into multiple sampling regions according to the hierarchical structure of the tree topology graph; sampling pixels in the multiple sampling regions respectively; generating a sampling image based on the obtained multiple sampling points and amplified sampling points, wherein the amplified sampling points are obtained by perturbing the sampling points, the perturbing process including translation and / or scaling; performing graph convolution processing on the sampling image to obtain classification results for multiple sampling points in the sampling image; and determining the classification results for multiple first pixels in the target image based on the classification results of the multiple sampling points.

[0012] In one possible implementation, extracting the centerline of the target object in the image to be processed includes: performing a first segmentation process on the image to be processed to obtain the target object; extracting the centerline of the target object; and generating a target image of the target object based on the centerline includes: generating a target image of the target object based on the centerline; and determining the starting endpoint of the target image based on the endpoint radius of the target object in the target image.

[0013] In one possible implementation, dividing the target object into multiple first regions based on the classification results of the multiple first pixels includes: aggregating first pixels with the same classification results to determine multiple first regions contained in the target object, and names corresponding to the multiple first regions.

[0014] In one possible implementation, the method further includes: performing a second segmentation process on the image to be processed to obtain the target part to which the target object belongs; determining the classification results of multiple second pixels in the target part using a nearest neighbor method based on the classification results of the multiple first pixels; and dividing the target part into multiple second regions based on the classification results of the multiple second pixels.

[0015] In one possible implementation, the image to be processed includes a medical image, and the target object includes a trachea object.

[0016] According to one aspect of this disclosure, an image processing apparatus is provided, comprising: a centerline extraction module for extracting the centerline of a target object in an image to be processed; a target image generation module for generating a target image of the target object based on the centerline; a classification result generation module for performing convolution processing on the target image to obtain classification results of a plurality of first pixels in the target image, wherein the classification results include the name of the region to which the first pixels belong; and a first region division module for dividing the target object into a plurality of first regions according to the classification results of the plurality of first pixels.

[0017] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the processing method described in any of the preceding claims.

[0018] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the processing method described in any one of the preceding claims.

[0019] This disclosure provides an image processing method that extracts the centerline of a target object in an image to be processed, generates a target image of the target object based on the centerline, performs convolution processing on the target image to obtain classification results for multiple first pixels in the target image, and then divides the target object into multiple first regions based on the classification results of the multiple first pixels. This disclosure, by performing convolution processing on the target image, can automatically name the target object, reducing manual costs. Furthermore, by performing convolution processing on the target image of the target object generated based on the centerline of the target object, the classification results obtained from the multiple first pixels in the target image are more consistent with the actual distribution characteristics of the target object, resulting in higher accuracy. Consequently, the accuracy of the multiple first regions divided based on the classification results is also higher.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0022] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0023] Figure 2 A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0024] Figure 3 A reference flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0025] Figure 4 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0026] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0027] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0028] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0029] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0031] In one related technology, doctors typically name the various tracheal segments in the lungs manually using CT images to simulate the actual tracheal distribution in a patient. This method is not only labor-intensive, but its accuracy is also highly dependent on the doctor's experience. Inexperienced or distracted doctors are prone to naming errors. Another related technology uses rule-based algorithms for tracheal segment naming. However, in special cases such as missing or malformed trachea, or blurred images, the accuracy is lower, often requiring manual verification by the user.

[0032] In view of this, embodiments of this disclosure provide an image processing method that can extract the centerline of a target object in an image to be processed, generate a target image of the target object based on the centerline, then perform convolution processing on the target image to obtain classification results of multiple first pixels in the target image, and then divide the target object into multiple first regions based on the classification results of the multiple first pixels. Embodiments of this disclosure, by performing convolution processing on the target image, can automatically name the target object, reducing manual costs. Furthermore, by performing convolution processing on the target image of the target object generated based on the centerline of the target object to obtain classification results of multiple first pixels in the target image, the classification results are more consistent with the actual distribution characteristics of the target object, resulting in higher accuracy, and consequently, higher accuracy in dividing the multiple first regions based on the classification results.

[0033] In one possible implementation, the image processing method provided in this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server. For example, the server can be a cloud server or a local server.

[0034] For example, when a user needs to segment and name the trachea, they can input a medical image (e.g., a CT image) into the aforementioned terminal device. The terminal device then executes the image processing method described above to automatically complete the segmentation and naming of the trachea. The user can then determine how to treat the diseased trachea based on the segmentation and naming results and their own medical experience. The user can also simulate and rehearse the treatment methods for the diseased trachea based on the segmentation and naming results.

[0035] See Figure 1 As shown, Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0036] Step S100: Extract the centerline of the target object in the image to be processed. Exemplarily, the image to be processed may include multi-layered (depending on the number of data acquisition systems in the hardware structure) medical images (such as CT images). The target object may include the trachea in the lungs. Exemplarily, the target object can be obtained using various trachea segmentation algorithms in related technologies (e.g., adaptive supervised region growing, etc.), which will not be elaborated upon in this embodiment.

[0037] In one example, the aforementioned centerline can be determined using centerline algorithms in related technologies, such as centerline extraction methods based on triangulation, topology refinement, and distance transformation. This embodiment of the present disclosure does not impose any limitations on these methods. In other words, this step may include: performing a first segmentation process on the image to be processed to obtain the target object; and then generating a target image of the target object based on the centerline, i.e., obtaining the centerline of each trachea using the aforementioned centerline algorithm.

[0038] In one example, the target object can be represented by a trachea mask image. The first segmentation process involves segmenting the image using the aforementioned trachea segmentation algorithm to obtain the trachea mask image. This trachea mask image includes multiple tracheas to mask non-trachea regions in the image to be processed, thereby reducing the impact of pixels in non-trachea regions on the trachea segmentation naming results during subsequent segmentation naming.

[0039] Step S200: Generate a target image of the target object based on the centerline. For example, the target image can be the topological structure of the trachea, or a tree structure of the trachea. For example, the topological structure of the trachea can be determined by the connection relationship between the centerlines of each trachea. For instance, if the centerline of trachea A is connected to tracheas B, C, and D respectively, then in the topological structure of the trachea, trachea A is the parent node of tracheas B, C, and D, and is displayed as trachea A being connected to tracheas B, C, and D respectively. By determining the centerlines between tracheas, the integrity of the trachea topological structure can be improved, thereby reducing the probability of tracheas being omitted in the subsequent segmentation and naming process. For example, this step may include: determining the starting endpoint of the target image based on the endpoint radius of the target object in the target image. For example, the endpoint radius can be used to represent the radius of the trachea, with the trachea with the thickest radius serving as the root node in the topological structure, i.e., the starting endpoint. For example, multiple images to be processed are usually arranged sequentially along the z-axis, which is the axis of the human body from head to toe. By comparing the tracheal radii in each image to be processed, the trachea with the largest radius is taken as the root node. Then, the topology formed by the root node can represent the level of each node (that is, the level of each trachea). In subsequent classification, the target image can be simplified into an undirected graph (that is, the classification result of the first pixel is highly correlated with the connection relationship (level) between nodes). The classification result represents the classification result of each node in the undirected graph. Then, in step S300, different classification effects are displayed at the corresponding nodes in the topology.

[0040] Step S300: Perform convolution processing on the target image to obtain classification results for multiple first pixels in the target image. The classification results include the name of the region to which the first pixel belongs. For example, the first pixel is a pixel in the topological structure of the trachea. The name of the region to which the first pixel belongs can be the segment name of the trachea, which can subsequently be matched with different lung segments (e.g., the posterior segment of the left upper lobe (LB1 / 2), the anterior segment of the left upper lobe (LB3), the superior lingular segment of the left upper lobe (LB4), etc.) at the display level to visually represent the distribution of the lungs and trachea.

[0041] See Figure 2 As shown, Figure 2 A flowchart of an image processing method according to an embodiment of the present disclosure is shown, such as... Figure 2 As shown, in one possible implementation, step S300 may include:

[0042] Step S310: Perform global graph convolution processing on the target image to obtain the global features and global adjacency matrix of the target image. The global adjacency matrix includes the feature distances between multiple first pixels in the target image. These global features can be represented as vectors to represent the pixel features of the first pixel at the global level. For example, the global graph convolution processing can be performed by a deep learning model (e.g., a convolutional neural network). In its EdgeConv GCN (EdgeConv Graph Convolutional Network) module, the KNN algorithm (K-Nearest Neighbors algorithm) is used to calculate the K nearest neighbors of each first pixel in the feature space, obtaining the adjacency matrix J(M, M), where M represents the total number of first pixels. The values ​​in matrix J represent the feature distances between the pixel features of two first pixels in the feature space. The aforementioned pixel features represent the abstract features of each first pixel, and the aforementioned feature distance represents the similarity between different first pixels (i.e., the degree of difference between pixel features between first pixels). For example, the higher the similarity between two first pixels, the smaller the feature distance between the two first pixels at the corresponding position in matrix J. In one example, the aforementioned first pixel may include four parameters, such as the x, y, and z axis coordinates of the first pixel in the topology of the trachea (which can correspond to the coordinate axes of a CT image, and will not be elaborated here) and its level in the topology, as input to the aforementioned deep learning model. For example, a coordinate matrix of size (N, 4) can be input, where N is the total number of first pixels and 4 is the number of the aforementioned parameters.

[0043] In one example, step S310 may include: performing K1 global graph convolution processes on the target image to obtain global features and a global adjacency matrix, wherein the global features are obtained through the K1th global graph convolution process, and the global adjacency matrix is ​​obtained through the first global graph convolution process, where K1 is a positive integer. The aforementioned global features are used to represent the arrangement features of multiple first pixels at the global level.

[0044] For example, if i is a value greater than 1 in K1, the i-th global graph convolution process may include: performing edge graph convolution on the intermediate global features (which may include the pixel features corresponding to each first pixel) output after the (i-1)-th global graph convolution process to obtain the first edge graph convolution result (i.e., the convolution result of the K pixel features corresponding to the K first pixel points with the smallest feature distance in the intermediate adjacency matrix of each first pixel point during the (i-1)-th global graph convolution process) and the intermediate adjacency matrix. Then, based on the intermediate adjacency matrix and the first edge graph convolution result, the intermediate global features output after the (i-1)-th global graph convolution process are fitted to obtain the intermediate global features output after the i-th global graph convolution process, wherein the fitting process includes convolution processing and / or regularization processing. For example, if the above-mentioned first edge map convolution processing result is a pixel feature A including N pixels, then the pixel features B of its K nearest neighbors are obtained through the intermediate adjacency matrix. Then, the N*K pixel features (i.e., N sets of pixel features A and B) are convolved with the intermediate global features at the i-1th time and / or regularized to fit the intermediate global features at the i-th time.

[0045] After K1 iterations, the intermediate global features obtained are used as the aforementioned global features. This embodiment of the present disclosure can increase the receptive field of the deep learning model when obtaining global features by performing multiple global graph convolution processes, thereby increasing the representativeness of the obtained global features and improving the accuracy of subsequent classification results.

[0046] For example, during each global graph convolution process, the K pixel features corresponding to the K first pixel points with the smallest feature distance in matrix J for each first pixel point A can be convolved. The convolution result is used as the new pixel feature corresponding to the first pixel point A. Based on the new pixel features corresponding to each first pixel point, a new intermediate adjacency matrix is ​​generated to highlight the global features between the first pixel points.

[0047] Step S320: Based on the global adjacency matrix, perform local graph convolution processing on the target image to obtain local features of the target image. In one example, step S320 may include: performing K2-fold local graph convolution processing on the target image based on the global adjacency matrix to obtain local features, wherein the local features are obtained through the K2-th local graph convolution processing, and K2 is a positive integer. The aforementioned local features are used to represent the arrangement features of multiple first pixels at the local level.

[0048] For example, if j is taken as a value greater than 1 in K2, the j-th local graph convolution process includes: performing edge graph convolution processing on the intermediate local features (which may include the pixel features corresponding to each first pixel) output after the (j-1)-th local graph convolution processing to obtain the second edge graph convolution processing result (that is, the convolution result of the K pixel features corresponding to the K first pixel points with the smallest feature distance in the global adjacency matrix for each first pixel point); and performing fitting processing on the intermediate local features output after the (j-1)-th local graph convolution processing according to the global adjacency matrix and the second edge graph convolution processing result to obtain the intermediate local features output after the j-th local graph convolution processing, wherein the fitting processing includes convolution processing and / or regularization processing. For example, if the above-mentioned second edge map convolution processing result is a pixel feature A including N pixels, then the pixel features B of its K nearest neighbors are obtained through the global adjacency matrix. Then, the N*K pixel features (i.e., N sets of pixel features A and B) are convolved with the intermediate local features at the j-1th time and / or regularized to fit the intermediate local features at the jth time.

[0049] After K2 iterations, the intermediate local features obtained are used as the aforementioned local features. This embodiment of the present disclosure can increase the receptive field of the deep learning model when obtaining local features by performing multiple local graph convolution processes, thereby increasing the representativeness of the obtained local features and improving the accuracy of the subsequent classification results.

[0050] For example, during each local graph convolution process, the K pixel features corresponding to the K first pixel points with the smallest feature distance in matrix J for each first pixel point A can be convolved (i.e., second edge graph convolution). The convolution result is used as the new pixel feature corresponding to the first pixel point A, and the above-mentioned global adjacency matrix is ​​reused to highlight the local features between the first pixel points.

[0051] Step S330: Based on global and local features, multiple first pixels in the target image are classified to obtain classification results for the multiple first pixels. For example, the aforementioned global and local features can be fused using fully connected layers in a deep learning model, enabling the deep learning model to correctly name trachea with local defects, abnormal imaging, or local deformities using both global and local features, thereby improving the accuracy of segmented naming. Furthermore, it has been verified that the deep learning model used in this embodiment can process one CT image every 3 seconds after obtaining the tracheal tree, which is more computationally efficient than the segmented naming method in related technologies, thus saving users' time.

[0052] In one possible implementation, when the target image includes a tree topology graph, step S300 may include: dividing the tree topology graph into multiple sampling regions according to the hierarchical structure in the tree topology graph; sampling pixels in each of the multiple sampling regions; generating a sampling image based on the obtained multiple sampling points; performing graph convolution processing on the sampling image to obtain the classification results of multiple sampling points in the sampling image; and determining the classification results of multiple first pixels in the target image based on the classification results of the multiple sampling points. For example, the number of the above sampling regions can be multiple, for example, the same number of sampling points can be collected in each trachea of ​​different levels. It can also be simply divided into two sampling regions with more than 3 levels and less than or equal to 3 levels, and then sampling is performed evenly in the two sampling regions. This embodiment of the present disclosure does not limit this. Then, the sampling image composed of the sampling points is input into a deep learning model for classification (that is, the sampling points are used to represent the first pixels) to perform the above steps S310 to S330. If set up in this way, on the one hand, it can save computing power, and on the other hand, it can reduce the impact of extreme cases (e.g., the number of sampling points is too close, resulting in the pixel features not being representative) on the subsequent classification results.

[0053] Continue reading Figure 1 In step S400, based on the classification results of multiple first pixels, the target object is divided into multiple first regions. For example, these different first regions are used to distinguish different tracheal segments. Different first regions can be distinguished by different identifiers (e.g., color, text) and displayed on the display screen of the terminal device. For example, this step may include: aggregating first pixels with the same classification result to determine the multiple first regions contained in the target object, and the names corresponding to the multiple first regions. For example, aggregation algorithms in related technologies (e.g., fuzzy C-means aggregation algorithm, also known as FCM method) can be used to aggregate first pixels with the same classification result, and then the target object is divided into multiple first regions to reduce the impact of misclassification of blurry or uncertain first pixels in the image on the division of first regions, thereby increasing the accuracy of tracheal segment naming.

[0054] Combination Figure 3 As shown, Figure 3 A reference schematic diagram of the image processing method according to an embodiment of the present disclosure is shown. The tracheal structure in the left figure is the topological structure of the trachea described above. Then, using the image processing method provided by the embodiment of the present disclosure, each trachea in the tracheal topological diagram can be divided into a first region to intuitively display the structural relationship of the trachea.

[0055] For example, after step S400, the process may further include: performing a second segmentation process on the image to be processed to obtain the target location to which the target object belongs. For example, the target location may be a lung lobe. The second segmentation process may be performed using a lung segmentation algorithm in related technologies, which will not be elaborated here in this embodiment. Then, based on the classification results of multiple first pixels, the classification results of multiple second pixels in the target location are determined by the nearest neighbor method. The second pixels are pixels in the lung region of the image to be processed. The classification results are the lung segments corresponding to each second pixel. By using the nearest neighbor algorithm as the algorithm used for lung segment segmentation, the boundaries of lung segments can be made more accurate and clear. Then, based on the classification results of multiple second pixels, the target location is divided into multiple second regions. The second regions are the lung segment regions. In practical application scenarios, the application value of tracheal segment naming is low. However, in this embodiment, by dividing each trachea into lung segment regions, users can intuitively understand the actual location of each trachea in the lung, which facilitates users in planning the corresponding processing methods.

[0056] For example, this disclosure also provides a training process for the deep learning model described above. In this training process, the data processing method is the same as that in actual use scenarios. This disclosure will not repeat the details here, but only describe the different parts.

[0057] Based on the hierarchical structure of the tree topology graph, the target object in the image to be processed is divided into multiple sampling regions. Pixel sampling is performed in each of the multiple sampling regions. Based on the obtained multiple sampling points and amplified sampling points, a sampled image is generated. The amplified sampling points are obtained by perturbing the sampling points, including translation and / or scaling, to augment the input data. The above sampling points can be represented and stored as a coordinate matrix of size (N, 4), where N is the total number of sampling points and 4 represents four parameters: the x, y, and z axis coordinates of the sampling point and its level in the tree topology graph. During the training phase, the category of each pixel can be sent to the deep learning model for subsequent updates to the network parameters, as detailed later. The above category refers to the first region mentioned above, i.e., the segmentation and naming result of the trachea, which can be manually labeled or labeled by a relevant algorithm (e.g., the nearest neighbor algorithm). This embodiment of the disclosure does not impose any limitations on this. The above translation and scaling operations can be performed on the coordinates of the sampling points, i.e., translating and scaling the coordinates corresponding to the sampling points. The sampled image is subjected to graph convolution processing (i.e., the sampled image is input into a deep learning model) to obtain the classification results of multiple sampling points in the sampled image (i.e., the classification results output by the deep learning model). Based on the classification results of multiple sampling points, the classification results of multiple first pixels in the target image are determined (i.e., the sampling points are used to represent the first pixels).

[0058] For example, the deep learning model described above may include two branch modules: a global branch module and a local branch module, which are used to obtain global features and local features, respectively. The two branch modules can also be computed in parallel. The global branch module obtains the global features of the sampling points (similar to step S310), and the local branch module obtains the local features of the sampling points (similar to step S320). Then, through a fully connected layer in the deep learning model, the global and local features are connected along the feature channel dimension (i.e., the channels in the feature space mentioned above) (similar to step S330), resulting in a probability vector for each sampling point. This probability vector can be used to obtain the probability that each sampling point belongs to each first region. The deep learning model can compare the loss value between the predicted first region and the true first region for each sampling point using a preset loss function. For example, the loss value can be positively correlated with the degree of difference between the two. In one example, the loss function can be the cross-entropy loss function, and the formula is as follows:

[0059]

[0060] Where H(X) is the loss value, and P(x) is the loss value. i ) represents sampling point x i The probability of a value belonging to any of the various classification results (which can be simply represented by 1 or 0), P`(x) i ) represents the sampling point x predicted by the deep learning model. i The probability of belonging to each classification result, where n is the total number of sampling points. The above loss function can be selected according to the actual situation, and will not be elaborated here in the embodiments of this disclosure.

[0061] Deep learning models can pass the loss value to each layer of the network by back-derivative calculation, and update the network parameters of the deep learning model through a preset weight update strategy (e.g., Adam (lr = 3e-4), which is also adaptive gradient descent) until the classification accuracy of the deep learning model converges.

[0062] See Figure 4 As shown, Figure 4 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown, such as Figure 4As shown in the embodiments of this disclosure, an image processing apparatus 100 is also provided. The image processing apparatus 100 includes: a centerline extraction module 110, used to extract the centerline of a target object in an image to be processed; a target image generation module 120, used to generate a target image of the target object based on the centerline; a classification result generation module 130, used to perform convolution processing on the target image to obtain classification results of a plurality of first pixels in the target image, wherein the classification results include the name of the region to which the first pixel belongs; and a first region division module 140, used to divide the target object into a plurality of first regions according to the classification results of the plurality of first pixels.

[0063] In one possible implementation, the convolution processing of the target image to obtain classification results for a plurality of first pixels in the target image includes: performing global graph convolution processing on the target image to obtain global features and a global adjacency matrix of the target image, wherein the global adjacency matrix includes feature distances between a plurality of first pixels in the target image; performing local graph convolution processing on the target image based on the global adjacency matrix to obtain local features of the target image; and performing classification processing on the plurality of first pixels in the target image based on the global features and the local features to obtain classification results for the plurality of first pixels.

[0064] In one possible implementation, the step of performing global graph convolution processing on the target image to obtain the global features and global adjacency matrix of the target image includes: performing K1 times of global graph convolution processing on the target image to obtain the global features and global adjacency matrix, wherein the global features are obtained through the K1th global graph convolution processing, and the global adjacency matrix is ​​obtained through the first global graph convolution processing, where K1 is a positive integer.

[0065] In one possible implementation, when i is less than or equal to K1 and greater than 1, the i-th global graph convolution process includes: performing edge graph convolution processing on the intermediate global features output after the (i-1)-th global graph convolution process to obtain a first edge graph convolution processing result and an intermediate adjacency matrix; and performing fitting processing on the intermediate global features output after the (i-1)-th global graph convolution process based on the intermediate adjacency matrix and the first edge graph convolution processing result to obtain the intermediate global features output after the i-th global graph convolution process, wherein the fitting processing includes convolution processing and / or regularization processing.

[0066] In one possible implementation, the step of performing local graph convolution processing on the target image based on the global adjacency matrix to obtain local features of the target image includes: performing K2-fold local graph convolution processing on the target image based on the global adjacency matrix to obtain local features, wherein the local features are obtained through the K2-fold local graph convolution processing, and K2 is a positive integer.

[0067] In one possible implementation, when j is less than or equal to K2 and greater than 1, the j-th local graph convolution process includes: performing edge graph convolution processing on the intermediate local features output after the (j-1)-th local graph convolution process to obtain a second edge graph convolution processing result; and performing fitting processing on the intermediate local features output after the (j-1)-th local graph convolution process according to the global adjacency matrix and the second edge graph convolution processing result to obtain the intermediate local features output after the j-th local graph convolution process, wherein the fitting processing includes convolution processing and / or regularization processing.

[0068] In one possible implementation, the target image includes a tree topology graph; the convolution processing of the target image to obtain classification results for multiple first pixels in the target image includes: dividing the tree topology graph into multiple sampling regions according to the hierarchical structure of the tree topology graph; sampling pixels in the multiple sampling regions respectively; generating a sampling image based on the obtained multiple sampling points; performing graph convolution processing on the sampling image to obtain classification results for multiple sampling points in the sampling image; and determining the classification results for multiple first pixels in the target image based on the classification results of the multiple sampling points.

[0069] In one possible implementation, the target image includes a tree topology graph; the convolution processing of the target image to obtain classification results for multiple first pixels in the target image includes: dividing the target object in the image to be processed into multiple sampling regions according to the hierarchical structure of the tree topology graph; sampling pixels in the multiple sampling regions respectively; generating a sampling image based on the obtained multiple sampling points and amplified sampling points, wherein the amplified sampling points are obtained by perturbing the sampling points, the perturbing process including translation and / or scaling; performing graph convolution processing on the sampling image to obtain classification results for multiple sampling points in the sampling image; and determining the classification results for multiple first pixels in the target image based on the classification results of the multiple sampling points.

[0070] In one possible implementation, extracting the centerline of the target object in the image to be processed includes: performing a first segmentation process on the image to be processed to obtain the target object; extracting the centerline of the target object; and generating a target image of the target object based on the centerline includes: generating a target image of the target object based on the centerline; and determining the starting endpoint of the target image based on the endpoint radius of the target object in the target image.

[0071] In one possible implementation, dividing the target object into multiple first regions based on the classification results of the multiple first pixels includes: aggregating first pixels with the same classification results to determine multiple first regions contained in the target object, and names corresponding to the multiple first regions.

[0072] In one possible implementation, the image processing apparatus further includes: a second region segmentation module, configured to perform: performing a second segmentation process on the image to be processed to obtain the target part to which the target object belongs; determining the classification results of multiple second pixels in the target part using a nearest neighbor method based on the classification results of the multiple first pixels; and dividing the target part into multiple second regions based on the classification results of the multiple second pixels.

[0073] In one possible implementation, the image to be processed includes a medical image, and the target object includes a trachea object.

[0074] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0075] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0076] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0077] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0078] Figure 5 This diagram illustrates a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other terminal devices.

[0079] Reference Figure 5 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0080] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0081] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0082] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0083] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0084] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0085] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0086] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0087] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G, LTE, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID), Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0088] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0089] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

Claims

1. An image processing method, characterized in that, include: Extract the centerline of the target object in the image to be processed; Based on the centerline, a target image of the target object is generated, wherein the target image is the topological structure of the trachea, or the tree structure of the trachea. The target image is subjected to convolution processing to obtain the classification results of multiple first pixels in the target image, wherein the classification results include the name of the region to which the first pixel belongs; Based on the classification results of the multiple first pixels, the target object is divided into multiple first regions; The step of performing convolution processing on the target image to obtain the classification results of multiple first pixels in the target image includes: Global graph convolution processing is performed on the target image to obtain the global features and global adjacency matrix of the target image, wherein the global adjacency matrix includes the feature distances between multiple first pixels in the target image; Based on the global adjacency matrix, local graph convolution processing is performed on the target image to obtain the local features of the target image; Based on the global features and the local features, multiple first pixels in the target image are classified to obtain the classification results of the multiple first pixels. The step of performing global graph convolution processing on the target image to obtain the global features and global adjacency matrix of the target image includes: The target image is subjected to K1 global graph convolution processes to obtain global features and a global adjacency matrix. The global features are obtained through the K1th global graph convolution process, and the global adjacency matrix is ​​obtained through the first global graph convolution process. K1 is a positive integer.

2. The method according to claim 1, characterized in that, When i is less than or equal to K1 and greater than 1, the i-th global graph convolution process includes: Perform edge graph convolution on the intermediate global features output after the (i-1)th global graph convolution to obtain the first edge graph convolution result and the intermediate adjacency matrix; Based on the intermediate adjacency matrix and the first edge graph convolution processing result, the intermediate global features output after the (i-1)th global graph convolution processing are fitted to obtain the intermediate global features output after the i-th global graph convolution processing, wherein the fitting processing includes convolution processing and / or regularization processing.

3. The method according to claim 1, characterized in that, The step of performing local graph convolution processing on the target image based on the global adjacency matrix to obtain the local features of the target image includes: Based on the global adjacency matrix, local features are obtained by performing K2 local graph convolutions on the target image, where the local features are obtained through the K2th local graph convolution, and K2 is a positive integer.

4. The method according to claim 3, characterized in that, When j is less than or equal to K2 and greater than 1, the j-th local graph convolution process includes: The intermediate local features output after the (j-1)th local graph convolution are processed by edge graph convolution to obtain the second edge graph convolution result. Based on the global adjacency matrix and the second edge graph convolution processing result, the intermediate local features output after the (j-1)th local graph convolution processing are fitted to obtain the intermediate local features output after the j-th local graph convolution processing, wherein the fitting processing includes convolution processing and / or regularization processing.

5. The method according to any one of claims 1 to 4, characterized in that, The target image includes a tree topology map; The step of performing convolution processing on the target image to obtain the classification results of multiple first pixels in the target image includes: Based on the hierarchical structure of the tree topology diagram, the tree topology diagram is divided into multiple sampling regions; Pixel sampling is performed in the multiple sampling regions, and a sampled image is generated based on the obtained multiple sampling points; The sampled image is subjected to graph convolution processing to obtain the classification results of multiple sampling points in the sampled image; Based on the classification results of the multiple sampling points, the classification results of multiple first pixels in the target image are determined.

6. The method according to any one of claims 1 to 4, characterized in that, The target image includes a tree topology map; The step of performing convolution processing on the target image to obtain the classification results of multiple first pixels in the target image includes: Based on the hierarchical structure in the tree topology diagram, the target object in the image to be processed is divided into multiple sampling regions; Pixel sampling is performed in the multiple sampling areas respectively. Based on the obtained multiple sampling points and amplified sampling points, a sampled image is generated. The amplified sampling points are obtained by perturbing the sampling points. The perturbing process includes translation and / or scaling. The sampled image is subjected to graph convolution processing to obtain the classification results of multiple sampling points in the sampled image; Based on the classification results of the multiple sampling points, the classification results of multiple first pixels in the target image are determined.

7. The method according to any one of claims 1 to 4, characterized in that, Extracting the center line of the target object in the image to be processed includes: The target object is obtained by performing a first segmentation process on the image to be processed; Extract the centerline of the target object; The step of generating a target image of the target object based on the centerline includes: Based on the center line, generate a target image of the target object; The starting endpoint of the target image is determined based on the endpoint radius of the target object in the target image.

8. The method according to any one of claims 1 to 4, characterized in that, The step of dividing the target object into multiple first regions based on the classification results of the multiple first pixels includes: The first pixels with the same classification result are aggregated to determine the multiple first regions contained in the target object, and the names corresponding to the multiple first regions.

9. The method according to any one of claims 1 to 4, characterized in that, The method further includes: A second segmentation process is performed on the image to be processed to obtain the target part to which the target object belongs; Based on the classification results of the plurality of first pixels, the classification results of the plurality of second pixels in the target region are determined by the nearest neighbor method; Based on the classification results of the multiple second pixels, the target area is divided into multiple second regions.

10. The method according to any one of claims 1 to 4, characterized in that, The image to be processed includes medical images, and the target object includes a trachea object.

11. An image processing apparatus, characterized in that, include: The centerline extraction module is used to extract the centerline of the target object in the image to be processed; The target image generation module is used to generate a target image of the target object based on the center line. The target image is the topological structure of the trachea, or the tree structure of the trachea. The classification result generation module is used to perform convolution processing on the target image to obtain the classification results of multiple first pixels in the target image, wherein the classification results include the name of the region to which the first pixel belongs; The first region division module is used to divide the target object into multiple first regions based on the classification results of the multiple first pixels; The classification result generation module is used to perform global graph convolution processing on the target image to obtain global features and a global adjacency matrix of the target image, wherein the global adjacency matrix includes feature distances between multiple first pixels in the target image; perform local graph convolution processing on the target image based on the global adjacency matrix to obtain local features of the target image; and perform classification processing on the multiple first pixels in the target image based on the global features and the local features to obtain classification results for the multiple first pixels. The classification result generation module is used to perform K1 global graph convolution processing on the target image to obtain global features and a global adjacency matrix. The global features are obtained through the K1th global graph convolution processing, and the global adjacency matrix is ​​obtained through the first global graph convolution processing. K1 is a positive integer.

12. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the processing method according to any one of claims 1 to 10.

Images