Image landmark recognition model training method and device

Abstract

The present application provides a method and device for training an image landmark point recognition model, by acquiring multiple training images, each of which contains multiple marking points, and each marking point has original coordinates. For each training image, multiple image blocks of different resolutions of the training image are obtained based on a Gaussian pyramid structure, and a feature map corresponding to each image block is obtained. Each feature map is imported into a constructed multi-layer neural network model, and the model is trained based on a loss function constructed based on the original coordinates of the marking points in each feature map and the coordinate information output by the model, until the loss function converges, and a recognition model obtained by training based on the multi-layer neural network model is obtained. The present application can obtain multiple image blocks of different resolutions through a Gaussian pyramid structure, so that the model can learn the feature information of images of different resolutions, thereby improving the accuracy of subsequent image recognition in response to different resolutions.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a method and device for training an image landmark recognition model. Background Art

[0002] Deep learning (DL) is a new research direction in the field of machine learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence (AI). Deep learning is the inherent laws and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies. Deep learning has achieved many results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as seeing, hearing and thinking, solves many complex pattern recognition problems, and makes great progress in artificial intelligence related technologies.

[0003] To date, deep convolutional networks have become the preferred architecture in the field of computer vision. The medical image analysis community has noticed these key developments. And is gradually turning to various deep learning network models. Using these technologies in medical image analysis methods can shorten the diagnosis process of clinicians. Therefore, there is still a lot of room for the application of deep learning in this field. One of the key problems that need to be solved in medical imaging applications is that the image quality and pixel distribution differences caused by the differences in various imaging devices lead to low recognition accuracy of the model under small error requirements. The main reason is that in the process of deep learning, the network has deficiencies in handling multi-precision pixel information, resulting in low accuracy of the obtained recognition model when dealing with images of different precisions. Summary of the invention

[0004] The purpose of the present application includes, for example, providing a method and device for training an image landmark point recognition model, which enables the trained model to learn the feature information of images of different resolutions, thereby improving the accuracy of image recognition corresponding to different resolutions.

[0005] The embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides an image landmark recognition model training method, the method comprising:

[0007] Acquire a plurality of training images, each of the training images comprising a plurality of marking points, each of the marking points having original coordinates;

[0008] For each training image, a plurality of image blocks of different resolutions of the training image are obtained based on a Gaussian pyramid structure, and a feature map corresponding to each of the image blocks is obtained;

[0009] Each of the feature maps is imported into the constructed multi-layer neural network model, and the model is trained based on a loss function constructed based on the original coordinates of the marked points in each of the feature maps and the coordinate information output by the model, until the loss function converges, thereby obtaining a recognition model trained based on the multi-layer neural network model.

[0010] In an optional implementation, before the step of obtaining a plurality of image blocks of different resolutions of the training image based on the Gaussian pyramid structure, the method further includes:

[0011] For each of the training images, normalization is performed on the pixel values ​​of all pixels in the training image so that the pixel value of each pixel after normalization is within a preset pixel distribution range.

[0012] In an optional implementation, the step of obtaining, for each training image, a plurality of image blocks of different resolutions of the training image based on a Gaussian pyramid structure includes:

[0013] For each training image, multiple processed images of different resolutions and sizes corresponding to the training image are obtained based on the Gaussian pyramid structure;

[0014] For each of the processed images, based on the position information of the marking point in the processed image, the images are cropped with the marking point as the center according to a preset uniform size to obtain a plurality of image blocks with different resolutions after the processing of the plurality of processed images.

[0015] In an optional implementation manner, the marking points in each of the training images are numbered and marked in a unified identification manner;

[0016] The step of cropping the image according to a preset uniform size with the marked point as the center based on the position information of the marked point in the processed image comprises:

[0017] For each marking point in each training image, based on the original coordinates of the marking points with the same number in the plurality of training images, calculate the average coordinates of the marking point with the number;

[0018] Based on the average coordinates of each marked point in the processed image, the image is cropped to a preset uniform size with the marked point as the center.

[0019] In an optional implementation, the step of obtaining a feature map corresponding to each of the image blocks includes:

[0020] Importing each of the obtained image blocks into a constructed improved human posture recognition model, wherein the improved human posture recognition model includes multiple parallel sub-networks, the multiple sub-networks have a unified input channel, and the multiple sub-networks have cross-layer connection channels;

[0021] Using multiple sub-networks to process image blocks of different resolutions, and fusing the image blocks of different resolutions to obtain image blocks with fused resolutions;

[0022] The image blocks with the fused resolution and the original multiple image blocks with different resolutions are processed into feature maps of set sizes and output.

[0023] In an optional embodiment, the multi-layer neural network model includes a first model and a second model, the first model includes a plurality of channels, and each of the feature maps corresponds to each of the channels;

[0024] The step of importing each of the feature graphs into the constructed multi-layer neural network model, and training the model based on the original coordinates of the marked points in each of the feature graphs and the loss function constructed by the coordinate information output by the model, comprises:

[0025] Importing each of the feature maps into the constructed first model, processing each of the feature maps using the corresponding channels according to the weights they have, and obtaining fused maps after fusion processing according to different weights;

[0026] The fusion graph is imported into the constructed second model for training, and under the guidance of the loss function constructed based on the original coordinates of the marked points in the fusion graph and the output coordinate information, the weights of each channel in the first model are adjusted, and the model parameters of the second model are adjusted, and then training is continued until the loss function converges.

[0027] In an optional embodiment, the loss function is constructed based on the absolute value of the difference between the original coordinates of the marked point and the coordinate information output by the model.

[0028] In an optional implementation, the original coordinates of each of the marking points are coordinates in a coordinate system constructed with the center point of the training image in which the marking point is located as the origin.

[0029] In an optional embodiment, the method further comprises:

[0030] Acquire an image to be recognized, input the image to be recognized into a pre-trained recognition model, and obtain coordinate information of each marker point contained in the image to be recognized, where the coordinate information is a relative position deviation of the marker point relative to the template position;

[0031] Based on the relative position deviation and the template position, the final coordinate value of the marking point is obtained.

[0032] In a second aspect, the present application provides an image landmark recognition model training device, the device comprising:

[0033] An acquisition module is used to acquire multiple training images, each of which contains multiple marking points, each of which has original coordinates, and for each training image, obtain multiple image blocks of different resolutions of the training image based on a Gaussian pyramid structure, and obtain a feature map corresponding to each image block;

[0034] The training module is used to import each of the feature maps into the constructed multi-layer neural network model, and train the model based on the loss function constructed based on the original coordinates of the marked points in each of the feature maps and the coordinate information output by the model, until the loss function converges, thereby obtaining a recognition model trained based on the multi-layer neural network model.

[0035] The beneficial effects of the embodiments of the present application include, for example:

[0036] The present application provides a method and device for training an image landmark point recognition model, by acquiring multiple training images, each of which contains multiple marking points, and each marking point has original coordinates. For each training image, multiple image blocks of different resolutions of the training image are obtained based on a Gaussian pyramid structure, and a feature map corresponding to each image block is obtained. Each feature map is imported into a constructed multi-layer neural network model, and the model is trained based on a loss function constructed based on the original coordinates of the marking points in each feature map and the coordinate information output by the model, until the loss function converges, and a recognition model obtained by training based on the multi-layer neural network model is obtained. The present application can obtain multiple image blocks of different resolutions through a Gaussian pyramid structure, so that the model can learn the feature information of images of different resolutions, thereby improving the accuracy of subsequent image recognition in response to different resolutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A flowchart of an image landmark recognition model training method provided in an embodiment of the present application;

[0039] Figure 2 for Figure 1 Flow chart of sub-steps included in step S102;

[0040] Figure 3 for Figure 2 Flow chart of sub-steps included in step S1022;

[0041] Figure 4 for Figure 1 Another flowchart of the sub-steps included in step S102;

[0042] Figure 5 A schematic diagram of a model architecture provided for an embodiment of the present application;

[0043] Figure 6 for Figure 1 Flow chart of sub-steps included in step S103;

[0044] Figure 7 A flowchart of a processing method in the image landmark recognition model training method provided in an embodiment of the present application;

[0045] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0046] Fig. 9 A functional module block diagram of the image landmark point recognition model training device provided in an embodiment of the present application.

[0047] Icon: 110 - storage medium; 120 - processor; 130 - image landmark recognition model training device; 131 - acquisition module; 132 - training module; 140 - communication interface. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0050] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0051] In the description of the present application, it should be noted that the terms "first", "second", etc., if used, are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0052] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0053] Figure 1 A flow chart of an image landmark recognition model training method provided in an embodiment of the present application is shown. The training method can be applied to terminal devices with specific image processing, data analysis and processing functions, such as personal computers, servers and other devices.

[0054] It should be understood that in other embodiments, the order of some steps in the image landmark recognition model training method of this embodiment can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of the image landmark recognition model training method are described as follows.

[0055] S101, acquiring a plurality of training images, each of the training images comprising a plurality of marking points, each of the marking points having original coordinates.

[0056] S102: For each training image, obtain multiple image blocks of different resolutions of the training image based on a Gaussian pyramid structure, and obtain a feature map corresponding to each image block.

[0057] S103, importing each of the feature maps into the constructed multi-layer neural network model, training the model based on a loss function constructed based on the original coordinates of the marked points in each of the feature maps and the coordinate information output by the model, until the loss function converges, thereby obtaining a recognition model trained based on the multi-layer neural network model.

[0058] In this embodiment, the acquired training image may be a lateral cranial image in the medical field, or may be an image for determining landmarks in other fields. In this embodiment, the training image is a lateral cranial image as an example for description.

[0059] Each of the collected training images contains multiple marker points, where the marker points are some special pixel points in the training image, which may be points containing relevant important information, points that can reflect certain representation information, etc. This embodiment does not impose specific restrictions on this. Among them, the marker points in the training image can be obtained by artificial marking. Each training image contains objects of the same type. Although there are certain differences between the objects, they should all have the same type of marker points, but the specific positions of the marker points are different.

[0060] For example, for training images of lateral skull films, each lateral skull film contains a skull image, and each skull image is obtained by taking pictures of different patients, so there are differences in size, shape, etc. However, when marking the landmarks of each lateral skull film, the jaw point, upper and lower lip edge points, upper and lower teeth edge points, etc. can be marked. That is, each training image can be marked with the same type of landmarks, but because the specific objects contained therein are different, the positions of the same type of landmarks also have certain differences.

[0061] In this embodiment, a Gaussian pyramid structure is used to obtain multiple image blocks of different resolutions for each training image. The Gaussian pyramid structure can process images into multiple images in the form of a pyramid. For example, from the bottom to the top of the structure, the resolution of each image can decrease from high to low. On this basis, each image can be divided into multiple image blocks. Thus, each training image can correspond to multiple image blocks of different resolutions.

[0062] On this basis, the feature map corresponding to each image block can be obtained, so that multiple feature maps with different resolutions can be obtained. The obtained feature map is input into the constructed multi-layer neural network model, so as to train the multi-layer neural network model using feature maps with different resolutions.

[0063] Among them, the marker points in each training image have original coordinates, that is, the coordinate values ​​determined when the marker points are manually marked. After each feature map is processed by the multi-layer neural network model, the output of the model can contain the coordinate information of the marker points. The purpose of model training is mainly to make the coordinate information of the marker points obtained after model processing as close as possible to or even consistent with the real original coordinates of the marker points. Therefore, a loss function can be constructed based on the original coordinates of the marker points in the feature map and the coordinate information output by the model, and the model can be iteratively trained multiple times under the guidance of the loss function. When the loss function reaches convergence, a trained recognition model can be obtained.

[0064] In this embodiment, a plurality of image blocks of different resolutions corresponding to each training image can be obtained through the Gaussian pyramid structure, so that the model can learn the feature information of images of different resolutions, thereby improving the accuracy of subsequent image recognition with different resolutions.

[0065] In a possible implementation, each training image can be a grayscale image. Since the grayscale image features themselves vary greatly, such as the maximum value, average value, and pixel distribution, etc. The range of variation of the features of different data may vary greatly. When processing the feature map, the difference in such numerical distribution may interfere with the convergence of the model. For example, features with large values ​​will play a decisive role, while features with small values ​​may be ignored. In order to eliminate the influence of unit and size differences between features, before processing each training image according to the Gaussian pyramid structure, the pixel values ​​of all pixels in the training image can also be normalized for each training image, so that the pixel value of each pixel after normalization is within the preset pixel distribution range.

[0066] For example, the pixel values ​​of all pixels in each training image can be mapped to a distribution with a mean value of 1.4255656 and a variance of 0.8835338. The normalized calculation formula can be shown as follows:

[0067]

[0068] Among them, I out Represents the processed image, I in Represents the input native image, I mean Represents the average value of the pixels in the current channel, I std Represents the variance of the current channel. Then the mean and variance are scaled to the specified range, so that the pixel distribution of the image is adjusted to a uniform distribution.

[0069] By normalizing the pixel values ​​of all pixels, interference in subsequent model training caused by excessive differences between pixel values ​​can be avoided.

[0070] In addition, in the prior art, when determining the specific coordinates of the marker point, the coordinate system is often constructed with the upper left corner of the training image as the origin. However, when the coordinate system is constructed with the upper left corner as the origin, the subsequently predicted marker point will obviously shift to the upper left. In order to prevent the marker point from shifting in a fixed direction, in this embodiment, the original coordinates of each marker point are the coordinates in the coordinate system constructed with the center point of the training image in which it is located as the origin.

[0071] Of course, in addition to the original coordinates being determined by a coordinate system constructed with the center point of the training image as the origin, subsequent coordinate information of various types of landmark points is also determined in the same coordinate system.

[0072] After constructing a coordinate system with the center point of the training image as the origin and normalizing the pixels of the training image, the step of constructing multiple image blocks with different resolutions using a Gaussian pyramid structure can be performed for each training image.

[0073] See also Figure 2 In this embodiment, in the above step of obtaining multiple image blocks with different resolutions, it can be achieved by the following method:

[0074] S1021, for each training image, obtaining multiple processed images with different resolutions and sizes corresponding to the training image based on the Gaussian pyramid structure.

[0075] S1022: For each of the processed images, based on the position information of the marking point in the processed image, cropping is performed with the marking point as the center according to a preset uniform size to obtain a plurality of image blocks with different resolutions after the processing of the plurality of processed images.

[0076] The Gaussian pyramid of each training image is a collection of images arranged in a pyramid shape with resolutions gradually decreasing from top to bottom. The bottom of the pyramid is a high-resolution identifier of the training image, while the top is a low-resolution representation. In this embodiment, a Gaussian low-pass filter is used for filtering, and when the pyramid moves to the upper layer, the size and resolution will decrease. That is, multiple processed images of different resolutions and sizes can be obtained. If the number of layers of the Gaussian pyramid is J, then plus the training image itself, J+1 processed images can be obtained in the end.

[0077] Among them, the size of the processed image at the bottom layer of the Gaussian pyramid structure can be (H / 2 J )×(W / 2 J ), where H represents the original length of the training image, W represents the original width of the training image, and J represents the number of layers of the Gaussian pyramid. Accordingly, the size of any layer j in the Gaussian pyramid structure can be expressed as (H / 2 j )×(W / 2 j ), ∫j∈[0,J].

[0078] For the obtained multiple processed images, each processed image may contain multiple marker points. The images can be cropped according to a preset uniform size with each marker point as the center, for example, they can be cropped into 64×64 image blocks. In this way, multiple image blocks with different resolutions and each containing a marker point at the center point can be obtained.

[0079] From the above, it can be seen that the marked landmarks in each training image include landmarks of the same type, and only the specific coordinate values ​​differ. In one implementation, the landmarks in each training image are numbered and marked in a unified identification method. For example, for training image 1, the landmarks in training image 1 can be numbered 1-1, 1-2, 1-3, etc. Similarly, for training image 2, the landmarks in training image 2 can be numbered 2-1, 2-2, 2-3, etc. Among them, 1-2 and 2-1 are landmarks of the same type, for example, both represent the jaw point, and 1-2 and 2-2 are also landmarks of the same type.

[0080] See also Figure 3 , based on the above, when cropping the processed image, it can be performed in the following way:

[0081] S10221, for each marking point in each training image, based on the original coordinates of the marking points with the same number in the multiple training images, calculate the average coordinates of the marking point with the number.

[0082] S10222, based on the average coordinates of each marked point in the processed image, cropping is performed at a preset uniform size with the marked point as the center.

[0083] In this embodiment, considering that the specific coordinate values ​​of the same type of markers in different training images are different, in order to avoid introducing too much unnecessary information in the subsequent model, when cropping the image block, the average coordinates of the marker points calculated can be used as the center for cropping. That is, the average coordinate position of the marker points in the training image is used as the expected attention area, and each processed image is cropped according to this expected attention area.

[0084] The average coordinates of each marker point are calculated based on the original coordinates of the marker points with the same number in multiple training images. For example, if the training images contain 100 images, there should be 100 marker points numbered 1. The average value is calculated based on the original coordinates of the 100 marker points, and the average value can be used as the average coordinate of the marker point numbered 1.

[0085] After obtaining multiple image blocks of different resolutions in the above manner, the corresponding feature maps can be obtained through model processing. In this embodiment, it is hoped that multi-precision feature information can be further extracted from the expected attention area. The problem of extracting multi-precision features is a common problem in deep learning. Please refer to Figure 4 In this embodiment, the feature image corresponding to each image block is obtained in the following way.

[0086] S1023, importing each of the obtained image blocks into a constructed improved human posture recognition model, where the improved human posture recognition model includes multiple parallel sub-networks, the multiple sub-networks have a unified input channel, and the multiple sub-networks have cross-layer connection channels.

[0087] S1024, using multiple sub-networks to process image blocks with different resolutions, and fusing the image blocks with different resolutions to obtain image blocks with fused resolution.

[0088] S1025, processing the image block with the fused resolution and the original multiple image blocks with different resolutions into feature maps of set sizes respectively and outputting them.

[0089] In order to further extract multi-precision features, an improved human posture recognition model HRNET (High-Resolution Net) is used in this embodiment. HRNET starts with a high-resolution subnetwork as the first stage, gradually increases the subnetwork from high resolution to low resolution, forms more stages, and connects the multi-resolution subnetworks in parallel. In this embodiment, the original HRNET is improved by changing the original multi-input channels to a single input channel, that is, multiple subnetworks have a unified input channel. In addition, there are cross-layer connection channels between multiple subnetworks. That is, based on the cross-layer connection channel, the high-resolution feature map can be downsampled to low resolution, and then restored to high resolution from the low-resolution feature map.

[0090] The network can perform multiple multi-scale fusions in the middle part, so that each high-resolution to low-resolution representation can repeatedly receive information from other parallel representations, thereby obtaining rich high-resolution feature maps.

[0091] See also Figure 5 , where the model architecture on the left is illustrated by taking the model containing four parallel sub-networks as an example, where the resolution of the network gradually decreases to half of the previous level, and the width, that is, the channel data, doubles. The first level can contain four residual units, each of which consists of a bottleneck layer with a width of 64 followed by a 3×3 convolution unit. The convolution unit is to reduce the width of the feature map to a set value. The 2nd, 3rd, and 4th levels contain 1, 4, and 3 exchange units, respectively. One exchange unit contains 4 residual units, and each residual unit contains two 3×3 convolutions. In this way, there are a total of 8 exchange units, that is, 8 multi-scale fusions are performed.

[0092] In this way, based on the resolution of the original image block, the feature maps of different scale stages are saved through cross-layer processing, which can retain the multi-precision features of the image to a greater extent. In this embodiment, the multi-scale feature map obtained by the cross-layer channel can be converted from the original 64×64 to 2×2, and then all are unified into 32×32 feature maps through bilinear interpolation, which is convenient for later processing of the feature map.

[0093] Based on the above, the feature map can be imported into the constructed multi-layer neural network model to train it. The multi-layer neural network model includes a first model and a second model. The first model contains multiple channels, and each feature map can correspond to each channel. Figure 6 When the model is constructed in the above step S103, it can be realized by the following steps:

[0094] S1031, importing each of the feature maps into the constructed first model, using the corresponding channel to process each of the feature maps according to the weights they have, to obtain a fused map after fusion processing according to different weights.

[0095] S1032, importing the fusion graph into the constructed second model for training, and under the guidance of the loss function constructed based on the original coordinates of the marked points in the fusion graph and the output coordinate information, adjusting the weights of each channel in the first model and adjusting the model parameters of the second model, and then continuing training until the loss function converges.

[0096] In this embodiment, the feature map obtained by the above method has a large number of layers, and direct post-processing will converge slowly. Therefore, it is necessary to further process the information contained in multiple feature maps to extract important information and reduce the weight of irrelevant information. The first model in this embodiment includes multiple channels, and the weight control of the feature map is equivalent to controlling the channel weight in the first model.

[0097] In this embodiment, two operations, Squeeze and Excitation, are mainly used in the first model. The Squeeze operation is used to encode the features on all channels into a more abstract and concise global feature, and the reduction factor used can be 16. The Excitation operation is used to learn the nonlinear mapping relationship between each channel. Since the existence of multi-channel features is allowed, the channel weights finally learned are not mutually exclusive.

[0098] Specifically, each channel may have an initial weight, and the sum of the weights of all channels may be 1. After multiple feature maps are imported into the first model, each feature map is processed in the corresponding channel. Each channel has its own weight. During processing, each channel processes each feature map according to its weight, so that a fusion map after processing with different weights can be output. That is, a more abstract and streamlined global feature map.

[0099] Please refer to Figure 5 , in the model architecture of the middle part, the first model may include a linear convolution layer, a ReLU activation layer, a linear convolution layer, and a Sigmoid activation layer connected in sequence. Among them, the two linear convolution layers are mainly used for dimensionality reduction processing to restore the original dimension of the feature image. The ReLU activation layer is in the middle of the linear layer and is used for sparse weights. The Sigmoid activation layer is used at the end to put the weights in the range of (0,1) to avoid the problem of excessive channel weights.

[0100] The above structure is helpful to reduce the complexity of the model and improve the generalization ability. The above channel attention mechanism can make the model better able to identify the features of each channel. Finally, the final feature map presented in the form of a matrix can be obtained, that is, the weights of each channel corresponding to the fusion map.

[0101] Based on the above, the obtained fusion graph can be imported into the constructed second model, such as Figure 5 As shown in the model architecture on the right side of the figure, the second model can be a D-MLP model. D-MLP is a three-layer linear regression network. After being processed by the D-MLP model, the coordinate information of the landmark points in the fusion image can be extracted. The landmark points in the fusion image have original coordinates, so the loss function can be constructed based on the original coordinates and the output coordinate information. The process of model training is also the process of minimizing the loss function. Model training can be repeated for multiple times, and the function value of the loss function obtained after each iteration can be used as a basis for judging whether the model training meets the requirements.

[0102] During multiple iterations of the model, the weights of each channel in the first model and the model parameters of the second model can be adjusted before each iteration, and training is continued on this basis until the loss function converges.

[0103] In this embodiment, by adjusting the weights of each channel of the first model, the weight distribution of the first model can set a larger weight for the feature map that is more helpful for the accuracy of the final landmark prediction, and set a smaller weight for the feature map that has a smaller impact on the final landmark. Thus, a fusion map can be further obtained according to a reasonable fusion weight.

[0104] In addition, in this embodiment, the second model may include a first linear layer, a first ReLU activation layer, a first Dropout layer, a second linear layer, a second ReLU activation layer, a second Dropout layer, and a third linear layer connected in sequence. The first linear layer has an output channel of 512, the second linear layer has an output channel of 128, and the third linear layer has an output channel of 2.

[0105] During the training process, the weights of D-MLP can be initialized by orthogonal regularization. Orthogonal regularization of weights can achieve faster and better loss convergence. In addition, a Dropout layer is added between layers, and its Dropout probability can be set to 0.3, that is, the weights are discarded with a probability of 0.3 to reduce the overfitting phenomenon of the model and improve the iteration efficiency.

[0106] In this embodiment, the coordinate information of the marker point output by the second model is actually an offset, which is relative to the average coordinate of the marker point. Therefore, when constructing the loss function, it is constructed based on the sum of the offset and the average coordinate and the original coordinate of the marker point.

[0107] During the training process of the model, the loss function directly affects the training depth of the network model. The most commonly used loss function in the depth estimation task is to directly compare the predicted value and the true value pixel by pixel and accumulate the difference. However, direct accumulation may cause positive and negative values ​​to cancel each other out. Therefore, in this embodiment, the constructed loss function can adopt the method of Least Absolute Deviations (LAD), that is, L1 generalization.

[0108] That is, the loss function is constructed based on the absolute value of the difference between the original coordinates of the landmark point and the coordinate information output by the model. In this embodiment, the constructed loss function can be as follows.

[0109]

[0110] Among them, Y i represents the target value, that is, the original coordinates of the marker point. i ) represents the estimated value, that is, the coordinate information output by the model. S represents the total loss, that is, the sum of the n landmarks.

[0111] Compared with the use of the mean squared error (MSE) of the commonly used regression model as the loss function, the size of the gradient propagated backward by LAD is only affected by the target direction, not the distance. When assuming that the error of linear regression obeys the Laplace distribution, the minimum absolute deviation regression is the maximum likelihood estimate of the parameters.

[0112] In this embodiment, under the guidance of the above-constructed loss function, the training optimization of the first model and the second model can be performed until the training requirements are met, and a recognition model trained by the first model and the second model is obtained.

[0113] The image landmark recognition model training method provided in this embodiment can first normalize the pixels to avoid interference caused by excessive pixel value differences. In addition, the origin of the coordinate system is changed to the center point of the image to avoid the problem of predicted landmark offset that originally took the upper left corner of the image as the origin.

[0114] On this basis, multiple image blocks with different resolutions can be obtained through the Gaussian pyramid structure, so that the model can learn the feature information of images with different resolutions. In addition, during the model training process, the model channel weights can be adjusted under the guidance of the loss function, so that feature maps that are more helpful for the accuracy of landmark prediction are given larger weights, feature maps that have less impact on the accuracy of landmark prediction are given smaller weights, and the fusion weights of feature maps are reasonably distributed.

[0115] The recognition model trained in the above way can be used to predict the landmarks in the image in the actual stage. Figure 7 The method provided in this embodiment may further include the following steps:

[0116] S201, obtaining an image to be recognized, inputting the image to be recognized into a pre-trained recognition model, and obtaining coordinate information of each marker point contained in the image to be recognized, wherein the coordinate information is a relative position deviation of the marker point relative to the template position.

[0117] S201, obtaining a final coordinate value of the marking point based on the relative position deviation and the template position.

[0118] The image to be identified can be any image that needs to be identified by landmarks, such as a lateral skull radiograph, or other images. The image to be identified is input into the recognition model obtained by prediction training for processing. Before importing, the image to be identified can also be processed to obtain its corresponding feature map, and the feature map is imported into the recognition model. Alternatively, the recognition model can also include a layer for processing the feature map, which is not specifically limited.

[0119] The recognition model can process the image to be recognized, and then output the coordinate information related to the marker points in the image to be recognized. The coordinate information can be a relative position deviation relative to the template position, and the template position can be the average position of the marker points already stored in the database. That is, the average position of each type of marker point can be calculated based on big data. However, the marker points in the image to be recognized that actually need to be predicted should have a certain deviation from the average position, that is, the recognition model can output the relative position deviation.

[0120] The final coordinate value of the marker point can be obtained by adding the relative position deviation obtained by the recognition model to the template position.

[0121] The method provided in this embodiment can learn the high-precision features of medical images and obtain automatic recognition results of landmarks with smaller errors when applied to landmark point prediction of lateral cranial radiographs, for example. Compared with traditional optimization methods, the method of the present application reduces the average error of landmark point recognition and enables all tasks to obtain satisfactory computational delays and lower energy consumption. Compared with most existing learning-based methods, the Gaussian pyramid preprocessing based on the attention mechanism reduces the processing of irrelevant information. This method decomposes the entire optimization problem into two sub-problems and solves them separately, effectively reducing the complexity of the algorithm. By improving the optimization action variable generation strategy and experience sampling strategy of the deep learning algorithm, the proposed algorithm is easy to converge.

[0122] See also Figure 8 , is a schematic diagram of an exemplary component of an electronic device provided in an embodiment of the present application, and the electronic device may be the above-mentioned terminal device with functions such as image processing and data analysis and processing. The electronic device may include a storage medium 110, a processor 120, an image landmark recognition model training device 130 and a communication interface 140. In the present embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the electronic device and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.

[0123] The image landmark point recognition model training device 130 can be understood as the above-mentioned electronic device, or the processor 120 of the electronic device, or can be understood as a software function module independent of the above-mentioned electronic device or processor 120 that implements the above-mentioned image landmark point recognition model training method under the control of the electronic device.

[0124] like Fig. 9As shown, the image landmark recognition model training device 130 may include an acquisition module and a training module 132. The functions of each functional module of the image landmark recognition model training device 130 are described in detail below.

[0125] The acquisition module 131 is used to obtain multiple training images, each of which contains multiple marking points, each of which has original coordinates. For each training image, multiple image blocks of different resolutions of the training image are obtained based on the Gaussian pyramid structure, and a feature map corresponding to each image block is obtained.

[0126] It can be understood that the obtaining module 131 can be used to execute the above steps S101 and S102. The detailed implementation of the obtaining module 131 can refer to the above contents related to steps S101 and S102.

[0127] The training module 132 is used to import each of the feature maps into the constructed multi-layer neural network model, and train the model based on the loss function constructed based on the original coordinates of the marked points in each of the feature maps and the coordinate information output by the model, until the loss function converges, thereby obtaining a recognition model trained based on the multi-layer neural network model.

[0128] It can be understood that the training module 132 can be used to execute the above step S103. The detailed implementation of the training module 132 can refer to the above content related to step S103.

[0129] In a possible implementation, the model training device further includes a normalization module, which can be used to:

[0130] For each of the training images, normalization is performed on the pixel values ​​of all pixels in the training image so that the pixel value of each pixel after normalization is within a preset pixel distribution range.

[0131] In a possible implementation, the obtaining module 131 may be specifically used for:

[0132] For each training image, multiple processed images of different resolutions and sizes corresponding to the training image are obtained based on the Gaussian pyramid structure;

[0133] For each of the processed images, based on the position information of the marking point in the processed image, the images are cropped with the marking point as the center according to a preset uniform size to obtain a plurality of image blocks with different resolutions after the processing of the plurality of processed images.

[0134] In a possible implementation manner, the marking points in each of the training images are numbered and marked in a unified identification manner, and the acquisition module 131 may be specifically used for:

[0135] For each marking point in each training image, based on the original coordinates of the marking points with the same number in the plurality of training images, calculate the average coordinates of the marking point with the number;

[0136] Based on the average coordinates of each marked point in the processed image, the image is cropped to a preset uniform size with the marked point as the center.

[0137] In a possible implementation, the acquisition module 131 may be used to obtain the feature map in the following manner:

[0138] Importing each of the obtained image blocks into a constructed improved human posture recognition model, wherein the improved human posture recognition model includes multiple parallel sub-networks, the multiple sub-networks have a unified input channel, and the multiple sub-networks have cross-layer connection channels;

[0139] Using multiple sub-networks to process image blocks of different resolutions, and fusing the image blocks of different resolutions to obtain image blocks with fused resolutions;

[0140] The image blocks with the fused resolution and the original multiple image blocks with different resolutions are processed into feature maps of set sizes and output.

[0141] In a possible implementation, the multi-layer neural network model includes a first model and a second model, the first model includes a plurality of channels, each of the feature graphs corresponds to each of the channels, and the training module 132 can be specifically used for:

[0142] Importing each of the feature maps into the constructed first model, processing each of the feature maps using the corresponding channels according to the weights they have, and obtaining fused maps after fusion processing according to different weights;

[0143] The fusion graph is imported into the constructed second model for training, and under the guidance of the loss function constructed based on the original coordinates of the marked points in the fusion graph and the output coordinate information, the weights of each channel in the first model are adjusted, and the model parameters of the second model are adjusted, and then training is continued until the loss function converges.

[0144] In a possible implementation, the loss function is constructed based on the absolute value of the difference between the original coordinates of the marking point and the coordinate information output by the model.

[0145] In a possible implementation manner, the original coordinates of each of the marking points are coordinates in a coordinate system constructed with the center point of the training image where the marking point is located as the origin.

[0146] In a possible implementation, the model training device may further include a processing module, which may be specifically used to:

[0147] Acquire an image to be recognized, input the image to be recognized into a pre-trained recognition model, and obtain coordinate information of each marker point contained in the image to be recognized, where the coordinate information is a relative position deviation of the marker point relative to the template position;

[0148] Based on the relative position deviation and the template position, the final coordinate value of the marking point is obtained.

[0149] In an embodiment of the present application, corresponding to the above-mentioned image landmark point recognition model training method, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is run, each step of the above-mentioned image landmark point recognition model training method is executed.

[0150] Among them, the steps executed when the aforementioned computer program is running will not be described one by one here, and reference can be made to the explanation of the image landmark point recognition model training method in the previous article.

[0151] In summary, the image landmark point recognition model training method and device provided in the embodiments of the present application obtain multiple training images, and each training image contains multiple marking points, and each marking point has original coordinates. For each training image, multiple image blocks of different resolutions of the training image are obtained based on the Gaussian pyramid structure, and a feature map corresponding to each image block is obtained. Each feature map is imported into the constructed multi-layer neural network model, and the model is trained based on the loss function constructed based on the original coordinates of the marking points in each feature map and the coordinate information output by the model. When the loss function converges, a recognition model obtained by training based on the multi-layer neural network model is obtained. The present application can obtain multiple image blocks of different resolutions through the Gaussian pyramid structure, so that the model can learn the feature information of images of different resolutions, thereby improving the accuracy of subsequent image recognition in response to different resolutions.

[0152] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for training an image landmark recognition model, characterized in that: The method comprises: Acquire a plurality of training images, each of the training images comprising a plurality of marking points, each of the marking points having original coordinates, and each of the training images having marking points of the same type; For each training image, a plurality of image blocks of different resolutions of the training image are obtained based on a Gaussian pyramid structure, and a feature map corresponding to each of the image blocks is obtained; Importing each of the feature graphs into the constructed multi-layer neural network model, training the model based on a loss function constructed based on the original coordinates of the marked points in each of the feature graphs and the coordinate information output by the model, until the loss function reaches convergence, thereby obtaining a recognition model trained based on the multi-layer neural network model; The step of obtaining, for each training image, a plurality of image blocks of different resolutions of the training image based on the Gaussian pyramid structure comprises: For each training image, multiple processed images of different resolutions and sizes corresponding to the training image are obtained based on the Gaussian pyramid structure, and each of the processed images contains multiple marking points; for each of the processed images, based on the position information of the marking points in the processed image, the images are cropped according to a preset uniform size with the marking points as the center, so as to obtain multiple image blocks of different resolutions after processing the multiple processed images, and each of the image blocks contains a marking point at the center point.

2. The image landmark recognition model training method according to claim 1, characterized in that: Before the step of obtaining a plurality of image blocks of different resolutions of the training image based on the Gaussian pyramid structure, the method further includes: For each of the training images, normalization is performed on the pixel values ​​of all pixels in the training image so that the pixel value of each pixel after normalization is within a preset pixel distribution range.

3. The image landmark recognition model training method according to claim 1, characterized in that: The marking points in each of the training images are numbered and marked in a unified identification manner; The step of cropping the image according to a preset uniform size with the marked point as the center based on the position information of the marked point in the processed image comprises: For each marking point in each training image, based on the original coordinates of the marking points with the same number in the plurality of training images, calculate the average coordinates of the marking point with the number; Based on the average coordinates of each marked point in the processed image, the image is cropped to a preset uniform size with the marked point as the center.

4. The image landmark recognition model training method according to claim 1, characterized in that: The step of obtaining the feature map corresponding to each of the image blocks comprises: Importing each of the obtained image blocks into a constructed improved human posture recognition model, wherein the improved human posture recognition model includes multiple parallel sub-networks, the multiple sub-networks have a unified input channel, and the multiple sub-networks have cross-layer connection channels; Using multiple sub-networks to process image blocks of different resolutions, and fusing the image blocks of different resolutions to obtain image blocks with fused resolutions; The image blocks with the fused resolution and the original multiple image blocks with different resolutions are processed into feature maps of set sizes and output.

5. The image landmark recognition model training method according to claim 1, characterized in that: The multi-layer neural network model includes a first model and a second model, the first model includes a plurality of channels, and each of the feature maps corresponds to each of the channels; The step of importing each of the feature graphs into the constructed multi-layer neural network model, and training the model based on the original coordinates of the marked points in each of the feature graphs and the loss function constructed by the coordinate information output by the model, comprises: Importing each of the feature maps into the constructed first model, processing each of the feature maps using the corresponding channels according to the weights they have, and obtaining fused maps after fusion processing according to different weights; The fusion graph is imported into the constructed second model for training, and under the guidance of the loss function constructed based on the original coordinates of the marked points in the fusion graph and the output coordinate information, the weights of each channel in the first model are adjusted, and the model parameters of the second model are adjusted, and then training is continued until the loss function converges.

6. The image landmark recognition model training method according to claim 1, characterized in that: The loss function is constructed based on the absolute value of the difference between the original coordinates of the marked point and the coordinate information output by the model.

7. The image landmark recognition model training method according to claim 1, characterized in that: The original coordinates of each of the marking points are coordinates in a coordinate system constructed with the center point of the training image where the marking point is located as the origin.

8. The image landmark recognition model training method according to any one of claims 1 to 7, characterized in that: The method further comprises: Acquire an image to be recognized, input the image to be recognized into a pre-trained recognition model, and obtain coordinate information of each marker point contained in the image to be recognized, where the coordinate information is a relative position deviation of the marker point relative to the template position; Based on the relative position deviation and the template position, the final coordinate value of the marking point is obtained.

9. An image landmark recognition model training device, characterized in that: The device is used to implement the image landmark recognition model training method according to any one of claims 1 to 8, comprising: An acquisition module is used to acquire multiple training images, each of which contains multiple marking points, each of which has original coordinates, and for each training image, obtain multiple image blocks of different resolutions of the training image based on a Gaussian pyramid structure, and obtain a feature map corresponding to each image block; The training module is used to import each of the feature maps into the constructed multi-layer neural network model, and train the model based on the loss function constructed based on the original coordinates of the marked points in each of the feature maps and the coordinate information output by the model, until the loss function converges, thereby obtaining a recognition model trained based on the multi-layer neural network model.

Images