Positioning model construction method, image landmark point positioning method, and electronic device

Abstract

This application provides a method for constructing a localization model and a method for locating image landmarks. First, the initial sample image is preprocessed and augmented. Then, an initial target region is selected, and the first round of landmark location is performed within this region. A second type of landmark with lower accuracy is then selected. Next, an automatic region optimization model is used to iteratively optimize the target region to improve overall localization accuracy. The iteration continues until the localization accuracy meets the requirements or reaches the preset maximum number of iterations, resulting in a localization model that meets the localization needs. The original image is input into the localization model to obtain its output data. Coordinate reconstruction is then performed based on the output data, ultimately yielding the original image with landmarks. This method achieves automatic landmark location on the image with a high level of accuracy, providing effective assistance for doctors in formulating treatment plans. It eliminates the need for doctors to repeatedly perform simple and tedious tasks, reducing the waste of human resources.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for constructing a localization model, a method for locating image landmarks, and an electronic device. Background Technology

[0002] The localization of landmarks on medical images is a prerequisite for many medical image analysis tasks, such as image segmentation and registration, and for determining clinical treatment plans, such as orthodontics. Taking orthodontics as an example, doctors need to mark anatomical landmarks on anteroposterior and lateral X-ray images obtained from cranial localization photography. Then, based on the marked landmarks, they need to measure angles and distances and analyze the geometric structure. After comparing with normal indicators, they can determine the mechanism of the deformity and formulate a correction plan. However, this requires doctors to frequently perform repetitive landmark localization work, resulting in a waste of human resources. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a method for constructing a localization model, a method for locating image landmarks, and an electronic device to solve the problem of doctors frequently performing repetitive landmark localization work, which results in a waste of human resources.

[0004] To achieve the above objectives, the first aspect of this application provides a method for constructing a positioning model, comprising:

[0005] Acquire an initial sample image, and perform preprocessing and augmentation on the initial sample image to obtain a sample image and sample coordinates;

[0006] In the sample image, an initial target region is determined for the first iteration;

[0007] Feature extraction is performed on each initial target region, and localization is performed within each initial target region based on the extracted features to determine at least one marker point;

[0008] Based on the sample coordinates, the positioning accuracy of each marker point is calculated to obtain the first type of marker points, the second type of marker points, and the average positioning accuracy.

[0009] Iterative positioning is performed based on the second type of marker points obtained in the previous positioning round until the average positioning accuracy reaches a preset threshold or the maximum number of iterations is reached, at which point the iterative positioning ends and the positioning model is obtained.

[0010] Optionally, the preprocessing and augmentation of the original image to obtain the feature image includes:

[0011] Separate the coordinates of the labeled real markers in the initial sample image, perform a validity check on the coordinates, and store the valid coordinates separately.

[0012] Based on the difference in grayscale values ​​between redundant and content areas in the sample image, the redundant areas of the sample image are removed to obtain a modified image.

[0013] The modified image is resized, and the resized modified image is then augmented to obtain the sample image.

[0014] The coordinates are scaled proportionally according to the scaling ratio of the size adjustment to obtain the sample coordinates of the real marker point in the sample image.

[0015] Optionally, determining the initial target region in the sample image for the first iteration includes:

[0016] The real landmarks in the sample image are classified based on a preset classification method.

[0017] The geometric center of each type of real marker is used as the real center coordinate, and a cross-entropy loss function is constructed based on the real center coordinate;

[0018] Based on the cross-entropy loss function, the geometric center of each type of marker point is predicted using a sliding window-based deep self-attention model, resulting in multiple predicted center coordinates.

[0019] Using the predicted center coordinates as the center, a central candidate range is selected according to the preset candidate side length;

[0020] Based on the candidate center range, the initial target region is created using the predicted center coordinates as the center by an automatic regional optimization model.

[0021] Optionally, the step of creating the initial target region using the region automatic optimization model with the predicted center coordinates as the center includes:

[0022] The initial side length is determined using an automatic regional optimization model, thus obtaining the current target region;

[0023] Based on the given step size, all points within the center candidate range are traversed, and the score for each point under the initial side length is calculated.

[0024] Using the highest-scoring point as the center point of the region, the initial side length of the current target region is optimized using the annealing algorithm to obtain the initial target region.

[0025] Optionally, after obtaining the initial target region, the method further includes:

[0026] Based on the coordinates of the center point of the region (C) x C y ) and the side length E of the initial target regionx E y The sample coordinates are updated; the update formula is:

[0027] lab x '=lab x -C x +E x

[0028] lab y '=lab y -C y +E y

[0029] Among them, (lab x ,lab y (lab) represents the coordinates of the original true landmark point in the sample coordinates. x ',lab y ') represents the updated coordinates of the actual marker point in the sample coordinates.

[0030] Optionally, the step of extracting features from each of the initial target regions and locating them within each initial target region based on the extracted features to determine at least one marker point includes:

[0031] The image features of the initial target region are extracted using a sliding window-based deep self-attention model, and the extracted features are analyzed using a fully connected layer based on a fully connected mechanism to generate at least one readable marker point.

[0032] Optionally, the step of calculating the positioning accuracy of each marker point based on the sample coordinates to obtain the first type of marker points, the second type of marker points, and the average positioning accuracy includes:

[0033] Calculate the Euclidean distance between each of the stated markers and the updated true markers, and calculate the mean of all the stated Euclidean distances;

[0034] All marker points whose Euclidean distance is greater than the mean are designated as the second type of marker points;

[0035] All markers whose Euclidean distance is less than or equal to the mean are designated as the first type of markers.

[0036] The ratio of the number of the first type of markers to the total number of markers is determined as the average positioning accuracy.

[0037] Optionally, the iterative positioning based on the second type of markers obtained in the previous round of positioning continues until the average positioning accuracy reaches a preset threshold or the maximum number of iterations is reached, at which point the iterative positioning ends, resulting in the positioning model, including:

[0038] Based on the second type of marker points, the target area in the previous iteration is re-divided using an automatic regional optimization model. The target area is then repositioned to obtain multiple new marker points. The average positioning accuracy of each new marker point in this iteration is calculated until the average positioning accuracy reaches a preset threshold or the maximum number of iterations is reached. The iterative positioning ends, and the coordinates of the marker points in the last iteration, the coordinates of the center point of the region in each iteration in the previous iteration, and the size of the target area in each iteration are output to obtain the positioning model.

[0039] A second aspect of this application provides a method for locating image landmarks, including:

[0040] Acquire the original image and input the original image into the localization model, and receive the coordinates of the marker points of the last iteration, the coordinates of the center point of the region in each iteration in the previous iteration, and the size of the target region in each iteration output by the localization model;

[0041] Based on the coordinates of the region center point in each iteration in the previous iteration and the size of the target region in each iteration, the coordinates of the marker points in the last iteration are restored to obtain the coordinates of the marker points on the original image.

[0042] Marker points are drawn on the original image based on the coordinates of the marker points to obtain the original image with the marker points.

[0043] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method provided in the first aspect of this application.

[0044] As can be seen from the above, the localization model construction method, image landmark localization method, and electronic device provided in this application, after initializing the initial target region of the first iteration, use a sliding window-based depth self-attention model to extract features from each target region, complete the localization of landmarks based on the features, and calculate the localization accuracy of this iteration; then, select the second type of landmarks with poor accuracy in this round, complete the re-division of the target region, and complete the iterative localization of landmarks in the new target region. The localization accuracy of the landmarks to be localized and the target region are continuously updated according to the localization accuracy of the current iteration until the localization accuracy of all landmarks is higher than a preset threshold or the iteration reaches the maximum number of iterations, obtaining the localization model output data. Then, coordinate reconstruction is performed based on the output data, and finally, the original image with landmarks is obtained. This achieves automatic localization of landmarks on the image with a high level of accuracy, providing effective assistance for doctors in formulating treatment plans, freeing doctors from repeating simple and tedious work, and reducing the waste of human resources. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of the image marker localization method according to an embodiment of this application;

[0047] Figure 2 This is a flowchart illustrating the method for constructing the positioning model in an embodiment of this application;

[0048] Figure 3 A flowchart illustrating the preprocessing and augmentation of embodiments of this application;

[0049] Figure 4 A flowchart illustrating the determination of the initial target region in this application embodiment;

[0050] Figure 5 A flowchart illustrating the creation of the initial target region for embodiments of this application;

[0051] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0053] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0054] As described in the background section, in current domestic clinical diagnosis, the localization of anatomical landmarks is often done manually by doctors based on their medical background knowledge and practical experience. Although manually locating anatomical landmarks is relatively easy for doctors, locating numerous landmarks remains a tedious and time-consuming task. Furthermore, image quality and individual patient differences further increase the workload of manual landmark localization. With the development of modern computer technology, the application of machine learning in medicine is becoming increasingly common and continues to demonstrate great potential. The excellent learning ability of neural networks allows them to learn from the clinical experience of professional doctors and apply it to medical diagnosis, reducing the manpower required for these complex tasks.

[0055] The positioning model construction method and image landmark positioning method provided in this application preprocess the initial sample image, balances the histogram of the skeletal and muscle tissue parts in the image to enhance detail, and expands the training data. Then, while determining the initial target area for the first iteration, the coordinate information of the real landmarks in the initial target area is updated. Then, the neural network is used to predict the coordinates of the landmarks in all the divided initial target areas and counts the positioning accuracy of the landmarks obtained in this iteration. Some second-class landmarks with poor accuracy are screened out. The target positioning area containing the second-class landmarks is redivided using an automatic region optimization model, and the coordinate labels of the real landmarks are updated. Using the updated labels, the coordinates of the landmarks are predicted again, and the iterative positioning is repeated to realize the iterative update of the positioning target area to improve the overall positioning accuracy. The iteration ends when the positioning accuracy meets the requirements or reaches the preset maximum number of iterations, and the positioning model output data is obtained. Then, the coordinates are restored according to the output data, and finally the original image with landmarks is obtained. This realizes the automatic positioning of landmarks on the image with a high level of accuracy, which provides effective assistance for doctors to formulate treatment plans, so that doctors do not have to repeat simple and tedious work and reduce the waste of human resources.

[0056] In some embodiments, such as Figure 1 As shown, the method for locating image landmarks includes:

[0057] Step 1: Acquire the original image and input it into the localization model. Receive the coordinates of the marker points from the last iteration, the coordinates of the region center point in each iteration in the previous iteration, and the size of the target region in each iteration, all output by the localization model.

[0058] In this step, the localization model is pre-built and trained. This localization model is an overall model composed of multiple models. For example, the localization model includes a sliding window-based deep self-attention model and a region automatic optimization model (a sliding window-based deep neural network self-attention model). The sliding window-based deep self-attention model is mainly used to predict the geometric center of similar marker points, that is, to determine the coordinates of the region center point in each iteration. The sliding window-based deep self-attention model can also be used to determine the coordinates of the marker points in each iteration. The region automatic optimization model is mainly used to determine the target region in each iteration and to optimize the target region to form a new target region for the next iteration. Therefore, the localization model uses the coordinates of the marker points in the last iteration output by the sliding window-based deep self-attention model and the coordinates of the region center point in the previous iteration, and uses the region automatic optimization model to output the size of the target region in each iteration.

[0059] Step 2: Based on the coordinates of the center point of the region in each iteration in the previous iteration and the size of the target region in each iteration, restore the coordinates of the marker points in the last iteration to obtain the coordinates of the marker points on the original image.

[0060] In this step, for example, if the coordinates of the marker point in the last iteration are (x, y), and the coordinates of the center point of the region obtained in the i-th iteration in the previous iteration are (C... xi C yi The size of the target region obtained in the i-th iteration is E. xi E yi The coordinates of the marker points (P) on the original image were calculated using the coordinate reconstruction formula. x ,P y ), where the coordinate restoration formula is:

[0061]

[0062]

[0063] Where N is the total number of iterations, and its value can be the maximum number of iterations or the number of iterations when the average positioning accuracy first reaches the preset threshold.

[0064] Step 3: Draw the marker points on the original image according to the coordinates of the marker points to obtain the original image with the marker points.

[0065] In this step, the coordinates of the marker points on the original image are calculated according to the coordinate restoration formula, and the marker points are drawn on the original image. This realizes the restoration of the marker point coordinates to the original image after N iterations, which facilitates the doctor's subsequent operation and disease diagnosis.

[0066] In summary, the image landmark positioning method provided in this application, after obtaining the output data of the positioning model, performs coordinate reconstruction based on the output data to finally obtain the original image with landmarks. This achieves automatic positioning of landmarks on the image with a high level of accuracy, providing effective assistance for doctors in formulating treatment plans, eliminating the need for doctors to repeatedly perform simple and tedious tasks, and reducing the waste of human resources.

[0067] In some embodiments, such as Figure 2 As shown, the method for constructing the localization model includes:

[0068] Step 100: Obtain the initial sample image, and perform preprocessing and augmentation on the initial sample image to obtain the sample image and sample coordinates.

[0069] In this step, sample images with anatomical landmark coordinates annotated by professional physicians are used as the dataset to train and test the performance of the sliding window-based depth self-attention model and the region automatic optimization model. Although the medical imaging process ensures the stability of the examinee's posture and minimizes interference from environmental factors, considering the significant individual differences among patients, this application embodiment performs preprocessing and augmentation operations on the initial medical sample images to enhance the generalization and feature extraction capabilities of the method and to address the problem of insufficient data volume to some extent.

[0070] The preprocessing process primarily involves cropping redundant backgrounds from the initial sample images to increase their information density. It also unifies the size of different initial sample images in the dataset, modifying them to a size suitable for neural network learning. Furthermore, taking the preprocessing of lateral head X-ray images as an example, adaptive histogram equalization can be performed to enhance dark areas and texture details, improving the feature extraction capability of the neural network. Additionally, various image augmentation methods can be used to expand the training set data, enhancing the model's robustness against interference and its performance on medical images of varying quality, ensuring accurate and reliable localization results.

[0071] Step 200: Determine the initial target region in the sample image for the first iteration.

[0072] This step primarily involves selecting the initial target region for the first iteration. The target region can be understood as the local area where the landmark to be located might appear. First, based on medical background knowledge, the landmarks to be located are pre-classified, and the geometric center coordinates of landmarks of the same class are calculated. For example, a model primarily composed of a deep neural network (a sliding window-based deep self-attention model) can be constructed to optimize the target region. Its input is a pre-processed sample image, and the label is the geometric center of each class of landmarks. The label represents the coordinate data of the landmark or geometric center point. Through supervised learning, the sliding window-based deep self-attention model can predict the center coordinates of the target region corresponding to landmarks of the same class. Subsequently, an automatic region optimization model based on classical machine learning methods is used to optimize the side length of each target region, completing the determination of the initial target region and the updating of the target region in subsequent iterations. This step segments the entire medical sample image into multiple local images where only a certain type of landmark appears. Each local image corresponds to an initial target region, so that independent sliding window-based deep self-attention models can be used to predict landmarks for different target regions, reducing the learning difficulty of the network and increasing localization efficiency and accuracy.

[0073] Step 300: Extract features from each initial target region and locate the region within each initial target region based on the extracted features to determine at least one landmark point.

[0074] This step aims to localize all landmarks in different initial target regions. A sliding window-based deep self-attention model is used to extract features from the images corresponding to the initial target regions. By defining and adjusting parameters such as model depth, window size, and loss function, the model can adaptively adjust for different types of target regions to achieve optimal localization performance and accurately identify at least one landmark in the initial target region. In this step, the model will undergo multiple rounds of training, and the best-performing model will be selected for statistical analysis of the localization results. The main goal is to filter out landmarks with poor localization results within each category based on their average localization accuracy, and use these for subsequent iterative regression of the target region. This allows for targeted optimization of the localization results, improves training speed, and reduces computational overhead.

[0075] Step 400: Calculate the positioning accuracy of each marker point based on the sample coordinates to obtain the first type of marker points, the second type of marker points, and the average positioning accuracy.

[0076] Step 500: Perform iterative positioning based on the second type of landmark points obtained in the previous positioning round until the average positioning accuracy reaches the preset threshold or the maximum number of iterations is reached, then end the iterative positioning and obtain the positioning model.

[0077] In practice, based on the point-by-point localization results of the markers in the initial target area, the average accuracy is first analyzed. If the average localization accuracy of such markers reaches a preset threshold or the maximum number of iterations for the target area is reached during the current iteration, the iterative optimization of the target area is stopped. Otherwise, a second category of markers below the threshold is selected from this category, these points are grouped together, their geometric centers are recalculated, and a new machine learning-based automatic region optimization model is used to reselect the target area for a new round of iteration. In this step, the newly obtained target area is input into the new, parameter-tuned marker localization model. The new localization model uses the parameters from the current iteration as the initial parameters for the next round of training. The localization model with these initial parameters becomes the new localization model for the next iteration. Then, the new localization model is used to re-obtain the localization results of the markers in the target area, and the target area is iterated based on the localization accuracy until the average localization accuracy reaches the preset threshold or the maximum number of iterations is reached, at which point the iterative localization ends, and the final localization model is obtained.

[0078] Then, based on the localization model trained in steps 100-500, after inputting the original image, the coordinates of all the markers on the target area corresponding to them in the last round can be obtained. By using the coordinates of the center point of the area retained in the target area regression process on the original image, the coordinates of the markers can be restored to the original image step by step, which enhances the readability of the results and facilitates the formulation of treatment plans by physicians.

[0079] In summary, the beneficial effects of the embodiments of this application include:

[0080] (1) The implementation methods of the original images and initial sample images in the embodiments of this application are mainly based on medical X-ray images. The data acquisition is relatively convenient and the cost is low. It can be applied to a variety of different medical landmark localization tasks and has strong versatility. The embodiments of this application mainly use deep neural networks and combine them with traditional machine learning methods to ensure batch processing of medical images. It has strong versatility and is suitable for landmark localization tasks in different parts of the body. It helps doctors complete the formulation of treatment plans while avoiding errors caused by subjective reasons that may occur during manual annotation.

[0081] (2) The embodiments of this application use a combination of deep neural network method and traditional machine learning method to effectively complete the division and extraction of target areas of similar medical landmarks, ensuring that the localization model can specifically enhance the localization accuracy of certain landmarks, effectively reducing the overall model complexity, and at the same time ensuring the generalization performance of the model to a certain extent.

[0082] (3) The embodiments of this application use transfer learning to achieve parameter sharing between landmark localization networks of different categories and different iteration rounds, thereby minimizing computational overhead and improving the convergence speed of the model. The construction and training process of the localization model involves multiple iterations, and each iteration requires a new model to be trained. Transfer learning uses the parameters from the previous iteration as the initial parameters for the current iteration, which can accelerate the convergence speed of the model to a certain extent.

[0083] In some embodiments, such as Figure 3 As shown, the original image undergoes preprocessing and augmentation to obtain a feature image, including:

[0084] Step 110: Separate the coordinates of the labeled real markers in the initial sample image, perform a validity check on the coordinates, and store the valid coordinates separately.

[0085] In this step, the labels of the landmarks in the sample images are marked by professional doctors. Each label corresponds to the coordinate information of a landmark, which provides learning data for the subsequent neural network. In order to facilitate the reading of the labels, it is necessary to separate the label corresponding to each initial sample image and store the coordinate information of the labeled landmarks in all initial sample images in a separate table file. At the same time, all coordinate data are checked to filter out samples with missing landmark coordinates, and the legality of the label data is checked in combination with the image size to ensure that there are no labels outside the image range.

[0086] Step 120: Based on the difference in grayscale values ​​between redundant and content areas in the sample image, remove the redundant areas of the sample image to obtain the modified image.

[0087] In this step, the initial sample image needs to be cropped to remove background portions in the upper half that do not contain skeletal structures. For example, the proportion of the skull in the overall initial sample image is estimated using the coordinate information of existing markers to determine the size of the cropping area in each image. Simultaneously, the integrity of the remaining structure is checked using image grayscale values. Since the skeletal structure of the skull exhibits characteristics of high grayscale values ​​and relatively smooth histogram changes in the grayscale histogram, if there are areas with high grayscale values ​​in the cropped area, it is considered that the image cropping has disrupted the overall integrity of the skull, and the cropping range will be reduced to ensure the integrity of the sample image. Furthermore, optionally, adaptive histogram equalization can be performed on the cropped modified image to balance the grayscale levels between different regions in the image and enhance the texture details of the bones and muscles.

[0088] Step 130: Resize the modified image and augment it to obtain the sample image.

[0089] Step 140: Scale the coordinates proportionally according to the scaling ratio of the size adjustment to obtain the sample coordinates of the real marker points in the sample image.

[0090] In practice, all modified images are resized, and the sizes of the modified skull images (front and side views) after background cropping are standardized. Coordinate information is updated according to the scaling ratio, and training and testing sets are divided proportionally for neural network training and performance verification. To enable the neural network to better perform feature extraction, augmentation processing is applied to all modified images in the training set. Augmentation methods mainly include flipping the modified images, scaling them up significantly, and adding Gaussian and salt-and-pepper noise to enhance the robustness of the neural network, thereby improving the accuracy of the sliding window-based deep self-attention model.

[0091] In some embodiments, such as Figure 4 As shown, the initial target region for the first iteration is determined in the sample image, including:

[0092] Step 210: Classify the real landmarks in the sample image based on the preset classification method.

[0093] Step 220: Use the geometric center of each class of real markers as the real center coordinates, and construct the cross-entropy loss function based on the real center coordinates.

[0094] Step 230: Based on the cross-entropy loss function, use a sliding window-based deep self-attention model to predict the geometric center of each type of marker point, and obtain multiple predicted center coordinates.

[0095] Step 240: Using the predicted center coordinates as the center, select the center candidate range according to the preset candidate side length.

[0096] Step 250: Based on the candidate center range, use the region automatic optimization model to create an initial target region centered on the predicted center coordinates.

[0097] In practical implementation, to reduce the prediction difficulty of a single model, improve convergence speed, and enhance model specificity, after acquiring the preprocessed sample images, optionally, all ground truth landmarks are first divided into K classes based on medical background knowledge and referencing doctors' clinical processing habits. For each class of ground truth landmarks, the ground truth center coordinates of the geometric center of this class of ground truth landmarks need to be calculated and saved as new labels. Then, a deep self-attention model based on a sliding window is used to predict the geometric center of each class of landmarks, obtaining multiple predicted center coordinates. The input of this model is the preprocessed sample image, the labels are the ground truth center coordinates of the geometric centers of the K classes of landmarks generated above, and the output of the model is K pairs of coordinates, each pair of coordinates corresponding to the predicted center coordinates of one class of landmarks. Compared to the number of landmarks to be located, the value of K is generally small. Therefore, the cross-entropy loss function Loss is used in the model training process. C Its definition is as follows:

[0098]

[0099] Where e is the natural constant and d is the Euclidean distance between the predicted center coordinates and the true center coordinates. Then, a moment estimation optimizer with a weight decay mechanism is used to minimize the cross-entropy loss function, and the network parameters are updated using the error backpropagation mechanism. After several rounds of training, the optimal set of parameters will be saved to achieve relatively accurate localization of the center coordinates of each type of marker point.

[0100] Furthermore, based on the predicted center coordinates output by the sliding window-based depth self-attention model using optimal parameters, a certain range is selected as the candidate center range for the target region. Then, an initial target region is created using the predicted center coordinates as the center, such as... Figure 5 As shown, it specifically includes:

[0101] Step 251: Use the region automatic optimization model to determine the initial side length and obtain the current target region.

[0102] Step 252: Based on the given step size, traverse all points within the center candidate range and calculate the score for each point under the initial side length.

[0103] Step 253: Using the highest score point as the center point of the region, optimize the initial side length of the current target region using the annealing algorithm to obtain the initial target region.

[0104] In practical implementation, to reduce time costs and computational overhead, the region automatic optimization model can directly determine the initial side lengths in the first iteration, with the initial side lengths in the horizontal and vertical directions being L respectively. x L y And restricting the side lengths in both directions to be the same, the calculation method is as follows:

[0105] L x =L y =d max +L region

[0106] In the formula, d max L represents the distance between the current predicted center coordinates and the farthest point among similar real marker points. region The preset candidate side length is given. Then, based on the given step size, all points within the central candidate range are traversed, and the score of each point under the initial side length is calculated. This application embodiment aims to ensure that all marker points within the target area are relatively evenly distributed, therefore the score calculation method is given as follows:

[0107]

[0108] In the formula, J represents the total number of markers of a certain type, and d i E is the distance from the i-th point to its nearest edge. iLet the initial side length be the nearest edge corresponding to the i-th point. Select the point with the highest score as the optimal center, and use it as the region center point. Optimize the side length of the current target region corresponding to the region center point to obtain the initial target region. Optionally, the optimization process can be completed using the annealing algorithm. This requires the number of iterations N, the number of repeated cooling cycles T, the cooling coefficient a, and the initial temperature T0 at each temperature. The restriction that the side lengths in both directions are the same during model initialization is removed. The optimization algorithm regresses on both side lengths simultaneously. Therefore, the final initial target region should be a rectangle with a variable proportion.

[0109] Optionally, during the side length optimization process, a check for the legality of the region selection is added to ensure that no marker points are missed or the region selection exceeds the range of the original image.

[0110] In some embodiments, after obtaining the initial target region, the method further includes:

[0111] Based on the coordinates (Cx, Cy) of the region's center point and the initial target region's side length E x E y The sample coordinates are updated; the update formula is:

[0112] lab x '=lab x -C x +E x

[0113] lab y '=lab y -C y +E y

[0114] Among them, (lab x ,lab y (lab) represents the coordinates of the original true landmark points in the sample coordinates. x ',lab y ') represents the updated coordinates of the actual marker points in the sample coordinates.

[0115] In some embodiments, feature extraction is performed on each initial target region, and localization is performed within each initial target region based on the extracted features to determine at least one marker point, including:

[0116] The image features of the initial target region are extracted using a sliding window-based deep self-attention model, and the extracted features are analyzed using a fully connected layer based on a fully connected mechanism to generate at least one readable marker point.

[0117] The deep self-attention model based on sliding windows consists of modules such as image segmentation, multi-head self-attention within windows, sliding window multi-head self-attention, and segmentation fusion. The image segmentation module divides the entire image into multiple pixel blocks of size H×W, where H and W represent the height and width of the pixel block, respectively. The segmentation fusion module then adjusts the overall number of channels to form a feature map of size H / 2×W / 2×4C, where C represents the number of channels. Finally, after layer regularization and a linear layer, the feature map size is adjusted to H / 2×W / 2×2C, essentially sampling the pixel blocks to half their original size while doubling the number of channels, enabling the subsequent self-attention module to extract features. The multi-head self-attention mechanism is used to calculate the inner product of each pixel within the current window with other pixels to acquire information. Then, by sliding the window, it includes pixels from different regions, enabling information interaction between windows and compensating for the small receptive field of window-based self-attention calculations. The extracted features are then used to directly regress the coordinates of the marker points using a fully connected layer. This involves multiplying the feature vector by the transposed fully connected weight matrix, adding a bias, and obtaining the coordinates of all marker points to be located within the region. To mitigate overfitting, a random dropout mechanism for neurons is introduced to improve overall anti-overfitting ability and generalization performance.

[0118] Furthermore, since there are generally multiple markers to be located within the same target area, a composite loss function, Loss, is used during the localization process. total Loss total =aLoss focal +bLoss wing Where a and b are the adjustable weights of the two types of losses, and the specific calculation methods for the two different loss functions are as follows:

[0119]

[0120]

[0121] In the above formula, γ, ω, and θ are all set hyperparameters, C is a constant, denoted as C = ω - ωln(1 + ωθ), and J is the total number of a certain type of marker points.

[0122] In some embodiments, the positioning accuracy of each marker point is statistically analyzed based on the sample coordinates to obtain a first type of marker point, a second type of marker point, and an average positioning accuracy, including:

[0123] Calculate the Euclidean distance between each marker and the updated true marker, and calculate the mean of all Euclidean distances.

[0124] All markers whose Euclidean distance is greater than the mean are classified as second-class markers.

[0125] All markers whose Euclidean distance is less than or equal to the mean are classified as Class I markers.

[0126] The ratio of the number of first-class markers to the total number of markers is determined as the average positioning accuracy.

[0127] Furthermore, based on the second type of marker points, the target area in the previous iteration is re-divided using an automatic regional optimization model. The target area is then re-located based on the re-divided area, resulting in multiple new marker points. The average positioning accuracy of each new marker point in this iteration is calculated until the average positioning accuracy reaches a preset threshold or the maximum number of iterations is reached. The iterative positioning ends, and the coordinates of the marker points in the last iteration, the coordinates of the center point of the area in each iteration in the previous iteration, and the size of the target area in each iteration are output, thus obtaining the positioning model.

[0128] In practice, an iterative approach was adopted to narrow down the target area for marker localization, while simultaneously locating multiple markers. This strategy aimed to improve localization accuracy while maximizing learning efficiency and minimizing computational load. The optimization process for the target area using an automatic region optimization model is as follows:

[0129] The primary basis for target region iteration is the localization performance of the existing model in the current region. First, it checks if the average localization accuracy of the markers in the target region during this iteration is higher than a set threshold. If so, the iteration ends; otherwise, it checks the number of iterations for the current region. If the maximum number of iterations is reached, the iteration also ends. When neither condition is met, second-class markers in the target region are selected during this iteration. When multiple second-class markers are selected, they are reclassified and the automatic region optimization model is reused for target region optimization iterations based on this classification. Considering the number of models and the time cost of model training, a separate sliding window-based deep self-attention model for predicting the geometric centers of the selected markers is no longer trained. Instead, the region center point from the previous iteration is used as the initialization center for the region iteration.

[0130] Furthermore, considering the poor predictive performance of the current model for these marker points, a larger candidate range and initial side length for the target region center were set in the new iteration to ensure that no marker points are missed.

[0131] When only one marker is selected as a second-type marker, in order to make full use of the influence information and to make mutual corrections when different markers are located, M markers that are close to the second-type marker will be selected to jointly iterate over the target area until the average positioning accuracy reaches the preset threshold or the maximum number of iterations is reached. Then the iterative positioning ends, and the coordinates of the marker in the last iteration, the coordinates of the center point of the area in each iteration in the previous iteration, and the size of the target area in each iteration are output to obtain the positioning model.

[0132] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0133] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0134] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image marker positioning method described in any of the above embodiments.

[0135] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0136] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0137] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0138] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0139] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0140] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0141] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0142] The electronic devices described above are used to implement the positioning method of the corresponding image markers in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0143] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method of constructing a positioning model, characterized by, The method comprises the following steps: obtaining an initial sample image, and preprocessing and augmenting the initial sample image to obtain a sample image and sample coordinates; wherein the step of obtaining an initial sample image, and preprocessing and augmenting the initial sample image to obtain a sample image and sample coordinates comprises: separating the coordinates of the labeled real landmark points in the initial sample image, and performing a legality check on the coordinates, and separately storing the coordinates after the legality check; removing the redundant regions of the initial sample image according to the gray value difference between the redundant regions and the content regions of the initial sample image to obtain a modified image; adjusting the size of the modified image, and augmenting the adjusted modified image to obtain the sample image; scaling the coordinates according to the scaling ratio of the size adjustment to obtain the sample coordinates of the real landmark points in the sample image; determining an initial target region for the first iteration in the sample image; wherein the step of determining an initial target region for the first iteration in the sample image comprises: classifying the real landmark points in the sample image based on a preset classification method; taking the geometric center of each type of real landmark point as a real center coordinate, and constructing a cross-entropy loss function according to the real center coordinates; predicting the geometric center of each type of landmark point based on a sliding window-based deep self-attention model based on the cross-entropy loss function to obtain a plurality of predicted center coordinates; centering on the predicted center coordinates, constructing a center candidate range according to a preset candidate side length; creating the initial target region with the predicted center coordinates as the center using a region automatic optimization model according to the center candidate range; wherein the step of creating the initial target region with the predicted center coordinates as the center using a region automatic optimization model according to the center candidate range comprises: determining an initial side length using a region automatic optimization model to obtain a current target region; traversing all points in the center candidate range according to a given step size, and calculating the score of each point under the initial side length; taking the point with the highest score as a region center point, and optimizing the initial side length of the current target region using an annealing algorithm to obtain the initial target region; extracting features from each initial target region, and positioning at least one landmark point in each initial target region according to the extracted features; statistically determining the positioning accuracy of each landmark point according to the sample coordinates to obtain a first type of landmark point, a second type of landmark point, and an average positioning accuracy; wherein the positioning accuracy of the first type of landmark point is greater than the positioning accuracy of the second type of landmark point; iteratively positioning the second type of landmark point obtained in the last iteration until the average positioning accuracy reaches a preset threshold or the maximum number of iterations is reached, ending the iterative positioning, and obtaining the positioning model.

2. The method of claim 1, wherein, After obtaining the initial target region, the method further comprises: According to the coordinates of the region center point and the side length of the initial target region , , the sample coordinates are updated; wherein the update formula is: wherein, is a coordinate of the original real landmark point in the sample coordinate, is a coordinate of the updated real landmark point in the sample coordinate.

3. The method of claim 1, wherein, the step of extracting features from each initial target region, and positioning at least one landmark point in each initial target region according to the extracted features comprises: The image features of the initial target region are extracted by using a deep self-attention model based on a sliding window, and the extracted features are analyzed by using a fully connected layer based on a full connection mechanism to generate at least one coordinate-readable landmark point.

4. The method of claim 2, wherein, The positioning accuracy of each landmark point is calculated according to the sample coordinates, and a first type of landmark point, a second type of landmark point, and an average positioning accuracy are obtained. The Euclidean distance between each landmark point and the updated real landmark point is calculated, and the mean value of all the Euclidean distances is calculated. All landmark points with a Euclidean distance greater than the mean value are taken as the second type of landmark point. All landmark points with a Euclidean distance less than or equal to the mean value are taken as the first type of landmark point. The ratio of the number of first type of landmark points to the number of all landmark points is determined as the average positioning accuracy.

5. The method of claim 4, wherein, The second type of landmark point obtained in the last round of positioning is used for iterative positioning until the average positioning accuracy reaches a preset threshold or the maximum number of iterations is reached, and the iterative positioning is ended to obtain the positioning model, including: Based on the second type of landmark point, the target region in the last round of iteration is re-divided using a regional automatic optimization model, and re-positioning is performed according to the re-divided target region to obtain a plurality of new landmark points, and the average positioning accuracy of each new landmark point in the current iteration is calculated until the average positioning accuracy reaches a preset threshold or the maximum number of iterations is reached, and the iterative positioning is ended, and the coordinates of the landmark points in the last round of iteration, the coordinates of the region center points in each round of iteration in the last round of iteration, and the size of the target region in each round of iteration are output to obtain the positioning model.

6. A method for locating a marker point in an image, characterized by including: Obtaining an original image, and inputting the original image into the positioning model of claims 1-5, and receiving the coordinates of the landmark points in the last round of iteration, the coordinates of the region center points in each round of iteration in the last round of iteration, and the size of the target region in each round of iteration output by the positioning model; According to the coordinates of the region center points in each round of iteration in the last round of iteration and the size of the target region in each round of iteration, the coordinates of the landmark points in the last round of iteration are restored to obtain the landmark point coordinates on the original image; According to the landmark point coordinates, the landmark points are drawn on the original image to obtain the original image with landmark points.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the method of claim 6.

Images