Local pelvic bone image rigid registration method, device, equipment and medium

Abstract

The application provides a local pelvic skeleton image rigid registration method, device, equipment and medium, comprising: extracting a 3D pelvic skeleton image from a 3D medical image; generating a 2D overall pelvic skeleton image corresponding to each group of candidate parameter combinations for the 3D pelvic skeleton image; performing feature point matching on a 2D medical image and the 2D overall pelvic skeleton image, and performing dynamic adaptive template matching on the 2D medical image and the 2D overall pelvic skeleton image based on the feature point matching result to determine a target 2D local pelvic skeleton image matched by the 2D medical image; and performing rigid registration on the 3D medical image based on the target 2D local pelvic skeleton image and the corresponding candidate parameter combination, so that the 3D medical image after the rigid registration is aligned with the 2D medical image. The application can effectively solve the rigid registration problem of local 2D X-ray imaging to global 3D CT data.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method, apparatus, device, and medium for rigid registration of local pelvic bone images. Background Technology

[0002] The pelvis is a very important skeletal structure in the human body, composed of the sacrum, coccyx, and the two hip bones. As a large skeletal group in the human body, the pelvis often presents challenges in pelvic surgery due to incomplete imaging. Furthermore, due to limitations in imaging, common registration methods become less accurate in clinical application for pelvic surgery.

[0003] Traditional feature extraction and segmentation methods require manual intervention, making automation difficult. Registration based on grayscale information, which determines transformation parameters by optimizing image grayscale similarity, performs poorly on images with deformation or significant grayscale changes.

[0004] Current deep learning-based methods struggle to train on localized skeletal data. Limited by medical imaging capabilities, intraoperative C-arm X-rays often only capture localized information of the pelvic bones, failing to provide a comprehensive view. Supervised networks typically train on global information. Because localized pelvic images exhibit greater uncertainty compared to complete pelvic bone imaging, it's difficult to construct sufficient datasets, making commonly used supervised deep learning methods challenging for localized pelvic images. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method, apparatus, device and medium for rigid registration of local pelvic bone images, which can effectively handle the problem of rigid registration from local 2D X-ray imaging to global 3D CT data.

[0006] In a first aspect, the present invention provides a method for rigid registration of local pelvic bone images, comprising:

[0007] Acquire the 3D medical image to be registered. The content displayed in the 3D medical image includes the overall pelvic information. Extract the 3D pelvic skeleton image from the 3D medical image.

[0008] Based on the fluctuation range of the 2D medical image corresponding to the 3D medical image, multiple sets of alternative parameter combinations are determined, and a 2D overall pelvic bone image corresponding to each set of alternative parameter combinations is generated for the 3D pelvic bone image; wherein, a set of alternative parameter combinations includes alternative parameter values ​​corresponding to multiple rigid registration parameter items.

[0009] Feature point matching is performed between 2D medical images and 2D whole pelvic bone images. Based on the feature point matching results, dynamic adaptive template matching is performed between 2D medical images and 2D whole pelvic bone images to determine the target 2D local pelvic bone image for matching 2D medical images.

[0010] Based on the target 2D local pelvic bone image and its corresponding alternative parameter combinations, rigid registration is performed on the 3D medical image to align the rigidly registered 3D medical image with the 2D medical image.

[0011] In one implementation, dynamic adaptive template matching is performed between a 2D medical image and a 2D overall pelvic bone image based on feature point matching results to determine the target 2D local pelvic bone image for matching the 2D medical image, including:

[0012] Based on the feature point matching results, the target size and position information of the cropping window corresponding to each 2D whole pelvic bone image are dynamically and adaptively adjusted to determine.

[0013] According to the target size and position information of the cropping window, the corresponding 2D whole pelvic bone image is cropped to obtain the 2D local pelvic bone image after cropping each 2D whole pelvic bone image.

[0014] The target 2D local pelvic bone image for matching the 2D medical image is determined based on the similarity between the 2D medical image and each 2D local pelvic bone image.

[0015] In one implementation, based on feature point matching results, the target size and position information of the cropping window corresponding to each of the 2D overall pelvic bone images are dynamically and adaptively determined, including:

[0016] For any of the 2D overall pelvic skeleton images described above, perform the following operations:

[0017] Based on the feature point matching results, the initial size and position information of the cropping window corresponding to the 2D overall pelvic skeleton image are determined, and a buffer area is set for the 2D overall pelvic skeleton image based on the initial size and position information.

[0018] Set an adaptive window set within the buffer area;

[0019] For each adaptive window in the adaptive window set, feature point pairs are extracted from the 2D medical image and the 2D whole pelvic bone image based on the feature point matching results, so as to determine the confidence level corresponding to the adaptive window using the feature point pairs.

[0020] Based on the confidence level of each adaptive window, determine the target size and position information of the clipping window.

[0021] In one implementation, determining the confidence level corresponding to the adaptive window using a set of feature point pairs includes:

[0022] Using a set of feature point pairs, the local similarity between the 2D medical image, the 2D overall pelvic skeleton image, and the adaptive window is determined; a confidence score is assigned to the adaptive window based on the local similarity; wherein, the confidence score is positively correlated with the local similarity.

[0023] Based on the confidence level corresponding to each adaptive window, determine the target size and position information of the cropping window, including:

[0024] Based on the confidence level of each adaptive window, target adaptive windows are selected, and the initial size and position information of the cropping window are adjusted using the size and position information of the target adaptive windows to obtain the target size and position information of the cropping window.

[0025] In one implementation, dynamically and adaptively determining the target size and position information of the cropping window corresponding to each 2D overall pelvic bone image further includes:

[0026] For any 2D overall pelvic skeleton image, perform the following operation:

[0027] Using a pre-trained cropping and localization model, based on feature point matching results, the complete image features of the 2D medical image and the local image features of the 2D overall pelvic bone image are extracted respectively. The similarity between the complete image features and the local image features is determined based on the cross-attention mechanism. Based on the similarity, the target size and position information of the cropping window of the 2D overall pelvic bone image are determined.

[0028] In one implementation, the HU value corresponding to each voxel in the 3D medical image is determined;

[0029] Based on multiple window transform parameters and the HU value corresponding to each voxel, the 3D medical image is divided into the background and non-skeleton parts, and the main pelvic skeleton structure.

[0030] Background and non-skeletal tissues in 3D medical images are removed by utilizing background and non-skeletal parts, and 3D pelvic skeleton images are extracted by combining the main pelvic skeleton structure.

[0031] In one implementation, feature point matching is performed between a 2D medical image and a 2D overall pelvic bone image, including:

[0032] The feature point matching model, based on self-attention and cross-attention mechanisms, extracts and matches feature points from 2D medical images and 2D whole pelvic bone images to obtain the feature point matching results.

[0033] Secondly, the present invention also provides a rigid registration device for local pelvic bone images, comprising:

[0034] The 3D image extraction module is used to acquire the 3D medical image to be registered. The content displayed in the 3D medical image includes the overall pelvic information. The 3D pelvic bone image is extracted from the 3D medical image.

[0035] The 2D overall image generation module is used to determine multiple sets of alternative parameter combinations based on the fluctuation range of the 2D medical image corresponding to the 3D medical image, and to generate a 2D overall pelvic bone image corresponding to each set of alternative parameter combinations for the 3D pelvic bone image; wherein, a set of alternative parameter combinations includes alternative parameter values ​​corresponding to multiple rigid registration parameter items.

[0036] The template matching module is used to perform feature point matching between 2D medical images and 2D whole pelvic bone images, and to perform dynamic adaptive template matching between 2D medical images and 2D whole pelvic bone images based on the feature point matching results, so as to determine the target 2D local pelvic bone image for matching the 2D medical image.

[0037] The rigid registration module is used to perform rigid registration on 3D medical images based on the target 2D local pelvic bone image and its corresponding alternative parameter combinations, so that the rigidly registered 3D medical image is aligned with the 2D medical image.

[0038] Thirdly, the present invention also provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0039] Fourthly, the present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0040] This invention provides a method, apparatus, device, and medium for rigid registration of local pelvic bone images. First, a 3D medical image to be registered is acquired. The 3D medical image displays overall pelvic information. A 3D pelvic bone image is extracted from the 3D medical image. Then, multiple sets of candidate parameter combinations are determined based on the fluctuation range of the corresponding 2D medical image. A 2D overall pelvic bone image corresponding to each set of candidate parameter combinations is generated for the 3D pelvic bone image. Each set of candidate parameter combinations includes candidate parameter values ​​corresponding to multiple rigid registration parameter items. Next, feature point matching is performed between the 2D medical image and the 2D overall pelvic bone image. Based on the feature point matching results, dynamic adaptive template matching is performed between the 2D medical image and the 2D overall pelvic bone image to determine the target 2D local pelvic bone image for matching. Finally, rigid registration is performed on the 3D medical image based on the target 2D local pelvic bone image and its corresponding candidate parameter combinations to align the rigidly registered 3D medical image with the 2D medical image. The above method addresses scenarios where it is difficult to obtain complete 3D medical images. After generating 2D global pelvic bone images corresponding to each set of alternative parameter combinations for the 3D pelvic bone image, feature point matching and dynamic adaptive template matching are performed sequentially on the 2D medical image and the 2D global pelvic bone image to obtain a target 2D local pelvic bone image that matches the 2D medical image. Based on this, combined with the alternative parameter combinations corresponding to the target 2D local pelvic bone image, rigid registration of the 3D medical image can be achieved. This invention can effectively handle the problem of rigid registration from local 2D X-ray imaging to global 3D CT data.

[0041] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0042] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 A schematic diagram of a rigid registration framework based on deep learning provided for an embodiment of the present invention;

[0045] Figure 2 A flowchart illustrating a method for rigid registration of local pelvic bone images provided in an embodiment of the present invention;

[0046] Figure 3 A technical framework diagram of a local pelvic bone image rigid registration method provided in an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the DRR imaging principle provided in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of an AspanFormer feature point matching result provided in an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of projection angle change provided in an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram illustrating template matching based on feature point matching relationships, provided by an embodiment of the present invention.

[0051] Figure 8 This is a schematic diagram of a rigid registration device for local pelvic bone images provided in an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Image-guided surgery (IGS) is a method that uses medical imaging technology to guide surgeons in real time during surgery. This method provides surgeons with greater precision and lower risk by precisely comparing preoperative imaging data (such as CT scans, MRI, ultrasound, etc.) with the patient's actual anatomical structures. Preoperative three-dimensional data obtained through methods like CT scans is richer in information and more accurate than two-dimensional data. However, it is often limited by time and difficulty in acquisition, making it impossible to acquire in real time within the surgical environment. Two-dimensional XR fluoroscopy data obtained using X-ray projection is acquired rapidly, and the technology and equipment are more advanced; acquiring XR images intraoperatively does not affect the surgery. For image-guided surgery, it is usually necessary to obtain three-dimensional information of the surgical site through CT or MRI scans before surgery. This information serves as a positioning standard during surgery and provides the surgeon with relevant information about the lesion to assess the patient's condition and plan the surgical procedure. Two-dimensional images obtained in real time during surgery, such as those obtained through X-rays, can reflect the current surgical situation and track the current location, so that doctors can understand the specific information at any given moment.

[0055] Registration is the process of aligning different digital images of the same lesion or tissue. Medical image registration technology aims to find a spatial transformation that allows corresponding points in two images acquired at different times and from different devices to match in spatial relationships and structure. It is a key technology in image-guided surgery. If registration technology can align intraoperative real-time 2D X-ray images with preoperative high-precision 3D CT images, the advantages of both can be combined to help doctors obtain richer data, thereby achieving higher-precision navigation surgery.

[0056] The accuracy and timeliness of matching 2D images with 3D model information are crucial for surgery. Registration between digital information of different dimensions often requires converting the two types of image information into a unified dimension. In medical image registration research and applications, there are already relatively mature methods that can project 3D volume information into 2D image information, and this method is highly accurate. Digitally reconstructed radiograph (DRR) generation is an important procedure in several medical imaging applications. This method can project 3D image information into a 2D image, thus achieving dimensionality reduction of 3D digital information. By using DRR technology, the registration problem between 2D and 3D image information is transformed into a registration problem between 2D images.

[0057] Traditional image registration techniques can be broadly categorized into two types: feature-based registration and grayscale-based registration. Feature-based registration extracts a small amount of feature information from the image to be registered and uses this extracted information for registration. Commonly used features include anatomical features of the human body and externally placed markers. Common anatomical features include anatomical markers, curve features, and surface features. Grayscale-based registration, on the other hand, compares grayscale values ​​to determine similarity and complete the registration process.

[0058] With the maturation of neural network technologies, deep learning methods are increasingly being used for image registration. ShunMiao et al. were the first to use a convolutional neural network (CNN) regression approach to register 2D / 3D medical images, regressing image transformation parameters using a neural network. Existing rigid registration methods based on deep learning largely rely on this structure, focusing on optimizing the network model to improve efficiency, specifically as follows... Figure 1 The diagram shows a rigid registration framework based on deep learning.

[0059] In this registration framework, CT data is first projected using DRR projection to obtain a 2D DRR projected image and the corresponding deformation parameter ground-truth as labels. The 2D image data and the corresponding ground-truth labels are then fed into the network for supervised learning, resulting in the registration network. Real X-ray images are then input into the learned registration model to predict spatial transformation parameters and complete the registration.

[0060] However, traditional feature extraction and segmentation methods require manual intervention, making automation difficult. Registration based on grayscale information, which optimizes image grayscale similarity to determine transformation parameters, performs poorly on images with deformation or significant grayscale changes. Current deep learning-based methods struggle to train on local skeletal data. Therefore, this invention provides a method, apparatus, device, and medium for rigid registration of local pelvic bone images, effectively addressing the rigid registration problem from local 2D X-ray imaging to global 3D CT data.

[0061] To facilitate understanding of this embodiment, a method for rigid registration of local pelvic bone images disclosed in this invention will first be described in detail. (See [link to relevant documentation]). Figure 2 The diagram shows a flowchart of a method for rigid registration of local pelvic bone images. The method mainly includes the following steps S202 to S208:

[0062] Step S202: Obtain the 3D medical image to be registered. The content displayed in the 3D medical image includes the overall pelvic information. Extract the 3D pelvic bone image from the 3D medical image.

[0063] The 3D medical image can be a CT image. In one example, the HU (Hounsfield Unit) value corresponding to each voxel in the 3D medical image can be determined. Combined with window transformation parameters, the background and non-skeletal parts, as well as the main structure of the pelvic skeleton in the 3D medical image, can be identified. Then, the 3D pelvic skeleton image can be extracted based on the identification results.

[0064] Step S204: Determine multiple sets of alternative parameter combinations based on the fluctuation range of the 2D medical image corresponding to the 3D medical image, and generate a 2D overall pelvic bone image corresponding to each set of alternative parameter combinations for the 3D pelvic bone image.

[0065] A set of alternative parameter combinations includes alternative parameter values ​​corresponding to multiple rigid registration parameter items. For example, a set of alternative parameter combinations may include alternative parameter values ​​corresponding to the rotation parameter item around the X-axis (referred to as the rotation angle around the X-axis) and alternative parameter values ​​corresponding to the rotation parameter item around the Y-axis (referred to as the rotation angle around the Y-axis). The 2D medical image can be an X-ray image, and the 2D overall pelvic skeleton image can be a DRR image. In one example, after determining multiple sets of alternative parameter combinations, multiple 2D overall pelvic skeleton images based on rotation along the X and Y axes can be generated from the 3D pelvic skeleton image.

[0066] Step S206: Perform feature point matching between the 2D medical image and the 2D whole pelvic bone image, and perform dynamic adaptive template matching between the 2D medical image and the 2D whole pelvic bone image based on the feature point matching results to determine the target 2D local pelvic bone image for matching the 2D medical image.

[0067] Dynamic adaptive template matching can be understood as dynamically and adaptively adjusting the cropping size. After cropping the 2D whole pelvic bone image according to the adjusted cropping size to obtain the corresponding 2D local pelvic bone image, the process of selecting the target 2D local pelvic bone image that matches the 2D medical image based on template matching. In one example, the ASpanFormer feature point matching network can be used to perform feature point matching between the 2D medical image and the 2D whole pelvic bone image. Then, for each 2D whole pelvic bone image, the size and position information of the cropping window of the 2D whole pelvic bone image are dynamically and adaptively adjusted based on the sliding window or neural network model. The 2D whole pelvic bone image is then cropped according to the adjusted cropping window to obtain the corresponding 2D local pelvic bone image. Finally, the 2D local pelvic bone image with the highest similarity to the 2D medical image is selected as the target 2D local pelvic bone image for matching.

[0068] Step S208: Based on the target 2D local pelvic bone image and its corresponding alternative parameter combinations, perform rigid registration on the 3D medical image so that the rigidly registered 3D medical image is aligned with the 2D medical image.

[0069] In one example, a complete set of rigid registration parameters can be solved based on a 2D medical image, a target 2D local pelvic bone image, and their corresponding alternative parameter combinations. This set of rigid registration parameters includes rotation angles around the X-axis, rotation angles around the Y-axis, rotation angles around the Z-axis, translation along the X-axis, translation along the Y-axis, and translation along the Z-axis. By rigidly registering the 3D medical image according to the complete set of parameters, the rigidly registered 3D medical image can be aligned with the 2D medical image.

[0070] The rigid registration method for local pelvic bone images provided in this invention addresses scenarios where it is difficult to obtain complete 3D medical images. After generating 2D global pelvic bone images corresponding to each set of alternative parameter combinations for the 3D pelvic bone image, feature point matching and dynamic adaptive template matching are sequentially performed on the 2D medical image and the 2D global pelvic bone image to obtain a target 2D local pelvic bone image that matches the 2D medical image. Based on this, and combined with the alternative parameter combinations corresponding to the target 2D local pelvic bone image, rigid registration of the 3D medical image can be achieved. This invention can effectively handle the rigid registration problem of local 2D X-ray imaging to global 3D CT data.

[0071] For ease of understanding, embodiments of the present invention provide, as follows: Figure 3 The diagram illustrates a technical framework for a rigid registration method for local pelvic bone images. This framework mainly consists of three parts: First, image preprocessing involves projecting a 3D CT image into a 2D global DRR image using DRR projection technology based on given alternative parameter combinations. Second, feature point matching is performed between the 2D global DRR image and the 2D X-ray image to obtain corresponding point pairs and their location information. Third, the relationship between the two images is calculated based on the feature point matching information. The 2D global DRR image is dynamically and adaptively cropped, and the cropped 2D local DRR image with the highest similarity to the 2D X-ray image is used as the target 2D local DRR image to solve for the rigid registration parameter combination, thereby performing rigid registration on the 3D CT image.

[0072] For ease of understanding, this embodiment of the invention provides a specific implementation of a method for rigid registration of local pelvic bone images.

[0073] Regarding the aforementioned step S202, this embodiment of the invention provides a specific implementation method for obtaining a 3D medical image to be registered, wherein the content displayed in the 3D medical image includes at least a local pelvic skeleton, and extracting a 3D pelvic skeleton image from the 3D medical image. The 3D medical image is a CT image.

[0074] include:

[0075] In the preprocessing section, the preoperative 3D CT images are projected into the required 2D overall DRR images (i.e., 2D overall pelvic skeleton images) according to the desired projection. See also Figure 4 The diagram illustrates a DRR imaging principle. The DRR program simulates X-ray imaging by simulating X-ray emission from a point source, passing through three-dimensional volume data, and projecting it onto a plane to generate a two-dimensional image, thus achieving the two-dimensional rendering of the three-dimensional volume data. In this process, the rigid registration parameter combination r can be set. x ,r y ,r z ,t x ,t y ,t z To determine the different two-dimensional projection results.

[0076] For DRR projection technology, windowing can be applied to CT images to filter bone information, resulting in cleaner DRR images. During CT imaging, each voxel along the X-ray path has its X-ray attenuation and scattering calculated based on its HU value and other physical properties. In medical imaging, HU units are used to distinguish different types of tissues, such as bone, muscle, and fat, based on their density. Windowing is a technique that processes CT data based on HU values. By setting an appropriate HU value range, it filters the CT data, focusing information on the bone and reducing extraneous information. Using CT data filtered using windowing for DRR projection yields cleaner DRR projected images.

[0077] In the specific implementation, please refer to steps A1 to A3 below:

[0078] Step A1: Determine the HU value corresponding to each voxel in the 3D medical image.

[0079] Step A2: Based on multiple window transformation parameters and the HU value corresponding to each voxel, the 3D medical image is divided into the background and non-skeleton parts and the main pelvic skeletal structure.

[0080] Step A3: Remove the background and non-skeletal tissues from the 3D medical image using the background and non-skeletal parts, and extract the 3D pelvic bone image by combining it with the main pelvic bone structure.

[0081] For example, the HU value (Hounsfield units) of CT images can be divided into the following three ranges to better extract the pelvic bones:

[0082] (1) Low HU value range: -1000HU to -500HU (or close to this range).

[0083] Characteristics: This range typically corresponds to low-density tissues such as air and fat. In pelvic CT images, while the pelvic bones themselves do not fall within this range, surrounding soft tissues (such as muscle and fat) and air within the pelvic cavity may. Identifying this range allows for the differentiation of non-skeletal tissues, providing background information for subsequent bone extraction.

[0084] (2) Medium HU value range: 200HU to 500HU (or adjusted according to specific circumstances).

[0085] Characteristics: This range typically corresponds to the transition zone from soft to harder tissues, including muscles, some cartilage, and thinner or looser portions of the pelvic bones. While the main body of the pelvic bones may not be within this range, some edges or transitional areas may have such a HU value. Identifying this range allows for further refinement of the boundaries for bone extraction.

[0086] (3) High HU value range: 500HU to +1000HU (or higher, depending on the density variation of the pelvic bones).

[0087] Characteristics: This range typically corresponds to high-density tissues, such as bone and calcified tissue. Major parts of the pelvic bones (such as the hip bones, ilium, pubis, and ischium) usually have this HU value. By identifying this range, the main structures of the pelvic bones can be accurately extracted.

[0088] In practical applications, window transformation techniques or multi-window edge transformation techniques can be used in conjunction with the aforementioned HU value range to extract the pelvic skeleton. First, background and non-skeletal tissues are removed based on the low HU value range; then, the boundaries and main structures of the skeleton are determined based on the medium and high HU value ranges; finally, the extraction results are optimized through morphological operations (such as dilation, erosion, smoothing, etc.) to obtain a more accurate 3D pelvic skeleton image.

[0089] In another implementation, the results of window transformation of CT images can be replaced with a deep learning-based method for segmentation, which is more flexible.

[0090] Regarding the aforementioned step S204, this embodiment of the invention provides a specific implementation method for determining multiple sets of alternative parameter combinations based on the fluctuation range of the 2D medical image corresponding to the 3D medical image, and generating a 2D overall pelvic bone image corresponding to each set of alternative parameter combinations for the 3D pelvic bone image, including: generating a 2D overall pelvic bone image based on the rotation of the X-axis and Y-axis on the CT image according to the fluctuation range of the X-ray image, for example, generating a 2D overall pelvic bone image every two degrees in the X-axis direction and the Y-axis direction respectively.

[0091] Regarding the aforementioned step S206, this embodiment of the invention provides a specific implementation method for feature point matching between a 2D medical image and a 2D whole pelvic bone image, including: using a feature point matching model, based on self-attention and cross-attention mechanisms, to extract and match feature points between the 2D medical image and the 2D whole pelvic bone image to obtain feature point matching results.

[0092] Specifically, by matching feature points between 2D whole pelvic skeleton images and X-ray images, the relationships between feature points can be obtained. In the experiment, the ASpanFormer feature point matching network was used as the feature point matching tool. This network model is a Transformer-based network structure that uses self-attention and cross-attention mechanisms to simultaneously complete the feature point extraction and matching process, obtaining results such as... Figure 5 The diagram shows a feature point matching result from the AspanFormer model. Compared to traditional methods, the deep learning-based AspanFormer model achieves better matching results in feature point extraction and matching for medical images.

[0093] Regarding the aforementioned step S206, this embodiment of the invention also provides a specific implementation method for dynamically adaptive template matching between a 2D medical image and a 2D overall pelvic bone image based on feature point matching results, in order to determine the target 2D local pelvic bone image for 2D medical image matching, including the following steps B1 to B3:

[0094] Step B1: Based on the feature point matching results, dynamically and adaptively determine the target size and position information of the cropping window corresponding to each 2D overall pelvic bone image.

[0095] In one implementation, the size and position information of the cropping window can be dynamically and adaptively adjusted based on a sliding window, or the size and position information of the cropping window can be dynamically and adaptively adjusted based on a neural network. Details are as follows:

[0096] Method 1: Based on a sliding window, the size and position of the cropping window are dynamically and adaptively adjusted. For any 2D whole pelvic skeleton image, the following operations are performed:

[0097] (1.1) Based on the feature point matching results, determine the initial size and position information of the clipping window corresponding to the 2D overall pelvic skeleton image, and set a buffer region for the 2D overall pelvic skeleton image based on the initial size and position information. For example, the initial size and position information of the minimum bounding box of the feature point matching results in the 2D overall pelvic skeleton image can be used to determine the initial size and position information of the clipping window corresponding to the 2D overall pelvic skeleton image. Then, using the clipping window edge represented by the initial size and position information of the clipping window, a buffer region is constructed with the clipping window edge as the center and based on a preset width.

[0098] (1.2) Set an adaptive window set within the buffer area. In one example, multiple adaptive sliding windows (referred to as adaptive windows) are set within the buffer area, and the set of adaptive windows is called the adaptive window set.

[0099] (1.3) For each adaptive window in the adaptive window set, extract a set of feature point pairs from the 2D medical image and the 2D whole pelvic bone image based on the feature point matching results, so as to determine the confidence level corresponding to the adaptive window using the set of feature point pairs.

[0100] In one implementation, a set of feature point pairs can be used to determine the local similarity of the 2D medical image, the 2D overall pelvic bone image, to the adaptive window, and a confidence score can be assigned to the adaptive window based on the local similarity, with the confidence score being positively correlated with the local similarity.

[0101] Optionally, similarity evaluation algorithms based on triangles, NCC (Normalized Cross-Correlation) algorithm, MSE (Mean-Square Error) algorithm, SSIM (Structural Similarity) algorithm, etc., can be used to determine the local similarity of the 2D medical image, the 2D overall pelvic bone image, and the adaptive window.

[0102] Taking a triangle-based similarity evaluation algorithm as an example: for the feature point relationship established between two images, the corresponding point positions can be found. However, considering that angles do not change with scaling and translation within a plane, but are only sensitive to rotational relationships in three dimensions, such as... Figure 6 The diagram shows a projection angle change. Therefore, corresponding triangle pairs can be formed according to the feature point matching relationship, and the similarity of the triangle pairs can be measured to obtain the local similarity of the 2D medical image and the 2D overall pelvic bone image with respect to the adaptive window.

[0103] Taking the SSIM algorithm as an example: The SSIM2D medical image, specifically the 2D overall pelvic bone image, is used to evaluate local similarity with respect to the adaptive window. SSIM is a commonly used image similarity metric that assesses image similarity based on three aspects: brightness, contrast, and structure.

[0104] (1.4) Determine the target size and position information of the clipping window based on the confidence level of each adaptive window. Specifically, target adaptive windows are selected based on the confidence level of each adaptive window, and the initial size and position information of the clipping window are adjusted using the size and position information of the target adaptive windows to obtain the target size and position information of the clipping window. In one example, if an adaptive window has a high confidence level, it can be used as the target adaptive window, and the clipping window can be enlarged or reduced according to the positional relationship between the target adaptive window and the clipping window; conversely, if an adaptive window has a low confidence level, it is prohibited to use that adaptive window to enlarge or reduce the clipping window.

[0105] Method 2: Based on a neural network, the size and position information of the cropping window are dynamically and adaptively adjusted. For any 2D whole pelvic bone image, the following operations are performed: using a pre-trained cropping localization model, based on the feature point matching results, the complete image features of the 2D medical image and the local image features of the 2D whole pelvic bone image are extracted respectively. The similarity between the complete image features and the local image features is determined based on the cross-attention mechanism. Based on the similarity, the target size and position information of the cropping window of the 2D whole pelvic bone image are determined.

[0106] In the specific implementation, firstly, the 2D medical image and the 2D overall pelvic skeleton image are fed into different convolutional neural networks (CNNs). The CNNs then extract the complete image features of the 2D medical image and the local image features of the 2D overall pelvic skeleton image based on feature point matching results. Next, the complete and local image features are fed into a cross-attention module, the core of this network. This module calculates the similarity between the local and complete image features and helps the model "focus" on the most matching region in the image. Here, Q (Query) represents the local image feature; K (Key) and V (Value) represent the complete image feature. The cross-attention module calculates the relevance based on Q (Query) and K (Key), and then performs a weighted summation of V (Value) based on the relevance to obtain a weighted feature representation. The specific steps are as follows: First, the relevance of Q (Query) and K (Key) is measured by calculating the dot product, for example, using dot product attention. Second, Softmax normalization is applied to transform the calculated relevance into a probability distribution, representing the matching weight at each position. Third, these matching weights are used to perform a weighted summation of V (Value) to obtain the final matching feature representation. Finally, the cropping region is further located based on similarity. Specifically, pre-selection is performed within areas of high similarity, and the bounding boxes are dynamically adjusted based on the similarity data to avoid truncating key structures.

[0107] In this embodiment of the invention, the network is trained using multiple sets of local and global information. In particular, a small-angle rotation of the overall image is incorporated to increase the network's robustness.

[0108] Step B2: According to the target size and position information of the cropping window, crop the corresponding 2D whole pelvic bone image to obtain a 2D local pelvic bone image after cropping each 2D whole pelvic bone image. For example, see... Figure 7 The diagram shows a template matching process based on feature point matching relationships. The left side is a schematic diagram of feature point matching, and the right side is a cropped 2D local pelvic bone image T'.

[0109] Step B3: Based on the similarity between the 2D medical image and each 2D local pelvic bone image, determine the target 2D local pelvic bone image for matching the 2D medical image. In one example, by comparing the target X-ray image with the template one by one, the 2D local pelvic bone image with the highest target similarity is selected as the target 2D local pelvic bone image for matching the 2D medical image, and the X-axis and Y-axis rotation parameters used when generating the target 2D local pelvic bone image are recorded as the calculated r. x r y .

[0110] Regarding the aforementioned step S208, this embodiment of the invention provides an implementation method for rigidly registering a 3D medical image based on a target 2D local pelvic bone image and its corresponding alternative parameter combinations, so that the rigidly registered 3D medical image is aligned with the 2D medical image, including: a known parameter combination r x r y The geometric relationship between the X-ray image and its matched target 2D local pelvic bone image can be solved step by step to obtain r. z ,t x ,t y ,t z That is, to obtain the complete rigid registration parameter combination r x ,r y ,r z ,t x ,t y ,t z For example, for any three feature points in an X-ray image, matching feature points are determined from a target 2D local pelvic bone image to form feature point pairs, and a rotation angle around the Z-axis is determined based on the vector pairs constructed from the feature point pairs; a scaling factor is determined based on the matching feature points between the X-ray image and the target 2D local pelvic bone image, and a translation along the Z-axis is determined based on the scaling factor and the distance between the target 2D local pelvic bone image and the radiation source; and a pixel movement distance coefficient corresponding to each unit physical distance movement is determined based on a template image with known physical translation relationships, and the translation along the X-axis and the translation along the Y-axis are determined based on the pixel movement distance coefficient and the scaling factor.

[0111] The CT image is projected according to the rigid registration parameter combination to form a new 2D local pelvic bone image. If the similarity between the new 2D local pelvic bone image and the X-ray image meets the threshold, then the CT image can be rigidly registered using the rigid registration parameter combination to align the CT image with the X-ray image. Otherwise, the rigid registration parameter combination is fine-tuned until the similarity between the new 2D local pelvic bone image and the X-ray image meets the threshold.

[0112] In summary, this invention incorporates deep learning technology and uses feature point matching results for registration. This method is primarily applicable to surgeries involving large bone groups such as the pelvis, where intraoperative X-ray images are registered with preoperative CT images. Using deep learning technology makes feature point matching more accurate and faster, yielding more comprehensive matching results. Registration based on feature point matching results allows for greater flexibility. Template matching of the global image based on feature point matching results focuses information on local areas, making registration more flexible. Furthermore, since the deep learning feature point matching process does not require manual setting of feature points, it allows for faster and more automated information processing.

[0113] Based on the foregoing embodiments, this invention provides a local pelvic bone image rigid registration device, see [link to previous embodiment]. Figure 8 The diagram shows a structural schematic of a rigid registration device for local pelvic bone images. The device mainly includes the following parts:

[0114] The 3D image extraction module 802 is used to acquire the 3D medical image to be registered. The content displayed in the 3D medical image includes the overall pelvic information. The 3D pelvic bone image is extracted from the 3D medical image.

[0115] The 2D overall image generation module 804 is used to determine multiple sets of alternative parameter combinations based on the fluctuation range of the 2D medical image corresponding to the 3D medical image, and to generate a 2D overall pelvic bone image corresponding to each set of alternative parameter combinations for the 3D pelvic bone image; wherein, a set of alternative parameter combinations includes alternative parameter values ​​corresponding to multiple rigid registration parameter items.

[0116] The template matching module 806 is used to perform feature point matching between a 2D medical image and a 2D whole pelvic bone image, and to perform dynamic adaptive template matching between the 2D medical image and the 2D whole pelvic bone image based on the feature point matching results, so as to determine the target 2D local pelvic bone image for matching the 2D medical image.

[0117] The rigid registration module 808 is used to perform rigid registration on a 3D medical image based on a target 2D local pelvic bone image and its corresponding alternative parameter combinations, so that the rigidly registered 3D medical image is aligned with the 2D medical image.

[0118] The rigid registration device for local pelvic bone images provided in this invention addresses scenarios where it is difficult to obtain complete 3D medical images. After generating 2D global pelvic bone images corresponding to each set of alternative parameter combinations for the 3D pelvic bone image, feature point matching and dynamic adaptive template matching are performed sequentially on the 2D medical image and the 2D global pelvic bone image to obtain a target 2D local pelvic bone image that matches the 2D medical image. Based on this, and combined with the alternative parameter combinations corresponding to the target 2D local pelvic bone image, rigid registration of the 3D medical image can be achieved. This invention can effectively handle the rigid registration problem of local 2D X-ray imaging to global 3D CT data.

[0119] In one implementation, the template matching module 806 is specifically used for:

[0120] Based on the feature point matching results, the target size and position information of the cropping window corresponding to each 2D whole pelvic bone image are dynamically and adaptively adjusted to determine.

[0121] According to the target size and position information of the cropping window, the corresponding 2D whole pelvic bone image is cropped to obtain the 2D local pelvic bone image after cropping each 2D whole pelvic bone image.

[0122] The target 2D local pelvic bone image for matching the 2D medical image is determined based on the similarity between the 2D medical image and each 2D local pelvic bone image.

[0123] In one implementation, the template matching module 806 is specifically used for:

[0124] For any of the 2D overall pelvic skeleton images described above, perform the following operations:

[0125] Based on the feature point matching results, the initial size and position information of the cropping window corresponding to the 2D overall pelvic skeleton image are determined, and a buffer area is set for the 2D overall pelvic skeleton image based on the initial size and position information.

[0126] Set an adaptive window set within the buffer area;

[0127] For each adaptive window in the adaptive window set, feature point pairs are extracted from the 2D medical image and the 2D whole pelvic bone image based on the feature point matching results, so as to determine the confidence level corresponding to the adaptive window using the feature point pairs.

[0128] Based on the confidence level of each adaptive window, determine the target size and position information of the clipping window.

[0129] In one implementation, the template matching module 806 is specifically used for:

[0130] Using a set of feature point pairs, the local similarity between the 2D medical image, the 2D overall pelvic bone image, and the adaptive window is determined; a confidence score is assigned to the adaptive window based on the local similarity; wherein the confidence score is positively correlated with the local similarity.

[0131] In one implementation, the template matching module 806 is specifically used for:

[0132] Based on the confidence level of each adaptive window, target adaptive windows are selected, and the initial size and position information of the cropping window are adjusted using the size and position information of the target adaptive windows to obtain the target size and position information of the cropping window.

[0133] In one implementation, the template matching module 806 is specifically used for:

[0134] For any 2D overall pelvic skeleton image, perform the following operation:

[0135] Using a pre-trained cropping and localization model, based on feature point matching results, the complete image features of the 2D medical image and the local image features of the 2D overall pelvic bone image are extracted respectively. The similarity between the complete image features and the local image features is determined based on the cross-attention mechanism. Based on the similarity, the target size and position information of the cropping window of the 2D overall pelvic bone image are determined.

[0136] In one embodiment, the 3D image extraction module 802 is specifically used for:

[0137] Determine the HU value for each voxel in a 3D medical image;

[0138] Based on multiple window transformation parameters and the HU value corresponding to each voxel, the 3D medical image is divided into the background and non-skeleton part and the main structure of the pelvic skeleton.

[0139] Background and non-skeletal tissues in 3D medical images are removed by utilizing background and non-skeletal parts, and combined with the main structure of the pelvic skeleton, a 3D pelvic skeleton image is extracted.

[0140] In one implementation, the multi-level template matching module 806 is specifically used for:

[0141] The feature point matching model, based on self-attention and cross-attention mechanisms, extracts and matches feature points from 2D medical images and 2D whole pelvic bone images to obtain the feature point matching results.

[0142] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0143] This invention provides an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, and the computer program, when run by the processor, executes the method described in any of the above embodiments.

[0144] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes: a processor 90, a memory 91, a bus 92, and a communication interface 93. The processor 90, the communication interface 93, and the memory 91 are connected through the bus 92. The processor 90 is used to execute executable modules, such as computer programs, stored in the memory 91.

[0145] The memory 91 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 93 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0146] Bus 92 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0147] The memory 91 is used to store programs. After receiving an execution instruction, the processor 90 executes the programs. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 90 or implemented by the processor 90.

[0148] The processor 90 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 90 or by instructions in software form. The processor 90 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 91. Processor 90 reads the information in memory 91 and, in conjunction with its hardware, completes the steps of the above method.

[0149] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0150] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for rigid registration of local pelvic bone images, characterized in that, include: Acquire a 3D medical image to be registered, the content of which includes overall pelvic information, and extract a 3D pelvic skeletal image from the 3D medical image; Based on the fluctuation range of the 2D medical image corresponding to the 3D medical image, multiple sets of alternative parameter combinations are determined, and a 2D overall pelvic bone image corresponding to each set of alternative parameter combinations is generated for the 3D pelvic bone image; wherein, a set of alternative parameter combinations includes alternative parameter values ​​corresponding to multiple rigid registration parameter items. Feature point matching is performed between the 2D medical image and the 2D whole pelvic bone image, and dynamic adaptive template matching is performed between the 2D medical image and the 2D whole pelvic bone image based on the feature point matching results to determine the target 2D local pelvic bone image for matching the 2D medical image. Based on the target 2D local pelvic bone image and its corresponding alternative parameter combination, the 3D medical image is rigidly registered so that the rigidly registered 3D medical image is aligned with the 2D medical image.

2. The method for rigid registration of local pelvic bone images according to claim 1, characterized in that, Based on the feature point matching results, dynamic adaptive template matching is performed between the 2D medical image and the 2D overall pelvic bone image to determine the target 2D local pelvic bone image for matching the 2D medical image, including: Based on the feature point matching results, the target size and position information of the cropping window corresponding to each 2D overall pelvic bone image are dynamically and adaptively adjusted to determine. According to the target size and position information of the cropping window, the corresponding 2D whole pelvic bone image is cropped to obtain a 2D local pelvic bone image after cropping each 2D whole pelvic bone image. The target 2D local pelvic bone image is determined based on the similarity between the 2D medical image and each of the 2D local pelvic bone images.

3. The method for rigid registration of local pelvic bone images according to claim 2, characterized in that, Based on the feature point matching results, the target size and position information of the cropping window corresponding to each of the 2D overall pelvic bone images are dynamically and adaptively determined, including: For any of the 2D overall pelvic skeleton images described above, perform the following operations: Based on the feature point matching results, the initial size and position information of the cropping window corresponding to the 2D overall pelvic skeleton image are determined, and a buffer area is set for the 2D overall pelvic skeleton image based on the initial size and position information. An adaptive window set is set within the buffer area; For each adaptive window in the set of adaptive windows, a set of feature point pairs is extracted from the 2D medical image and the 2D whole pelvic bone image based on the feature point matching results, so as to determine the confidence level corresponding to the adaptive window using the set of feature point pairs. The target size and position information of the cropping window are determined based on the confidence level corresponding to each adaptive window.

4. The method for rigid registration of local pelvic bone images according to claim 3, characterized in that, Determining the confidence level corresponding to the adaptive window using the set of feature point pairs includes: Using the feature point pair set, the local similarity between the 2D medical image, the 2D overall pelvic bone image, and the adaptive window is determined; a confidence score is assigned to the adaptive window based on the local similarity; wherein the confidence score is positively correlated with the local similarity. Based on the confidence level corresponding to each adaptive window, the target size and position information of the cropping window are determined, including: Based on the confidence level corresponding to each adaptive window, target adaptive windows are selected, and the initial size and position information of the cropping window are adjusted using the size and position information of the target adaptive windows to obtain the target size and position information of the cropping window.

5. The method for rigid registration of local pelvic bone images according to claim 2, characterized in that, The method of dynamically and adaptively determining the target size and position information of the cropping window corresponding to each of the 2D overall pelvic bone images also includes: For any of the 2D overall pelvic skeleton images described above, perform the following operations: Using a pre-trained cropping and localization model, based on the feature point matching results, the complete image features of the 2D medical image and the local image features of the 2D overall pelvic bone image are extracted respectively. The similarity between the complete image features and the local image features is determined based on the cross-attention mechanism. Based on the similarity, the target size and position information of the cropping window of the 2D overall pelvic bone image are determined.

6. The method for rigid registration of local pelvic bone images according to claim 1, characterized in that, Extracting a 3D pelvic bone image from the 3D medical image includes: Determine the HU value corresponding to each voxel in the 3D medical image; Based on multiple window transformation parameters and the HU value corresponding to each voxel, the 3D medical image is divided into the background and non-skeleton part, and the main pelvic skeleton structure part. The background and non-skeletal parts are used to remove the background and non-skeletal tissues from the 3D medical image, and the 3D pelvic bone image is extracted by combining the main pelvic bone structure.

7. The method for rigid registration of local pelvic bone images according to claim 1, characterized in that, Feature point matching is performed between the 2D medical image and the 2D overall pelvic bone image, including: The feature point matching model, based on self-attention and cross-attention mechanisms, extracts and matches feature points between the 2D medical image and the 2D overall pelvic bone image to obtain the feature point matching results.

8. A rigid registration device for local pelvic bone images, characterized in that, include: A 3D image extraction module is used to acquire a 3D medical image to be registered. The content displayed in the 3D medical image includes overall pelvic information. A 3D pelvic bone image is extracted from the 3D medical image. The 2D overall image generation module is used to determine multiple sets of alternative parameter combinations based on the fluctuation range of the 2D medical image corresponding to the 3D medical image, and to generate a 2D overall pelvic bone image corresponding to each set of alternative parameter combinations for the 3D pelvic bone image; wherein, a set of alternative parameter combinations includes alternative parameter values ​​corresponding to multiple rigid registration parameter items. The template matching module is used to perform feature point matching between the 2D medical image and the 2D whole pelvic bone image, and to perform dynamic adaptive template matching between the 2D medical image and the 2D whole pelvic bone image based on the feature point matching results, so as to determine the target 2D local pelvic bone image to be matched with the 2D medical image. A rigid registration module is used to perform rigid registration on the 3D medical image based on the target 2D local pelvic bone image and its corresponding alternative parameter combination, so that the rigidly registered 3D medical image is aligned with the 2D medical image.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images