Image fusion methods, apparatus and computer equipment

Abstract

This application relates to the fields of medical devices and puncture technology, and more particularly to an image fusion method, apparatus, and computer equipment. The method includes: acquiring a first transverse process image segmented from an ultrasound image of the vertebral region of a target object, and a second transverse process image segmented from a medical image of the vertebral region; registering the ultrasound image and the medical image based on the three-dimensional coordinate information of each transverse process in the first and second transverse process images to obtain a registration matrix, and then performing image fusion based on the registration matrix. This method can correctly guide puncture.

Description

Technical Field

[0001] This application relates to the fields of medical devices and puncture technology, and in particular to an image fusion method, apparatus and computer equipment. Background Technology

[0002] Anesthesia is an indispensable part of modern medicine, greatly improving the safety and effectiveness of medical procedures while also enhancing patient comfort and treatment experience. Ultrasound-guided lumbar plexus block and sacral plexus block are two effective regional anesthesia methods, primarily used for lower limb surgeries. Compared to traditional general anesthesia, spinal or epidural anesthesia offers numerous advantages, such as fewer side effects, longer duration of pain control, and faster recovery.

[0003] However, traditional anesthesia-guided puncture systems typically involve punctures under the guidance of ultrasound images. However, due to the low resolution of ultrasound and the limited imaging effect on specific tissues, the puncture guidance effect is poor, increasing the puncture risk. Summary of the Invention

[0004] Therefore, it is necessary to provide an image fusion method, apparatus, and computer device that can correctly guide puncture in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides an image fusion method, the method comprising:

[0006] Acquire a first transverse process image segmented from an ultrasound image of the vertebral region of the target object, and a second transverse process image segmented from a medical image of the vertebral region;

[0007] Based on the three-dimensional coordinate information of each transverse process in the first transverse process image and the second transverse process image, the ultrasound image and the medical image are registered to obtain a registration matrix, and image fusion is performed based on the registration matrix.

[0008] In one embodiment, the step of registering the ultrasound image and the medical image based on the three-dimensional coordinate information of each transverse process in the first transverse process image and the second transverse process image to obtain a registration matrix includes:

[0009] Identify the vertebrae in which each transverse process is located in the first transverse process image and the vertebrae in which each transverse process is located in the second transverse process image;

[0010] Transverse processes in the same vertebra are identified as a pair of transverse processes, based on the first and second transverse process images.

[0011] Based on the three-dimensional coordinate information of each transverse process in the transverse process alignment, the ultrasound image and the medical image are registered to obtain a registration matrix.

[0012] In one embodiment, determining the vertebrae where each transverse process in the first transverse process image is located includes:

[0013] Obtain the center coordinates of each transverse process in each vertebra of the target object;

[0014] Based on the center coordinates and the three-dimensional coordinate information of each transverse process in the first transverse process image, determine the target center that is closest to each transverse process in the first transverse process image from each center;

[0015] The vertebrae corresponding to the target center are identified as the vertebrae where each transverse process is located in the first transverse process image.

[0016] In one embodiment, determining the vertebrae where each transverse process in the second transverse process image is located includes:

[0017] Based on the coordinates of each transverse process in the second transverse process image on the target axis, the transverse processes in the second transverse process image are sorted; the direction of the target axis is from the foot to the head of the target object;

[0018] Based on the sorting results of each transverse process in the second transverse process image, the vertebrae in which each transverse process is located in the second transverse process image are determined.

[0019] In one embodiment, acquiring the first transverse process image segmented from the ultrasound image of the vertebral region of the target object includes:

[0020] An ultrasound probe is used to scan the vertebral region of the target object in the sagittal plane to obtain an ultrasound image sequence with two-dimensional coordinate information;

[0021] A first reconstruction model is obtained by performing three-dimensional reconstruction based on the ultrasound image sequence.

[0022] The first transverse process image is segmented from the first reconstruction model; the first reconstruction model includes the three-dimensional coordinate information of each transverse process in the first transverse process image.

[0023] In one embodiment, the method further includes:

[0024] A second reconstruction model is obtained by performing three-dimensional reconstruction based on the image sequence containing the medical images.

[0025] The first coordinates of the ultrasound image scanned in real time by the ultrasound probe are obtained; the first coordinates are three-dimensional coordinates.

[0026] Based on the registration matrix, the first coordinates are converted into first transformed coordinates in the second reconstruction model, and based on the first transformed coordinates, the cross-sectional image of the ultrasound image in the second reconstruction model is determined;

[0027] The cross-sectional image and the ultrasound image are fused to obtain a fused image, which is then used for puncture guidance based on the fused image, the cross-sectional image, and the ultrasound image.

[0028] In one embodiment, the method further includes:

[0029] Obtain the planned puncture markers in the second reconstruction model after the three-dimensional reconstruction of the medical image; the puncture markers include at least one of puncture route coordinate information and puncture point coordinate information.

[0030] When the ultrasound probe scans the puncture marker, the puncture marker is displayed in the ultrasound image to guide the puncture.

[0031] Secondly, this application provides an image fusion apparatus, the apparatus comprising:

[0032] The image acquisition module is used to acquire a first transverse process image segmented from an ultrasound image of the vertebral region of the target object and a second transverse process image segmented from a medical image of the vertebral region of the target object.

[0033] The registration and fusion module is used to register the ultrasound image and the medical image based on the three-dimensional coordinate information of each transverse process in the first transverse process image and the second transverse process image to obtain a registration matrix.

[0034] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0035] The aforementioned image fusion method, apparatus, and computer equipment acquire a first transverse process image segmented from an ultrasound image of the vertebral region of the target object and a second transverse process image segmented from a medical image of the vertebral region. Based on the three-dimensional coordinate information of each transverse process in the first and second transverse process images, the ultrasound image and the medical image are registered. This allows for high-precision spatial alignment based on the coordinate information of multiple transverse processes, reducing registration errors and making the registration matrix more accurate. Consequently, in scenarios such as puncture, the image fused based on the registration matrix can be used to correctly guide the puncture. Attached Figure Description

[0036] Figure 1 This is an application environment diagram of the image fusion method in one embodiment;

[0037] Figure 2 This is a flowchart illustrating an image fusion method in one embodiment;

[0038] Figure 3 This is a schematic diagram of the first transverse process image in one embodiment;

[0039] Figure 4 This is a schematic diagram of the second transverse process image in one embodiment;

[0040] Figure 5 This is a schematic diagram of the lumbar spine arrangement in one embodiment;

[0041] Figure 6 This is a schematic diagram of a cross-sectional image in one embodiment;

[0042] Figure 7 This is a schematic diagram of the image interface in one embodiment;

[0043] Figure 8 This is a schematic diagram of the image interface in another embodiment;

[0044] Figure 9 This is a flowchart illustrating the image fusion method in another embodiment;

[0045] Figure 10 This is a structural block diagram of an image fusion device in one embodiment;

[0046] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The image fusion method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown is illustrated. Console 102 and terminal 104 are connected. Console 102 can be a device or a system for obtaining a registration matrix. Specifically, console 102 acquires a first transverse process image segmented from an ultrasound image of the vertebral region of the target object, and a second transverse process image segmented from a medical image of the vertebral region. Console 102 registers the ultrasound image and the medical image based on the three-dimensional coordinate information of each transverse process in the first and second transverse process images to obtain a registration matrix, which is then used for image fusion.

[0049] In one embodiment, such as Figure 2 As shown, an image fusion method is provided, which can be applied to... Figure 1 Taking the console in the browser as an example, the following steps are included:

[0050] S202, acquire the first transverse process image segmented from the ultrasound image of the vertebral region of the target object, and the second transverse process image segmented from the medical image of the vertebral region.

[0051] The vertebral region can be either the lumbar or thoracic vertebrae of the target object. Ultrasound images are obtained by scanning the vertebral region of the target object using an ultrasound probe. Methods for segmenting the first transverse process image from the ultrasound image include, but are not limited to, threshold segmentation, edge detection, region generation, and model segmentation. A combination of these methods can also be used. Threshold segmentation divides pixels in the image into several parts by setting one or more thresholds, thus segmenting the first transverse process image from other tissues in the ultrasound image based on pixel values. Edge detection uses edge detection algorithms to identify the edge contours of the transverse processes in the image, locating and segmenting the first transverse process image based on these contours. Model-based methods use a trained segmentation model to segment the first transverse process image from the ultrasound image. An example of a first transverse process image segmented from an ultrasound image of the lumbar vertebrae of the target object is shown below. Figure 3 As shown.

[0052] Medical imaging refers to static image data of the vertebral region of a target object at a specific moment. Medical imaging includes, but is not limited to, computed tomography (CT) imaging, magnetic resonance imaging (MRI), and positron emission tomography (PET). Methods for segmenting the second transverse process from medical images include, but are not limited to, thresholding, edge detection, region generation, and model segmentation. Combinations of these methods can also be used. An example of segmenting the second transverse process from medical images of the lumbar spine is shown below. Figure 4 As shown.

[0053] Optionally, after scanning the vertebral region of the target object with ultrasound, the control console acquires the ultrasound image scanned by the ultrasound probe and segments the first transverse process image from the ultrasound image using one or more of the following methods: threshold segmentation, edge detection, and model segmentation. After obtaining medical images of the vertebral region of the target object using one or more of the following methods: computed tomography, magnetic resonance imaging, and positron emission tomography, the control console segments the second transverse process image from the medical images using one or more of the following methods: threshold segmentation, edge detection, and model segmentation.

[0054] S204. Based on the three-dimensional coordinate information of each transverse process in the first and second transverse process images, the ultrasound image and medical image are registered to obtain a registration matrix, and image fusion is performed based on the registration matrix.

[0055] In the process of registering ultrasound images and medical images, it is first determined which transverse processes in the first and second transverse process images correspond to each other, that is, which transverse processes in the first and second transverse process images belong to the same vertebra. Then, based on the three-dimensional coordinate information of the transverse processes belonging to the same vertebra, the ultrasound images and medical images are registered.

[0056] The registration matrix can be used to convert the three-dimensional coordinate information in ultrasound images into the three-dimensional coordinate information in medical images. The inverse matrix of the registration matrix can be used to convert the three-dimensional coordinate information in medical images into the three-dimensional coordinate information in ultrasound images, thereby enabling the fusion between ultrasound images and medical images.

[0057] Optionally, the console first determines which transverse processes in the first and second transverse process images belong to the same vertebra. Then, based on the three-dimensional coordinate information of the transverse processes belonging to the same vertebra, the ultrasound image and the medical image are registered to obtain a registration matrix. The registration matrix is ​​used to convert the three-dimensional coordinate information in the ultrasound image into the three-dimensional coordinate information in the medical image, or to convert the three-dimensional coordinate information in the medical image into the three-dimensional coordinate information in the ultrasound image, thereby achieving the fusion between the ultrasound image and the medical image.

[0058] In the aforementioned image fusion method, a first transverse process image segmented from an ultrasound image of the vertebral region of the target object and a second transverse process image segmented from a medical image of the vertebral region are acquired. Based on the three-dimensional coordinate information of each transverse process in the first and second transverse process images, the ultrasound image and the medical image are registered. This allows for high-precision spatial alignment based on the coordinate information of multiple transverse processes, reducing registration errors and making the registration matrix more accurate. Consequently, in scenarios such as puncture, the image fused based on the registration matrix can be used to correctly guide the puncture.

[0059] In one embodiment, the ultrasound image and the medical image are registered based on the three-dimensional coordinate information of each transverse process in the first and second transverse process images to obtain a registration matrix, including:

[0060] Identify the vertebrae in which each transverse process is located in the first transverse process image and the vertebrae in which each transverse process is located in the second transverse process image.

[0061] Transverse processes in the same vertebra are identified as a pair of transverse processes by comparing the first and second images of the transverse process.

[0062] Based on the three-dimensional coordinate information of each transverse process in the midline, the ultrasound image and medical image are registered to obtain the registration matrix.

[0063] The term "vertebra" refers to the specific vertebra in which the transverse process is located within the target object. Specifically, the human body has 12 thoracic vertebrae, designated as T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, and T12. It also has 5 lumbar vertebrae, designated as L1, L2, L3, L4, and L5, which are connected sequentially as follows: Figure 5 As shown, L1 is the first lumbar vertebra, L2 is the second lumbar vertebra, L3 is the third lumbar vertebra, L4 is the fourth lumbar vertebra, and L5 is the fifth lumbar vertebra.

[0064] The first and second images of the transverse process refer to transverse processes that are in the same vertebra. For example, if transverse process A in the first image is a transverse process in the first lumbar vertebra, and transverse process B in the second image is also a transverse process in the first lumbar vertebra, then transverse processes A and B are in the same lumbar vertebra and belong to a pair of transverse processes.

[0065] In some embodiments, since the second transverse process image may include each transverse process of the target object's vertebral region, the transverse processes in the second transverse process image can be sorted according to the three-dimensional coordinate information of each transverse process in the second transverse process image, and the vertebra where each transverse process in the second transverse process image is located can be obtained according to the sorting result.

[0066] In some embodiments, the vertebrae in which each transverse process in the first / second transverse process image is located can be determined based on the morphological characteristics of the transverse process and / or adjacent structures. For example, the vertebrae are in the lumbar region, and the transverse process of the third lumbar vertebra is usually the longest, being the longest transverse process of all lumbar vertebrae. Therefore, the longest transverse process in the first / second transverse process image can be identified as the transverse process of the third lumbar vertebra. The transverse process of the fifth lumbar vertebra is usually thicker and extends laterally, forming the lumbosacral joint with the sacrum. Therefore, the thicker transverse process extending laterally in the first / second transverse process image can be identified as the transverse process of the fifth lumbar vertebra. The transverse process of the fifth lumbar vertebra connects to the sacrum, forming the lumbosacral angle. Therefore, the transverse process connected to the sacrum can also be identified as the transverse process of the fifth lumbar vertebra.

[0067] Optionally, the console sorts the transverse processes in the second transverse process image based on their three-dimensional coordinates, and determines the vertebrae where each transverse process is located based on the sorting result. The console also determines the vertebrae where each transverse process is located in the first transverse process image based on its morphological features and / or adjacent structures. The console identifies transverse processes in the same vertebra from both the first and second transverse process images as a transverse process pair, and registers the ultrasound image and medical image based on the three-dimensional coordinates of each transverse process in the pair to obtain a registration matrix.

[0068] In this embodiment, ultrasound images and medical images are registered based on the three-dimensional coordinate information of each transverse process. This ensures that the registration is performed only within the anatomical structure of the same vertebral body, avoiding cross-segment mismatches. For example, it can prevent the transverse process of the third lumbar vertebra in the first transverse process image of the lumbar spine from being aligned with the transverse process of the fourth lumbar vertebra in the second transverse process image, thereby improving the accuracy of the registration matrix.

[0069] In one embodiment, determining the vertebrae in which each transverse process is located in the first transverse process image includes:

[0070] Obtain the center coordinates of each transverse process in each vertebra of the target object.

[0071] Based on the center coordinates and the three-dimensional coordinates of each transverse process in the first transverse process image, the target center that is closest to each transverse process in the first transverse process image is determined from each center.

[0072] The vertebrae corresponding to the target center are identified as the vertebrae where each transverse process is located in the first transverse process image.

[0073] The center coordinates of each transverse process refer to the coordinates of its center. The transverse process center can be obtained from a target image including each vertebra of the target. Specifically, the target image is binarized to obtain a binarized image, and the edge contours of each transverse process in the binarized image are identified. The centroid of the edge contour is determined as the center of the transverse process of the target object, or a bounding rectangle containing the edge contour is determined, and the center of the bounding rectangle is determined as the center of the transverse process of the target object. Furthermore, the vertebra corresponding to each center can be determined based on the coordinate sorting results of each center. Specifically, the sorting results of each center are obtained based on the magnitude of its coordinate values ​​on the target axis, and the vertebra corresponding to each center is determined based on the sorting results. The direction of the target axis is from the foot to the head of the target object.

[0074] In the first transverse process image, each transverse process corresponds to a target center. For example, in the first transverse process image, there are transverse processes 1, 2, and 3. Calculations show that center 1 is closest to transverse process 1, center 2 is closest to transverse process 2, and center 3 is closest to transverse process 3. Therefore, center 1 is the target center of transverse process 1, center 2 is the target center of transverse process 2, and center 3 is the target center of transverse process 3. The vertebra corresponding to center 1 is the vertebra where transverse process 1 is located, the vertebra corresponding to center 2 is the vertebra where transverse process 2 is located, and the vertebra corresponding to center 3 is the vertebra where transverse process 3 is located.

[0075] In some embodiments, the center coordinates of each transverse process can also be the coordinates of the cluster center of the transverse processes. Specifically, the transverse process centers of each vertebra of the target object are scanned from multiple directions to obtain the first coordinates of each transverse process center in each direction; principal component analysis and cluster analysis are performed based on each first coordinate to obtain the cluster center of each transverse process and the coordinates of the cluster center.

[0076] Optionally, the console performs binarization processing on the target image including each vertebra of the target object to obtain a binarized image, and identifies the edge contours of each transverse process in the binarized image. The centroid of the edge contour is determined as the center of the transverse process of the target object vertebra, or a bounding rectangle containing the edge contour is determined, and the center of the bounding rectangle is determined as the center of the transverse process of the target object vertebra, and the coordinates of the transverse process center are obtained. The console determines the vertebra corresponding to each center based on the coordinate sorting results. The console calculates the distance between each center and each transverse process based on the center coordinates and the three-dimensional coordinate information of each transverse process in the first transverse process image, and determines the target center from each center that is closest to each transverse process in the first transverse process image based on the calculation results. The console determines the vertebra corresponding to the target center as the vertebra where each transverse process in the first transverse process image is located.

[0077] In this embodiment, by using the center coordinates and the three-dimensional coordinate information of each transverse process in the first transverse process image, the target center that is closest to each transverse process in the first transverse process image is determined from each center, so that the vertebra corresponding to each transverse process can be accurately determined.

[0078] In one embodiment, determining the vertebrae in which each transverse process is located in the second transverse process image includes:

[0079] Based on the coordinates of each transverse process in the second transverse process image on the target axis, the transverse processes in the second transverse process image are sorted.

[0080] Based on the sorting results of the transverse processes in the second transverse process image, the vertebrae in which each transverse process is located in the second transverse process image are determined.

[0081] The target axis is oriented from the feet to the head of the target object. Since the vertebrae are arranged sequentially from the head to the feet of the target object, the target axis is used as the direction from the feet to the head. The transverse processes in the second transverse process image are sorted according to their coordinates on the target axis to determine the vertebrae where each transverse process is located in the second transverse process image, which improves accuracy.

[0082] Furthermore, based on the coordinate values ​​of each transverse process on the target axis, they are sorted from low to high to obtain the sorting result of each transverse process. Transverse processes with smaller coordinate values ​​are located at the bottom, closer to the feet of the target object, while transverse processes with larger coordinate values ​​are located at the top, closer to the head of the target object. Specifically, if the vertebrae are lumbar vertebrae, then the lumbar vertebrae where each transverse process is located in the sorting result are the fifth, fourth, third, second, and first lumbar vertebrae, respectively. That is, the transverse process ranked first is located in the fifth lumbar vertebra, the second in the fourth lumbar vertebra, the third in the third lumbar vertebra, the fourth in the second lumbar vertebra, and the fifth in the first lumbar vertebra.

[0083] Optionally, the console sorts each transverse process from low to high according to its coordinate value on the target axis to obtain the sorting result of each transverse process.

[0084] In this embodiment, by sorting the coordinates of the transverse processes on the target axis, the transverse processes of different vertebrae can be more clearly identified and distinguished, which helps to avoid confusion between the transverse processes of adjacent vertebrae. This allows for accurate guidance of the puncture needle or anesthetic needle during puncture.

[0085] In one embodiment, acquiring a first transverse process image segmented from an ultrasound image of the vertebral region of the target object includes:

[0086] An ultrasound probe is used to scan the vertebral region of the target object in the sagittal plane to obtain an ultrasound image sequence with two-dimensional coordinate information;

[0087] A first reconstruction model is obtained by performing three-dimensional reconstruction based on ultrasound image sequences;

[0088] The first transverse process image is segmented from the first reconstruction model; the first reconstruction model includes the three-dimensional coordinate information of each transverse process in the first transverse process image.

[0089] The sagittal plane is a cross-sectional view of the human body, referring to a plane that cuts along the target object from front to back or from back to front. Performing ultrasound scans of the vertebrae in the sagittal plane helps to observe the overall orientation of the transverse processes and their relative positions to the anterior and posterior aspects of the spine.

[0090] An ultrasound image sequence consists of multiple ultrasound images. These images are obtained by scanning the vertebral region of the target object from a sagittal plane using an ultrasound probe. The ultrasound image sequence is scanned by the ultrasound probe and transmitted to a control console.

[0091] The three-dimensional coordinate information of each transverse process in the first transverse process image can be obtained through two-dimensional coordinate transformation. Specifically, the pose information of the ultrasound probe during the ultrasound image acquisition process is acquired, and the transformation matrix is ​​determined based on the pose information. The transformation matrix is ​​used to characterize the transformation relationship from the two-dimensional image coordinate system to the global three-dimensional coordinate system. The two-dimensional coordinate information of each transverse process in the first transverse process image is transformed using the transformation matrix to obtain the three-dimensional coordinate information of each transverse process in the first transverse process image.

[0092] Optionally, the operator controls the ultrasound probe to scan the vertebral region of the target object from a sagittal position and sends the scanned multiple ultrasound images to a control console. The control console receives the ultrasound image sequence with two-dimensional coordinate information and performs three-dimensional reconstruction on the ultrasound image sequence to obtain a first reconstructed model. The control console segments the first transverse process image from the first reconstructed model and obtains the pose information of the ultrasound probe during the ultrasound image acquisition process. Based on the pose information, the control console determines a transformation matrix and uses the transformation matrix to transform the two-dimensional coordinate information of each transverse process in the first transverse process image to obtain the three-dimensional coordinate information of each transverse process in the first transverse process image.

[0093] In this embodiment, a first reconstruction model is obtained by performing three-dimensional reconstruction based on ultrasound image sequences. This allows for observation of the structure of the vertebrae and their transverse processes from multiple angles, which helps to more comprehensively understand the structural relationship between the vertebrae and their transverse processes.

[0094] In some embodiments, the image fusion method further includes performing data preprocessing on each frame of the acquired ultrasound image sequence. This can improve the image quality and accuracy of the ultrasound image sequence. Data preprocessing includes, but is not limited to, denoising and contrast enhancement. Denoising can be achieved using filtering algorithms, and contrast enhancement can be achieved through histogram equalization or contrast boosting.

[0095] In some embodiments, before performing 3D reconstruction on the ultrasound image sequence, image feature matching can be used to align adjacent frames of the ultrasound image sequence to improve the accuracy of 3D reconstruction. Image feature matching refers to detecting and matching key feature points in different ultrasound images, determining the correspondence between these key feature points, and aligning adjacent frames of the ultrasound image sequence based on these correspondences.

[0096] In some embodiments, after obtaining the first reconstruction model, the first reconstruction model can be smoothed to further improve the accuracy of the first reconstruction model.

[0097] In some embodiments, the method for obtaining the three-dimensional coordinate information of the first transverse process image further includes: obtaining the three-dimensional coordinate information of the first transverse process image segmented from the ultrasound image through an electromagnetic positioning system.

[0098] In some embodiments, acquiring a first transverse process image segmented from an ultrasound image of the vertebral region of a target object includes: scanning the vertebral region of the target object from a sagittal plane using an ultrasound probe to obtain an ultrasound image sequence with two-dimensional coordinate information; segmenting two-dimensional transverse process images from each frame of the ultrasound image sequence; and reconstructing the two-dimensional transverse process images in three dimensions to obtain the first transverse process image.

[0099] In one embodiment, the image fusion method further includes:

[0100] A second reconstruction model is obtained by performing three-dimensional reconstruction based on the image sequence containing the medical image.

[0101] Obtain the first coordinate of the ultrasound image scanned in real time by the ultrasound probe. The first coordinate is a three-dimensional coordinate.

[0102] Based on the registration matrix, the first coordinates are transformed into the first transformed coordinates in the second reconstruction model, and based on the first transformed coordinates, the cross-sectional image of the ultrasound image in the second reconstruction model is determined.

[0103] By fusing cross-sectional images and ultrasound images, a fused image is obtained, which is then used for puncture guidance based on the fused image, cross-sectional image, and ultrasound image.

[0104] Here, the image sequence containing the medical images refers to an image sequence composed of multiple frames of medical images. Specifically, a second reconstruction model is obtained by performing 3D reconstruction using multiple frames of medical images. The first coordinates can be obtained through a magnetic positioning system.

[0105] During ultrasound scanning, the control console can acquire cross-sectional images in the second reconstruction model in real time based on the first coordinates of the scanned ultrasound image. This provides more comprehensive and accurate imaging information. Specifically, ultrasound images provide real-time soft tissue contrast, while computed tomography (CT) images provide detailed bone structure and density information. Combining the two helps doctors obtain more complete and precise structural location information. The coordinates of the cross-sectional image in the second reconstruction model are the first transformed coordinates. A cross-sectional image corresponding to the lumbar spine is shown below. Figure 6 As shown.

[0106] In some embodiments, the 3D reconstruction process includes: preprocessing multiple frames of medical images in an image sequence to obtain a preprocessed image sequence; and using a 3D reconstruction algorithm to perform 3D reconstruction on the preprocessed image sequence to obtain a second reconstruction model. The data preprocessing includes at least one of denoising, image registration, and normalization. Denoising refers to reducing noise in the medical images; normalization refers to adjusting the grayscale range in the medical images; and image registration refers to registering multiple frames of medical images to the same coordinate system. The 3D reconstruction methods include, but are not limited to, voxel interpolation and surface reconstruction methods.

[0107] In some embodiments, the segmentation of the first transverse process image and the second transverse process image can both be achieved using a segmentation model. For example, a trained 3D U-Net model can be used to segment the first reconstruction model to obtain the first transverse process image, and a trained 3D U-Net model can be used to segment the second reconstruction model to obtain the second transverse process image.

[0108] Optionally, the console preprocesses multiple frames of medical images to obtain a preprocessed image sequence, and then uses a 3D reconstruction algorithm to perform 3D reconstruction on the preprocessed image sequence to obtain a second reconstruction model. The console acquires the first coordinates of the ultrasound image scanned in real-time by the ultrasound probe through a magnetic positioning system, and converts the first coordinates into the first transformed coordinates in the second reconstruction model based on a registration matrix. The console then searches for the cross-sectional image corresponding to the first transformed coordinates in the second reconstruction model. The console fuses the cross-sectional image and the ultrasound image to obtain a fused image, which is used for puncture guidance based on the fused image, cross-sectional image, and ultrasound image.

[0109] In one embodiment, the console can also receive a contrast adjustment request to adjust the contrast of the fused image, thereby significantly distinguishing the cross-sectional image from the ultrasound image.

[0110] In this embodiment, by fusing ultrasound images and cross-sectional images, the soft tissue contrast and detailed skeleton of the target vertebral region can be displayed simultaneously, which helps to more comprehensively understand the target vertebral region and its surrounding environment, thereby reducing puncture risk and improving puncture success rate.

[0111] In one embodiment, the image fusion method further includes:

[0112] Obtain the planned puncture markers in the second reconstruction model after 3D reconstruction of medical images; the puncture markers include at least one of puncture route coordinate information and puncture point coordinate information.

[0113] When the ultrasound probe detects the puncture marker, the puncture marker is displayed in the ultrasound image to guide the puncture.

[0114] Puncture markers can be determined by staff based on the nerve tissue at the vertebral region of the target patient. These markers guide the staff during the anesthesia puncture, providing more precise pain control and reducing risks. Puncture markers can also be determined based on anatomical fixation points, such as the iliac crest or spinous process, to help pinpoint the puncture site. For example, during spinal anesthesia, the space between the third and fourth vertebrae, or the space between the fourth and fifth vertebrae, can be used as the puncture point.

[0115] The puncture point refers to the point where the anesthetic needle or puncture needle will eventually reach, and the puncture path refers to the route taken by the anesthetic needle or puncture needle in the target object.

[0116] In some embodiments, puncture markers can also be automatically planned by the control console based on preset marker information. The preset marker information includes pre-defined candidate puncture points and / or candidate puncture routes. For example, when performing spinal anesthesia on the lumbar spine, medical staff typically use the gap between the third and fourth lumbar vertebrae or the gap between the fourth and fifth lumbar vertebrae as the puncture point. Therefore, the gap between the third and fourth lumbar vertebrae or the gap between the fourth and fifth lumbar vertebrae can be used as preset marker information, allowing the control console to automatically plan the gap between the third and fourth lumbar vertebrae or the gap between the fourth and fifth lumbar vertebrae as the puncture marker.

[0117] During ultrasound scanning, the fusion of ultrasound images and medical images is performed in real time. Therefore, when an ultrasound image is scanned, its positional information is converted into its positional information in the medical image coordinate system using a registration matrix. The puncture marker is planned in the second reconstruction model, meaning it has its own positional information in the medical image coordinate system. When the converted cross-sectional image of the ultrasound image intersects with the position of the puncture marker in the medical image coordinate system, it indicates that the puncture marker has been scanned in the ultrasound image. Determining whether the ultrasound probe has scanned the puncture marker is done in real time.

[0118] Displaying puncture markers in ultrasound images can be achieved through coordinate transformation. Specifically, based on the inverse of the registration matrix, the coordinate information of the puncture markers is converted into coordinate information in the ultrasound coordinate system, and the scanned puncture markers are displayed in the ultrasound image according to the transformed coordinate information.

[0119] In some embodiments, a puncture marker may also be displayed in the fused image obtained by fusing the ultrasound image and the cross-sectional image.

[0120] Optionally, after the staff plans the puncture markers based on the neural tissue of the target vertebra, the console automatically acquires the planned puncture markers in the second reconstruction model after the 3D reconstruction of the medical image. The puncture markers include at least one of puncture route coordinates and puncture point coordinates. During ultrasound probe scanning, the console converts the positional information of the ultrasound image into positional information in medical image coordinates in real time using a registration matrix. When the converted cross-sectional image of the ultrasound image intersects with the position of the puncture marker in the medical image coordinate system, it is determined that the puncture marker has been scanned in the ultrasound image. Based on the inverse matrix of the registration matrix, the console converts the coordinate information of the puncture marker into coordinates in the ultrasound coordinate system and displays the scanned puncture marker in the ultrasound image according to the converted coordinates to guide the puncture.

[0121] In this embodiment, by displaying the puncture marker in the ultrasound image, doctors can more intuitively see the position of the needle tip and its position relative to the puncture marker, which facilitates real-time adjustment of the puncture angle and depth and improves the controllability of the operation.

[0122] In one embodiment, when the ultrasound probe detects a puncture marker, displaying the puncture marker in the ultrasound image to guide the puncture includes:

[0123] The second coordinate of the ultrasound image scanned in real time by the ultrasound probe is obtained. The second coordinate is a three-dimensional coordinate.

[0124] Based on the registration matrix, the second coordinates are transformed into the second transformed coordinates in the second reconstruction model.

[0125] When the second transformation coordinates match at least one of the puncture route coordinates and the puncture point coordinates, the ultrasound probe is identified as having scanned the puncture marker.

[0126] The puncture marker is displayed in the ultrasound image to guide the puncture.

[0127] The second coordinate can be obtained through a magnetic positioning system. Matching the second transformed coordinate with at least one of the puncture route coordinate information and the puncture point coordinate information means that the second transformed coordinate intersects with at least one of the puncture route coordinate information and the puncture point coordinate information.

[0128] When displaying puncture markers in ultrasound images, only the puncture markers scanned in real time by the ultrasound probe are shown.

[0129] Optionally, the console acquires the second coordinates of the ultrasound image scanned in real time by the ultrasound probe through a magnetic positioning system. Based on the registration matrix, the console converts the second coordinates into second transformed coordinates in the second reconstruction model. When the second transformed coordinates match at least one of the puncture route coordinates and puncture point coordinates, the console determines that the ultrasound probe has scanned a puncture marker and displays the scanned puncture marker in the ultrasound image to guide the puncture.

[0130] In this embodiment, by acquiring the second coordinates of the ultrasound image scanned in real time by the ultrasound probe and converting them into the second transformed coordinates in the second reconstruction model based on a pre-determined registration matrix, accurate matching between different modal image data is achieved. This ensures that the structures in the ultrasound image can accurately correspond to the second reconstruction model after the medical image is reconstructed in three dimensions, improving positioning accuracy. When the second transformed coordinates match the puncture route coordinates or puncture point coordinates, it is determined that the ultrasound probe has scanned the puncture marker, and the puncture marker scanned on the ultrasound image is detected. This real-time feedback mechanism greatly enhances visualization and intuitiveness, facilitating more precise puncture operations.

[0131] In some embodiments, the console includes a display screen that can be used to display an image interface, which includes a first region 10, a second region 20, and a third region 30. The first region 10 can simultaneously display ultrasound images and fused images, and the display positions of the fused images and ultrasound images within the first region 10 can be switched. The second region 20 displays a three-dimensional model for guiding puncture, and the third region 30 can be used to display acquired medical images and / or fused images. A schematic diagram of the image interface is shown below. Figure 7 and Figure 8 As shown. The three-dimensional model used to guide the puncture can be either a first reconstruction model or a second reconstruction model.

[0132] This application also provides an application scenario in which the above-described image fusion method is applied. Specifically, the image fusion method is applied in this scenario as follows:

[0133] The operator controls an ultrasound probe to scan the vertebrae of the target object in a sagittal plane and sends multiple frames of ultrasound images to a control console. The control console receives the ultrasound image sequence with two-dimensional coordinate information and performs three-dimensional reconstruction of the ultrasound image sequence to obtain a first reconstructed model. The control console segments the first transverse process image from the first reconstructed model and obtains the pose information of the ultrasound probe during the ultrasound image acquisition process. Based on the pose information, the control console determines a transformation matrix and uses the transformation matrix to transform the two-dimensional coordinate information of each transverse process in the first transverse process image to obtain the three-dimensional coordinate information of each transverse process in the first transverse process image. After obtaining medical images of the vertebrae of the target object through one or more of the methods of computed tomography, magnetic resonance imaging, and positron emission tomography, the control console uses one or more of the methods of threshold segmentation, edge detection, and model segmentation to segment the second transverse process image from the medical images.

[0134] The console sorts the transverse processes from lowest to highest coordinate value on the target axis, obtaining the sorting results. Based on the morphological characteristics and / or adjacent structures of each transverse process in the first transverse process image, the console determines the vertebra where each transverse process is located. The console identifies the transverse processes in the first and second transverse process images within the same vertebra as a transverse process pair, and registers the ultrasound image and medical image based on the three-dimensional coordinate information of each transverse process in the pair, obtaining a registration matrix.

[0135] The console preprocesses multiple frames of medical images to obtain a preprocessed image sequence. A 3D reconstruction algorithm is then used to reconstruct the preprocessed image sequence in 3D, resulting in a second reconstruction model. The console acquires the first coordinates of the ultrasound image scanned in real-time by the ultrasound probe via a magnetic positioning system. Based on the registration matrix, the first coordinates are converted into the first transformed coordinates in the second reconstruction model. The console then locates the cross-sectional image corresponding to the first transformed coordinates within the second reconstruction model. Finally, the console fuses the cross-sectional image and the ultrasound image to obtain a fused image, which is used for puncture guidance.

[0136] After staff plan the puncture markers based on the neural tissue of the target vertebra, the control console automatically acquires the planned puncture markers in the second reconstruction model after 3D reconstruction of the medical image. The puncture markers include at least one of the following: puncture route coordinates and puncture point coordinates. During ultrasound probe scanning, the control console converts the positional information of the ultrasound image into positional information in medical image coordinates in real time using a registration matrix. When the converted cross-sectional image of the ultrasound image intersects with the position of the puncture marker in the medical image coordinate system, it is determined that the puncture marker has been scanned in the ultrasound image. Based on the inverse matrix of the registration matrix, the control console converts the coordinate information of the puncture marker into coordinates in the ultrasound coordinate system and displays the scanned puncture marker in the ultrasound image according to the converted coordinates to guide the puncture. A detailed flowchart is shown below. Figure 9 As shown.

[0137] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0138] Based on the same inventive concept, this application also provides an image fusion apparatus for implementing the image fusion method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image fusion apparatus embodiments provided below can be found in the limitations of the image fusion method described above, and will not be repeated here.

[0139] In one embodiment, such as Figure 10 As shown, an image fusion apparatus is provided, comprising:

[0140] The image acquisition module 1002 is used to acquire a first transverse process image segmented from an ultrasound image of the vertebral region of the target object and a second transverse process image segmented from a medical image of the vertebral region of the target object.

[0141] The registration and fusion module 1004 is used to register the ultrasound image and the medical image based on the three-dimensional coordinate information of each transverse process in the first transverse process image and the second transverse process image to obtain a registration matrix.

[0142] In one embodiment, the image fusion device is further configured to: determine the vertebrae in which each transverse process is located in the first transverse process image and the vertebrae in which each transverse process is located in the second transverse process image; determine the transverse processes in the first and second transverse process images in the same vertebra as a transverse process pair; and register the ultrasound image and the medical image according to the three-dimensional coordinate information of each transverse process in the transverse process pair to obtain a registration matrix.

[0143] In one embodiment, the image fusion apparatus is further configured to: obtain the center coordinates of each transverse process in each vertebra of the target object; determine the target center that is closest to each transverse process in the first transverse process image from each center based on the center coordinates and the three-dimensional coordinate information of each transverse process in the first transverse process image; and determine the vertebra corresponding to the target center as the vertebra where each transverse process in the first transverse process image is located.

[0144] In one embodiment, the image fusion apparatus is further configured to: sort the transverse processes in the second transverse process image according to the coordinate size of each transverse process in the target axis; the direction of the target axis is from the foot to the head of the target object; and determine the vertebra where each transverse process in the second transverse process image is located according to the sorting result of each transverse process in the second transverse process image.

[0145] In one embodiment, the image fusion apparatus is further configured to: scan the vertebral region of the target object from a sagittal position using an ultrasound probe to obtain an ultrasound image sequence with two-dimensional coordinate information; perform three-dimensional reconstruction based on the ultrasound image sequence to obtain a first reconstruction model; segment a first transverse process image from the first reconstruction model; the first reconstruction model includes the three-dimensional coordinate information of each transverse process in the first transverse process image.

[0146] In one embodiment, the image fusion device is further configured to: perform three-dimensional reconstruction based on the image sequence containing the medical image to obtain a second reconstruction model; acquire the first coordinates of the ultrasound image scanned in real time by the ultrasound probe; the first coordinates are three-dimensional coordinates; based on the registration matrix, convert the first coordinates into the first transformed coordinates in the second reconstruction model, and based on the first transformed coordinates, determine the cross-sectional image of the ultrasound image in the second reconstruction model; fuse the cross-sectional image and the ultrasound image to obtain a fused image, for puncture guidance based on the fused image, the cross-sectional image, and the ultrasound image.

[0147] In one embodiment, the image fusion device is further configured to: acquire a planned puncture marker in a second reconstruction model after three-dimensional reconstruction of the medical image; the puncture marker includes at least one of puncture route coordinate information and puncture point coordinate information; and display the puncture marker in the ultrasound image when the ultrasound probe scans the puncture marker to guide the puncture.

[0148] In one embodiment, the image fusion device is further configured to: acquire a second coordinate of an ultrasound image scanned in real time by an ultrasound probe; the second coordinate is a three-dimensional coordinate; convert the second coordinate into a second transformed coordinate in a second reconstruction model based on a registration matrix; determine a puncture marker scanned by the ultrasound probe when the second transformed coordinate matches at least one of the puncture route coordinate information and the puncture point coordinate information; and display the puncture marker in the ultrasound image to guide the puncture.

[0149] Each module in the aforementioned image fusion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0150] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11As shown. The computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores ultrasound images, first transverse process images, medical images, second transverse process images, three-dimensional coordinate information of the transverse processes, registration matrices, transverse process pairs, center coordinates, target centers, sorting results of each transverse process, a first reconstruction model, a second reconstruction model, first coordinates, first transformed coordinates, fused images, puncture markers, second coordinates, and second transformed coordinates. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements an image fusion method.

[0151] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0153] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0154] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0156] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0157] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image fusion method, characterized in that, The method includes: Acquire a first transverse process image segmented from an ultrasound image of the vertebral region of the target object, and a second transverse process image segmented from a medical image of the vertebral region; Based on the three-dimensional coordinate information of each transverse process in the first transverse process image and the second transverse process image, the ultrasound image and the medical image are registered to obtain a registration matrix; A second reconstruction model is obtained by performing three-dimensional reconstruction based on the image sequence containing the medical images. The first coordinates of the ultrasound image scanned in real time by the ultrasound probe are obtained; the first coordinates are three-dimensional coordinates. Based on the registration matrix, the first coordinates are converted into first transformed coordinates in the second reconstruction model, and based on the first transformed coordinates, the cross-sectional image of the ultrasound image in the second reconstruction model is determined; The cross-sectional image and the ultrasound image are fused to obtain a fused image, which is then used for puncture guidance based on the fused image, the cross-sectional image, and the ultrasound image.

2. The method according to claim 1, characterized in that, The process of registering the ultrasound image and the medical image based on the three-dimensional coordinate information of each transverse process in the first and second transverse process images to obtain a registration matrix includes: Identify the vertebrae in which each transverse process is located in the first transverse process image and the vertebrae in which each transverse process is located in the second transverse process image; Transverse processes in the same vertebra are identified as a pair of transverse processes, based on the first and second transverse process images. Based on the three-dimensional coordinate information of each transverse process in the transverse process alignment, the ultrasound image and the medical image are registered to obtain a registration matrix.

3. The method according to claim 2, characterized in that, The step of determining the vertebrae where each transverse process is located in the first transverse process image includes: Obtain the center coordinates of each transverse process in each vertebra of the target object; Based on the center coordinates and the three-dimensional coordinate information of each transverse process in the first transverse process image, determine the target center that is closest to each transverse process in the first transverse process image from each center; The vertebrae corresponding to the target center are identified as the vertebrae where each transverse process is located in the first transverse process image.

4. The method according to claim 2, characterized in that, Determining the vertebrae in which each transverse process is located in the second transverse process image includes: Based on the coordinates of each transverse process in the second transverse process image on the target axis, the transverse processes in the second transverse process image are sorted; the direction of the target axis is from the foot to the head of the target object; Based on the sorting results of each transverse process in the second transverse process image, the vertebrae in which each transverse process is located in the second transverse process image are determined.

5. The method according to claim 1, characterized in that, The acquisition of the first transverse process image segmented from the ultrasound image of the vertebral region of the target object includes: An ultrasound probe is used to scan the vertebral region of the target object in the sagittal plane to obtain an ultrasound image sequence with two-dimensional coordinate information; A first reconstruction model is obtained by performing three-dimensional reconstruction based on the ultrasound image sequence. The first transverse process image is segmented from the first reconstruction model; the first reconstruction model includes the three-dimensional coordinate information of each transverse process in the first transverse process image.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the planned puncture markers in the second reconstruction model after the three-dimensional reconstruction of the medical image; the puncture markers include at least one of puncture route coordinate information and puncture point coordinate information. When the ultrasound probe scans the puncture marker, the puncture marker is displayed in the ultrasound image to guide the puncture.

7. The method according to claim 6, characterized in that, The step of displaying the puncture marker in the ultrasound image when the ultrasound probe detects the puncture marker to guide the puncture includes: The second coordinates of the ultrasound image scanned in real time by the ultrasound probe are obtained; the second coordinates are three-dimensional coordinates. Based on the registration matrix, the second coordinates are transformed into the second transformed coordinates in the second reconstruction model; When the second transformed coordinates match at least one of the puncture route coordinates and the puncture point coordinates, it is determined that the ultrasound probe has scanned the puncture marker; The puncture marker is displayed in the ultrasound image to guide the puncture.

8. An image fusion apparatus, characterized in that, The device includes: The image acquisition module is used to acquire a first transverse process image segmented from an ultrasound image of the vertebral region of the target object and a second transverse process image segmented from a medical image of the vertebral region of the target object. The registration and fusion module is used to register the ultrasound image and the medical image based on the three-dimensional coordinate information of each transverse process in the first transverse process image and the second transverse process image to obtain a registration matrix; The method includes: performing three-dimensional reconstruction based on the image sequence containing the medical image to obtain a second reconstruction model; acquiring the first coordinates of the ultrasound image scanned in real time by the ultrasound probe; the first coordinates being three-dimensional coordinates; converting the first coordinates into first transformed coordinates in the second reconstruction model based on the registration matrix, and determining the cross-sectional image of the ultrasound image in the second reconstruction model based on the first transformed coordinates; fusing the cross-sectional image and the ultrasound image to obtain a fused image, and performing puncture guidance based on the fused image, the cross-sectional image, and the ultrasound image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images