Image stitching method, image fusion method, device and computer equipment

Abstract

The application relates to an image splicing method, an image fusion method, an image splicing device and a computer device. The image fusion method comprises the following steps: acquiring a region with information distribution satisfying a preset distribution condition in a first medical image as a reference image, acquiring a region in a second medical image satisfying a similarity requirement with the reference image as a target image according to the reference image, and then acquiring a conversion relationship between the reference image and the target image, so that the reference image and the target image are spliced according to the conversion relationship. Through the above method, the reference image and the target image for splicing can be screened in a targeted manner, a large amount of interference information is excluded while the effectiveness of the image content is ensured, redundant calculation is avoided, and the efficiency and accuracy of image splicing are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image stitching method, an image fusion method, an apparatus, and a computer device. Background Technology

[0002] Medical ultrasound diagnostic technology is an indispensable part of modern medical imaging technology, and the information content of ultrasound images obtained based on ultrasound diagnostic technology occupies an important position among various medical information.

[0003] Due to limitations in the width and scanning angle of ultrasound probes, real-time ultrasound scanning has a limited field of view, resulting in narrow images that cannot fully reflect the observed tissue. To address this issue, traditional techniques typically segment the real-time ultrasound image into small image blocks. These small image blocks are then registered with multiple image blocks cut from the next frame of ultrasound image to stitch together frame-by-frame ultrasound images, forming a wide-view ultrasound image.

[0004] However, traditional image stitching techniques are time-consuming and have low accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide an image stitching method, an image fusion method, an apparatus, and a computer device to address the aforementioned technical problems.

[0006] Firstly, this application provides an image stitching method, including:

[0007] Obtain a reference image of the first medical image, wherein the reference image represents the region in the first medical image whose information distribution satisfies a preset distribution condition;

[0008] The target image in the second medical image is obtained from the reference image. The target image represents the region in the second medical image that meets the similarity requirement with the reference image.

[0009] Obtain the transformation relationship between the reference image and the target image;

[0010] Based on the transformation relationship, the reference image and the target image are stitched together.

[0011] In one embodiment, obtaining the transformation relationship between the reference image and the target image includes:

[0012] Construct a first objective function; the first objective function characterizes the registration error after the registration processing of the reference image and the target image, and the functional relationship between the reference image and the target image after transformation by the transformation relation;

[0013] When the registration error is minimized, the first objective function is solved based on the target image and the reference image to obtain at least one candidate transformation relationship between the reference image and the target image.

[0014] Based on at least one candidate transformation relationship, determine the transformation relationship between the reference image and the target image.

[0015] In one embodiment, the registration error includes angular scale error, and the transformation relationship includes rotation angle and scaling scale; when multiple candidate transformation relationships exist, determining the transformation relationship between the reference image and the target image based on at least one candidate transformation relationship includes:

[0016] Construct a second objective function relating angular scale error to the transformation relationship;

[0017] When the angular scale error and the registration error are both minimized, the second objective function is solved to obtain the rotation angle of the target image, the rotation angle of the reference image, and the scaling factor of the target image relative to the reference image.

[0018] Based on the rotation angle values ​​of the target image, the rotation angle values ​​of the reference image, and the scaling factor, a transformation relationship is selected from multiple candidate transformation relationships.

[0019] In one embodiment, if the first medical image is the first frame of the medical image during the image acquisition process, then the reference image is the middle region image of the first medical image;

[0020] If the first medical image is not the first frame of the medical image during the image acquisition process, then the reference image is the target image in the first medical image.

[0021] In one embodiment, the method further includes:

[0022] Based on the conversion relationship between each frame of medical images and other frames of medical images acquired during the image acquisition process, all medical images are stitched together to obtain a preliminary stitched image.

[0023] The noise information of the initial stitched image is optimized to obtain the optimized stitched image.

[0024] Secondly, this application also provides an image fusion method, including:

[0025] Obtain candidate medical image sequences; the candidate medical image sequences include multiple candidate medical images;

[0026] The medical images that match each target medical image in the candidate medical image sequence are obtained, resulting in multiple matching candidate medical images; among them, the target medical images and the candidate medical images are medical images of different types.

[0027] Multiple matching candidate medical images are stitched together to obtain a candidate stitched image;

[0028] The target stitched image is fused with the candidate stitched image to obtain a fused stitched image; wherein, the target stitched image is obtained by stitching together each target medical image using any of the above image stitching methods.

[0029] In one embodiment, the medical images that match each acquired target medical image in the candidate medical image sequence are obtained, resulting in multiple matching candidate medical images, including:

[0030] For each target medical image, preliminary candidate medical images that meet the similarity requirements with the target medical image are identified from the candidate medical image sequence;

[0031] Match and validate the target medical image with the preliminary candidate medical images;

[0032] If the matching verification is successful, the preliminary candidate medical image is determined as a matching candidate medical image for the target medical image.

[0033] In one embodiment, matching and verification of the target medical image and preliminary candidate medical images includes:

[0034] Determine the conversion relationship between the target medical image and the preliminary candidate medical images;

[0035] The current registration error is determined based on the conversion relationship between the target medical image and the preliminary candidate medical images;

[0036] If the current registration error is less than or equal to the error threshold, then the matching verification is considered successful.

[0037] Thirdly, this application also provides an image stitching device, comprising:

[0038] The reference determination module is used to acquire a reference image of the first medical image, wherein the reference image represents the region in the first medical image whose information distribution satisfies a preset distribution condition;

[0039] The target determination module is used to obtain the target image in the second medical image based on the reference image. The target image represents the region in the second medical image that meets the similarity requirement with the reference image.

[0040] The transformation determination module is used to obtain the transformation relationship between the reference image and the target image;

[0041] The stitching processing module is used to stitch the reference image and the target image according to the transformation relationship.

[0042] Fourthly, this application also provides an image fusion apparatus, comprising:

[0043] The candidate acquisition module is used to acquire candidate medical image sequences; the candidate medical image sequence includes multiple candidate medical images.

[0044] The candidate matching module is used to obtain the medical images that match each target medical image in the candidate medical image sequence, resulting in multiple matching candidate medical images; among them, the target medical images and candidate medical images are different types of medical images;

[0045] The candidate stitching module is used to obtain candidate stitching images between multiple matching candidate medical images;

[0046] The fusion processing module is used to fuse the target stitched image with the candidate stitched image to obtain a fused stitched image; wherein, the target stitched image is obtained by stitching together each target medical image using any of the above image stitching methods.

[0047] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0048] In the aforementioned image stitching method, image fusion method, apparatus, and computer device, a region in a first medical image whose information distribution meets preset distribution conditions is obtained as a reference image. Then, a region in a second medical image that meets similarity requirements with the reference image is obtained as a target image. Furthermore, a transformation relationship between the reference image and the target image is obtained, and the reference image and target image are stitched together based on this transformation relationship. This method enables targeted selection of the reference and target images for stitching, ensuring the validity of the image content while eliminating a large amount of interfering information, avoiding redundant calculations, and improving the efficiency and accuracy of image stitching. Attached Figure Description

[0049] Figure 1 This is an internal structural diagram of a computer device in one embodiment;

[0050] Figure 2 This is a flowchart illustrating an image stitching method in one embodiment;

[0051] Figure 3 This is a flowchart illustrating the process of determining the conversion relationship between a reference image and a target image in one embodiment;

[0052] Figure 4 This is a flowchart illustrating the process of determining the transformation relationship between a reference image and a target image in another embodiment;

[0053] Figure 5 This is a flowchart illustrating the image stitching method in another embodiment;

[0054] Figure 6 This is a flowchart illustrating an image fusion method in one embodiment;

[0055] Figure 7 This is a schematic diagram of the ultrasound wide-view fusion imaging process in one embodiment;

[0056] Figure 8 This is a schematic diagram of the process for obtaining matching candidate medical images in one embodiment;

[0057] Figure 9 This is a schematic diagram of the matching verification process in one embodiment;

[0058] Figure 10 This is a structural block diagram of an image stitching device in one embodiment;

[0059] Figure 11 This is a structural block diagram of an image fusion device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] The image stitching method provided in this application embodiment can be applied to, for example... Figure 1 The computer device shown can be a terminal. It includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image stitching or image fusion method. The display screen can be an LCD screen or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0062] Those skilled in the art will understand that Figure 1The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0063] In one embodiment, such as Figure 2 As shown, an image stitching method is provided, which can be applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0064] S210. Obtain a reference image of the first medical image, wherein the reference image represents the region in the first medical image whose information distribution satisfies a preset distribution condition.

[0065] The first medical image is the medical image to be stitched together. Optionally, the first medical image can be any of the common medical images such as CR (Computed Radiography), CT (Computed Tomography), MR (Magnetic Resonance), and ultrasound. In this embodiment, the type of the first medical image is not specifically limited.

[0066] Optionally, the computer device can communicate with the imaging device that generates the first medical image, receive the first medical image sent by the imaging device, and then extract the image from the first medical image to obtain the region in the first medical image whose information distribution meets the preset distribution conditions, and use it as a reference image of the first medical image.

[0067] The region whose information distribution meets the preset distribution conditions is the region in the first medical image with a large and dense distribution of information of interest. Optionally, the computer device can perform region of interest identification on the first medical image to extract the region of interest as the aforementioned reference image. Generally, the central region of the image is the region with a large and dense distribution of information of interest, and the computer device can directly extract the central region of the first medical image as the aforementioned reference image.

[0068] S220. Obtain the target image in the second medical image based on the reference image. The target image represents the region in the second medical image that meets the similarity requirement with the reference image.

[0069] The second medical image is also a medical image to be stitched together. Optionally, the second medical image can be any of the common medical images such as CR images, CT images, MR images, and ultrasound images. In this embodiment, the type of the second medical image is not specifically limited. It should be noted that in this embodiment, the first medical image and the second medical image are the same type of medical image obtained by scanning the same test object.

[0070] Optionally, the first and second medical images can be independent medical images, with no sequential relationship between them; alternatively, they can be medical images with a sequential relationship. Optionally, the acquisition probe on the imaging device moves while acquiring tissue images, and the acquired medical images are sequentially sent to the computer device, with the second medical image being the next frame after the first medical image. During image acquisition, due to the movement of the acquisition probe on the imaging device and / or varying pressure applied by the operator to the acquisition probe, the image content in the first and second medical images may not be entirely identical.

[0071] Optionally, after obtaining the second medical image, the computer device can divide the second medical image into multiple regions of the same size as the reference image, and calculate the similarity between each region in the reference image and the second medical image, so as to select the region that meets the similarity requirement as the target image in the second medical image. For example, the computer device can directly determine the region in the second medical image that obtains the highest similarity as the aforementioned target image.

[0072] Optionally, the computer device can also use the normalization cross-correlation (NCC) algorithm to determine the target image in the second medical image. The NCC algorithm is used to normalize the pixel correlation between the images to be registered. This method establishes an objective function to measure the correlation by constructing a 3*3 neighborhood registration window at the pixel position p to be matched and constructing a neighborhood matching window at the target pixel position p′ in the same way.

[0073] When applying the above-mentioned NCC algorithm to determine the target image in the second medical image, the pixel to be matched is the pixel in the reference image in the first medical image, and the target pixel is the pixel in the second medical image.

[0074] The NCC algorithm satisfies the following formula (1):

[0075]

[0076] Among them, E(S) i,jS represents the average gray value of the candidate target image at (i, j) in the second medical image, and E(T) represents the average gray value of the reference image in the first medical image. The candidate target image is the image determined in the second medical image based on the neighborhood registration window constructed at pixel (i, j) in the second medical image. i,j (s,t) represents the pixel value variance of the candidate target image, M and N represent the width and height of the candidate target image, respectively, T(s,t) represents the pixel value at pixel (s,t) in the reference image in the first medical image, and R(i,j) represents the similarity between the candidate target image and the reference image. The similarity range is [-1,1]. The closer the similarity is to 1, the higher the similarity between the two.

[0077] S230. Obtain the transformation relationship between the reference image and the target image.

[0078] The transformation relationship between the reference image and the target image reflects the transformation operations performed on them. Optionally, the transformation operation includes at least one of translation, rotation, and scaling. Optionally, the transformation relationship between the reference image and the target image can be represented by a transformation matrix that includes specific parameters for translation, rotation, and scaling.

[0079] Optionally, the computer device performs feature recognition on the reference image and the target image respectively to obtain the feature points in each of the reference image and the target image, and performs feature point matching to determine the successfully matched feature point pairs. Then, it analyzes and compares the successfully matched feature point pairs to obtain the transformation relationship between the reference image and the target image. The feature points in the reference image and the target image can be points where the image grayscale value changes drastically, or points with large curvature at the contour edge, and can be obtained using appropriate feature point extraction algorithms.

[0080] S240. Based on the transformation relationship, stitch the reference image and the target image together.

[0081] Optionally, after obtaining the above transformation relationship, the computer device performs corresponding transformation operations on the reference image / target image according to the transformation relationship and then merges and stitches them together. For example, the transformation relationship includes: the target image is translated by a distance +L (the symbol +, - indicates direction) and rotated by an angle +θ relative to the reference image, and its size is reduced by a factor of S. The computer device then performs the transformation operation of translating the target image by a distance -L, rotating it by an angle -θ, and enlarging it by a factor of S relative to the reference image, and then directly performs linear merging and stitching with the reference image to obtain a stitched image of the reference image in the first medical image and the target image in the second medical image.

[0082] In this embodiment, the computer device acquires a region in a first medical image whose information distribution meets a preset distribution condition as a reference image, and acquires a region in a second medical image that meets a similarity requirement with the reference image as a target image. Then, it obtains the transformation relationship between the reference image and the target image, and stitches the reference image and target image according to this transformation relationship. This method enables targeted selection of the reference and target images for stitching, ensuring the validity of the image content while eliminating a large amount of interfering information, avoiding redundant calculations, and improving the efficiency and accuracy of image stitching.

[0083] To improve the accuracy of the conversion relationship between the acquired reference image and the target image, in one embodiment, such as Figure 3 As shown, the above-mentioned S230, obtaining the transformation relationship between the reference image and the target image, includes:

[0084] S310. Construct the first objective function.

[0085] The first objective function represents the registration error between the reference image and the target image after registration processing, and the functional relationship between the reference image and the target image after transformation.

[0086] It should be noted that the more accurate the transformation relationship is, the higher the degree of overlap between the reference image and the target image after transformation, and the smaller the registration error between the reference image and the target image.

[0087] Optionally, the first objective function is specifically as shown in formula (2):

[0088]

[0089] Where f represents the transformation relationship between the reference image and the target image. T represents the registration error of the overlapping area between the reference image and the target image. i,j Represents the reference image, T j,i The target image is represented by i and j, which represent the sequential numbers of the medical images to be stitched together.

[0090] Furthermore, by performing least squares processing on the first objective function above, we obtain the following formula (3):

[0091]

[0092] Among them, E A This represents the registration error after least squares processing. f(T) i,j That is, through the transformation relation f to T i,j The image obtained after conversion processing. τ i,jTo adjust the parameter, 0 < τ i,j <1, when the transformation relation f is determined solely based on the first objective function, τ i,j Take values ​​close to 0; conversely, when the transformation relationship f is determined simultaneously based on both the first and second objective functions, τ i,j Take a value close to 1.

[0093] S320. When the registration error is minimized, solve the first objective function based on the target image and the reference image to obtain at least one candidate transformation relationship between the reference image and the target image.

[0094] Optionally, the computer device can construct a first objective function to characterize the registration error between the reference image and the target image after transformation by the transformation relationship, and minimize the registration error. By solving the first objective function, at least one candidate transformation relationship that satisfies the minimum registration error between the reference image and the target image can be obtained.

[0095] S330. Determine the transformation relationship between the reference image and the target image based on at least one candidate transformation relationship.

[0096] Optionally, if there is only one candidate transformation relationship among the obtained at least one, the computer device directly determines the candidate transformation relationship as the transformation relationship between the reference image and the target image; if there are two or more (i.e., multiple) candidate transformation relationships among the obtained at least one, the computer device can select any set of candidate transformation relationships as the transformation relationship between the reference image and the target image from the multiple candidate transformation relationships, and can also select the transformation relationship between the reference image and the target image from the multiple sets of candidate transformation relationships according to further constraints.

[0097] In this embodiment, the computer device constructs a first objective function representing the registration error after registration processing of the reference image and the target image, as well as the functional relationship between the reference image and the target image after transformation. When the registration error is minimized, the first objective function is solved based on the target image and the reference image to obtain at least one candidate transformation relationship between the reference image and the target image. Then, based on at least one candidate transformation relationship, the transformation relationship between the reference image and the target image is determined. By determining the transformation relationship between the reference image and the target image by minimizing the registration error, the computer device can quickly and accurately obtain the transformation relationship between the reference image and the target image using mathematical methods, thus improving the accuracy of the obtained transformation relationship between the reference image and the target image.

[0098] Registration errors include angular scale errors, and the aforementioned transformation relationships include rotation angle and scaling scale. When at least one candidate transformation relationship includes two or more (i.e., multiple) such relationships, i.e., when multiple candidate transformation relationships exist, the computer device can use constraints on the rotation angle and scaling scale as further constraints to select the transformation relationship between the reference image and the target image from multiple sets of candidate transformation relationships. In an optional embodiment, such as... Figure 4 As shown, S330 above, determining the transformation relationship between the reference image and the target image based on at least one candidate transformation relationship, includes:

[0099] S410. Construct a second objective function relating angular scale error to the transformation relationship.

[0100] Alternatively, the second objective function is specifically defined as formula (4):

[0101] E S =∑ t ω k ||c(θ t )-s t-1 c(θ t-1 )|| 2 Formula (4)

[0102] Among them, E S ω represents the angular scale error. k The parameter represents the local confidence level between the current image t and the previous frame image t-1, where θ represents the rotation angle, and S... t-1 c(θ) represents the scaling factor of the current image t relative to the previous frame image t-1. t ) and c(θ) t-1 ω represents the similarity transformation parameters determined based on the rotation angle. k The following formula (4) is satisfied:

[0103]

[0104] Where, d k This represents the center distance between the current image and the previous frame image. σ is a spacing scalar parameter that can be used to constrain adjacent images.

[0105] It should be noted that in this embodiment, the current image is the target image in the second medical image, and the previous frame image is the reference image in the first medical image.

[0106] S420. When the angular scale error and the registration error are both at their minimum values, solve the second objective function to obtain the rotation angle of the target image, the rotation angle of the reference image, and the scaling factor of the target image relative to the reference image.

[0107] S430. Based on the rotation angle value of the target image, the rotation angle value of the reference image, and the scaling value, filter the transformation relationship from multiple candidate transformation relationships.

[0108] Optionally, the computer device can construct a second objective function representing the angular scale error between the reference image and the target image after the rotation angle and scaling scale in the transformation relationship are transformed, and minimize the angular scale error. Solving the second objective function yields the rotation angle value of the target image, the rotation angle value of the reference image, and the scaling scale value of the target image relative to the reference image. Then, from multiple candidate transformation relationships, candidate transformation relationships that match (e.g., are the same) the obtained rotation angle value of the target image, the rotation angle value of the reference image, and the scaling scale value of the target image relative to the reference image are selected as the transformation relationship between the reference image and the target image.

[0109] In this embodiment, the registration error includes angular scale error, and the transformation relationship includes rotation angle and scaling scale. The computer device constructs a second objective function relating the angular scale error and the transformation relationship. When both the angular scale error and the registration error are minimized, the second objective function is solved to obtain the rotation angle values ​​of the target image, the rotation angle values ​​of the reference image, and the scaling scale value of the target image relative to the reference image. Then, based on the rotation angle values ​​of the target image, the rotation angle values ​​of the reference image, and the scaling scale value, a transformation relationship is selected from multiple candidate transformation relationships. In the above method, the constraints on the rotation angle and scaling scale are used as further constraints to ensure that the rotation angle and scaling scale between images are as consistent as possible. This allows for the selection of a transformation relationship between the reference image and the target image that ensures a better final stitched image effect, avoiding image distortion or artifacts caused by tissue deformation or large rotation angles.

[0110] When the above image stitching method is applied to real-time stitching of continuously acquired medical images, the first medical image and the second medical image are medical images acquired in a continuous time sequence, and the second medical image is the next frame of the first medical image.

[0111] Optionally, if the first medical image is the first frame of the medical image during the image acquisition process, then the reference image is the middle region image of the first medical image. The first frame of the medical image is the earliest acquired medical image among the continuously acquired medical images.

[0112] Optionally, the intermediate region image is an image formed by taking a region of a preset size centered on the center point of the first medical image. For example, taking a region of 30% to 40% of the length and 30% to 40% of the width of the first medical image as the center point, specifically, taking a region of 1 / 3 of the length and 1 / 3 of the width of the first medical image as the reference image.

[0113] Optionally, if the first medical image is not the first frame of the medical image during the image acquisition process, then the reference image is the target image in the first medical image. For example, if the first medical image is the second frame of the medical image during the image acquisition process, the reference image in the second frame of the medical image is the target image determined based on the reference image in the first frame of the medical image.

[0114] It should be noted that the image stitching method provided in this embodiment is not limited to stitching the first medical image and the second medical image, but can be applied to stitching multiple frames of medical images acquired continuously. Optionally, after receiving the first medical image, the computer device receives the second medical image, and then uses the image stitching method described in the foregoing embodiment to stitch the reference image in the first medical image with the target image in the second medical image to obtain a stitched image of the reference image and the target image. This stitched image is a wide-view image with an increased field of view. The computer device continues to receive the third medical image, and uses the target image in the second medical image as a new reference image. It then uses the method described in the foregoing embodiment to determine a new target image in the third medical image, and determines the conversion relationship between the new reference image and the new target image. Based on this conversion relationship, it stitches the previously stitched image (including the target image, i.e., the new reference image) with the new target image to obtain a new stitched image, i.e., a wide-view image with a larger field of view... After receiving the fourth medical image, a similar process is used to obtain a wide-view image with a larger field of view until the computer device stops receiving medical images, at which point the stitching process ends.

[0115] In this embodiment, if the first medical image is the first frame of the medical image during the image acquisition process, then the reference image is the middle region image of the first medical image; if the first medical image is not the first frame of the medical image during the image acquisition process, then the reference image is the target image in the first medical image. This method allows for simultaneous acquisition and stitching of medical images, suitable for online stitching applications, and significantly improves stitching efficiency.

[0116] In addition to the online stitching method described above, which involves stitching images while they are being acquired, the image stitching method provided in this application can also be applied to offline stitching where all medical images are acquired first and then stitched together. In one embodiment, such as... Figure 5 As shown, the above image stitching method also includes:

[0117] S510. Based on the conversion relationship between each frame of medical images and other frames of medical images acquired during the image acquisition process, all medical images are stitched together to obtain a preliminary stitched image.

[0118] Optionally, after the computer device obtains all medical images acquired during the image acquisition process, it can stitch all the medical images together in response to the user's selection. For example, the computer device can perform online stitching of the medical images acquired during the image acquisition process as described in the aforementioned embodiment and present the stitched wide-view image. The user can choose whether to re-stitch based on the presented wide-view image, that is, to perform offline stitching based on all the acquired medical images.

[0119] If the user finds the wide-view image obtained through online stitching unsatisfactory, such as having severe artifacts or excessive interference, they can choose to re-stitch. The computer device then responds to the user's selection by acquiring the transformation relationship between every two medical images acquired during the image acquisition process—that is, the transformation relationship between each frame of a medical image and other frames. Based on this transformation relationship, all medical images are stitched together to obtain a preliminary stitched image. Optionally, the process of determining the transformation relationship is detailed above. Figure 3 and Figure 4 For details on the splicing process according to the conversion relationship in the corresponding embodiments, please refer to the embodiments corresponding to S240 above, which will not be repeated here.

[0120] Optionally, prior to S510 above, the computer device may further screen all obtained medical images to remove abnormal images, determine the transformation relationship between pairs of medical images based on the screened medical images, and stitch the screened medical images together according to the transformation relationship to obtain a preliminary stitched image. Specifically, the correlation between all medical images can be determined based on the correspondence between information such as content, features, structure, relationship, texture, and grayscale, and medical images with weak correlation can be identified as abnormal images.

[0121] S520. Optimize the noise information of the initial stitched image to obtain the optimized stitched image.

[0122] Specifically, computer equipment can employ image stitching optimization algorithms to optimize the noise information of the initially stitched image, resulting in an optimized stitched image. This image stitching optimization algorithm can be Laplacian pyramid fusion, used to eliminate artifacts and process stitching seams in the overlapping parts of the stitched medical images, thereby obtaining an optimized stitched image with a natural and accurate imaging effect.

[0123] In this embodiment, the method further includes stitching all medical images together according to the transformation relationship between each frame of medical images acquired during the image acquisition process and other frames of medical images, to obtain a preliminary stitched image, and optimizing the noise information of the preliminary stitched image to obtain an optimized stitched image. This achieves one-click optimized stitching, effectively solving the problems of stitching artifacts and distortion, improving the accuracy of stitching, and obtaining a natural and accurate optimized stitched image.

[0124] Typically, users not only need image stitching, but also require image fusion of the stitched medical images with other types of medical images to obtain a fused image with rich information content. Based on this, one embodiment provides an image fusion method, such as... Figure 6 As shown, the method includes:

[0125] S610, Obtain candidate medical image sequences.

[0126] The candidate medical image sequence includes multiple candidate medical images. Each candidate medical image is a specific type of medical image, such as an ultrasound, CR, or MR image.

[0127] Optionally, the computer device may communicate with the corresponding imaging device to obtain a sequence of candidate medical images, including multiple candidate medical images, from the corresponding imaging device.

[0128] S620. Obtain the medical images that match each target medical image in the candidate medical image sequence, and obtain multiple matching candidate medical images.

[0129] In this context, the target medical image and candidate medical images are different types of medical images of the same test subject. For example, current radiofrequency ablation (RFA) techniques are primarily ultrasound-guided, with CT images used as a reference during the process. When applied to RFA, the target medical image is an ultrasound image, while the candidate medical images are CT images. Matching candidate medical images are those among the candidate medical images whose similarity to the target medical image meets the similarity requirements. Matching candidate medical images and target medical images represent different types of medical images for the same cross-section of the test subject.

[0130] Optionally, the computer device calculates the similarity between the target medical image and each candidate medical image in the candidate medical image sequence, compares the magnitudes of the obtained similarities, and selects the candidate medical image with the highest similarity as the matching candidate medical image. Multiple matching candidate medical images can be obtained for different target medical images. The process of determining the similarity can be achieved using the NCC algorithm described in the aforementioned embodiments, or it can be achieved using other template matching, optical flow methods, or feature point matching methods, which will not be elaborated further here.

[0131] S630. Multiple matching candidate medical images are stitched together to obtain a candidate stitched image.

[0132] S640. The target stitched image and the candidate stitched image are fused to obtain a fused stitched image.

[0133] The target stitched image is obtained by stitching together the target medical images using the stitching method described in any of the foregoing embodiments.

[0134] Optionally, the candidate stitched image can also be obtained by stitching together the matching candidate medical images using the stitching method described in any of the foregoing embodiments. That is, the candidate stitched image and the fused stitched image are wide-view images obtained using the same stitching method. Optionally, the computer device can also use a different stitching method than that described in any of the foregoing embodiments to stitch together the matching candidate medical images. In this embodiment, the stitching method for obtaining the candidate stitched image is not specifically limited. The stitching method for obtaining the target stitched image is described above. Figures 2-5 The corresponding implementation examples will not be described in detail here.

[0135] Specifically, after obtaining the candidate stitched image and the target stitched image, the computer device performs image weighted fusion on the candidate stitched image and the target stitched image according to the weights corresponding to each image to obtain the fused stitched image. The sum of the weights corresponding to each image is 1, and the specific value can be set by the user according to their needs.

[0136] Optionally, the weights corresponding to the candidate stitched image and the target stitched image can be the same, both being 1 / 2; or the weight corresponding to the candidate stitched image can be greater than the weight corresponding to the target stitched image; or the weight corresponding to the candidate stitched image can be less than the weight corresponding to the target stitched image.

[0137] When applied to RFA, such as Figure 7 The ultrasound wide-view fusion imaging process shown describes the target stitched image obtained by stitching together ultrasound images (i.e., Figure 7 The ultrasound stitched image (in the image) and the candidate stitched image obtained by stitching together CT images (i.e., Figure 7The process involves fusing the input modal images (input modal stitching), that is, loading the three-dimensional data information of the CT image into the ultrasound image, and simultaneously displaying the fused stitched image of the ultrasound image and CT image on the computer screen (i.e.,...). Figure 7 (The resulting image of the wide-view fusion).

[0138] In this embodiment, the computer device acquires a candidate medical image sequence including multiple candidate medical images, and acquires medical images that match each acquired target medical image in the candidate medical image sequence, resulting in multiple matching candidate medical images. These multiple matching candidate medical images are then stitched together to obtain a candidate stitched image. Finally, the target stitched image and the candidate stitched images are fused to obtain a fused stitched image. The target medical images and candidate medical images are of different types, and the target stitched image is obtained by stitching together the target medical images using the stitching method described in any of the preceding embodiments. This method enables the fusion of different types of medical images to obtain a fused stitched image with rich information content, providing a basis for subsequent diagnosis and treatment.

[0139] To improve the accuracy of the identified matching candidate medical images, in one embodiment, such as Figure 8 As shown, in step S620 above, the medical images that match each acquired target medical image in the candidate medical image sequence are obtained, resulting in multiple matching candidate medical images, including:

[0140] S810. For each target medical image, determine preliminary candidate medical images in the candidate medical image sequence that meet the similarity requirements with the target medical image.

[0141] Optionally, the maximum similarity is achieved by satisfying the similarity requirement. The computer calculates the similarity between each target medical image and each candidate medical image in the candidate medical image sequence, compares the magnitudes of the obtained similarities, and selects the candidate medical image with the highest similarity as the initial candidate medical image for the corresponding target medical image. Multiple initial candidate medical images can be obtained for different target medical images.

[0142] Optionally, the process of obtaining the preliminary candidate medical images corresponding to the target medical image described above can be performed synchronously with the real-time acquisition of the target medical image. After receiving the first medical image, the computer device uses the first medical image as the target medical image and uses the above method to determine preliminary candidate medical images that meet the similarity requirements with the first medical image in the candidate medical image sequence. After receiving the second medical image, the second medical image is used as the new target medical image, and the process continues to determine preliminary candidate medical images that meet the similarity requirements with the second medical image in the candidate medical image sequence... thus obtaining the preliminary candidate medical image corresponding to each target medical image.

[0143] S820. Perform matching verification between the target medical image and the preliminary candidate medical images.

[0144] Among them, the matching verification is to verify whether the target medical image with the highest similarity is truly matched with the corresponding preliminary candidate image.

[0145] Optionally, the computer device may calculate multiple times whether the similarity between the target medical image and the preliminary candidate medical images is the maximum similarity among the similarities between the target medical image and the remaining candidate medical images. If yes, the matching verification is successful; otherwise, the matching verification fails.

[0146] S830. If the matching verification is successful, the preliminary candidate medical image is determined as the matching candidate medical image of the target medical image.

[0147] Optionally, after the preliminary candidate medical image and the target medical image have been successfully matched, the computer device determines the preliminary candidate medical image as a matching candidate medical image of the target medical image.

[0148] Optionally, when the process of obtaining the preliminary candidate medical image corresponding to the target medical image can be performed synchronously with the real-time acquisition of the target medical image, if the matching verification is successful, the computer device will retain the corresponding target medical image and the preliminary candidate medical image for subsequent stitching and fusion; otherwise, if the matching verification is successful, the computer device will discard the corresponding target medical image and will not perform subsequent stitching and fusion, while receiving new medical images.

[0149] In this embodiment, for each target medical image, preliminary candidate medical images that meet the similarity requirements with the target medical image are determined from the candidate medical image sequence. The target medical image and the preliminary candidate medical images are then matched and verified. If the matching verification is successful, the preliminary candidate medical image is determined as a matching candidate medical image of the target medical image. In this method, preliminary candidate medical images that meet the similarity requirements are first determined based on similarity, and then further matching verification is performed to verify the determined preliminary candidate medical images. Finally, a successfully verified matching candidate medical image that matches the target medical image is obtained, improving the accuracy of the determined matching candidate medical images and providing an accurate stitching foundation for subsequent image stitching, thus improving the overall accuracy of image stitching.

[0150] In addition to the matching verification method that involves multiple similarity calculations as described above, matching verification can also be performed based on the transformation relationship between the target medical image and the preliminary candidate medical images. In one embodiment, such as... Figure 9 As shown, S820 above, which involves matching and verifying the target medical image and the preliminary candidate medical images, includes:

[0151] S910. Determine the conversion relationship between the target medical image and the preliminary candidate medical images.

[0152] Optionally, the computer device can pre-determine the conversion relationship between the target medical image and preliminary candidate medical images. The process of determining this conversion relationship is detailed above. Figure 3 and Figure 4 The corresponding embodiments can obtain the conversion relationship between the target medical image and the preliminary candidate medical image based on the same method.

[0153] S920. Determine the current registration error based on the conversion relationship between the target medical image and the preliminary candidate medical images.

[0154] Optionally, after obtaining the conversion relationship between the target medical image and the preliminary candidate medical images, the computer device further obtains the current registration error between the converted target medical image and the preliminary candidate medical images after conversion processing based on this conversion relationship. The process for determining this current registration error is detailed above. Figure 3 In the corresponding embodiment, the current registration error between the converted target medical image and the preliminary candidate medical image can be obtained using the same method.

[0155] S930. If the current registration error is less than or equal to the error threshold, then the matching verification is successful.

[0156] Optionally, after obtaining the current registration error between the converted target medical image and the preliminary candidate medical image, the computer device can directly compare the magnitude of the obtained current registration error with an error threshold to determine whether the matching verification between the target medical image and the preliminary candidate medical image is successful. Specifically, if the current registration error is less than or equal to the error threshold, the matching verification is considered successful; otherwise, if the current registration error is greater than the error threshold, the matching verification is considered unsuccessful.

[0157] In this embodiment, the computer device pre-determines the transformation relationship between the target medical image and the preliminary candidate medical image. Then, based on this transformation relationship, it determines the current registration error. If the current registration error is less than or equal to an error threshold, the matching verification is deemed successful. The transformation relationship between the target medical image and the preliminary candidate medical image reflects the transformation operation between them. After performing the corresponding transformation operation based on this relationship, the target medical image and the preliminary candidate medical image can be transformed to the same pose. Based on the registration error between the two in the same pose, it can accurately determine whether the target medical image and the preliminary candidate medical image truly match, effectively preventing mismatches caused by the test subject's breathing movements, thereby improving the accuracy of the matching verification and further enhancing the overall image stitching accuracy.

[0158] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0159] In one embodiment, such as Figure 10 As shown, an image stitching device is provided, including: a reference determination module 1001, a target determination module 1002, a conversion determination module 1003, and a stitching processing module 1004, wherein:

[0160] The reference determination module 1001 is used to acquire a reference image of the first medical image, wherein the reference image represents the region in the first medical image whose information distribution satisfies a preset distribution condition;

[0161] The target determination module 1002 is used to obtain a target image in the second medical image based on the reference image. The target image represents the region in the second medical image that meets the similarity requirement with the reference image.

[0162] The conversion determination module 1003 is used to obtain the conversion relationship between the reference image and the target image;

[0163] The stitching processing module 1004 is used to stitch the reference image and the target image according to the transformation relationship.

[0164] In one embodiment, the conversion determination module 1003 is specifically used for:

[0165] Construct a first objective function; the first objective function characterizes the registration error between the reference image and the target image after registration processing, and the functional relationship between the reference image and the target image after transformation by the transformation relationship; when the registration error takes the minimum value, solve the first objective function based on the target image and the reference image to obtain at least one candidate transformation relationship between the reference image and the target image; determine the transformation relationship between the reference image and the target image based on at least one candidate transformation relationship.

[0166] In one embodiment, the registration error includes angular scale error, and the transformation relationship includes rotation angle and scaling scale; when multiple candidate transformation relationships exist, the transformation determination module 1003 is specifically used for:

[0167] A second objective function is constructed to establish the relationship between angular scale error and transformation relationship. When the angular scale error and registration error are minimized, the second objective function is solved to obtain the rotation angle of the target image, the rotation angle of the reference image, and the scaling factor of the target image relative to the reference image. Based on the rotation angle of the target image, the rotation angle of the reference image, and the scaling factor, transformation relationships are selected from multiple candidate transformation relationships.

[0168] In one embodiment, if the first medical image is the first frame of the medical image during the image acquisition process, then the reference image is the middle region image of the first medical image; if the first medical image is not the first frame of the medical image during the image acquisition process, then the reference image is the target image in the first medical image.

[0169] In one embodiment, the splicing processing module 1004 is further configured to:

[0170] Based on the transformation relationship between each frame of medical images and other frames of medical images acquired during the image acquisition process, all medical images are stitched together to obtain a preliminary stitched image; the noise information of the preliminary stitched image is optimized to obtain an optimized stitched image.

[0171] Each module in the aforementioned image stitching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0172] In one embodiment, such as Figure 11As shown, an image fusion device is provided, including: a candidate acquisition module 1101, a candidate matching module 1102, a candidate stitching module 1103, and a fusion processing module 1104, wherein:

[0173] The candidate acquisition module 1101 is used to acquire a sequence of candidate medical images; the sequence of candidate medical images includes multiple candidate medical images.

[0174] The candidate matching module 1102 is used to obtain the medical images that match each target medical image in the candidate medical image sequence, and obtain multiple matching candidate medical images; wherein, the target medical images and candidate medical images are different types of medical images;

[0175] Candidate stitching module 1103 is used to obtain candidate stitched images between multiple matching candidate medical images;

[0176] The fusion processing module 1104 is used to fuse the target stitched image with the candidate stitched image to obtain a fused stitched image; wherein, the target stitched image is obtained by stitching together each target medical image using any of the above image stitching methods.

[0177] In one embodiment, the candidate matching module 1102 is specifically used for:

[0178] For each target medical image, preliminary candidate medical images that meet the similarity requirements with the target medical image are identified in the candidate medical image sequence; the target medical image and the preliminary candidate medical images are matched and verified; if the matching verification is successful, the preliminary candidate medical image is determined as the matching candidate medical image of the target medical image.

[0179] In one embodiment, the candidate matching module 1102 is specifically used for:

[0180] Determine the conversion relationship between the target medical image and the preliminary candidate medical images; determine the current registration error based on the conversion relationship between the target medical image and the preliminary candidate medical images; if the current registration error is less than or equal to the error threshold, then the matching verification is successful.

[0181] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0182] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0183] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0184] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0185] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image stitching method, characterized in that, The method includes: A reference image is obtained for the first medical image, wherein the reference image represents a region in the first medical image whose information distribution satisfies a preset distribution condition; the region whose information distribution satisfies the preset distribution condition is a region in the first medical image where the amount of information of interest is large and dense. A target image in a second medical image is obtained based on the reference image, wherein the target image represents the region in the second medical image that has the greatest similarity to the reference image; A first objective function is constructed; the first objective function characterizes the registration error between the reference image and the target image after registration processing, and the functional relationship between the reference image and the target image after transformation by the transformation relation; the transformation relation is a transformation matrix including translation, rotation and scaling parameters; When the registration error reaches its minimum value, the first objective function is solved based on the target image and the reference image to obtain at least one candidate transformation relationship between the reference image and the target image. Construct a second objective function relating the angular scale error to the transformation relationship; When the angular scale error and the registration error are at their minimum values, the second objective function is solved to obtain the rotation angle value of the target image, the rotation angle value of the reference image, and the scaling value of the target image relative to the reference image. Based on the rotation angle value of the target image, the rotation angle value of the reference image, and the scaling scale value, a transformation relationship is selected from multiple candidate transformation relationships; The reference image and the target image are stitched together according to the transformation relationship.

2. The method according to claim 1, characterized in that, If the first medical image is the first frame of the medical image during the image acquisition process, then the reference image is the middle region image of the first medical image; If the first medical image is not the first frame of the medical image during the image acquisition process, then the reference image is the target image in the first medical image.

3. The method according to claim 1, characterized in that, The method further includes: Based on the conversion relationship between each frame of medical images and other frames of medical images acquired during the image acquisition process, all medical images are stitched together to obtain a preliminary stitched image. The noise information of the initial stitched image is optimized to obtain an optimized stitched image.

4. An image fusion method, characterized in that, The method includes: Obtain a candidate medical image sequence; the candidate medical image sequence includes multiple candidate medical images; The medical images that match each target medical image in the candidate medical image sequence are obtained to obtain multiple matching candidate medical images; wherein, the target medical images and the candidate medical images are medical images of different types; The multiple matching candidate medical images are stitched together to obtain a candidate stitched image; The target stitched image is fused with the candidate stitched image to obtain a fused stitched image; wherein the target stitched image is obtained by stitching the target medical images using the stitching method described in any one of claims 1-3.

5. The method according to claim 4, characterized in that, The process involves acquiring medical images that match each target medical image in the candidate medical image sequence, resulting in multiple matching candidate medical images, including: For each target medical image, preliminary candidate medical images that meet the similarity requirements with the target medical image are determined from the candidate medical image sequence; The target medical image and the preliminary candidate medical images are matched and verified. If the matching verification is successful, the preliminary candidate medical image is determined as a matching candidate medical image of the target medical image.

6. The method according to claim 5, characterized in that, The matching and verification of the target medical image and the preliminary candidate medical images includes: Determine the conversion relationship between the target medical image and the preliminary candidate medical image; The current registration error is determined based on the conversion relationship between the target medical image and the preliminary candidate medical images; If the current registration error is less than or equal to the error threshold, then the matching verification is considered successful.

7. An image stitching device, characterized in that, The device includes: A reference determination module is used to acquire a reference image of the first medical image, wherein the reference image represents a region in the first medical image whose information distribution satisfies a preset distribution condition; the region whose information distribution satisfies the preset distribution condition is a region in the first medical image where the amount of information of interest is large and dense. The target determination module is used to obtain a target image in the second medical image based on the reference image, wherein the target image represents the region in the second medical image that has the greatest similarity to the reference image; A transformation determination module is used to construct a first objective function. When the registration error is minimized, the first objective function is solved based on the target image and the reference image to obtain at least one candidate transformation relationship between the reference image and the target image. A second objective function is constructed between the angular scale error and the transformation relationship. When the angular scale error and the registration error are both minimized, the second objective function is solved to obtain the rotation angle value of the target image, the rotation angle value of the reference image, and the scaling scale value of the target image relative to the reference image. Based on the rotation angle value of the target image, the rotation angle value of the reference image, and the scaling scale value, transformation relationships are selected from multiple candidate transformation relationships. The first objective function characterizes the registration error after the registration processing of the reference image and the target image, and the functional relationship between the reference image and the target image after transformation by the transformation relationship. The transformation relationship is a transformation matrix including translation, rotation, and scaling parameters. The registration error includes angular scale error. The stitching processing module is used to stitch the reference image and the target image according to the transformation relationship.

8. An image fusion apparatus, characterized in that, The device includes: A candidate acquisition module is used to acquire a sequence of candidate medical images; the sequence of candidate medical images includes multiple candidate medical images. The candidate matching module is used to obtain medical images that match each acquired target medical image in the candidate medical image sequence, thereby obtaining multiple matching candidate medical images; wherein, the target medical images and the candidate medical images are medical images of different types; A candidate stitching module is used to obtain candidate stitching images between the multiple matching candidate medical images; A fusion processing module is used to fuse the target stitched image with the candidate stitched image to obtain a fused stitched image; wherein the target stitched image is obtained by stitching the target medical images together using the image stitching method according to any one of claims 1-3.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images