A method and apparatus for prostate image fusion using a posture sensor

Abstract

The application relates to the technical field of image processing, and discloses a method and device for prostate image fusion by adopting a posture sensor, which comprises the following steps: fixing the posture sensor on an ultrasonic probe; locking the depth position of the ultrasonic probe when the ultrasonic probe detects a prostate position; collecting ultrasonic images by rotating the ultrasonic probe; recording the rotation information of the ultrasonic probe by the posture sensor; obtaining multi-angle ultrasonic images of the prostate and corresponding posture sensor data; rotating the ultrasonic images of each angle according to the coordinate alignment rotation angle of the multi-angle ultrasonic images and the corresponding posture sensor data; obtaining a two-dimensional ultrasonic image sequence and performing interpolation processing on the ultrasonic images; and constructing a three-dimensional ultrasonic image; and fusing the nuclear magnetic image and the real-time two-dimensional ultrasonic image according to the pre-obtained nuclear magnetic image to obtain a fusion image; the application is a new method for realizing multi-modal fusion images by adopting a posture sensor.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a method and device for prostate image fusion using a posture sensor. BACKGROUND

[0002] At present, prostate biopsy is an important standard for diagnosing prostate cancer. In the operation process, the prostate is imaged by a transrectal ultrasound probe. The ultrasound image obtained by imaging cannot reflect the high-risk area of the prostate, resulting in missed detection and false negatives, low detection rate, and affecting the treatment of patients. At the same time, the existing prostate image acquisition equipment adopts a multi-joint positioning method, which is expensive and easily disturbed by the environment. The mechanical connecting rod multi-joint positioning navigation equipment is too complex and time-consuming to arrange and maintain, and is not easy to operate, and cannot adapt to the situation of patient body position moving during operation.

[0003] The prior art has the following problems: during ultrasound image acquisition, the position and angle of the ultrasound probe are difficult to control, and adjusting the position and angle of the probe will cause discomfort to the patient, and the control difficulty is large, resulting in angle deviation and incomplete problems of the acquired ultrasound image, which cannot fully reflect the actual situation of the prostate; during the registration and fusion of the ultrasound image and the nuclear magnetic image, a direct image fusion method is used, without combining the ultrasound image pose data collected by the posture sensor, resulting in a complex image fusion process, ignoring the angle information provided by the posture sensor data, resulting in poor multi-modal image registration and fusion effect, affecting the judgment result; in order to solve at least one of the above problems, the present application proposes a method and device for prostate image fusion using a posture sensor. SUMMARY

[0004] In view of the deficiencies of the prior art, the main purpose of the present application is to provide a method and device for prostate image fusion using a posture sensor, which can effectively solve the problems in the background art. The specific technical scheme of the present application is as follows:

[0005] A method for prostate image fusion using a posture sensor, comprising:

[0006] The posture sensor is fixed on the ultrasound probe. When the ultrasound probe detects the position of the prostate, the depth position of the ultrasound probe is locked, the ultrasound image is collected by rotating the ultrasound probe, the rotation information of the ultrasound probe is recorded by the posture sensor, and the multi-angle ultrasound image of the prostate and the corresponding posture sensor data obtained by the posture sensor are obtained;

[0007] According to the multi-angle ultrasound image and the corresponding posture sensor data, each angle of the ultrasound image is rotated according to the rotation angle of the coordinate alignment, and a two-dimensional ultrasound image sequence is obtained;

[0008] interpolating the two-dimensional ultrasound images in the two-dimensional ultrasound image sequence according to the attitude sensor data in the three-dimensional space according to a preset grid to construct three-dimensional ultrasound images;

[0009] extracting the attitude sensor data and determining an ultrasound plane of current ultrasound probe imaging, extracting a two-dimensional slice image in the nuclear magnetic image according to the pre-acquired nuclear magnetic image and the ultrasound plane, fusing the two-dimensional ultrasound image and the corresponding two-dimensional slice image of the nuclear magnetic image to obtain a fused image.

[0010] Specifically, according to the multi-angle ultrasound image and the corresponding attitude sensor data, the ultrasound image of each angle is rotated according to the coordinate alignment rotation angle to obtain a two-dimensional ultrasound image sequence, including:

[0011] According to the multi-angle ultrasound image and the corresponding attitude sensor data, the rotation angle of each ultrasound image is obtained;

[0012] By analyzing the attitude sensor data corresponding to each ultrasound image, the rotation angle is compensated to construct an angle matrix;

[0013] According to the corresponding angle value in the angle matrix, the ultrasound image is rotated according to the rotation center of the three-dimensional space to obtain a two-dimensional ultrasound image sequence.

[0014] Specifically, the compensation of the rotation angle by analyzing the attitude sensor data corresponding to each ultrasound image to construct an angle matrix includes:

[0015] By a preset first feature extraction model, a first ultrasound image feature is obtained by extracting features from each ultrasound image;

[0016] From the first ultrasound image feature, a prostate edge feature in the ultrasound image is extracted, and a second ultrasound image feature is obtained by rotating according to the attitude sensor data in the three-dimensional space;

[0017] By edge matching the second ultrasound image feature with the prostate in the pre-acquired nuclear magnetic image, the corresponding spatial relationship between the attitude sensor data and the nuclear magnetic image is calculated to construct an angle matrix.

[0018] Specifically, the two-dimensional ultrasound images in the two-dimensional ultrasound image sequence are interpolated according to the attitude sensor data in the three-dimensional space according to a preset grid to construct three-dimensional ultrasound images, including:

[0019] By a preset second feature extraction model, an ultrasound image texture feature is obtained by extracting texture features in each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence;

[0020] According to the texture features of the ultrasound images, an overlapping region in each ultrasound image is identified according to a preset grid, to obtain an overlapping region;

[0021] When the corresponding overlapping region in the ultrasound image is fused by pixels, the non-overlapping region is used as a splicing region;

[0022] The splicing region is spliced by analyzing the image texture matching degree of the texture features of the ultrasound images in the splicing region, to obtain a second ultrasound image set;

[0023] The ultrasound images in the second ultrasound image set are sampled and interpolated in three-dimensional space according to the attitude sensor data, to construct a three-dimensional ultrasound image.

[0024] Specifically, according to the texture features of the ultrasound images, an overlapping region in each ultrasound image is identified according to a preset grid size, to obtain an overlapping region, including:

[0025] According to the preset grid size, the ultrasound image texture features in the corresponding first grid region of the plurality of ultrasound images are compared, to obtain a feature difference value;

[0026] When the feature difference value is less than a preset feature difference threshold, a circle of grid is expanded outward according to the grid size, to obtain a second grid region;

[0027] When the ultrasound image texture feature difference value in the second grid region is less than a preset feature difference threshold, a circle of grid region is expanded outward according to the grid size, until the feature difference value in the grid region is greater than or equal to the preset feature difference threshold, and the expanded grid region is used as an overlapping region.

[0028] Specifically, the splicing region is spliced by analyzing the image texture matching degree of the texture features of the ultrasound images in the splicing region, to obtain a second ultrasound image set, including:

[0029] According to the texture features of the ultrasound images in the splicing region, the texture in the ultrasound images is analyzed, to obtain a texture trend;

[0030] By analyzing the continuity between the texture trends of the edges of the splicing region, the angle and position of the splicing region are adjusted and spliced, to obtain a second ultrasound image set.

[0031] Specifically, the three-dimensional ultrasound image is constructed, further including:

[0032] The spatial characteristics of the rotation of the ultrasound probe are used to determine the rotation center, the real-time ultrasound image corresponding to the reconstructed three-dimensional ultrasound is compared and optimized on the section defined by the attitude sensor, to obtain the motion calculation result of the real-time two-dimensional ultrasound relative to the three-dimensional ultrasound;

[0033] According to the mapping relationship between the three-dimensional ultrasound image and the nuclear magnetic image, and in combination with the motion calculation result, a fusion error of the real-time ultrasound image and the nuclear magnetic image caused by motion is compensated through a compensation formula, and the compensation formula is as follows:

[0034]

[0035] In the formula, is a coordinate matching result of real-time two-dimensional ultrasound and reconstructed three-dimensional ultrasound based on image processing; is a coordinate mapping result of three-dimensional ultrasound and nuclear magnetic image; is a coordinate conversion result of fusion of real-time ultrasound image and nuclear magnetic image;

[0036] The sensor is used to collect the ultrasound probe tracking data, including: converting the motion of the ultrasound probe into gesture control, changing the running parameters of the preset surgical navigation software through the rotation of the ultrasound probe; using one or more posture sensors independent of the ultrasound probe to control the running mode of the surgical navigation software; using an acceleration sensor to detect a tapping event and converting it into a corresponding mouse function.

[0037] Specifically, in response to fusing the corresponding two-dimensional slice images of the two-dimensional ultrasound image and the nuclear magnetic image to obtain a fusion image, including:

[0038] According to the pre-acquired nuclear magnetic image, image processing and feature extraction are performed through a preset third feature extraction model to obtain a nuclear magnetic image feature of the prostate in the nuclear magnetic image;

[0039] The positions of the three-dimensional ultrasound image and the nuclear magnetic image are corresponded to obtain a position correspondence relationship of each two-dimensional ultrasound image and a two-dimensional slice image in the nuclear magnetic image;

[0040] In combination with the nuclear magnetic image feature and the posture sensor data, a mapping relationship between the nuclear magnetic image and the ultrasound image is established;

[0041] In combination with the mapping relationship and the position correspondence relationship, a feature fusion is performed on the corresponding two-dimensional slice image of each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence and the nuclear magnetic image to obtain a fusion feature;

[0042] According to the fusion feature, a fusion image is obtained to fuse the prostate image;

[0043] The combination of the mapping relationship and the position correspondence relationship, the feature fusion of the corresponding two-dimensional slice image of each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence and the nuclear magnetic image to obtain a fusion feature, includes:

[0044] According to the position relationship and position correspondence of the corresponding feature points in the mapping relationship, each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence is mapped into the coordinate system of the corresponding two-dimensional slice image of the magnetic resonance image;

[0045] The mapped two-dimensional ultrasound image feature values and the corresponding two-dimensional slice image feature values are weighted and averaged to obtain a fusion feature.

[0046] Specifically, the mapping relationship between the magnetic resonance image and the ultrasound image is established in combination with the magnetic resonance image features and the attitude sensor data, including:

[0047] The magnetic resonance image features are subjected to key feature point extraction to obtain magnetic resonance image feature points;

[0048] According to the magnetic resonance image feature points, corresponding ultrasound image features of ultrasound image feature points at corresponding positions are extracted;

[0049] The mapping relationship between the magnetic resonance image and the ultrasound image is established by matching the magnetic resonance image features and the ultrasound image features of the corresponding feature points in combination with the attitude sensor data.

[0050] Specifically, in response to obtaining the fusion image, the image obtained by CT or PET is fused with the ultrasound image.

[0051] An apparatus for prostate image fusion using an attitude sensor, comprising:

[0052] A fixing device for connecting an ultrasound probe, the fixing device being provided with an attitude sensor, the fixing device being used to fuse the attitude sensor with the ultrasound probe, and the fixing device comprising:

[0053] An inner support adjusting mechanism, the inner support adjusting mechanism comprising an inner support, a clamping seat, and an adjusting piece, the inner support being connected with the clamping seat and forming a clamping area for mounting the ultrasound probe, the adjusting piece being arranged in the clamping seat and being used to adjust the clamping area of the inner support to lock or unlock the ultrasound probe;

[0054] The attitude sensor is used to record the rotation information of the ultrasound probe, to obtain multi-angle ultrasound images of the prostate and corresponding attitude sensor data, and to rotate each angle of ultrasound image according to the coordinate-aligned rotation angle to obtain a two-dimensional ultrasound image sequence and construct a three-dimensional ultrasound image;

[0055] The attitude sensor data is extracted and the ultrasound plane of the current ultrasound probe imaging is determined, according to the pre-obtained magnetic resonance image, in combination with the ultrasound plane, two-dimensional slice images in the magnetic resonance image are extracted, each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence is fused with the corresponding two-dimensional slice image of the magnetic resonance image to obtain a fusion image.

[0056] Specifically, the fixing device further comprises a movement adjusting mechanism, the movement adjusting mechanism comprises a linear structure and a rotating structure, and the rotating structure is rotationally connected with the linear structure.

[0057] The linear structure comprises an outer support and a locking seat, the locking seat is slidingly arranged on the outer support, a clamping groove is formed on the outer support, the locking seat is matched with the clamping groove and is used for adjusting the position of the locking seat along the outer support, the locking seat is further connected with an inner support, the rotating structure is rotationally connected with one end of the outer support, and an opening part is formed on the rotating structure for the ultrasonic probe to extend out.

[0058] Compared with the prior art, the application has the following beneficial effects:

[0059] The application searches for the prostate position through the ultrasonic probe, locks the deep position of the ultrasonic probe, collects the ultrasonic images of the prostate at different angles through the rotation of the ultrasonic probe, records the rotation angle in real time through the attitude sensor, corrects the angle of the ultrasonic image in combination with the nuclear magnetic image and the attitude sensor data, fuses the ultrasonic images at multiple angles, fuses the multi-modal data of the nuclear magnetic image and the ultrasonic image, and obtains a fusion image; through the locking of the deep position and the rotation of the ultrasonic probe, the influence on the patient in the image collection process can be reduced, the complete multi-angle ultrasonic image is collected, the multi-angle ultrasonic image fusion and the multi-modal image fusion are respectively performed in combination with the image attitude data, and the comprehensive prostate image data at multiple angles can be obtained after the fusion, which is helpful for doctors to more accurately identify the prostate lesions and improve the accuracy and reliability of the prostate disease diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 It is a structure schematic view of the device for prostate image fusion by using the attitude sensor in embodiment 1 of the application;

[0061] Figure 2 It is a structure schematic view of the linear structure and the inner support in embodiment 1 of the application;

[0062] Figure 3 It is an explosion view of the locking seat and the inner support in embodiment 1 of the application;

[0063] Figure 4 It is a structure schematic view of the inner support in embodiment 1 of the application;

[0064] Figure 5 It is a work flow chart of a method for prostate image fusion by using the attitude sensor in embodiment 2 of the application;

[0065] Figure 6 It is a schematic view of the fusion of the overlapping regions of the ultrasonic images in embodiment 2 of the application;

[0066] Figure 7 Fig. 2 is a schematic diagram of ultrasound image overlap region identification in the embodiment 2 of the present application;

[0067] Figure 8 Fig. 3 is a schematic diagram of ultrasound image splicing region position adjustment in the embodiment 2 of the present application.

[0068] Reference signs:

[0069] 1, mobile adjusting mechanism; 2, inner support adjusting mechanism; 3, linear structure; 4, rotating structure; 41, opening part; 42, arc-shaped protrusion; 5, outer support; 51, clamping groove; 52, claw head; 6, locking clamping seat; 61, clamping seat base body; 62, locking part; 63, locking groove; 64, clamping claw; 65, clamping hook; 7, inner support; 71, mounting seat; 72, hoop chain; 73, threaded seat; 74, groove; 8, hoop seat; 9, adjusting part. DETAILED DESCRIPTION

[0070] In order to make the above objectives, features and advantages of the present application more apparent, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0071] In the following description, a lot of specific details are set forth in order to facilitate a full understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the spirit of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0072] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments.

[0073] Embodiment 1:

[0074] At present, prostate biopsy is an important standard for diagnosing prostate cancer. During the biopsy operation, the doctor performs ultrasound imaging of the prostate through a transrectal ultrasound probe and completes the prostate biopsy operation under the guidance of the real-time image, which plays the real-time advantage of ultrasound imaging. Single ultrasound imaging is limited by the resolution of ultrasound imaging and the ability to identify prostate cancer areas, and cannot reflect the high-risk areas of the prostate, resulting in missed detection and false negatives, low detection rate, and affecting patient treatment. Currently, real-time ultrasound images and non-real-time but high-resolution MRI images of each region of the prostate and prostate cancer are fused to obtain accurate prostate images by fusing multi-modal data information. However, directly fusing ultrasound images and MRI images in the image fusion process ignores the rotation information of ultrasound images at different angles, which can lead to a complex fusion process and poor fusion results. The embodiment provides a method for prostate image fusion using a posture sensor, which combines the position information of the ultrasound probe in the posture sensor to fuse multi-angle prostate ultrasound images and multi-modal prostate image data.

[0075] Among them, the existing technology of transrectal prostate detection is a six-degree-of-freedom detection method, which constantly adjusts the angles and positions of multiple degrees of freedom, which not only causes the detection process to be complicated, but also makes the obtained image data complex and not conducive to fusion image to obtain the lesion position of the lesion.

[0076] As Figures 1 to 4As shown, the present application provides a device for prostate image fusion using a posture sensor, which is used to realize a method for prostate image fusion using a posture sensor, which comprises a fixing device for connecting an ultrasonic probe, a posture sensor is arranged on the fixing device to obtain posture sensor data collected by the posture sensor by rotating the ultrasonic probe, the fixing device is used to fuse the posture sensor and the ultrasonic probe, and the fixing device comprises a moving adjusting mechanism 1 and an inner support adjusting mechanism 2. The moving adjusting mechanism 1 comprises a linear structure 3 and a rotating structure 4, and the rotating structure 4 is rotationally connected with the linear structure 3. The linear structure 3 comprises an outer support 5 and a locking seat 6, the locking seat 6 is slidably arranged on the outer support 5, the locking seat 6 comprises a seat base 61 and a locking piece 62, the outer support 5 is formed with a clamping groove 51, the seat base 61 is slidably connected with the outer support 5, the seat base 61 is formed with a locking groove 63, the locking groove 63 is formed with a clamping jaw 64 for clamping or separating with the clamping groove 51, the locking piece 62 is slidably arranged in the locking groove 63 and is slidably connected with the seat base 61, and the locking piece 62 moves along the locking groove 63 to keep sliding on the seat base 61, so that the locking piece 62 is not easy to slide out of the seat base 61. The locking piece 62 comprises a locking direction and an unlocking direction, when the locking piece 62 is controlled to move along the locking direction, the locking piece 62 moves along the locking groove 63 and presses the clamping jaw 64 to be clamped with the locking groove 63, so as to relatively fix and lock the seat base 61 with the outer support 5, when the locking piece 62 is controlled to move along the unlocking direction, the locking piece 62 moves along the locking groove 63 and separates from the clamping jaw 64, and the clamping jaw 64 separates from the clamping groove 51, so as to unlock the locking seat 6 from the outer support 5, unlock the seat base 61 from the outer support 5, and adjust the position of the ultrasonic probe along the length direction of the outer support 5.

[0077] As shown in Figure 2 and Figure 3 , the rotating structure 4 is rotationally connected with one end of the outer support 5, the rotating structure 4 is formed with an opening part 41 for the ultrasonic probe to extend out, the end of the outer support 5 is formed with a claw head 52, the rotating structure 4 is formed with an arc-shaped protrusion 42, the claw head 52 is tightly connected with the arc-shaped protrusion 42, so that the outer support 5 can rotate along the direction of the arc-shaped protrusion 42, thereby adjusting the angular position of the posture sensor.

[0078] As shown in Figure 2 , Figure 3 and Figure 4As shown, the inner support adjustment mechanism 2 includes an inner support 7, a clamp seat 8, and an adjustment member 9. The inner support 7 connects the locking seat 6 and the clamp seat 8, forming a clamping area for installing the ultrasonic probe. The inner support 7 includes a mounting base 71 and a clamp chain 72. A hook 65 is formed on the side of the seat base 61 away from the locking member 62. A groove 74 is formed on the mounting base 71 for engaging with the hook 65. The mounting base 71 is connected to the seat base 61 by engaging the hook 65 with the groove 74 on the mounting base 71. The clamp chain 72 is set on both sides of the mounting base 71 and is connected to a threaded seat 73 respectively. The threaded seat 73 is set in the clamp seat 8 and is threadedly connected to the adjustment member 9. An installation cavity is formed in the clamp seat 8 for installing the threaded seat 73 and the adjustment member 9. The position of the threaded seat 73 is adjusted by rotating the adjustment member 9, thereby adjusting the clamp chain 72 to lock or unlock the ultrasonic probe. The attitude sensor is set inside the clamp 8. When the ultrasonic probe is locked, the angle of the attitude sensor is adjusted by rotating the ultrasonic probe to obtain images detected at different angles.

[0079] During prostate examination, the ultrasound probe is first inserted into the inner support 7, with one end protruding from the opening 41 of the rotating structure 4. The rotation of the adjusting component 9 moves the threaded seat 73 along the clamp seat 8, thereby moving the clamp chain 72 and locking the ultrasound probe, thus installing the attitude sensor and the ultrasound probe. When the ultrasound probe is inserted into the body via the rectum, the side of the rotating structure 4 opposite to the outer support 5 is aligned with the patient's skin. The locking seat 6 is pushed along the outer support 5, causing the ultrasound probe to extend to the location of the prostate lesion. The locking component 62 then locks the seat base 61 to the outer support 5. At this point, the ultrasound probe is positioned at the appropriate location for prostate lesion detection. Rotating the ultrasound probe causes the outer support 5 to rotate along the arc-shaped protrusion 42 of the rotating structure 4. During rotation, the attitude sensor records the rotation information of the ultrasound probe, acquiring multi-angle ultrasound images of the prostate and corresponding attitude sensor data.

[0080] like Figure 3 As shown, it also includes a data acquisition module, which fixes the attitude sensor on the ultrasound probe. When the ultrasound probe detects the prostate position, it locks the depth position of the ultrasound probe and acquires ultrasound images by rotating the ultrasound probe. The attitude sensor records the rotation information of the ultrasound probe and obtains multi-angle ultrasound images of the prostate and corresponding attitude sensor data.

[0081] The coordinate alignment module rotates the ultrasound image at each angle according to the coordinate alignment rotation angle based on the multi-angle ultrasound images and the corresponding attitude sensor data to obtain a two-dimensional ultrasound image sequence.

[0082] The ultrasound image fusion module interpolates the two-dimensional ultrasound images in the two-dimensional ultrasound image sequence according to the preset grid in the three-dimensional space according to the attitude sensor data, and constructs a three-dimensional ultrasound image;

[0083] The multi-modal data fusion module extracts the attitude sensor data and determines an ultrasound plane of the current ultrasound probe imaging, extracts a two-dimensional slice image in the nuclear magnetic image according to the pre-acquired nuclear magnetic image in combination with the ultrasound plane, fuses each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence with the corresponding two-dimensional slice image of the nuclear magnetic image, and obtains a fusion image, so as to fuse the prostate image.

[0084] In this embodiment, the acquisition module acquires the prostate image, the attitude sensor is fixed on the ultrasound probe, after the depth position of the prostate detected by the ultrasound probe is locked, the probe is rotated to acquire multi-angle ultrasound images, and the attitude sensor records the corresponding rotation information, so that the ultrasound images containing the prostate at different angles and the attitude sensor data are obtained, thereby providing a data basis for image processing and fusion; specifically, the coordinate alignment module calculates the rotation angle of the image, compensates the angle according to the image features, constructs an angle matrix, performs a rotation operation with the image center as the rotation center, corrects the angle deviation of the image caused by acquisition, aligns the ultrasound images in the spatial coordinates, and obtains a two-dimensional ultrasound image sequence after coordinate alignment, thereby providing a basis for image stitching and multi-modal fusion.

[0085] Specifically, the ultrasound image fusion module identifies the overlapping area between the images by using the image texture features and the grid comparison for the two-dimensional ultrasound image sequence after coordinate alignment, fuses the pixels in the overlapping area, adjusts the angle and position of the stitching area according to the continuity of the edge texture trend of the stitching area, stitches the ultrasound images into a complete image, obtains a second ultrasound image set, interpolates the ultrasound images in the second ultrasound image set, and constructs a three-dimensional ultrasound image in the three-dimensional space, thereby improving the integrity and quality of the fused ultrasound image; the multi-modal data fusion module combines the pre-acquired nuclear magnetic image and the attitude sensor data, determines the position correspondence relationship between the two-dimensional ultrasound image and the two-dimensional slice image in the nuclear magnetic image according to the position relationship between the three-dimensional ultrasound image and the nuclear magnetic image in the three-dimensional space, extracts the nuclear magnetic image features and establishes a mapping relationship with the ultrasound image, maps each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence to the coordinate system of the two-dimensional slice image in the corresponding nuclear magnetic image, performs weighted average fusion on the feature values at the corresponding positions, generates a fusion image according to the fused features, integrates the advantages of the nuclear magnetic image and the ultrasound image, and provides a more accurate image for prostate disease diagnosis.

[0086] The hoop seat 8 is used to fix the posture sensor on the ultrasonic probe, generally inside the ultrasonic probe disinfection cover, and can also be outside the disinfection cover. After the inner support 7 is fixed, the rotation relationship of the posture sensor relative to the scan plane coordinate system of the ultrasonic probe is also fixed. If the relationship of the ultrasonic image relative to the rotation center can also be determined, the posture sensor can track the movement of the scan plane of the ultrasonic probe. And by rotating the transrectal ultrasonic probe, a three-dimensional image of the prostate is reconstructed.

[0087] Embodiment 2:

[0088] As Figure 5 The application provides a method for detecting based on an apparatus for fusing prostate images by using a posture sensor, comprising:

[0089] S101, fix the posture sensor on the ultrasonic probe, when the ultrasonic probe detects the position of the prostate, lock the depth position of the ultrasonic probe, collect ultrasonic images by rotating the ultrasonic probe, the posture sensor records the rotation information of the ultrasonic probe, and obtain multi-angle ultrasonic images of the prostate and corresponding posture sensor data obtained by the posture sensor;

[0090] S102, according to the multi-angle ultrasonic images and the corresponding posture sensor data, rotate each angle of the ultrasonic image according to the rotation angle of the coordinate alignment, and obtain a two-dimensional ultrasonic image sequence;

[0091] S103, perform interpolation processing on the two-dimensional ultrasonic images in the two-dimensional ultrasonic image sequence, and construct a three-dimensional ultrasonic image;

[0092] S104, extract the posture sensor data and determine the ultrasonic plane of the current ultrasonic probe imaging, according to the pre-acquired magnetic resonance image, combine the ultrasonic plane to extract a two-dimensional slice image in the magnetic resonance image, fuse each two-dimensional ultrasonic image in the two-dimensional ultrasonic image sequence with the corresponding two-dimensional slice image of the magnetic resonance image, and obtain a fused image.

[0093] In the diagnosis process of prostate diseases, accurate image information is crucial for disease detection and treatment plan. In this embodiment, the depth position and rotation angle of the ultrasonic probe are controlled respectively, multi-angle ultrasonic images are collected, the information obtained by the posture sensor is used, and multi-angle ultrasonic images and magnetic resonance images are accurately fused through coordinate alignment and coordinate mapping. Compared with the single image fusion method in the prior art, this scheme respectively controls the depth and angle of the ultrasonic probe, can realize the rapid acquisition of ultrasonic images, reduces the discomfort caused to patients by the traditional multi-joint ultrasonic imaging device, and through multi-angle ultrasonic image acquisition, accurate coordinate alignment, high-quality ultrasonic image splicing and multi-modal image fusion, the fused image can more clearly and comprehensively show the morphological structure and lesion characteristics of the prostate, and improve the accuracy and reliability of the diagnosis result.

[0094] In the embodiment, the posture sensor is fixed on the ultrasonic probe, the operator inserts the ultrasonic probe device into the rectum, operates the ultrasonic probe, and observes the environment in the rectum in real time by using ultrasonic imaging. When the image display reaches the prostate position, the depth position of the ultrasonic probe inserted into the rectum is locked, the ultrasonic probe is slowly rotated around the posture sensor at the depth position, the ultrasonic probe continuously collects images during the rotation, the posture sensor synchronously records the posture sensor information of each rotation, the posture sensor information includes the angle data of each rotation for collecting images, and the multi-angle ultrasonic images of the prostate and the corresponding posture sensor data are obtained until the ultrasonic image collection process of each angle of the prostate is completed.

[0095] Preferably, by locking the depth position of the ultrasonic probe first, keeping the front and back positions of the ultrasonic probe unchanged, and rotating the ultrasonic probe around the posture sensor for multi-angle ultrasonic image collection, the discomfort caused to the patient during the rotation can be reduced, the complete morphological structure of the prostate is obtained through the multi-angle ultrasonic images, and information loss caused by single-angle collection is avoided. The posture sensor records the posture sensor data in real time, which provides position information reference for coordinate alignment and fusion of different-angle ultrasonic images, and improves the accuracy and reliability of image fusion.

[0096] Specifically, for the collected multi-angle ultrasonic images, the ultrasonic images of different angles are aligned in coordinates, the rotation angle of each ultrasonic image relative to the initial position is determined according to the posture sensor data recorded by the posture sensor, the ultrasonic images are rotated, the image features and edge contours of the rotated ultrasonic images are analyzed, the angle of the inaccurate area in the splicing process is analyzed and compensated, the accurate angle of each image should be rotated is obtained combining the rotation angle of each ultrasonic image and the angle compensation value, the center of each ultrasonic image is taken as the rotation center, the ultrasonic images are rotated according to the corresponding rotation angle, and a two-dimensional ultrasonic image sequence after coordinate alignment is obtained. By aligning the coordinates by combining the posture sensor data recorded by the posture sensor and the ultrasonic image features, the image angle deviation in the collection process can be effectively corrected, the quality and accuracy of image fusion are improved, and more reliable diagnosis basis is provided for doctors.

[0097] Specifically, in the coordinate-aligned two-dimensional ultrasound image sequence, the overlapping regions of the ultrasound images are identified, and according to the texture features of each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence, the regions with small texture feature value differences are identified as overlapping regions. By fusing the pixel information of the overlapping regions and splicing the pixel information of the non-overlapping regions, the ultrasound images of different angles are fused and spliced, and the multi-angle ultrasound image fused image is obtained by combining the ultrasound image information of different angles. The overlapping region identification and splicing based on image texture features can make full use of the texture information of the image, improve the accuracy of image fusion and splicing, and the spliced image can show the overall information of the prostate, providing high-quality ultrasound image data for subsequent fusion with magnetic resonance images. The image set after splicing is subjected to interpolation processing, so that the two-dimensional image is mapped to a three-dimensional space, three-dimensional image reconstruction is realized, and a three-dimensional ultrasound image is constructed. The three-dimensional ultrasound image can provide accurate position information for the fusion process of the ultrasound image and the magnetic resonance image, and improve the accuracy of the fusion process.

[0098] The three-dimensional ultrasound image and the pre-acquired magnetic resonance image are compared in position in a three-dimensional space to obtain the position correspondence between each two-dimensional ultrasound image and the slice image of the magnetic resonance image. After fusing the multi-angle ultrasound images, the multi-modal prostate image data is fused in combination with the pre-acquired magnetic resonance image, the mapping relationship between the two modal images is established in combination with the attitude sensor data recorded by the attitude sensor, the real-time two-dimensional ultrasound image features are fused with the slice image features at the corresponding position in the magnetic resonance image according to the mapping relationship, and a fused image combining the advantages of the two modal images is obtained. Through multi-modal image fusion, the advantages of magnetic resonance images and ultrasound images can be combined, wherein the magnetic resonance images can clearly show the tissue structure and lesion details of the prostate, and the ultrasound images have the advantages of real-time and convenience. The fused image combines the advantages of the two modal data, provides richer and more accurate prostate image information for doctors, and helps to improve the diagnosis accuracy of prostate diseases and the rationality and effectiveness of treatment plans.

[0099] The application searches for the prostate position through an ultrasonic probe, locks the depth position of the ultrasonic probe, collects ultrasonic images of the prostate at different angles through rotating the ultrasonic probe, and records the rotation angle in real time by using a posture sensor, corrects the angle of the ultrasonic images in combination with the nuclear magnetic image and the posture sensor data, fuses the ultrasonic images at multiple angles, fuses the multi-modal data of the nuclear magnetic image and the ultrasonic image, and obtains a fused image; by locking the depth position first and then rotating the ultrasonic probe, the influence on the patient during image collection can be reduced, complete multi-angle ultrasonic images are collected, multi-angle ultrasonic image fusion and multi-modal image fusion are performed in combination with the image posture data, and comprehensive prostate image data at multiple angles can be obtained after fusion, which helps doctors to more accurately identify prostate lesions and improves the accuracy and reliability of prostate disease diagnosis.

[0100] Further, according to the multi-angle ultrasonic images and the corresponding posture sensor data, each ultrasonic image at an angle is rotated according to the rotation angle of coordinate alignment, to obtain a two-dimensional ultrasonic image sequence, including:

[0101] S201, obtaining a rotation angle of each ultrasonic image according to multi-angle ultrasonic images and corresponding posture sensor data;

[0102] S202, compensating the rotation angle by analyzing the image features of each ultrasonic image, and constructing an angle matrix;

[0103] S203, rotating each angle ultrasonic image center as a rotation center in combination with the corresponding angle value in the angle matrix, to obtain a two-dimensional ultrasonic image sequence.

[0104] This embodiment combines multi-angle ultrasonic images and rotation angle data in the corresponding posture sensor data of each ultrasonic image, calculates the rotation angle, compensates the rotation angle by analyzing the image features, constructs an angle matrix, rotates the corresponding ultrasonic image based on the angle matrix, and realizes coordinate alignment of the multi-angle ultrasonic images; the image is rotated in combination with the posture sensor data recorded by the posture sensor in real time, which can quickly adjust the angle of the multi-angle ultrasonic image data, and the angle compensation based on the image features can improve the accuracy of coordinate alignment.

[0105] In this embodiment, the rotation angle of each ultrasound image is calculated according to the multi-angle ultrasound image and the attitude sensor data recorded by the attitude sensor in real time when the ultrasound probe rotates to collect the image; the rotation data of each ultrasound image is compared with the angle when the first ultrasound image is collected, and the rotation angle value is obtained. For example, the rotation angle recorded by the attitude sensor when the first ultrasound image is collected is 0°, and the rotation angle recorded when the second image is collected is 10° to the right. Therefore, it is clear that the rotation angle of the first image is 0°, and the rotation angle of the second image is 10° to the right. The rotation angle of each ultrasound image can be quickly calculated according to the attitude sensor data collected by the attitude sensor in real time, thereby providing a data basis for angle compensation.

[0106] Specifically, the image features of the ultrasound image are analyzed, including the edge profile of the prostate, the internal texture structure and other feature information, and the misalignment after rotation by the rotation angle is identified. When the texture features of the rotated image do not correspond, the rotation angle is compensated, so that the texture feature difference of the multi-angle ultrasound image after rotation is within the error range, the corresponding rotation angle compensation value is calculated, the rotation angle is compensated, and the angle matrix is constructed according to the compensated rotation angle. By compensating the rotation angle and constructing the angle matrix, the rotation angle of the image is more accurate, the accuracy of image coordinate alignment is improved, the image angle deviation caused by collection error can be effectively corrected, the accuracy of multi-angle ultrasound image splicing and the quality of the fused image can be improved, and the doctor can more accurately observe the morphological structure of the prostate.

[0107] Specifically, after the angle matrix is constructed, the center of each ultrasound image is taken as the rotation center, and the ultrasound image is rotated according to the corresponding rotation angle value in the matrix, so that the ultrasound images of different angles are adjusted to the same coordinate direction in the coordinate alignment, and a two-dimensional ultrasound image sequence after coordinate alignment is obtained. By aligning the coordinates of the multi-angle ultrasound image, the coordinate direction is unified, so that the ultrasound images of different angles are in the same coordinate in space, the consistency and splicing property of the images of different angles are improved, the accuracy of the image splicing process is improved, the overlapping area between the images can be accurately matched, the splicing error is reduced, and the quality of the fused image is improved, thereby providing the doctor with clearer and more accurate diagnostic images.

[0108] Further, by analyzing the image features of each ultrasound image, the rotation angle is compensated, and the angle matrix is constructed, including:

[0109] S301, feature extraction is performed on each ultrasound image by using a pre-set first feature extraction model to obtain first ultrasound image features;

[0110] S302, the prostate edge features in the ultrasound image are extracted from the first ultrasound image features to obtain second ultrasound image features;

[0111] S303, calculate the corresponding angle compensation value by edge matching of the second ultrasound image feature;

[0112] S304, compensate the corresponding rotation angle according to the angle compensation value of each ultrasound image, and construct an angle matrix.

[0113] In this embodiment, the image information in the multi-angle ultrasound image is extracted for feature extraction, the prostate edge is analyzed based on the image features, the matching degree of the prostate edge features corresponding to the ultrasound image data at different angles is analyzed, the corresponding angle compensation value is calculated, the corresponding rotation angle is compensated based on the angle compensation value, and an angle matrix is constructed. By combining the ultrasound image features to match and analyze the prostate edge features, the corresponding rotation angle is compensated, so that the compensated rotation angle can align the rotation coordinates of the ultrasound images at different angles. By combining the feature information of the ultrasound image, the rotation angle recorded by the attitude sensor is compensated, which can effectively eliminate the angle error caused by interference in the image acquisition process, and make the rotation angle of the ultrasound image more accurate.

[0114] In this embodiment, first, a first feature extraction model is used to extract features from each ultrasound image to obtain first ultrasound image features. The first feature extraction model includes a convolutional neural network model based on deep learning, a texture feature extraction model based on a gray level co-occurrence matrix, and the like. In this embodiment, the first feature extraction model uses a ResNet model. The ResNet model is trained using a large amount of historical ultrasound image data to obtain a pre-trained first feature extraction model. Each ultrasound image collected is input into the pre-trained first feature extraction model, and the model extracts various features in the image, including gray scale features, texture features, edge features, and the like. By extracting image features, key feature information in the ultrasound image can be obtained, and image alignment based on feature data can improve the accuracy of image matching.

[0115] Specifically, the prostate edge-related feature information is filtered and extracted from the first ultrasound image features to obtain more targeted second ultrasound image features. The prostate edge-related feature information is filtered by analyzing the regions with obvious gray scale changes, the boundaries with dramatic texture changes, and the like. The prostate edge is recognized considering the dramatic change of the prostate edge features. The filtered feature information is processed by using an edge detection algorithm, specifically including a Canny edge detection algorithm, a Sobel operator, and the like. The edge information of the prostate, such as the edge contour feature points and the edge curve, is extracted to obtain the second ultrasound image features. The analysis based on the prostate edge information can accurately grasp the shape and position information of the prostate in the ultrasound image. The angle change during the image acquisition process is analyzed to provide a data basis for calculating the angle compensation value, thereby improving the accuracy of the image coordinate alignment.

[0116] Specifically, the edge matching is performed on the second ultrasound image features to calculate the corresponding angle compensation value. The differences between the images at different angles caused by the angle deviation are analyzed by comparing the shape, position, and direction of the edge contour and other edge information. The angle compensation value is calculated based on the differences. The second ultrasound image features of different ultrasound images are matched by using an edge matching algorithm. The edge matching algorithm includes a SIFT algorithm based on feature point matching, an ORB algorithm, and the like. The SIFT algorithm is used in this embodiment. The scale-invariant feature descriptor is extracted for the prostate edge feature points of each image. The corresponding relationship of the prostate edge feature points in different images is found by calculating the similarity between the descriptors. The position difference of the edge feature points between adjacent images caused by the angle deviation is calculated based on the corresponding relationship of the edge feature points obtained by matching. The position difference is converted into an angle value to obtain the corresponding angle compensation value. The angle compensation value that enables the corresponding edges to be matched by rotation is calculated based on the position difference. The angle compensation value is calculated by the edge matching, which can perform image edge matching based on the feature information of the image itself to calculate an accurate angle compensation value.

[0117] For each ultrasound image, the corresponding angle compensation value is combined with the rotation angle recorded by the attitude sensor to calculate the compensated rotation angle. For example, a right rotation is defined as positive. If the angle compensation value is positive, it is added to the rotation angle. If the angle compensation value is negative, it is subtracted from the original rotation angle to obtain the compensated rotation angle. The compensated rotation angles are arranged in the order of the ultrasound image acquisition to form a two-dimensional matrix, and an angle matrix is constructed. The rotation angle information is provided for the corresponding ultrasound image rotation by the angle matrix, so that the image can be accurately aligned after rotation, the quality of image stitching and image fusion is improved, and it is helpful for doctors to obtain more accurate prostate lesion information from the fused image.

[0118] Further, the two-dimensional ultrasound images in the two-dimensional ultrasound image sequence are subjected to interpolation processing to construct a three-dimensional ultrasound image, comprising:

[0119] S401, extracting texture features in each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence by a preset second feature extraction model to obtain ultrasound image texture features;

[0120] S402, identifying overlapping regions in each ultrasound image according to the ultrasound image texture features and in accordance with a preset grid size to obtain the overlapping regions;

[0121] S403, when performing pixel fusion on the corresponding overlapping regions in the ultrasound images, the non-overlapping regions are used as stitching regions;

[0122] S404, stitching the stitching regions by analyzing the image texture matching degree of the ultrasound image texture features in the stitching regions to obtain a second ultrasound image set;

[0123] S405, sampling and interpolating the ultrasound images in the second ultrasound image set in the three-dimensional space according to the attitude sensor data to construct a three-dimensional ultrasound image.

[0124] In this embodiment, the texture information is extracted by the preset second feature extraction model based on the texture features of the ultrasound images, the overlapping regions are identified by using the grid comparison method, the pixel fusion of the overlapping regions is performed and the stitching regions are calibrated in combination with the texture matching degree to obtain the second ultrasound image set. The image overlapping regions are fused and stitched based on the image texture features, which can improve the accuracy and reliability of image stitching, and the stitched image can more truly reflect the actual morphology of the prostate. The two-dimensional images are mapped to the three-dimensional space by interpolating the ultrasound images in the second ultrasound image set, and a three-dimensional ultrasound image is constructed, thereby providing a position reference for multi-modal image fusion.

[0125] In this embodiment, the texture features in each two-dimensional ultrasound image are extracted by the preset second feature extraction model to obtain ultrasound image texture features. The texture features in the ultrasound images can reflect the structural information and physical characteristics of the internal tissues, and the textures of different tissues or the same tissue in different states are different. The regions with similar textures in the images are identified as overlapping regions based on the texture features, thereby improving the accuracy of region identification. The second feature extraction model includes a texture feature extraction model based on a gray level co-occurrence matrix, a local binary pattern model, etc. The second feature extraction model in this embodiment uses the texture feature extraction model based on the gray level co-occurrence matrix. A large number of historical ultrasound images are used to train the texture feature extraction model based on the gray level co-occurrence matrix to obtain a pre-trained second feature extraction model.

[0126] The second feature extraction model is pre-trained, and each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence is sequentially input into the pre-trained second feature extraction model. The model extracts features capable of representing image texture by analyzing information such as the distribution, variation law and relationship between adjacent pixels of the pixel gray value in the image, including texture features such as roughness, contrast and directionality of the texture. The model outputs the texture features of the ultrasound image. By extracting the texture features through the model, key information related to the texture in the image can be efficiently and accurately obtained, providing a data basis for identifying the overlapping area and image stitching, and helping to improve the image fusion quality and image stitching quality.

[0127] Specifically, according to the texture features of the ultrasound image, the image is divided into a 10x10 pixel grid according to a preset grid size. The texture feature difference of different ultrasound images in the grid area is analyzed. When the texture feature difference is less than a certain threshold, the area is identified as an overlapping area. The texture feature difference of different ultrasound images in the expanded grid area is analyzed by expanding the grid outward until the texture feature difference in the grid area is greater than or equal to the corresponding threshold, and the corresponding grid area is determined as an overlapping area. Identifying the overlapping area in combination with the similarity and continuity of the texture features can make full use of the intrinsic information of the image and accurately find the overlapping part between images. Compared with the traditional method of identifying the overlapping area based on image gray scale, the present scheme can adapt to ultrasound images of different angles, improve the accuracy and reliability of the overlapping area identification.

[0128] After identifying the overlapping area of the ultrasound image, the pixels in the overlapping area are fused, the non-overlapping area is taken as a stitching area, and the stitching area and the overlapping area are adjusted and stitched to obtain a complete ultrasound image. As shown in Figure 6 As shown in FIG. 6, there are two overlapping areas, an overlapping area a and an overlapping area b, between the ultrasound image A, the ultrasound image B and the ultrasound image C. The pixels at corresponding positions of the ultrasound image A, the ultrasound image B and the ultrasound image C in the overlapping area are fused. There is an overlapping area c between the ultrasound image B and the ultrasound image C. The pixels at corresponding positions of the ultrasound image B and the ultrasound image C in the overlapping area c are fused. The non-overlapping area is taken as a stitching area and is subjected to stitching processing.

[0129] Specifically, the texture features of the overlapped region are combined with the texture information of the multi-angle ultrasound image, the texture trend and the initial ultrasound image are deviated, the ultrasound image texture features of the edge of the splicing region are analyzed, the texture trend and continuity are analyzed, the angle and position of the splicing region are adjusted according to the continuity of the image texture between the overlapped region and the splicing region, the texture of the spliced image is continuous, and a complete second ultrasound image set is obtained; based on the texture features of the splicing region, the image splicing can avoid the problems such as texture misplacement and fracture in the spliced image, the angle and position of the splicing region are adjusted, the quality of the image splicing is improved, and the spliced image can accurately reflect the morphological structure of the prostate.

[0130] Further, the ultrasound images in the second ultrasound image set are subjected to interpolation processing, the ultrasound images are sequentially filled, the ultrasound images are mapped from two-dimensional space to three-dimensional space, the pixel values of four pixel points around the interpolation point are weighted and averaged by using a bilinear interpolation algorithm between adjacent ultrasound images, the pixel value of the interpolation point is calculated, the interpolated ultrasound image is obtained, the calculated interpolated ultrasound image is sequentially inserted between adjacent ultrasound images, and a three-dimensional ultrasound image is constructed; by constructing the three-dimensional ultrasound image, when performing multi-modal image fusion, the corresponding two-dimensional ultrasound image and the magnetic resonance image slice at the corresponding position of the magnetic resonance image can be fused according to the position correspondence between the three-dimensional ultrasound image and the magnetic resonance image, so that the real-time two-dimensional ultrasound image and the magnetic resonance image slice are sequentially correspondingly fused, and the accuracy of the multi-modal image fusion process is improved.

[0131] Further, according to the texture features of the ultrasound image, the overlapped region in each ultrasound image is identified according to a preset grid size, and the overlapped region is obtained, including:

[0132] S501, comparing the ultrasound image texture features in the first grid region corresponding to the plurality of ultrasound images according to a preset grid size, to obtain a feature difference value;

[0133] S502, when the feature difference value is less than a preset feature difference threshold, a grid is expanded outward by one circle according to the grid size, to obtain a second grid region;

[0134] S503, when the ultrasound image texture feature difference value in the second grid region is less than the preset feature difference threshold, the grid region is expanded outward by one circle according to the grid size, until the feature difference value in the grid region is greater than or equal to the preset feature difference threshold, and the expanded grid region is taken as the overlapped region.

[0135] In this embodiment, first, set the grid size according to the actual calculation accuracy requirement, and set the grid size to a 10x10 pixel size grid in this embodiment. The texture of the ultrasound image in the corresponding size grid is compared, the difference of the texture feature values of multiple ultrasound images in each grid area is calculated, the smaller the feature difference value, the more similar the texture of the corresponding grid area. The calculated feature difference value is compared with the preset feature difference threshold, and the feature difference threshold is set according to the calculation accuracy requirement. When the calculated feature difference value is less than the preset feature difference threshold, the texture similarity of the corresponding grid area is high, the grid area is taken as the overlap area, and a circle of grids is expanded outward from the center of the grid area according to the corresponding grid size. After one circle of expansion, a second grid area is obtained. The texture feature difference value comparison and analysis process is repeated, and the difference of the texture feature values of multiple ultrasound images in the second grid area is analyzed. When the texture feature difference value in the second grid area is less than the preset feature difference threshold, the expanded second grid area is taken as the overlap area.

[0136] The operation of expanding the grid and comparing the texture feature difference value is repeated. When the texture feature difference value in the expanded grid area is greater than or equal to the preset feature difference threshold, it indicates that the overlap area range has been exceeded. The grid area after the last expansion is taken as the final overlap area. The boundary of the overlap area is determined according to the change of the texture feature through grid iterative expansion. The overlap area of the ultrasound image is identified through grid division. Based on the texture feature comparison and the gradual expansion of the grid, the maximum overlap area can be determined by using the texture information of the ultrasound image. Compared with the traditional overlap area identification method based on image gray scale or simple shape matching, this scheme can effectively cope with the differences in ultrasound images under different acquisition conditions, and improve the accuracy and reliability of the overlap area identification.

[0137] As shown in FIG. 1, the ultrasound image is divided into grids according to the preset grid size. Taking each grid as the center, the first grid area is obtained. The difference of the texture feature values of multiple ultrasound images in the first grid area is compared with the preset feature difference threshold. When the difference is less than the preset feature difference threshold, a circle of grids is expanded outward from the center of the grid according to the grid size, and the second grid area is obtained. The difference of the texture feature values of multiple ultrasound images in the second grid area is compared with the preset feature difference threshold. When the difference is less than the preset feature difference threshold, a circle of grids is expanded outward from the center of the grid according to the grid size, and the difference of the texture feature values of multiple ultrasound images in the expanded grid area (the area enclosed by the dashed line in the figure) is compared with the preset feature difference threshold. When the difference is greater than or equal to the preset feature difference threshold, the texture feature difference of multiple ultrasound images in the expanded grid area is large, and the grid area cannot be taken as the overlap area. The second grid area is taken as the overlap area. The grid expansion and feature difference value comparison process is repeated until the feature difference value is greater than or equal to the preset feature difference threshold. The grid area in the last comparison process is taken as the overlap area. Figure 7 As shown in FIG. 1, the ultrasound image is divided into grids according to the preset grid size. Taking each grid as the center, the first grid area is obtained. The difference of the texture feature values of multiple ultrasound images in the first grid area is compared with the preset feature difference threshold. When the difference is less than the preset feature difference threshold, a circle of grids is expanded outward from the center of the grid according to the grid size, and the second grid area is obtained. The difference of the texture feature values of multiple ultrasound images in the second grid area is compared with the preset feature difference threshold. When the difference is less than the preset feature difference threshold, a circle of grids is expanded outward from the center of the grid according to the grid size, and the difference of the texture feature values of multiple ultrasound images in the expanded grid area (the area enclosed by the dashed line in the figure) is compared with the preset feature difference threshold. When the difference is greater than or equal to the preset feature difference threshold, the texture feature difference of multiple ultrasound images in the expanded grid area is large, and the grid area cannot be taken as the overlap area. The second grid area is taken as the overlap area. The grid expansion and feature difference value comparison process is repeated until the feature difference value is greater than or equal to the preset feature difference threshold. The grid area in the last comparison process is taken as the overlap area.

[0138] Further, the image texture matching degree is analyzed by analyzing the texture features of the ultrasonic images in the splicing area, and the second ultrasonic image set is obtained by splicing the splicing area, including:

[0139] S601, according to the texture features of the ultrasonic images in the splicing area, analyzing the texture in the ultrasonic images to obtain the texture trend;

[0140] S602, by analyzing the continuity between the texture trends of the edges of the splicing area, adjusting the angle and position of the splicing area and splicing to obtain the second ultrasonic image set.

[0141] In this embodiment, the texture features of the ultrasonic images in the splicing area are analyzed, the texture trend information is analyzed according to the texture features of the ultrasonic images, and the texture trend is obtained. The texture trend analysis algorithm includes a texture direction analysis algorithm based on a direction gradient histogram, a texture direction encoding algorithm based on a local binary pattern, etc. In this embodiment, the texture direction analysis algorithm based on the direction gradient histogram is used, the splicing area image is divided into a plurality of small unit cells, the direction histogram of the pixel gradient in each unit cell is calculated, the proportion of different directions in the histogram is counted, and the main direction of the texture in the unit cell is determined. The texture trend information of each unit cell is integrated to obtain the texture trend of the ultrasonic images in the entire splicing area. By analyzing the texture trend in the splicing area, the position and angle of the splicing area can be adjusted in combination with the continuity of the texture trend in the ultrasonic images, the accuracy of image splicing is improved, and the spliced image more truly reflects the texture of the prostate tissue.

[0142] For the image of the splicing area edge, the texture trend of the splicing area edge is analyzed, which is compared with the angle, direction and change trend of the texture trend of the adjacent image edge. The difference between the texture trends of the adjacent splicing image edges is analyzed by calculating the angle difference of the texture trend and the similarity of the direction vector. When the texture trend of the splicing area edge is discontinuous, the angle and position of the splicing area are adjusted. The texture between the adjacent images is spliced together through rotation and translation operations. When there is an angle deviation between the texture trends of the adjacent image edges, one of the images is rotated. When there is a position misalignment, the position of the image is adjusted through translation operation, so that the texture trends of the splicing images after position adjustment are consistent. After the angle and position of the splicing area are adjusted, the images of the splicing area are fused and spliced. For the overlapping part, the fusion is performed by averaging the pixel values. After all the splicing areas are fused and spliced, a second ultrasonic image set is obtained. The adjustment and splicing of the splicing area based on the continuity of the texture trend can make the spliced image more reasonable, improve the overall quality of the spliced image, and help doctors more accurately observe the prostate tissue structure and lesion characteristics, thereby improving the accuracy and efficiency of prostate disease diagnosis.

[0143] As shown in Figure 8 By analyzing the texture trend of texture d in splicing area n and texture e in splicing area m, the texture trends of texture d and texture e are consistent, but the connection point has deviation. At this time, the splicing area m is translated to the left, so that the textures d and e are continuous.

[0144] Further, in response to fusing the corresponding two-dimensional slice images of the two-dimensional ultrasonic image and the nuclear magnetic image to obtain a fused image, comprising:

[0145] S701, according to the pre-acquired nuclear magnetic image, performing feature extraction through a pre-set third feature extraction model to obtain a nuclear magnetic image feature;

[0146] S702, corresponding the positions of the three-dimensional ultrasonic image and the nuclear magnetic image to obtain the position correspondence relationship between each two-dimensional ultrasonic image and the two-dimensional slice image in the nuclear magnetic image;

[0147] S703, combining the nuclear magnetic image feature and the attitude sensor data to establish a mapping relationship between the nuclear magnetic image and the ultrasonic image;

[0148] S704, combining the mapping relationship and the position correspondence relationship, performing feature fusion on each two-dimensional ultrasonic image in the two-dimensional ultrasonic image sequence and the corresponding two-dimensional slice image of the nuclear magnetic image to obtain a fused feature;

[0149] S705, according to the fused feature, obtaining a fused image to fuse the prostate image.

[0150] The embodiment corresponds the positions of the three-dimensional ultrasound image and the magnetic resonance image, obtains the position correspondence relationship of each two-dimensional ultrasound image and the two-dimensional slice image in the magnetic resonance image, fuses the corresponding two-dimensional slice image in the magnetic resonance image with each real-time two-dimensional ultrasound image in turn, extracts the magnetic resonance image features, combines the posture sensor data during the ultrasound image acquisition, establishes the mapping relationship between the magnetic resonance image and the ultrasound image, fuses the feature values corresponding to the mapping relationship, and generates a fusion image. The fusion image combines the advantages of the magnetic resonance image and the ultrasound image, has the high-resolution display capability of the magnetic resonance image on the tissue anatomical structure and lesion details, has the advantages of the ultrasound image on the real-time dynamic observation of the tissue and the clear display of the boundary, provides comprehensive and accurate prostate image data, can improve the identification accuracy of the lesion site, reduces misdiagnosis and missed diagnosis, and improves the accuracy of prostate disease diagnosis.

[0151] In the embodiment, the third feature extraction model is used for feature extraction according to the pre-acquired magnetic resonance image, the rich tissue information in the magnetic resonance image is extracted, including the signal intensity of different tissues, anatomical structure details and other information, the third feature extraction model includes a convolutional neural network model based on deep learning, a traditional image feature extraction model and the like, the convolutional neural network model based on deep learning is used in the embodiment, the convolutional neural network model is trained by using a large number of historical magnetic resonance images, a pre-trained third feature extraction model is obtained, the pre-acquired magnetic resonance image is input into the pre-trained third feature extraction model, the model extracts the key features in the magnetic resonance image, the key features in the image are efficiently and accurately extracted by automatically extracting the magnetic resonance image features through the model.

[0152] Specifically, the positions of the three-dimensional ultrasound image and the magnetic resonance image are corresponded, the position correspondence relationship of each two-dimensional ultrasound image and the two-dimensional slice image in the magnetic resonance image is obtained, the corresponding two-dimensional slice image in the magnetic resonance image is fused according to each real-time two-dimensional ultrasound image; the mapping relationship between the magnetic resonance image and the ultrasound image is established by combining the extracted magnetic resonance image features and the posture sensor data; wherein the posture sensor data records the spatial position information during the ultrasound image acquisition, the magnetic resonance image features reflect the structure information of the prostate, the mapping relationship between the magnetic resonance image and the ultrasound image in the spatial position and the structure features is analyzed according to the posture sensor data, the image data of the two modalities is mapped into the same spatial coordinate system, the two modalities of images can be fused through the established mapping relationship, the basis is provided for the multi-modal data feature fusion, the structure information of the prostate can be accurately reflected in the fused image, and the quality of the image fusion is improved.

[0153] After the corresponding mapping relationship is constructed, the two-dimensional slice image in the nuclear magnetic image and the two-dimensional ultrasound image are mapped into the same coordinate space according to the mapping relationship, the feature values of the corresponding positions are fused, and a fused feature is obtained. Through the feature value fusion based on the mapping relationship, the information of the nuclear magnetic image and the ultrasound image can be integrated, the fused feature can more comprehensively describe the state of the prostate, more rich diagnostic information can be provided for doctors, and the diagnostic accuracy for the prostate disease can be improved.

[0154] Specifically, according to the fused feature, an image is generated by using an image reconstruction algorithm, including an interpolation-based algorithm, a deep learning-based image generation algorithm, etc. In this embodiment, a corresponding fused image is generated by using a pre-trained generative adversarial network. The fused image combines the advantages of the nuclear magnetic image and the ultrasound image, doctors can obtain the anatomical structure details of the prostate and the real-time tissue boundary information from the fused image at the same time, the position, size and nature of the prostate lesion can be more accurately judged, and a corresponding treatment plan can be formulated.

[0155] Further, the mapping relationship between the nuclear magnetic image and the ultrasound image is established by combining the nuclear magnetic image feature and the attitude sensor data, including:

[0156] S801, key feature points of the nuclear magnetic image feature are extracted to obtain nuclear magnetic image feature points.

[0157] S802, the nuclear magnetic image feature points and the attitude sensor data are combined to extract ultrasound image feature points at corresponding positions in the ultrasound image.

[0158] S803, the mapping relationship between the nuclear magnetic image and the ultrasound image is established by analyzing the positional relationship between the nuclear magnetic image feature points and the ultrasound image feature points.

[0159] In this embodiment, key feature points are extracted according to the extracted nuclear magnetic image feature to obtain nuclear magnetic image feature points. The extracted nuclear magnetic image feature points can reflect important structural information of the prostate, including the edge profile of the prostate, internal blood vessel branch points, feature points of the lesion area and the like. The key feature points in the image are extracted by a feature point extraction algorithm, which includes a scale-invariant feature transform algorithm, a speeded-up robust features algorithm and the like. In this embodiment, the scale-invariant feature transform algorithm is used. According to the calculation process, the parameters of the scale-invariant feature transform algorithm are set, the nuclear magnetic image feature is input into the scale-invariant feature transform algorithm with the set parameters, the key feature points in the nuclear magnetic image are extracted by the algorithm, and a vector describing the local feature of each feature point is generated. By extracting the key feature points, the data reflecting the key structural information can be screened, the mapping relationship can be established based on the key feature points, the accuracy and efficiency of establishing the mapping relationship can be improved, and the increase of the calculation complexity and the matching error caused by using too much redundant information can be avoided.

[0160] Specifically, according to the attitude sensor data, the coordinates of the nuclear magnetic image feature points are converted from the coordinate system of the nuclear magnetic image to the spatial coordinate system during the ultrasonic image acquisition, and according to the positions of the key feature points, the ultrasonic image feature points at the corresponding positions are extracted. The combination of the attitude sensor data and the nuclear magnetic image feature points can accurately locate the feature points corresponding to the nuclear magnetic image in the ultrasonic image. The combination of the attitude sensor data and the nuclear magnetic image feature points extracts the ultrasonic image feature points at the corresponding positions, uses the spatial rotation angle information during the ultrasonic image acquisition, and can accurately locate the feature points corresponding to the nuclear magnetic image in the ultrasonic image. This avoids directly searching for corresponding feature points in the entire ultrasonic image, reduces the amount of calculation, and improves the efficiency and accuracy of feature point matching.

[0161] Specifically, after the nuclear magnetic image feature points and the ultrasonic image feature points are extracted, the horizontal distance, the vertical distance, the angle difference and other position information between the nuclear magnetic image feature points and the ultrasonic image feature points are analyzed, and the angle difference is calculated by the arctangent function. A mapping relationship is established between the nuclear magnetic image feature points and the ultrasonic image feature points by using a geometric transformation model, which includes an affine transformation model, a perspective transformation model, etc. The geometric transformation model used in this embodiment is an affine transformation model. The coordinates of the nuclear magnetic image feature points and the ultrasonic image feature points are input into the affine transformation model. The model establishes a mapping relationship between the corresponding nuclear magnetic image and ultrasonic image according to the coordinate corresponding position relationship between the corresponding feature points. The mapping relationship is established by analyzing the position relationship of the feature points. Based on the structural characteristics of the image, the two kinds of modal images can be aligned, which provides an accurate corresponding relationship for image fusion and improves the quality and accuracy of the fused image.

[0162] Further, the corresponding relationship between the mapping relationship and the position is combined to perform feature fusion on each two-dimensional ultrasonic image in the two-dimensional ultrasonic image sequence and the corresponding two-dimensional slice image of the nuclear magnetic image to obtain a fusion feature, including:

[0163] S901, according to the position relationship of the corresponding feature points in the mapping relationship and the position corresponding relationship, mapping each two-dimensional ultrasonic image in the two-dimensional ultrasonic image sequence to the coordinate system of the corresponding two-dimensional slice image of the nuclear magnetic image;

[0164] S902, weighted average of the mapped two-dimensional ultrasonic image feature value and the corresponding two-dimensional slice image feature value is obtained.

[0165] In the embodiment, according to the corresponding position relationship in the mapping relationship and the position corresponding relationship between the two-dimensional ultrasound images and the two-dimensional slice images in the nuclear magnetic image, all pixel points of each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence are converted into the coordinate system of the nuclear magnetic image according to the corresponding relationship between the nuclear magnetic image feature points in the mapping relationship; for each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence, each pixel point in the image is traversed, the coordinates after the coordinate system conversion are calculated according to the mapping relationship, the pixel points in the ultrasound image are rearranged according to the converted coordinates, and are mapped to the nuclear magnetic image coordinate system; each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence is mapped to the coordinate system of the corresponding two-dimensional slice image of the nuclear magnetic image, the coordinate alignment of the two different modal images in the spatial position is realized, the pixel misalignment problem caused by the coordinate system difference is eliminated, an accurate coordinate corresponding relationship is provided for image fusion, the accuracy and effectiveness of the fusion features are improved, and thus the quality of the fused image is improved.

[0166] Specifically, the mapped ultrasound image feature values and the nuclear magnetic image feature values at the corresponding positions are weighted and averaged to obtain the fusion features; the weights of the ultrasound image feature values and the nuclear magnetic image feature values are preset according to the importance of the nuclear magnetic image and the ultrasound image in the diagnosis of the prostate disease; for example, in the diagnosis of prostate tumors, the qualitative diagnosis of tumors by the nuclear magnetic image is more important, the weight of the nuclear magnetic image feature value is set to 0.7, and the weight of the ultrasound image feature value is set to 0.3; for the mapped ultrasound image and the nuclear magnetic image, each pixel point is traversed to obtain the ultrasound image feature value and the nuclear magnetic image feature value of each corresponding position pixel point; the feature values at each corresponding position are weighted and averaged according to the preset weights to obtain the fusion feature value at each corresponding position; the fusion feature values at each position are integrated to obtain the fusion features; the weights can be flexibly adjusted by the weighted average fusion of the feature values at the corresponding positions, the information of the corresponding modal image is highlighted according to different diagnostic requirements, the fusion features can more accurately reflect the actual situation of the prostate, the quality of the fused image is improved, and thus the accuracy and reliability of the doctor in the diagnosis of the prostate disease are improved.

[0167] Embodiment 3:

[0168] The embodiment provides a method for detection based on a device for prostate image fusion using a posture sensor, and the method comprises the following steps: fixing the posture sensor on an ultrasonic probe, locking a depth position of the ultrasonic probe when the ultrasonic probe detects a prostate position, collecting ultrasonic images by rotating the ultrasonic probe, recording rotation information of the ultrasonic probe by the posture sensor, and acquiring multi-angle ultrasonic images of the prostate and posture sensor data acquired by the posture sensor; rotating each angle of the ultrasonic images according to a coordinate alignment rotation angle, so as to obtain a two-dimensional ultrasonic image sequence; performing interpolation processing on the two-dimensional ultrasonic images in the two-dimensional ultrasonic image sequence according to the posture sensor data in a three-dimensional space according to a preset grid, so as to construct a three-dimensional ultrasonic image; extracting the posture sensor data and determining an ultrasonic plane of current ultrasonic probe imaging, extracting a two-dimensional slice image in a nuclear magnetic image according to the ultrasonic plane and the pre-acquired nuclear magnetic image, fusing the two-dimensional ultrasonic image and the corresponding two-dimensional slice image of the nuclear magnetic image, and obtaining a fusion image.

[0169] Specifically, the three-dimensional ultrasonic image is constructed, and the method further comprises the following steps: determining a rotation center by using a spatial characteristic of the rotation of the ultrasonic probe, comparing and optimizing the three-dimensional ultrasonic image reconstructed according to the real-time ultrasonic image on a section defined by the posture sensor, and obtaining a motion calculation result of the real-time two-dimensional ultrasonic image relative to the three-dimensional ultrasonic image; combining the motion calculation result, compensating fusion errors of the real-time ultrasonic image and the nuclear magnetic image caused by motion according to a mapping relationship between the three-dimensional ultrasonic image and the nuclear magnetic image by using a compensation formula, and the compensation formula is as follows:

[0170]

[0171] In the formula, wherein, is a coordinate matching result of the real-time two-dimensional ultrasonic image and the reconstructed three-dimensional ultrasonic image based on image processing; is a coordinate mapping result of the three-dimensional ultrasonic image and the nuclear magnetic image; is a coordinate conversion result of the fusion of the real-time ultrasonic image and the nuclear magnetic image.

[0172] Specifically, the position of a point on the rotation axis of the ultrasonic probe remains unchanged in space during the rotation of the ultrasonic probe, based on this spatial characteristic, the rotation center can be accurately positioned by analyzing a series of images collected during the rotation of the ultrasonic probe, and the ultrasonic probe is fixed on a mechanical device that can accurately control the rotation angle; the collected ultrasonic images are preprocessed, the rotation center is calculated by fitting according to the coordinate change in the images of different angles; an accurate spatial reference origin is provided for the reconstruction of the three-dimensional ultrasonic image, the three-dimensional reconstruction error caused by the uncertainty of the rotation center is reduced, and the spatial positioning accuracy of the three-dimensional image is improved.

[0173] The posture sensor acquires spatial posture information of the ultrasonic probe in real time, including position and angle, etc., determines corresponding sections in the three-dimensional ultrasonic image according to the posture information, and the sections are associated with the sections corresponding to the real-time two-dimensional ultrasonic image; calculates the position deviation and angle deviation between the real-time two-dimensional ultrasonic image and the corresponding section image of the three-dimensional ultrasonic image, adjusts and optimizes the motion parameters through the least square method according to the calculated deviation, and obtains the motion calculation result of the real-time two-dimensional ultrasonic image relative to the three-dimensional ultrasonic image; through analyzing the motion of the real-time two-dimensional ultrasonic image relative to the three-dimensional ultrasonic image, accurate motion parameter basis is provided for image fusion error compensation.

[0174] Specifically, the coordinate mapping relationship between the three-dimensional ultrasonic image and the nuclear magnetic image is established through the image registration technology, the coordinates in the three-dimensional ultrasonic image are converted into the nuclear magnetic image coordinate system, the coordinate corresponding relationship of the image features of the real-time two-dimensional ultrasonic image and the three-dimensional ultrasonic image is obtained based on the coordinate matching result of the real-time two-dimensional ultrasonic image and the reconstructed three-dimensional ultrasonic image, the corrected coordinate conversion result of the real-time ultrasonic image and the nuclear magnetic image fusion is calculated according to the compensation formula, the real-time ultrasonic image and the nuclear magnetic image are fused according to the corrected coordinate conversion result, and the fused image is obtained; through the coordinate compensation, the error caused by the real-time ultrasonic probe motion and the nuclear magnetic image fusion can be effectively compensated, and the accuracy and reliability of the two image fusion are improved.

[0175] Further, the sensor is used to collect the ultrasonic probe tracking data, including: converting the motion of the ultrasonic probe into the gesture control of the operator, changing the running parameters of the preset surgical navigation software through the rotation of the ultrasonic probe; using one or more posture sensors independent of the ultrasonic probe to control the operation mode of the surgical navigation software; using an acceleration sensor to detect a tapping event and converting it into a corresponding mouse function.

[0176] Specifically, the sensor is used to collect the ultrasonic probe tracking data, the surgical navigation software is preset, the running parameters that need to be changed by gesture control are determined, including image zoom ratio, image rotation angle, imaging depth, etc., and the corresponding probe rotation gesture mode is set for each parameter, including clockwise rotation corresponding to image enlargement, counterclockwise rotation corresponding to image reduction, rotation around the X-axis corresponding to adjusting the imaging depth, etc.; a large amount of gesture data is used to train the preset gesture recognition model, the gesture recognition model includes but is not limited to a graph convolutional neural network model, a pre-trained gesture recognition model is obtained, the gestures of the operator are recognized through the pre-trained gesture recognition model, the recognized gesture instructions are converted into corresponding software running parameter adjustment signals according to the preset mapping relationship, and the signals are sent to the surgical navigation software; the software automatically adjusts the corresponding running parameters according to the signals; through gesture control, the operator does not need to be distracted to use additional input devices, and the adjustment of the software parameters can be realized through natural probe rotation gestures, which improves the continuity and efficiency of the surgical operation and reduces the operation interruption time.

[0177] Specifically, one or more high-precision independent attitude sensors are selected, including but not limited to a nine-axis attitude sensor, which is fixed on a position easy to be controlled by the operator, such as a handle of a surgical instrument, a ring on the operator's hand or a special control rod, to ensure that the sensor can stably sense the attitude control action of the operator; attitude data of the independent attitude sensor under different preset attitudes are collected, including acceleration, angular velocity, magnetic field strength and other information, an attitude sample database is established, the real-time collected self-attitude data is matched in the attitude sample database, and the current attitude instruction is judged; according to the recognized attitude instruction, the control unit sends a mode switching signal to the surgical navigation software, and the software switches to the corresponding running mode after receiving the switching signal; through attitude recognition and mode switching, remote and contactless control of the software running mode is realized, which avoids the operator to frequently switch between the probe and other control devices, improves the convenience and accuracy of mode switching, and also reduces the risk of cross infection.

[0178] An acceleration sensor is installed on an ultrasonic probe or a related device which is easy to be hit by an operator, the acceleration sensor is calibrated, the acceleration range which can be generated in a normal operation process is determined, and interference signals generated by non-hitting actions are excluded; an acceleration threshold of a hitting event is set, when the acceleration value detected by the sensor exceeds the acceleration threshold, it is judged that a hitting event occurs; a corresponding relationship between the hitting event and the mouse function is defined, including but not limited to single hitting corresponding to mouse left button single click, continuous two times of rapid hitting corresponding to mouse left button double click, larger force hitting corresponding to mouse right button single click, etc.; according to the real-time collected acceleration data, the hitting event type is recognized, and it is converted into the corresponding mouse function instruction; the mouse function instruction recognized by the operator provides a simple and intuitive mouse function replacement operation mode, without additional mouse device, the number of devices on the operating table is reduced, and the operation complexity is also reduced.

[0179] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for prostate image fusion using an attitude sensor, characterized in that, include: An attitude sensor is fixed to an ultrasound probe. When the ultrasound probe detects the prostate, the depth of the ultrasound probe is locked. Ultrasound images are acquired by rotating the ultrasound probe. The attitude sensor records the rotation information of the ultrasound probe, thereby obtaining multi-angle ultrasound images of the prostate and the attitude sensor data obtained by the corresponding attitude sensor. Based on the multi-angle ultrasound images and the corresponding attitude sensor data, the ultrasound images at each angle are rotated according to the coordinate-aligned rotation angle to obtain a two-dimensional ultrasound image sequence. The two-dimensional ultrasound images in the two-dimensional ultrasound image sequence are interpolated in three-dimensional space according to a preset grid based on the attitude sensor data to construct a three-dimensional ultrasound image. The attitude sensor data is extracted and the ultrasonic plane of the current ultrasonic probe imaging is determined. Based on the pre-acquired MRI image, a two-dimensional slice image is extracted from the MRI image in combination with the ultrasonic plane. The two-dimensional ultrasound image and the corresponding two-dimensional slice image of the MRI image are fused to obtain a fused image, including: Based on the pre-acquired MRI images, image processing and feature extraction are performed using a preset third feature extraction model to obtain the MRI image features of the prostate in the MRI images; By mapping the positions of the three-dimensional ultrasound images to those of the MRI images, the positional correspondence between each two-dimensional ultrasound image and the two-dimensional slice image in the MRI image is obtained. By combining the nuclear magnetic resonance imaging features and attitude sensor data, a mapping relationship between nuclear magnetic resonance imaging and ultrasound images is established; Combining the mapping relationship and positional correspondence, feature fusion is performed on each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence and the corresponding two-dimensional slice image of the MRI image to obtain fused features; Based on the fusion features, a fused image is obtained to fuse the prostate image; The process involves combining the mapping relationship and positional correspondence to fuse the features of each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence with the corresponding two-dimensional slice image of the MRI image, resulting in fused features, including: Based on the positional relationship and positional correspondence of the corresponding feature points in the mapping relationship, each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence is mapped to the coordinate system of the corresponding two-dimensional slice image of the MRI image; The fused features are obtained by weighted averaging the feature values ​​of the mapped two-dimensional ultrasound image and the corresponding two-dimensional slice image.

2. The method for prostate image fusion using an attitude sensor according to claim 1, characterized in that, Based on the multi-angle ultrasound images and corresponding attitude sensor data, the ultrasound images at each angle are rotated according to a coordinate-aligned rotation angle to obtain a two-dimensional ultrasound image sequence, including: The rotation angle of each ultrasound image is obtained based on the multi-angle ultrasound images and the corresponding attitude sensor data. By analyzing the posture sensor data corresponding to each ultrasound image, the rotation angle is compensated, and an angle matrix is ​​constructed. By combining the corresponding angle values ​​in the angle matrix, the ultrasound image is rotated around the rotation center in three-dimensional space to obtain a two-dimensional ultrasound image sequence.

3. The method for prostate image fusion using an attitude sensor according to claim 2, characterized in that, The step of compensating for the rotation angle and constructing an angle matrix by analyzing the posture sensor data corresponding to each ultrasound image includes: By using a preset first feature extraction model, feature extraction is performed on each ultrasound image to obtain the first ultrasound image features; The prostate edge features in the ultrasound image are extracted from the first ultrasound image features and rotated in three-dimensional space according to the attitude sensor data to obtain the second ultrasound image features; By performing edge matching between the second ultrasound image features and the prostate in the pre-acquired MRI image, the corresponding spatial relationship between the attitude sensor data and the MRI image is calculated, and an angle matrix is ​​constructed.

4. The method for prostate image fusion using an attitude sensor according to claim 1, characterized in that, The two-dimensional ultrasound images in the sequence are interpolated in three-dimensional space according to a preset grid based on attitude sensor data to construct a three-dimensional ultrasound image, including: The texture features of each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence are extracted using a pre-defined second feature extraction model to obtain the ultrasound image texture features. Based on the texture features of the ultrasound images, overlapping regions in each ultrasound image are identified according to a preset grid to obtain the overlapping regions; When performing pixel fusion on corresponding overlapping regions in ultrasound images, the non-overlapping regions are used as the stitching regions. By analyzing the texture matching degree of the ultrasound images in the stitched region, the stitched regions are stitched together to obtain a second set of ultrasound images. The ultrasound images in the second ultrasound image set are sampled and interpolated in three-dimensional space according to the attitude sensor data to construct a three-dimensional ultrasound image.

5. The method for prostate image fusion using an attitude sensor according to claim 4, characterized in that, The step of identifying overlapping regions in each ultrasound image according to the texture features of the ultrasound image and a preset grid size to obtain the overlapping regions includes: According to the preset grid size, the ultrasound image texture features in the first grid area corresponding to multiple ultrasound images are compared to obtain the feature difference value; When the feature difference is less than the preset feature difference threshold, the grid is expanded outward by one ring according to the grid size to obtain the second grid region; When the difference in texture features of the ultrasound image within the second grid region is less than a preset feature difference threshold, the grid region is expanded outward by a ring according to the grid size until the feature difference within the grid region is greater than or equal to the preset feature difference threshold, and the expanded grid region is taken as the overlapping region.

6. The method for prostate image fusion using an attitude sensor according to claim 4, characterized in that, The step involves analyzing the texture matching degree of the ultrasound images in the stitched regions, stitching the stitched regions together to obtain a second set of ultrasound images, including: Based on the texture features of the ultrasound image in the stitched area, the texture in the ultrasound image is analyzed to obtain the texture trend; By analyzing the continuity between the texture trends at the edges of the splicing areas, the angles and positions of the splicing areas are adjusted and spliced ​​to obtain a second set of ultrasound images.

7. The method for prostate image fusion using an attitude sensor according to claim 1, characterized in that, Constructing three-dimensional ultrasound images also includes: By utilizing the spatial characteristics of the ultrasonic probe rotation to determine the rotation center, the three-dimensional ultrasound reconstructed from the real-time ultrasonic image is compared and optimized on the cross-section defined by the attitude sensor to obtain the motion calculation results of the real-time two-dimensional ultrasound relative to the three-dimensional ultrasound. Based on the mapping relationship between three-dimensional ultrasound images and MRI images, and combined with the motion calculation results, the fusion error of real-time ultrasound images and MRI images caused by motion is compensated using a compensation formula as follows: ； In the formula, It is the coordinate matching result of real-time two-dimensional ultrasound and reconstructed three-dimensional ultrasound based on image processing; It is the coordinate mapping result of three-dimensional ultrasound and MRI images; It is the coordinate transformation result of real-time ultrasound image and MRI image fusion.

8. The method for prostate image fusion using an attitude sensor according to claim 1, characterized in that, The system utilizes sensors to collect tracking data from the ultrasound probe, including: converting the movement of the ultrasound probe into gesture control; changing the operating parameters of the preset surgical navigation software by rotating the ultrasound probe; controlling the operating mode of the surgical navigation software using one or more attitude sensors independent of the ultrasound probe; and detecting tapping events using an accelerometer and converting them into corresponding mouse functions.

9. A method for prostate image fusion using an attitude sensor according to claim 1, characterized in that, By combining the MRI image features and attitude sensor data, a mapping relationship between MRI images and ultrasound images is established, including: Key feature points are extracted from the MRI images to obtain the MRI image feature points; Based on the nuclear magnetic resonance imaging feature points, the corresponding ultrasound image features of the corresponding ultrasound image feature points in the ultrasound image are extracted; By matching the features of MRI and ultrasound images at corresponding feature point locations, and combining them with attitude sensor data, a mapping relationship between MRI and ultrasound images is established.

10. A method for prostate image fusion using an attitude sensor as described in any one of claims 1 to 9, characterized in that, In response to obtaining fused images, the method also includes fusing images obtained using CT or PET with ultrasound images.

11. An apparatus for prostate image fusion using an attitude sensor, for implementing the method for prostate image fusion using an attitude sensor as described in any one of claims 1 to 9, characterized in that, include: A fixing device for connecting an ultrasonic probe, the fixing device having an attitude sensor mounted thereon, the fixing device being used to fuse the attitude sensor with the ultrasonic probe, the fixing device comprising: The inner support adjustment mechanism (2) includes an inner support (7), a clamp (8) and an adjustment member (9). The inner support (7) is connected to the clamp (8) and forms a clamping area for installing an ultrasonic probe. The adjustment member (9) is located inside the clamp (8) and is used to adjust the clamping area of ​​the inner support (7) to lock or unlock the ultrasonic probe. The attitude sensor is used to record the rotation information of the ultrasound probe, acquire multi-angle ultrasound images of the prostate and corresponding attitude sensor data, and rotate the ultrasound images of each angle according to the coordinate-aligned rotation angle to obtain a two-dimensional ultrasound image sequence and construct a three-dimensional ultrasound image. The attitude sensor data is extracted and the ultrasonic plane of the current ultrasonic probe imaging is determined. Based on the pre-acquired MRI image, the two-dimensional slice image in the MRI image is extracted in combination with the ultrasonic plane. Each two-dimensional ultrasound image in the two-dimensional ultrasound image sequence is fused with the corresponding two-dimensional slice image of the MRI image to obtain a fused image.

12. The apparatus for prostate image fusion using an attitude sensor as described in claim 11, characterized in that: The fixing device further includes a moving adjustment mechanism (1), which includes a linear structure (3) and a rotating structure (4), and the rotating structure (4) is rotatably connected to the linear structure (3); The linear structure (3) includes an outer bracket (5) and a locking seat (6). The locking seat (6) is slidably disposed on the outer bracket (5). A slot (51) is formed on the outer bracket (5). The locking seat (6) cooperates with the slot (51) and is used to adjust the position of the locking seat (6) along the outer bracket (5). The locking seat (6) is also connected to the inner bracket (7). The rotating structure (4) is rotatably connected to one end of the outer bracket (5). An opening (41) for the ultrasonic probe to extend is formed on the rotating structure (4).

Images