Medical image processing method and device, computing equipment and storage medium

Abstract

The invention provides a medical image processing method and device, computing equipment and a storage medium. The method may include: in response to determining to enter a three-dimensional reconstruction mode, obtaining an ultrasound image sequence from the ultrasound probe; obtaining attitude information of the ultrasonic probe related to the ultrasonic image sequence; obtaining at least one intermediate processing result based on the ultrasonic image sequence; and obtaining three-dimensional reconstruction data of the ultrasonic image sequence based on the attitude information and the intermediate processing result.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a medical image processing method, apparatus, computing device and storage medium. Background Art

[0002] Currently, doctors often use medical image sequences scanned by medical scanning equipment to make medical diagnoses. Among them, ultrasound acquisition equipment is one of the most common medical scanning equipment. A method for effectively reconstructing ultrasound images in three dimensions is desired.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] According to one aspect of the present disclosure, a medical image processing method is provided, comprising: in response to determining to enter a three-dimensional reconstruction mode, obtaining an ultrasound image sequence from the ultrasound probe; obtaining posture information of the ultrasound probe related to the ultrasound image sequence; obtaining at least one intermediate processing result based on the ultrasound image sequence; and obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing result.

[0005] According to another aspect of the present disclosure, a medical image processing device is provided, comprising: an image acquisition unit for obtaining an ultrasound image sequence from the ultrasound probe in response to determining to enter a three-dimensional reconstruction mode; a posture acquisition unit for obtaining posture information of the ultrasound probe related to the ultrasound image sequence; a processing unit for obtaining at least one intermediate processing result based on the ultrasound image sequence; and a reconstruction unit for obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing result.

[0006] According to another aspect of the present disclosure, a computing device is provided, comprising: a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement a medical image processing method according to one or more embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, a medical image processing method according to one or more embodiments of the present disclosure is implemented.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the medical image processing method according to one or more embodiments of the present disclosure when executed by a processor.

[0009] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a schematic diagram illustrating an example system in which the various methods described herein may be implemented according to an exemplary embodiment;

[0012] Figure 2 is a flow chart illustrating a medical image processing method according to an exemplary embodiment;

[0013] Figure 3A is a schematic diagram illustrating medical image processing according to an exemplary embodiment;

[0014] Figure 3B is a schematic block diagram illustrating a medical image processing apparatus according to an exemplary embodiment;

[0015] Figure 4 is a block diagram illustrating an exemplary computer device that can be applied to the exemplary embodiments. DETAILED DESCRIPTION

[0016] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0017] The terms used in the description of various examples described in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. As used herein, the term "plurality" means two or more, and the term "based on" should be interpreted as "based at least in part on". In addition, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations.

[0018] Exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0019] Figure 1 is a schematic diagram illustrating an example system 100 in which the various methods described herein may be implemented, according to an example embodiment.

[0020] refer to Figure 1 The system 100 includes a client device 110 , a server 120 , and a network 130 that communicatively couples the client device 110 and the server 120 .

[0021] The client device 110 includes a display 114 and a client application (APP) 112 that can be displayed via the display 114. The client application 112 can be an application that needs to be downloaded and installed before running or a small program (liteapp) as a lightweight application. In the case where the client application 112 is an application that needs to be downloaded and installed before running, the client application 112 can be pre-installed on the client device 110 and activated. In the case where the client application 112 is a small program, the user 102 can directly run the client application 112 on the client device 110 by searching for the client application 112 in the host application (for example, by the name of the client application 112, etc.) or scanning the graphic code of the client application 112 (for example, a bar code, a QR code, etc.), without installing the client application 112. In some embodiments, the client device 110 can be any type of mobile computer device, including a mobile computer, a mobile phone, a wearable computer device (for example, a smart watch, a head-mounted device, including smart glasses, etc.) or other types of mobile devices. In some embodiments, the client device 110 may alternatively be a stationary computer device, such as a desktop computer, a server computer or other types of stationary computer devices. In some optional embodiments, the client device 110 may also be or include a medical image printing device.

[0022] The server 120 is typically a server deployed by an Internet Service Provider (ISP) or an Internet Content Provider (ICP). The server 120 may represent a single server, a cluster of multiple servers, a distributed system, or a cloud server that provides basic cloud services (such as cloud databases, cloud computing, cloud storage, and cloud communications). It will be understood that although Figure 1 The server 120 is shown in FIG. 1 in communicating with only one client device 110 , but the server 120 may provide background services for multiple client devices simultaneously.

[0023] Examples of network 130 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet. Network 130 can be a wired or wireless network. In some embodiments, the data exchanged through network 130 is processed using technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. In addition, encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet protocol security (IPsec) can also be used to encrypt all or some links. In some embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0024] The system 100 may also include an image acquisition device 140. In some embodiments, Figure 1 The image acquisition device 140 shown can be a medical scanning device, including but not limited to scanning or imaging devices used in positron emission tomography (PET), positron emission tomography with computerized tomography (PET / CT), single photon emission computed tomography with computerized tomography (SPECT / CT), computerized tomography (CT), medical ultrasonography, nuclear magnetic resonance imaging (NMRI), magnetic resonance imaging (MRI), cardiovascular angiography (CA), digital radiography (DR), etc. For example, the image acquisition device 140 may include a digital subtraction angiography scanner, a magnetic resonance angiography scanner, a tomography angiography scanner, a positron emission tomography scanner, a positron emission computed tomography scanner, a single photon emission computed tomography scanner, a computed tomography scanner, a medical ultrasound examination device, a nuclear magnetic resonance imaging scanner, a magnetic resonance imaging scanner, a digital radiography scanner, etc. The image acquisition device 140 may communicate with a server (e.g., Figure 1The image processing system is connected to the server 120 in the figure or a separate server of the imaging system (not shown in the figure) to realize image data processing, including but not limited to converting the scan data (for example, converting it into a medical image sequence), compressing it, correcting its pixels, and reconstructing it in three dimensions.

[0025] The image acquisition device 140 may be connected to the client device 110 , for example, via the network 130 , or directly connected to the client device in other ways to communicate with the client device.

[0026] Optionally, the system may further include an intelligent computing device or computing card 150. The image acquisition device 140 may include or be connected (e.g., removably connected) to such a computing card 150, etc. As an example, the computing card 150 may implement image data processing, including but not limited to conversion, compression, pixel correction, reconstruction, etc. As another example, the computing card 150 may implement a medical image processing method according to an embodiment of the present disclosure.

[0027] The system may also include other parts not shown, such as a data storage unit. The data storage unit may be a database, a data repository, or one or more devices for data storage in other forms, and may be a conventional database, or may include a cloud database, a distributed database, etc. For example, direct image data formed by the image acquisition device 140 or a medical image sequence or three-dimensional image data obtained through image processing may be stored in the data storage unit for subsequent retrieval by the server 120 and the client device 110 from the data storage unit. In addition, the above-mentioned image acquisition device 140 may also directly provide the image data or the medical image sequence or three-dimensional image data obtained through image processing to the server 120 or the client device 110, etc.

[0028] The user can use the client device 110 to control the acquisition of images or videos, view the acquired images or videos (including preliminary image data or images that have been analyzed and processed, etc.), view the analysis results, interact with the acquired images or analysis results, input acquisition instructions, configure data, etc. The client device 110 can send configuration data, instructions or other information to the image acquisition device 140 to control the acquisition of the image acquisition device, process data, etc.

[0029] For the purpose of the embodiments of this disclosure, Figure 1In the example of , the client application 112 may be an image sequence management application, which may provide various functions, such as storage management, indexing, sorting, and classification of the acquired image sequences. Correspondingly, the server 120 may be a server used together with the image sequence management application. The server 120 may provide image sequence management services to the client application 112 running in the client device 110 based on user requests or instructions generated according to the embodiments of the present disclosure, such as managing the image sequence storage in the cloud, storing and classifying the image sequence according to a specified index (including, for example, but not limited to, sequence type, patient identification, body part, acquisition target, acquisition stage, acquisition machine, whether lesions are detected, severity, etc.), and retrieving and providing the image sequence to the client device according to the specified index, etc. Alternatively, the server 120 may also provide or allocate such service capabilities or storage space to the client device 110, and the client application 112 running in the client device 110 may provide corresponding image sequence management services according to user requests or instructions generated according to the embodiments of the present disclosure, etc. It is to be understood that the above is only an example, and the present disclosure is not limited thereto.

[0030] Figure 2 2 is a flowchart illustrating a medical image processing method 200 according to an exemplary embodiment. The method 200 may be performed on a client device (eg, Figure 1 The execution of each step of the method 200 may be performed at the client device 110 shown in FIG. Figure 1 In some embodiments, the method 200 may be performed on a server (e.g., Figure 1 In some embodiments, method 200 may be performed by a client device (eg, client device 110) and a server (eg, server 120) in combination.

[0031] In the following, each step of the method 200 is described in detail.

[0032] refer to Figure 2 At step 210 , in response to determining to enter the three-dimensional reconstruction mode, an ultrasound image sequence is obtained from the ultrasound probe.

[0033] At step 220 , posture information of the ultrasound probe related to the ultrasound image sequence is obtained.

[0034] At step 230 , at least one intermediate processing result is obtained based on the ultrasound image sequence.

[0035] At step 240 , three-dimensional reconstruction data of the ultrasound image sequence is obtained based on the posture information and the intermediate processing result.

[0036] Through the above method, after entering the three-dimensional reconstruction mode, the intermediate processing results related to the image can be used to assist the three-dimensional reconstruction process. On the one hand, additional information is only acquired after the three-dimensional reconstruction mode, which will not bring excessive information processing and resource burden. On the other hand, accurate three-dimensional data can be obtained using the intermediate processing results.

[0037] At least one intermediate processing result may be an intermediate processing result related to at least one object in the image. Exemplarily, at least one intermediate processing result can identify the location information of at least one object in the image. Exemplarily, at least one object may be a lesion, tissue, organ, blood vessel, edge or other object that can be developed and / or identified. As an example, the intermediate processing result related to the image may be image segmentation information or segmentation data, but the present disclosure is not limited thereto. As a non-limiting specific example, the intermediate processing result may be a segmentation boundary, object recognition information, feature information, texture, etc., which will be further described below in conjunction with the accompanying drawings.

[0038] Due to the particularity of ultrasound data, especially the dependence of ultrasound data on user operations and the two-dimensional imaging capability of ultrasound, it has always been a difficult problem to perform three-dimensional reconstruction of ultrasound data. In addition, for ultrasound data to follow user operations, a large number of ultrasound images will be obtained during ultrasound scanning, and processing all such image sequences will also consume a lot of computing resources. According to an embodiment of the present disclosure, a three-dimensional reconstruction mode can be entered based on a three-dimensional reconstruction instruction, and the prior knowledge of the user (for example, a doctor) can be effectively utilized to perform a three-dimensional reconstruction process based on the three-dimensional reconstruction instructions issued by the doctor, thereby reducing the amount of information that needs to be processed. For example, after the doctor recognizes the target that needs three-dimensional segmentation with the naked eye, or believes that the current area is of interest, a three-dimensional reconstruction instruction is issued. The posture information is obtained based on the three-dimensional reconstruction instruction, and the three-dimensional reconstruction process is started, thereby saving computing resources and being able to obtain the three-dimensional reconstruction results required by the doctor in a targeted manner.

[0039] Exemplarily, the three-dimensional reconstruction instruction may be an instantaneous instruction, such as pressing a specific button to trigger entering the three-dimensional reconstruction mode. Exemplarily, the posture information may be obtained after receiving the three-dimensional reconstruction instruction. Exemplarily, the ultrasound image sequence to be processed may also be an ultrasound image sequence obtained after the three-dimensional reconstruction instruction, or an ultrasound image sequence that traces back a certain amount of time (e.g., 10s, 30s) or a certain amount of data (e.g., 10 frames, 100 frames, etc.) at the moment of receiving the three-dimensional reconstruction instruction, or both. Exemplarily, the three-dimensional reconstruction mode may end when the user operates again, for example, when the user presses the same button again, clicks the same button with a different gesture, or presses another button indicating the end of the three-dimensional reconstruction state, the three-dimensional reconstruction mode is exited, and the posture information and intermediate processing results are no longer obtained.

[0040] Exemplarily, the 3D reconstruction instruction may be a continuous instruction, for example, when the user presses a specific button or touches a specific touch area, the 3D reconstruction mode is entered, posture information and intermediate processing results are started to be obtained, and when the user releases the button or leaves the touch area, the 3D reconstruction mode is exited.

[0041] In some other embodiments, the 3D reconstruction mode may be triggered based on other means, such as after detecting that a 3D reconstruction condition is met, such as but not limited to after detecting a significant lesion, without requiring user instructions.

[0042] According to some embodiments, obtaining the posture information related to the ultrasound probe may include obtaining the posture information based on a posture detection device associated with the ultrasound probe in response to determining to enter a three-dimensional reconstruction mode.

[0043] In one embodiment, the position detection device may be enabled after entering the three-dimensional reconstruction mode, for example, after receiving a three-dimensional reconstruction instruction from the user. The position detection device is capable of recording the posture information related to the ultrasound probe. In other embodiments, the position detection device may continuously record the posture information, but only after entering the three-dimensional reconstruction mode, for example, after receiving a three-dimensional reconstruction instruction, performs three-dimensional reconstruction on the associated image sequence based on the posture information.

[0044] According to some embodiments, the posture detection device may be a gyroscope. For example, the gyroscope may measure spatial parameters of the ultrasound probe, such as rotation angles (α, β, γ), at the time of acquisition of the corresponding ultrasound image sequence. Some ultrasound probes are already equipped with gyroscopes, and such parameters may be used in combination with segmentation for three-dimensional reconstruction. In other more traditional probe scenarios that do not have gyroscopes attached, the posture detection device may be attached to the probe, thereby obtaining posture information related to the ultrasound image without modifying the existing ultrasound probe.

[0045] Exemplarily, obtaining at least one intermediate processing result based on the ultrasound image sequence may be obtaining an intermediate processing result regarding at least one identified target or at least one image region involved in the ultrasound image sequence. According to some embodiments, obtaining at least one intermediate processing result based on the ultrasound image sequence includes: in response to determining to enter a three-dimensional reconstruction mode, enabling an image segmentation process to obtain the intermediate processing result.

[0046] According to some embodiments, the image segmentation process may be a real-time segmentation algorithm. By introducing a real-time segmentation algorithm, the target area may be segmented in real time while acquiring the ultrasound image, and the segmentation results may be spliced ​​into three-dimensional image data in combination with the posture information of the probe. As further described below in conjunction with some exemplary embodiments, the real-time segmentation may also be used to correct the posture data to further improve the accuracy of reconstruction.

[0047] According to some embodiments, the image segmentation process can be used to determine a segmentation boundary of at least one object in the ultrasound image sequence. Exemplarily, the method further comprises correcting the posture information based on the segmentation boundary.

[0048] At least one object may be an identification target related to the current medical analysis purpose, such as a lesion or a pathology. The one or more objects used for separation may also be other objects, such as tissues, organs or other objects with boundaries that can be used for reference.

[0049] According to some embodiments, the three-dimensional reconstruction instruction comes from an operating accessory removably attached to the ultrasound probe. Exemplarily, the operating accessory includes a button or other interactive operating interface.

[0050] In actual operation, doctors can easily identify the target area when scanning with an ultrasound probe. To achieve this goal, a specific sector scan can be performed for the target area, and the three-dimensional reconstruction process can be triggered by the user clicking a button. In this process, only three parameters of the probe in space, namely α, β and γ, need to be measured by a gyroscope to complete the acquisition of posture information. For example, the user only needs to press a button to identify the start of the three-dimensional reconstruction operation and start acquiring the posture information of the ultrasound probe. At the same time, the system will start the real-time segmentation process of the target area and process it in combination with the angle parameters of the intermediate processing results. Through these intermediate processing results, the features in the intermediate processing results, such as segmentation or other object features, can be spliced ​​together in three-dimensional space to complete the three-dimensional reconstruction of the target area. In addition, the intermediate processing results can also be used to correct the posture information of the probe. Through the above method, the system can establish a three-dimensional model of the lesion or organ in real time and display missing or complete information at different angles.

[0051] According to the embodiments of the present disclosure, reconstruction parameters can be obtained only by posture information, for example, without position information, and reliable three-dimensional reconstruction parameters are generated by combining image segmentation algorithm and gyroscope posture data, thereby achieving three-dimensional reconstruction of the target area.

[0052] The segmentation algorithm can identify the boundary information of the target area from the ultrasound image, such as the outline of the lesion or organ. These intermediate processing results can not only be used to splice and generate a 3D model, but also to correct the probe's posture information and improve the accuracy of reconstruction.

[0053] The gyroscope provides the angle information of the probe in space (such as α, β, γ), and these three parameters can describe the rotation state of the probe. Through these parameters, the position and posture of the probe in three-dimensional space can be determined, reducing the complexity and hardware cost of traditional six-degree-of-freedom position and posture measurement.

[0054] Therefore, by combining the intermediate processing results with the gyroscope data, key parameters for 3D reconstruction can be generated under the condition of simple hardware and low cost. The reliability and accuracy of these parameters are directly related to the effect of 3D reconstruction, which is the core innovation and technical advantage of the present invention.

[0055] Exemplarily, the technical solution can be implemented by adding a small accessory. For example, the operating accessory is tied to the ultrasound probe, and the user can manually rotate the probe and issue a reconstruction instruction by long pressing, double clicking or single clicking a button. If the ultrasound probe already includes a gyroscope, the required hardware configuration only includes a button, and the entire 3D reconstruction process can be completed by combining a real-time processing algorithm such as a segmentation algorithm. The method has extremely high applicability and can be used with any existing ultrasound probe.

[0056] According to some embodiments, obtaining at least one intermediate processing result based on the ultrasound image sequence is performed by an image processing device, and wherein the operating accessory is communicatively connected to the image processing device.

[0057] For example, the operating accessory may have a wireless connection capability to connect to an image processing device. As a specific non-limiting embodiment, in the case where the ultrasound probe itself has a posture detection capability or has been attached with a posture detection device such as a gyroscope, the operating accessory may be a simple button with wireless communication capability, which is pressed or otherwise triggered when the user wants to perform three-dimensional reconstruction. After receiving the instruction, the image processing device starts to execute the image processing process and the three-dimensional reconstruction process, and uses the recorded posture data and processing result data of the associated image sequence to obtain a three-dimensional reconstruction result, such as an incomplete intermediate result or a final three-dimensional reconstruction result, for output to the user.

[0058] According to some embodiments, obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing result includes: correcting the posture information based on the intermediate processing result; and obtaining the three-dimensional reconstruction data based on the corrected posture information and the ultrasound image sequence.

[0059] According to some embodiments, obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing results may include: generating prompt information based on the posture information and at least one of the intermediate processing results, the prompt information being used to guide the acquisition of additional ultrasound images, and the additional ultrasound images being used for the three-dimensional reconstruction data.

[0060] For example, the method may include obtaining the three-dimensional reconstruction data based on the ultrasound image sequence and the additional ultrasound image. For another example, the method may include updating or improving the three-dimensional reconstruction data based on the additional ultrasound image.

[0061] According to some embodiments, the intermediate processing result may be related to a first medical analysis target, and the prompt information identifies a missing angle or position of the first medical analysis target in the ultrasound image sequence.

[0062] Exemplarily, the three-dimensional reconstruction data generated during the three-dimensional reconstruction process includes spatial distribution information of lesions or target organs, and provides real-time prompts to identify missing angles or view areas.

[0063] According to some embodiments, the method may further include stitching an ultrasound image sequence according to the intermediate processing results and posture information to generate an intermediate reconstruction result, and dynamically updating a visual prompt in an intermediate reconstruction result display area to guide the user to complete the remaining scanning action.

[0064] According to such an exemplary embodiment, the user can be prompted in real time when the image quality is average, and the current scanning status and even the missing angle area can be indicated in a visual manner.

[0065] In addition, in terms of data storage, the system can save only the key image segments used for 3D reconstruction, thereby reducing storage requirements and optimizing data processing efficiency.

[0066] According to some embodiments, the method may further include, after obtaining the three-dimensional reconstruction data, discarding image portions of the ultrasound image sequence that are not used to generate the three-dimensional reconstruction data.

[0067] Exemplarily, only ultrasound image segments related to the three-dimensional reconstruction may be saved to reduce data storage requirements. For example, only part of the data after the 3D reconstruction is triggered may be saved because this part of the data indicates the user's interest.

[0068] According to one or more embodiments of the present disclosure, an ultrasonic imaging method and apparatus for realizing target three-dimensional volume data reconstruction at low cost can be provided in the field of ultrasonic imaging technology.

[0069] In medical ultrasound imaging, 3D reconstruction technology usually requires precise scanning of the target area and recording of a large amount of spatial information. However, existing technologies often require the use of complex equipment and multiple sensors to measure the complete spatial posture parameters of the ultrasound probe. This not only increases the complexity and cost of the equipment, but also places high demands on actual operation. In addition, the lack of real-time feedback in existing methods may result in reconstruction results that do not fully cover the target area, affecting diagnostic accuracy.

[0070] According to one or more embodiments of the present disclosure, a three-dimensional reconstruction method and device based on an ultrasonic probe are provided to achieve inexpensive and efficient three-dimensional reconstruction of a target area. By introducing real-time processing such as segmentation algorithms and simplified posture measurement, only three posture parameters (α, β, γ) of the probe need to be measured to complete the three-dimensional reconstruction of the target area. At the same time, the reconstruction process is started by a trigger button, making the operation easier and more efficient.

[0071] According to one or more embodiments of the present disclosure, three-dimensional data reconstruction can be performed only for the target area that the user is interested in. When the doctor operates the ultrasound probe, the target area can be identified by a simple scan and the reconstruction process can be triggered.

[0072] According to one or more embodiments of the present disclosure, by introducing real-time processing such as segmentation algorithms, real-time segmentation and splicing can be achieved. For example, the target area can be segmented in real time while acquiring the ultrasound image, and the segmentation results can be spliced ​​into three-dimensional image data in combination with the posture information of the probe. Real-time segmentation can also be used to correct posture data to further improve the accuracy of reconstruction.

[0073] Figure 3A A schematic diagram of an exemplary embodiment according to the present disclosure is shown, in which the image is aligned using a characteristic signal to obtain the position of a specified angle. Because gyro gives the posture, that is, the spatial angle surface angle of each video frame is given. The image information can be used to determine the appropriate position in the slice.

[0074] As an example, the basis for alignment can be to use a certain frame as a reference system, and the adjacent frames will be aligned and positioned with it through image segmentation, image feature matching, etc. Segmentation can determine boundary alignment. Image features can determine content alignment. After the alignment operation, the adjacent frames obtain the appropriate position parameters of their sections. And so on.

[0075] The core of segmentation and image feature matching is to process the characteristic signals in the image to achieve spatial alignment between multiple frames of images, thereby determining the positional relationship at a specific angle. This is an important basic step in the 3D reconstruction process. The gyroscope is used to obtain the posture information of the ultrasound probe, including the spatial angle and direction of each frame of ultrasound image. Through this posture information, a reference direction in 3D space can be provided for each frame of the image, helping to locate the image to the accurate slice position.

[0076] For example, an image segmentation algorithm can be used to align the boundary areas of an image (boundary alignment). For another example, an image feature matching algorithm is used to align the content in an image (content alignment). Both methods can achieve the alignment operation of image frames. After the alignment is completed, adjacent image frames can obtain their accurate slice position parameters in three-dimensional space. Based on this method, the alignment is performed frame by frame and the positional relationship of the remaining frames is derived.

[0077] According to some embodiments, object segmentation is used to identify a region of interest (e.g., a lesion or an organ) and extract features of the region. By identifying and analyzing the image features of the target region, the appropriate position of the target region in a specific posture can be determined, thereby providing an accurate positioning basis for subsequent three-dimensional reconstruction.

[0078] According to one or more embodiments of the present disclosure, segmentation and image feature matching can accurately align and position image frames by combining the posture information provided by the gyroscope. This method can efficiently obtain key parameters for three-dimensional reconstruction, significantly simplify the reconstruction process, and improve reconstruction accuracy and reliability.

[0079] According to one or more embodiments of the present disclosure, the position parameters of the section can be obtained through image alignment and then reconstructed.

[0080] Regarding image alignment, in ultrasound 3D reconstruction, different frames of ultrasound images are collected from different angles, and each frame represents a section. If these sections do not have accurate spatial position information, they cannot be directly used for 3D reconstruction. Image alignment can include using feature information in the image (such as boundaries, textures, or other significant points) or using segmentation algorithms to positionally match images of different sections so that they can be arranged in the correct relative position in 3D space.

[0081] The position of each frame of ultrasound image in three-dimensional space is represented by the position parameters of the slice. These parameters may include, for example: spatial coordinates (X, Y, Z), indicating the position of the slice in three-dimensional space; angle information (such as alpha, beta, gamma), indicating the rotation direction of the slice in space. These parameters can be determined by the posture information provided by the gyroscope and the result of image alignment.

[0082] Once the accurate position parameters of all sections are obtained, these sections can be spliced ​​and superimposed in three-dimensional space to form a three-dimensional reconstruction model of the target object. For example, by three-dimensionally stacking the segmented areas of each section (such as the target organ or lesion), the three-dimensional structure of the target can be reproduced.

[0083] According to one or more embodiments of the present disclosure, by aligning the images (combining image features or segmentation results), the position parameters (position and angle) of each slice in three-dimensional space can be accurately determined. These position parameters are the basis of three-dimensional reconstruction, and they can be used to accurately combine multiple frame image slices into a complete three-dimensional ultrasound model.

[0084] Some examples of intermediate processing results are given above. It is understandable that the intermediate processing results are not limited to the above examples. The intermediate processing results can be any geometry, boundary, object feature, texture, etc. that can be extracted or processed in the image.

[0085] According to one or more embodiments of the present disclosure, a small operating accessory can be provided. According to some exemplary technical solutions, the hardware structure is simple in design, and only a small accessory needs to be installed on the ultrasound probe, and the user can trigger the reconstruction process by long pressing, double-clicking, or clicking a button. The operating accessory is compatible with various existing ultrasound probes, and there is no need to modify the probe itself, so it has strong applicability.

[0086] According to one or more embodiments of the present disclosure, real-time prompts and storage optimization can be achieved. When the image quality of the target area is poor or some angles are not covered, the system can provide real-time visual prompts to guide the doctor to adjust the probe angle to complete the scan. In addition, when storing reconstruction data, the system only saves the ultrasound image segments related to the reconstruction, thereby reducing storage requirements and data processing costs.

[0087] During use, the doctor identifies the target area through a simple scan and presses the button on the probe to start the 3D reconstruction process. At this time, the system uses the gyroscope to record the probe's posture parameters (such as α, β, γ, etc.) and calls the real-time segmentation algorithm to segment the target area. With the support of intermediate processing results and posture information, the system splices the ultrasound image sequence into 3D data and dynamically updates the reconstruction results on the display device. The doctor can adjust the scanning angle according to the real-time prompts to ensure complete coverage of the target area. Finally, the system saves the ultrasound image segments used for reconstruction and generates a complete 3D reconstruction result.

[0088] According to one or more embodiments of the present disclosure, three-dimensional reconstruction of the target area can be completed efficiently and at low cost, providing doctors with clear and intuitive three-dimensional ultrasound images, greatly improving the accuracy and convenience of diagnosis and treatment.

[0089] According to one or more embodiments of the present disclosure, by introducing a real-time segmentation algorithm as a reference, only three parameters of the probe need to be recorded to complete the three-dimensional reconstruction. According to one or more embodiments of the present disclosure, a three-dimensional reconstruction instruction is issued to the system by a user button trigger, thereby significantly simplifying the operation steps and improving the user experience.

[0090] Although the various operations are depicted in the drawings as being in a particular order, this should not be understood as requiring that these operations must be performed in the particular order shown or in a sequential order, nor should it be understood as requiring that all the operations shown must be performed to obtain the desired results. For example, two steps described in order herein may be performed in reverse order, or may be performed concurrently. For another example, one or more steps in the various embodiments of the present disclosure may be omitted.

[0091] In addition, it is understood that the method for predicting or determining data involved in one or more embodiments of the present disclosure is not a method for a doctor to directly determine the diagnosis result, but involves data processing or information processing in the medical process, and the data processing result can be used for reference by the doctor, thereby assisting the doctor's medical operation. It is understood that the information processing method, data prediction method, determination method, decision-making method, etc. involved in one or more embodiments of the present disclosure are executed by a computer or a device including a computer.

[0092] Figure 3B is a schematic block diagram illustrating a medical image processing apparatus 300 according to an exemplary embodiment. The medical image processing apparatus 300 may include an image acquisition unit 310, a posture acquisition unit 320, a processing unit 330, and a reconstruction unit 320. The image acquisition unit 310 may be used to obtain an ultrasound image sequence from the ultrasound probe in response to determining to enter a three-dimensional reconstruction mode. The posture acquisition unit 320 may be used to obtain posture information of the ultrasound probe related to the ultrasound image sequence. The processing unit 330 may be used to obtain at least one intermediate processing result based on the ultrasound image sequence. The reconstruction unit 320 may be used to obtain three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing result.

[0093] It should be understood that Figure 3B The various modules of the apparatus 300 shown in FIG. Figure 2 The steps in the method 200 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 200 and its variants are also applicable to the device 300 and the modules included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0094] According to an embodiment of the present disclosure, a computing device is also disclosed, including a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the medical image processing method according to the embodiment of the present disclosure and its variant examples.

[0095] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is also disclosed, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the medical image processing method according to the embodiment of the present disclosure and its variant examples are implemented.

[0096] According to an embodiment of the present disclosure, a computer program product is also disclosed, including a computer program, wherein when the computer program is executed by a processor, the steps of the medical image processing method according to the embodiment of the present disclosure and its variant examples are implemented.

[0097] Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein may be divided into multiple modules, and / or at least some functions of multiple modules may be combined into a single module. The specific module discussed herein performs an action including the specific module itself performing the action, or alternatively the specific module calls or otherwise accesses another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, the specific module that performs the action may include the specific module itself that performs the action and / or another module that the specific module calls or otherwise accesses to perform the action. For example, the various modules or units described in accordance with one or more embodiments of the present disclosure may be combined into a single module or unit in some embodiments. For another example, two or more modules or units may be described in parallel in one or more embodiments of the present disclosure, while in some other embodiments, there may be one or more inclusion relationships between these modules and units. As used herein, the phrase "entity A initiates action B" or "entity A causes action B to be performed" may refer to entity A issuing an instruction to perform action B, but entity A itself does not necessarily perform the action B. For example, the phrase “the display module causes display ..." may mean that the display module instructs a display (not shown) or other possible display device to display, while the display module itself does not need to perform the action of “displaying”.

[0098] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 3BThe various modules described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of the modules or units described in accordance with one or more embodiments of the present disclosure can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0099] According to one aspect of the present disclosure, a computing device is provided, which includes a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any method embodiment described above.

[0100] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method embodiment described above are implemented.

[0101] According to one aspect of the present disclosure, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the steps of any method embodiment described above are implemented.

[0102] In the following, combined Figure 4 Illustrative examples of such a computer device, non-transitory computer-readable storage medium, and computer program product are described.

[0103] Figure 4 An example configuration of a computer device 400 that can be used to implement the methods described herein is shown. For example, Figure 1 The server 120 and / or the client device 110 shown in FIG. 4 may include an architecture similar to the computer device 400. The above-mentioned medical image processing device / apparatus may also be implemented in whole or at least in part by the computer device 400 or a similar device or system.

[0104] Computer device 400 can be a variety of different types of devices, such as a server of a service provider, a device associated with a client (e.g., a client device), a system on a chip, and / or any other suitable computer device or computing system. Examples of computer device 400 include, but are not limited to, a desktop computer, a server computer, a laptop or netbook computer, a mobile device (e.g., a tablet computer, a cellular or other wireless phone (e.g., a smart phone), a notepad computer, a mobile station), a wearable device (e.g., glasses, a watch), an entertainment device (e.g., an entertainment appliance, a set-top box communicatively coupled to a display device, a game console), a television or other display device, a car computer, and the like. Therefore, computer device 400 can range from a full-resource device with a large amount of memory and processor resources (e.g., a personal computer, a game console) to a low-resource device with limited memory and / or processing resources (e.g., a traditional set-top box, a handheld game console).

[0105] Computer device 400 may include at least one processor 402, memory 404, communication interface(s) 406, a display device 408, other input / output (I / O) devices 410, and one or more mass storage devices 412, all capable of communicating with one another, such as via a system bus 414 or other appropriate connections.

[0106] Processor 402 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. Processor 402 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 402 may be configured to obtain and execute computer-readable instructions stored in memory 404, mass storage device 412, or other computer-readable media, such as program code for operating system 416, program code for application program 418, program code for other programs 420, and the like.

[0107] The memory 404 and the mass storage device 412 are examples of computer-readable storage media for storing instructions that are executed by the processor 402 to implement the various functions described above. For example, the memory 404 may generally include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the mass storage device 412 may generally include a hard drive, a solid-state drive, a removable medium, including external and removable drives, a memory card, a flash memory, a floppy disk, an optical disk (e.g., a CD, a DVD), a storage array, a network attached storage, a storage area network, etc. The memory 404 and the mass storage device 412 may all be collectively referred to herein as memory or computer-readable storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by the processor 402 as a specific machine configured to implement the operations and functions described in the examples herein.

[0108] A number of program modules may be stored on mass storage device 412. These programs include operating system 416, one or more application programs 418, other programs 420, and program data 422, and they may be loaded into memory 404 for execution. Examples of such applications or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing components / functionality including method 200 (including any suitable steps of method 200) and / or additional embodiments described herein.

[0109] Although in Figure 4 404 of the computer device 400, but modules 416, 418, 420, and 422, or portions thereof, may be implemented using any form of computer-readable media accessible by the computer device 400. As used herein, "computer-readable media" includes at least two types of computer-readable media, namely, computer storage media and communication media.

[0110] Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data. Computer storage media includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computer device.

[0111] In contrast, communication media may embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism. Computer storage media as defined herein does not include communication media.

[0112] The computer device 400 may also include one or more communication interfaces 406 for exchanging data with other devices, such as through a network, direct connection, etc., as discussed above. Such communication interfaces may be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), a wired or wireless (such as IEEE802.11 wireless LAN (WLAN)) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a wireless network interface. TM The communication interface 406 may facilitate communication within a variety of network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. The communication interface 406 may also provide for communication with external storage devices (not shown) such as in a storage array, network attached storage, storage area network, etc.

[0113] In some examples, a display device 408 such as a monitor may be included for displaying information and images to the user. Other I / O devices 410 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and the like.

[0114] Although the present disclosure has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative and exemplary rather than restrictive; the present disclosure is not limited to the disclosed embodiments. Variations to the disclosed embodiments will be understood and implemented by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps that are not listed, and the word "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A medical image processing method, comprising: In response to determining to enter the three-dimensional reconstruction mode, obtaining an ultrasound image sequence from the ultrasound probe; Obtaining posture information of the ultrasound probe related to the ultrasound image sequence; Obtaining at least one intermediate processing result based on the ultrasound image sequence; and Three-dimensional reconstruction data of the ultrasound image sequence is obtained based on the posture information and the intermediate processing result.

2. The method according to claim 1, wherein: Obtaining the posture information related to the ultrasound probe includes obtaining the posture information based on a posture detection device associated with the ultrasound probe in response to determining to enter a three-dimensional reconstruction mode.

3. The method according to claim 2, wherein: The posture detection device is a gyroscope.

4. The method according to any one of claims 1 to 3, wherein: Obtaining at least one intermediate processing result based on the ultrasound image sequence includes, in response to determining to enter a three-dimensional reconstruction mode, enabling an image processing process to obtain the at least one intermediate processing result.

5. The method according to claim 4, wherein: The image processing process is a real-time segmentation algorithm.

6. The method according to claim 4 or 5, wherein: The image processing process can be used to determine a segmentation boundary or at least one image feature of at least one object in the sequence of ultrasound images.

7. The method according to any one of claims 1 to 6, wherein: The three-dimensional reconstruction instructions come from an operating accessory removably attached to the ultrasound probe. The method according to claim 7 , wherein the operating attachment comprises a button.

9. The method according to claim 7 or 8, wherein: Obtaining at least one intermediate processing result based on the ultrasound image sequence is performed by an image processing device, and wherein the operating accessory is communicably connected to the image processing device.

10. The method according to any one of claims 1 to 9, wherein: Obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing result includes: Correcting the posture information based on the intermediate processing result; and The three-dimensional reconstruction data is obtained based on the corrected posture information and the ultrasound image sequence.

11. The method according to any one of claims 1 to 10, wherein: Obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing results includes generating prompt information based on at least one of the posture information and the intermediate processing results, wherein the prompt information is used to guide the acquisition of additional ultrasound images, and the additional ultrasound images are used for the three-dimensional reconstruction data.

12. The method according to claim 11, wherein: The intermediate processing result is related to a first medical analysis target, and the prompt information identifies a missing angle or position of the first medical analysis target in the ultrasound image sequence.

13. The method according to any one of claims 1-12, further comprising stitching an ultrasound image sequence according to the intermediate processing results and posture information to generate an intermediate reconstruction result, and dynamically updating a visual prompt in an intermediate reconstruction result display area to guide the user to complete the remaining scanning action. 14 . The method according to claim 1 , further comprising, after obtaining the three-dimensional reconstruction data, discarding image portions in the ultrasound image sequence that are not used to generate the three-dimensional reconstruction data.

15. The method according to any one of claims 1 to 13, wherein: Entering the three-dimensional reconstruction mode is determined based on a three-dimensional reconstruction instruction from a user operating the ultrasound probe.

16. A medical image processing device, comprising: an image acquisition unit, configured to acquire an ultrasound image sequence from the ultrasound probe in response to determining to enter a three-dimensional reconstruction mode; A posture obtaining unit, used for obtaining posture information of the ultrasound probe related to the ultrasound image sequence; a processing unit, configured to obtain at least one intermediate processing result based on the ultrasound image sequence; and A reconstruction unit is used to obtain three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing result.

17. A computing device comprising: a memory, a processor and a computer program stored on the memory, The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images