Medical image processing method and device, computing equipment and storage medium

Abstract

The invention provides a medical image processing method and device, computing equipment and a storage medium. The method may include: in response to determining to enter a three-dimensional reconstruction mode, obtaining an ultrasound image sequence from the ultrasound probe; obtaining attitude information of the ultrasonic probe related to the ultrasonic image sequence; obtaining at least one intermediate processing result based on the ultrasonic image sequence; and obtaining three-dimensional reconstruction data of the ultrasonic image sequence based on the attitude information and the intermediate processing result.

Description

Cross-reference to applications related to medical image processing methods, devices, computing equipment, and storage media

[0001] This application claims priority to Chinese patent application 2025101042902, filed on January 22, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of data processing, and in particular to a medical image processing method, apparatus, computing device, and storage medium. Background Technology

[0003] Currently, doctors often use medical image sequences obtained from medical scanning equipment for medical diagnosis. Among these, ultrasound acquisition equipment is one of the most common medical scanning devices. A method for effectively reconstructing three-dimensional ultrasound images is desired.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] According to one aspect of this disclosure, a medical image processing method is provided, comprising: in response to determining that a three-dimensional reconstruction mode has been entered, obtaining an ultrasound image sequence from the ultrasound probe; obtaining attitude information of the ultrasound probe related to the ultrasound image sequence; obtaining at least one intermediate processing result based on the ultrasound image sequence; and obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the attitude information and the intermediate processing result.

[0006] According to another aspect of this disclosure, a medical image processing apparatus is provided, comprising: an image acquisition unit for acquiring an ultrasound image sequence from the ultrasound probe in response to determining entry into a three-dimensional reconstruction mode; a pose acquisition unit for acquiring pose information of the ultrasound probe related to the ultrasound image sequence; a processing unit for acquiring at least one intermediate processing result based on the ultrasound image sequence; and a reconstruction unit for acquiring three-dimensional reconstruction data of the ultrasound image sequence based on the pose information and the intermediate processing result.

[0007] According to another aspect of this disclosure, a computing device is provided, comprising: a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement a medical image processing method according to one or more embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements a medical image processing method according to one or more embodiments of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, implements a medical image processing method according to one or more embodiments of this disclosure.

[0010] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0011] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: FIG1 is a schematic diagram illustrating an example system in which the various methods described herein may be implemented according to exemplary embodiments; FIG2 is a flowchart illustrating a medical image processing method according to exemplary embodiments; FIG3A is a schematic diagram illustrating medical image processing according to exemplary embodiments; FIG3B is a schematic diagram illustrating cross-sections, lesions, image patches, and pose data according to exemplary embodiments; FIG3C is a schematic diagram illustrating an image processing process according to exemplary embodiments; FIG3D is a schematic diagram illustrating geometric constraints according to exemplary embodiments; FIG3E is a schematic diagram illustrating medical image processing according to exemplary embodiments; FIG3F is a schematic block diagram illustrating a medical image processing apparatus according to exemplary embodiments; and FIG4 is a block diagram illustrating an exemplary computer device applicable to exemplary embodiments. Detailed Implementation

[0012] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0013] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0014] Exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0015] Figure 1 is a schematic diagram illustrating an example system 100 in which various methods described herein may be implemented according to exemplary embodiments.

[0016] Referring to Figure 1, the system 100 includes a client device 110, a server 120, and a network 130 that communicatively couples the client device 110 and the server 120.

[0017] Client device 110 includes a display 114 and a client application (APP) 112 that can be displayed on the display 114. Client application 112 can be an application that needs to be downloaded and installed before running, or a lightweight application (liteapp). If client application 112 is an application that needs to be downloaded and installed before running, client application 112 can be pre-installed on client device 110 and activated. If client application 112 is a mini-app, user 102 can run client application 112 directly on client device 110 without installing it, by searching for client application 112 in the host application (e.g., by the name of client application 112) or by scanning the graphic code of client application 112 (e.g., barcode, QR code, etc.). In some embodiments, client device 110 can be any type of mobile computing device, including mobile computers, mobile phones, wearable computing devices (e.g., smartwatches, head-mounted devices including smart glasses, etc.), or other types of mobile devices. In some embodiments, the client device 110 may alternatively be a stationary computer device, such as a desktop computer, server computer, or other type of stationary computer device. In some alternative embodiments, the client device 110 may also be or may include a medical image printing device.

[0018] Server 120 is typically a server deployed by an Internet Service Provider (ISP) or Internet Content Provider (ICP). Server 120 can represent a single server, a cluster of multiple servers, a distributed system, or a cloud server providing basic cloud services such as cloud databases, cloud computing, cloud storage, and cloud communications. It will be understood that although server 120 is shown communicating with only one client device 110 in Figure 1, server 120 can provide background services to multiple client devices simultaneously.

[0019] Examples of network 130 include combinations of local area networks (LANs), wide area networks (WANs), personal area networks (PANs), and / or communication networks such as the Internet. Network 130 can be wired or wireless. In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to process data exchanged through network 130. Furthermore, encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some of the links. In some embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0020] System 100 may also include an image acquisition device 140. In some embodiments, the image acquisition device 140 shown in FIG1 may be a medical scanning device, including but not limited to scanning or imaging devices used in positron emission tomography (PET), positron emission tomography with computerized tomography (PET / CT), single photon emission computed tomography with computerized tomography (SPECT / CT), computerized tomography (CT), medical ultrasonography, nuclear magnetic resonance imaging (NMRI), magnetic resonance imaging (MRI), cardiovascular angiography (CA), digital radiography (DR), etc. For example, the image acquisition device 140 may include a digital subtraction angiography scanner, a magnetic resonance angiography scanner, a computed tomography angiography scanner, a positron emission tomography (PET) scanner, a positron emission tomography (PET) scanner, a single photon emission tomography (SPT) scanner, a computed tomography (CT) scanner, a medical ultrasound examination device, a magnetic resonance imaging (MRI) scanner, a digital radiography (DRA) scanner, etc. The image acquisition device 140 may be connected to a server (e.g., server 120 in Figure 1 or a separate server of the imaging system, not shown in the figure) to process image data, including but not limited to converting scan data (e.g., converting it into a medical image sequence), compressing it, pixel correction, and three-dimensional reconstruction.

[0021] Image acquisition device 140 may be connected to client device 110, for example, via network 130, or otherwise directly connected to client device to communicate with client device.

[0022] Optionally, the system may also include an intelligent computing device or a computing card 150. The image acquisition device 140 may include or be connected (e.g., detachably connected) to such a computing card 150. As an example, the computing card 150 can perform image data processing, including but not limited to conversion, compression, pixel correction, reconstruction, etc. As another example, the computing card 150 can implement a medical image processing method according to embodiments of the present disclosure.

[0023] The system may also include other components not shown, such as a data storage unit. The data storage unit may be a database, data repository, or other form of device for data storage; it may be a conventional database, or it may include a cloud database, a distributed database, etc. For example, direct image data generated by the image acquisition device 140, or medical image sequences or three-dimensional image data obtained through image processing, may be stored in the data storage unit for subsequent retrieval by the server 120 and client device 110. Furthermore, the image acquisition device 140 may also directly provide direct image data or medical image sequences or three-dimensional image data obtained through image processing to the server 120 or client device 110, etc.

[0024] Users can use client device 110 to control the acquisition of images or videos, view the acquired images or videos (including preliminary image data or images after analysis and processing), view analysis results, interact with the acquired images or analysis results, input acquisition commands, configure data, etc. Client device 110 can send configuration data, commands, or other information to image acquisition device 140 to control image acquisition device acquisition, data processing, etc.

[0025] For the purposes of this embodiment of the disclosure, in the example of FIG1, the client application 112 can be an image sequence management application that can provide various functions, such as storage management, indexing, sorting, and classification of acquired image sequences. Correspondingly, the server 120 can be a server used in conjunction with the image sequence management application. The server 120 can provide image sequence management services to the client application 112 running on the client device 110 based on user requests or instructions generated according to embodiments of this disclosure. For example, it can manage image sequence storage in the cloud, store and classify image sequences according to specified indexes (including, but not limited to, sequence type, patient identifier, body part, acquisition target, acquisition stage, acquisition machine, whether lesions are detected, severity, etc.), and retrieve and provide image sequences to the client device according to specified indexes, etc. Alternatively, the server 120 can also provide or allocate such service capabilities or storage space to the client device 110, and the client application 112 running on the client device 110 can provide corresponding image sequence management services according to user requests or instructions generated according to embodiments of this disclosure, etc. It is understood that the above is only one example, and this disclosure is not limited thereto.

[0026] Figure 2 is a flowchart illustrating a medical image processing method 200 according to an exemplary embodiment. Method 200 can be executed at a client device (e.g., client device 110 shown in Figure 1), that is, the execution entity for each step of method 200 can be the client device 110 shown in Figure 1. In some embodiments, method 200 can be executed at a server (e.g., server 120 shown in Figure 1). In some embodiments, method 200 can be executed in combination by a client device (e.g., client device 110) and a server (e.g., server 120).

[0027] The steps of method 200 are described in detail below.

[0028] Referring to Figure 2, at step 210, in response to determining that the three-dimensional reconstruction mode has been entered, an ultrasound image sequence from the ultrasound probe is obtained.

[0029] At step 220, the orientation information of the ultrasound probe related to the ultrasound image sequence is obtained.

[0030] At step 230, at least one intermediate processing result is obtained based on the ultrasound image sequence.

[0031] At step 240, the three-dimensional reconstruction data of the ultrasound image sequence is obtained based on the posture information and the intermediate processing results.

[0032] The above method allows for the use of intermediate processing results related to the image to assist the 3D reconstruction process after entering the 3D reconstruction mode. On the one hand, additional information is acquired only after entering the 3D reconstruction mode, thus avoiding excessive information processing and resource burden. On the other hand, accurate 3D data can be obtained using the intermediate processing results.

[0033] At least one intermediate processing result may be an intermediate processing result related to at least one object in the image. Exemplarily, at least one intermediate processing result can identify the location information of at least one object in the image. Exemplarily, at least one object may be a lesion, tissue, organ, blood vessel, edge, or other object that can be visualized and / or identified. As an example, the intermediate processing result related to the image may be image segmentation information or segmentation data, but this disclosure is not limited thereto. As a non-limiting specific example, the intermediate processing result may be segmentation boundaries, object identification information, feature information, texture, etc., which will be further described below in conjunction with the accompanying drawings. Due to the special nature of ultrasound data, especially its dependence on user operation and the two-dimensional imaging capability of ultrasound, three-dimensional reconstruction of ultrasound data has always been a challenge. Furthermore, since ultrasound data follows user operation, a large number of ultrasound images are acquired during ultrasound scanning, and processing all such image sequences consumes significant computational resources. According to embodiments of this disclosure, a three-dimensional reconstruction mode can be entered based on a three-dimensional reconstruction command, effectively utilizing the user's (e.g., a doctor's) prior knowledge and performing the three-dimensional reconstruction process based on the doctor's issued three-dimensional reconstruction command, reducing the amount of information that needs to be processed. For example, once a doctor visually identifies a target requiring 3D segmentation or determines that the current area is a region of interest, they issue a 3D reconstruction command. Based on the 3D reconstruction command, pose information is obtained, and the 3D reconstruction process begins. This saves computational resources and allows for the targeted acquisition of the 3D reconstruction results needed by the doctor.

[0034] Exemplarily, the 3D reconstruction command can be an instantaneous command, such as being triggered by pressing a specific button to enter the 3D reconstruction mode. Exemplarily, attitude information can be acquired starting after receiving the 3D reconstruction command. Exemplarily, the ultrasound image sequence to be processed can be an ultrasound image sequence acquired after the 3D reconstruction command, or an ultrasound image sequence that traces back a certain amount of time (e.g., 10s, 30s) or a certain amount of data (e.g., 10 frames, 100 frames, etc.) before the moment the 3D reconstruction command is received, or both. Exemplarily, the 3D reconstruction mode can end upon further user interaction, for example, when the user presses the same button again, clicks the same button with a different gesture, or presses another button indicating the end of the 3D reconstruction state, the 3D reconstruction mode exits, and attitude information and intermediate processing results are no longer acquired.

[0035] For example, the 3D reconstruction command can be a continuous command, such as entering the 3D reconstruction mode when the user presses a specific button or touches a specific touch area, starting to obtain posture information and intermediate processing results, and exiting the 3D reconstruction mode when the user releases the button or leaves the touch area.

[0036] In some other embodiments, the 3D reconstruction mode can be triggered in other ways, such as after detecting that the 3D reconstruction conditions are met, for example, but not limited to, after detecting important lesions, without requiring user instructions.

[0037] According to some embodiments, obtaining attitude information related to the ultrasound probe may include obtaining the attitude information based on a pose detection device associated with the ultrasound probe in response to determining entry into a three-dimensional reconstruction mode.

[0038] In one embodiment, the pose detection device can be activated after entering the 3D reconstruction mode, for example, after receiving a 3D reconstruction command from the user. The pose detection device is capable of recording the pose information related to the ultrasound probe. In other embodiments, the pose detection device can continuously record pose information, but only performs 3D reconstruction of the associated image sequence based on the pose information after entering the 3D reconstruction mode, for example, after receiving a 3D reconstruction command.

[0039] According to some embodiments, the pose detection device can be a gyroscope. For example, a gyroscope can measure spatial parameters of the ultrasound probe, such as rotation angles (α, β, γ), at the acquisition time of the corresponding ultrasound image sequence. Some ultrasound probes already have gyroscopes attached, and such parameters can be used in conjunction with segmentation for 3D reconstruction. In other, more traditional probe scenarios that do not inherently have gyroscopes attached, the pose detection device can be attached to the probe, thereby obtaining pose information related to the ultrasound images without modifying the existing ultrasound probe.

[0040] For example, obtaining at least one intermediate processing result based on the ultrasound image sequence may be obtaining an intermediate processing result regarding at least one identified target or at least one image region involved in the ultrasound image sequence. According to some embodiments, obtaining at least one intermediate processing result based on the ultrasound image sequence includes: in response to determining entry into a three-dimensional reconstruction mode, enabling an image processing process to obtain the intermediate processing result.

[0041] According to some embodiments, the image processing process can be a real-time segmentation algorithm. By introducing a real-time segmentation algorithm, the target region can be segmented in real time while acquiring ultrasound images, and the segmentation results can be stitched together into three-dimensional image data by combining the probe's pose information. As further described below with reference to some exemplary embodiments, real-time segmentation can also be used to correct pose data, further improving the accuracy of reconstruction.

[0042] According to some embodiments, the image processing procedure can be used to determine the segmentation boundary or at least one image feature of at least one object in the ultrasound image sequence. Exemplarily, the method further includes correcting the pose information based on the segmentation boundary.

[0043] At least one object can be an identification target relevant to the current medical analysis purpose, such as a lesion or lesion. The one or more objects used for separation can also be other objects, such as tissues, organs, or other objects with boundaries that can be used for reference.

[0044] Figure 3B is a schematic diagram of cross-sections, lesions, image patches, and attitude data according to an embodiment of the present disclosure. Schematively, four scanning cross-sections with respect to the lesion, corresponding image patches, and corresponding gyroscope attitude data alpha1, alpha2, alpha3, and alpha4 are shown. It will be understood that three-dimensional reconstruction can be performed using the illustrated gyroscope data.

[0045] For example, 3D reconstruction using pose data or gyroscope data may include imposing one or more constraints to avoid jumps in the image sequence. For example, constraints may include geometric feature constraints. Geometric feature constraints may mean that features involved in the image sequence or image patch follow geometric rules, such as features (e.g., lesion boundaries, tissue textures) on adjacent image cross-sections being precisely matched / aligned. For example, constraints may include smoothness constraints on spatial location to avoid jumps.

[0046] As a specific, non-limiting example, the image processing process can include a neural network. Figure 3C illustrates a specific, non-limiting example. As shown in Figure 3C, the neural network can be a two-sub-network, where the input layer receives an image sequence and an angle sequence, the intermediate layers extract and fuse features respectively, and the output layer directly generates reconstructed data or correction parameters. The input of the illustrated two-stream network architecture can include a first video stream input, which may correspond to an ultrasound image sequence, and the input can include a second angle stream input, which may include gyroscope angle data (alpha1, alpha2, …, alphan) corresponding one-to-one with video frames. An exemplary network structure can include a first sub-network and a second sub-network, where the first sub-network can receive the video stream input and extract deep features in the image dimension. The second sub-network can receive the angle stream input and extract features in the motion / pose dimension. The network can also include a fusion and output section for fusing the features of the two sub-networks to output the final reconstructed volume data or corrected position sequence.

[0047] For example, referring to Figure 3C, the first input can be an image input, such as n image patches or an image sequence as input. The second input can be attitude data, such as attitude data from a gyroscope. For example, each slice can have corresponding gyroscope data (alpha_1, alpha_2, alpha_3, dots) indicating the acquisition angle of the image slice.

[0048] The first feature extraction subnetwork, or video stream subnetwork, can extract high-dimensional features such as texture and tissue boundaries within the image through convolutional layers. The second feature extraction subnetwork, or angle stream subnetwork, can process angle sequences through an encoder to extract the motion trend features of the probe. The features from the two branches can be combined in a fusion layer, comprehensively considering both "what the image looks like" and "how the probe rotates." The final output is the planar displacement (dx, dy) of each frame relative to the previous frame or reference frame.

[0049] The output or optimization objective can be (dx, dy) for each image patch, such as a correction offset in the XY plane, or a translation correction parameter of an image slice relative to its gyroscope pose, for precise alignment. Exemplarily, as previously described, the loss function or network optimization objective can include a smoothing term and a matching loss. The smoothing term can be used to constrain the smoothness of spatial locations, ensuring smooth reconstructed edges, avoiding abrupt changes, and constraining the output (dx, dy) to prevent abrupt changes. The matching loss can be used to ensure that features (such as lesion boundaries and tissue textures) on adjacent image slices can be precisely matched / aligned, and can be constrained at the continuous frame level and at the image feature level. As an example, the matching loss can be characterized as a feature matching score, used to calculate the overlap of image features after spatially transforming the original image using the predicted (dx, dy), as a supervision signal.

[0050] According to some embodiments, the 3D reconstruction commands originate from an operating accessory removably attached to the ultrasound probe. Exemplarily, the operating accessory includes buttons or other interactive interfaces.

[0051] In practice, doctors can easily identify target areas when scanning with an ultrasound probe. To achieve this, a specific sector scan can be performed on the target area, and the 3D reconstruction process can be triggered by the user clicking a button. During this process, only three parameters of the probe in space, namely α, β, and γ, need to be measured using a gyroscope to obtain attitude information. For example, the user only needs to press a button to recognize and start the 3D reconstruction operation, and at the same time begin to acquire the attitude information of the ultrasound probe. Simultaneously, the system will initiate a real-time segmentation process of the target area and process it in conjunction with the angle parameters of the intermediate processing results. Through these intermediate processing results, features such as segmentation or other object features can be stitched together in 3D space to complete the 3D reconstruction of the target area. In addition, the intermediate processing results can also be used to correct the probe's attitude information. Through the above method, the system can build a 3D model of lesions or organs in real time and display missing or complete information from different angles.

[0052] According to embodiments of this disclosure, reconstruction parameters can be obtained using only pose information, such as without position information. Reliable 3D reconstruction parameters are generated by combining image segmentation algorithms and gyroscope pose data, thereby achieving 3D reconstruction of the target region.

[0053] Segmentation algorithms can identify the boundary information of target regions, such as the outline of lesions or organs, from ultrasound images. These intermediate processing results can not only be used to stitch together and generate 3D models, but also to correct the probe's pose information, improving the accuracy of reconstruction.

[0054] According to one or more embodiments, a gyroscope provides the probe's angular information in space (e.g., α, β, γ), and these three parameters describe the probe's rotational state. These parameters allow for the determination of the probe's pose in three-dimensional space, reducing the complexity and hardware cost of traditional six-degree-of-freedom (α, β, γ, x, y, z) pose measurements. Therefore, by combining intermediate processing results with gyroscope data, key parameters for 3D reconstruction can be generated under conditions of simple hardware and low cost. The reliability and accuracy of these parameters directly affect the effectiveness of 3D reconstruction and are the core innovation and technical advantage of this invention.

[0055] For example, this technical solution can be implemented by adding a small accessory. For instance, by attaching the operating accessory to the ultrasound probe, the user can issue reconstruction commands by manually rotating the probe and by pressing, double-clicking, or single-clicking a button. If the ultrasound probe already includes a gyroscope, the required hardware configuration consists only of buttons, and the entire 3D reconstruction process can be completed by combining it with real-time processing algorithms such as segmentation algorithms. This method has extremely high applicability and can be used with any existing ultrasound probe.

[0056] According to one or more embodiments, displacement data in at least the x and y planes can also be supervised by one or more constraints such as smoothing constraints, local co-construction search, cross-image constraints, elastic deformation compensation, etc., thereby further reducing the computational cost of the algorithm and enabling more efficient and accurate acquisition of 3D reconstruction results.

[0057] According to some embodiments, obtaining at least one intermediate processing result based on the ultrasound image sequence is performed by an image processing device, wherein the operating accessory is communicatively connected to the image processing device.

[0058] For example, the operating accessory can have wireless connectivity to connect to an image processing device. As a specific, non-limiting embodiment, if the ultrasound probe itself has pose detection capabilities or is already attached to a pose detection device such as a gyroscope, the operating accessory can be a simple button with wireless communication capabilities. When the user wants to perform 3D reconstruction, pressing the button or triggering it in other ways will cause the image processing device to receive the instruction and begin executing the image processing and 3D reconstruction processes. Using the recorded pose data and processing result data from the associated image sequence, it obtains the 3D reconstruction result, which may be an incomplete intermediate result or the final 3D reconstruction result, for output to the user.

[0059] According to some embodiments, obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing results includes: correcting the posture information based on the intermediate processing results; and obtaining the three-dimensional reconstruction data based on the corrected posture information and the ultrasound image sequence.

[0060] According to some embodiments, obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing results may include: generating prompting information based on at least one of the posture information and the intermediate processing results, the prompting information being used to guide the acquisition of additional ultrasound images, the additional ultrasound images being used for the three-dimensional reconstruction data.

[0061] For example, the method may include obtaining the three-dimensional reconstruction data based on the ultrasound image sequence and the additional ultrasound images. As another example, the method may include updating or improving the three-dimensional reconstruction data based on the additional ultrasound images.

[0062] Figure 3D illustrates a specific embodiment of a mechanism for assisting 3D reconstruction using prior geometric constraints according to an embodiment of the present disclosure. In such an embodiment, prior geometric constraints may be additionally or alternatively used to correct displacement. On the user side, a reference scan may be performed first, whereby the user scans once along a certain direction (e.g., longitudinal) to obtain one or more reference sections (e.g., section P0). The reference section does not need to be strictly perpendicular to the main section, as long as it forms a certain angle with the subsequent main scan direction to obtain section contour information. The user can then perform a main scan, advancing the scan along another direction (e.g., transverse, approximately perpendicular to or at a significant angle to the first step direction) to obtain a sequence of images (e.g., P1, P2, ..., P5). It is understood that the execution order of the reference scan and the main scan is not limited to this; for example, the main scan may be performed before the reference scan, or the orientation of the ultrasound probe may be changed during the main scan to obtain one or more reference scan results, and the present disclosure is not limited thereto.

[0063] At the algorithmic level, feature intersections can be calculated. For example, a P0 section can be obtained first, which may contain the longitudinal contour of anatomical structures (e.g., but not limited to tissues, organs, feature curves). An exemplary constraint mechanism could be to intersect the P0 section in space with transverse sections (P1 to P5) in the main scan. This allows for displacement correction. For example, one or more features can be extracted from the image of P0, such as feature curves (e.g., the wavy lines shown in the lower half of the figure), or features of other shapes. The extracted features can be tissues, lesions, organs, blood vessels, or any other texture or feature of interest. Thus, if the algorithm-calculated P1 position (x, y, z) causes its feature points to deviate from the reference curve provided by P0, one or more of x, y, and z can be adjusted accordingly until they match. According to such an embodiment, a complex degree-of-freedom search problem can be simplified to a "connect-the-dots" problem—that is, mounting a sequence of images onto a known reference contour line—enabling accurate 3D reconstruction with a simplified algorithm.

[0064] In another preferred embodiment of this disclosure, the method utilizes the cross-geometric constraints generated by multi-directional scanning to obtain or correct the 3D reconstruction data. This is a reconstruction strategy with "low computational cost and high accuracy." First, reference information can be obtained to guide the user to perform a first scanning action, obtaining at least one reference section image (e.g., P0). This reference section contains the contour features of the target object (e.g., lesion, organ) in a first direction (e.g., the feature curve shown in the illustration). Next, sequence information can be obtained to guide the user to perform a second scanning action, obtaining a sequence of main scan images (e.g., P1-P5). The advancement direction of the second scan forms a non-zero angle with the direction of the first scan (preferably approximately orthogonal / perpendicular, e.g., an angle of 60-90 degrees). Next, a displacement calculation system based on cross constraints can identify feature trajectories (e.g., tissue boundary curves) in the reference section image. Exemplarily, for each frame in the main scan image sequence, the system determines its spatial intersection point with the reference section.

[0065] An exemplary position correction logic could be to match the corresponding feature point in the main scan image with the feature trajectory in the reference section. If the two do not coincide spatially, the spatial coordinate parameters (x, y, z) of the current frame image are adjusted until the feature point falls on the feature trajectory.

[0066] According to embodiments of this disclosure, this "scan twice with an angle" operation utilizes the scan data from the first step as the "skeleton" or "ruler" for the second step of sequence stitching. This allows the system to significantly reduce the search range for (dx, dy) estimation and significantly improve stitching accuracy by relying solely on gyroscope angle data, while eliminating the need for users to perform extremely precise, slow scans.

[0067] According to one or more embodiments of this disclosure, the method further includes outputting a prompt message to a user, the prompt message guiding the user to perform at least two scans with an angle. The method may also include constraining the spatial position of an image sequence obtained from a second scan based on feature contours in a reference section obtained from a first scan, and calculating or correcting 3D reconstructed data by spatially aligning feature points in the image sequence with the feature contours.

[0068] Referring back to Figure 2, according to one or more embodiments of this disclosure, the step of obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing result may include: performing non-rigid registration on the ultrasound image sequence or the intermediate processing result to compensate for or correct tissue elastic deformation caused by ultrasound probe pressure, thereby eliminating the influence of tissue deformation differences on the accuracy of three-dimensional reconstruction.

[0069] Referring back to Figure 2, according to one or more embodiments of this disclosure, the obtained ultrasound image sequence may include at least one reference ultrasound image scanned along a first direction and at least two scanning ultrasound images scanned along a second direction, wherein the first direction and the second direction are not parallel. Exemplarily, the first direction is perpendicular to the second direction. Exemplarily, the first direction and the second direction may be approximately perpendicular, for example, within a range of ±5 degrees. Exemplarily, the first direction and the second direction may form an angle, for example, an angle greater than 30 degrees, greater than 60 degrees, or greater than 80 degrees. Exemplarily, the angle between the first direction and the second direction may be determined based on the shape of the region of interest.

[0070] In some embodiments, obtaining at least one intermediate processing result based on the ultrasound image sequence may include identifying at least one region of interest (ROI) related to the target human body. The ROI may be an organ, tissue, or other object with a certain contour, geometric features, or shape. It is understood that the ROI may be applicable as an alignment reference and may not necessarily be the object to be analyzed in the ultrasound image sequence. For example, in medical scans targeting small lesions (such as breast or thyroid nodules), the tissue to be identified may be the boundary of the small lesion itself; however, the ROI may be other features that facilitate alignment, such as, but not limited to, the contours of the chest wall and ribs, the boundaries of abdominal organs, or stable anatomical structures such as large blood vessels or deep fascia lines.

[0071] In such an example, obtaining the 3D reconstruction data of the ultrasound image sequence based on the pose information and the intermediate processing results may include obtaining spatial position constraints of the at least two scanned ultrasound images based on the positions of at least one geometric feature of the region of interest in the reference ultrasound image and the positions of corresponding geometric features of the region of interest in the at least two scanned ultrasound images; and obtaining the 3D reconstruction data based on the spatial position constraints and the pose information. It is understood that the geometric feature can be a geometric contour. Obtaining the 3D reconstruction data based on the spatial position constraints and the pose information may involve constraining the spatial position of each frame in the main scanned image sequence based on the feature contour in the reference cross-sectional image, and calculating or correcting the 3D reconstruction data by spatially aligning the corresponding feature points in the main scanned image sequence with the feature contour.

[0072] In some embodiments, the method may further include: determining the first ultrasound image as the reference ultrasound image in response to determining that at least a first ultrasound image has been obtained along the first direction and that the first ultrasound image contains the outline of at least one object of interest; and outputting a visual cue for indicating scanning along a second direction different from the first direction.

[0073] In some embodiments, the 3D reconstruction data is obtained based on cross-geometric constraints generated by multi-directional scanning. Exemplarily, the method may include outputting prompts to a user, guiding the user to perform at least two scans with non-zero angles to obtain: a. at least one reference section image (P0) containing contour features of the target object in a first direction; and b. a sequence of master scan images (P1-P5) whose advancing direction forms a non-zero angle (preferably approximately orthogonal) with the first direction. The method may include constraining the spatial position of each frame in the master scan image sequence based on the feature contours in the reference section image. The method may include calculating or correcting the 3D reconstruction data by spatially aligning corresponding feature points in the master scan image sequence with the feature contours.

[0074] According to one or more embodiments of this disclosure, the step of calculating or correcting spatial position by aligning feature points with feature contours aims to simplify a complex multi-degree-of-freedom search problem into a contour trajectory-based matching problem, thereby reducing the search range for spatial coordinate parameters (x, y, z) and significantly improving stitching accuracy.

[0075] Figure 3E illustrates a schematic diagram of a visual cue according to an exemplary embodiment of the present disclosure. As shown in Figure 3E, the blue scan area shown in the figure can be output to the user as a reference first. In this case, when fanning the red area in the direction of the arrow, the difficulty of "aligning" the image to reconstruct the 3D data is greatly reduced, and the probability of errors is reduced. It is understood that the figures shown are merely non-limiting examples, and the present disclosure is not limited thereto.

[0076] According to one or more embodiments of this disclosure, a method for 3D reconstruction based on gyroscope angles and real-time segmentation / intermediate results is provided.

[0077] According to one or more embodiments of this disclosure, the accuracy and rationality of 3D reconstruction can be guaranteed by constraining the predicted displacement (dx, dy) based on the neural network with smoothing and / or matching loss functions.

[0078] According to one or more embodiments of this disclosure, geometric intersection constraints can be applied based on two scans with an existing angle, thereby achieving accurate and desired 3D reconstruction. Exemplarily, a set of image sections with angular variations (pose variations) can be obtained through simple fan-shaped sweeps, and aligned and stitched using the geometric constraints between them.

[0079] According to some embodiments, the intermediate processing result may be related to a first medical analysis target, and the prompt information indicates the angle or position of the missing first medical analysis target in the ultrasound image sequence.

[0080] For example, the three-dimensional reconstruction data generated during the three-dimensional reconstruction process includes spatial distribution information of lesions or target organs, and provides real-time prompts to identify missing angles or view areas.

[0081] According to some embodiments, the method may further include stitching together an ultrasound image sequence based on the intermediate processing results and posture information to generate an intermediate reconstruction result, and dynamically updating visual prompts in the intermediate reconstruction result display area to guide the user to complete the remaining scanning actions.

[0082] According to such an exemplary embodiment, it is possible to provide real-time prompts to the user even when the image quality is average, and to visually indicate the current scan status or even missing angular areas.

[0083] In addition, in terms of data storage, the system can save only the key image segments used for 3D reconstruction, thereby reducing storage requirements and optimizing data processing efficiency.

[0084] According to some embodiments, the method may further include discarding image portions of the ultrasound image sequence that were not used to generate the three-dimensional reconstruction data after obtaining the three-dimensional reconstruction data.

[0085] For example, only the ultrasound image segments relevant to 3D reconstruction can be saved to reduce data storage requirements. For instance, only the portion of data triggered after 3D reconstruction can be saved, as this portion indicates the user's interest.

[0086] According to one or more embodiments of this disclosure, an ultrasound imaging method and apparatus for inexpensively reconstructing three-dimensional volume data of a target can be provided in the field of ultrasound imaging technology.

[0087] In medical ultrasound imaging, 3D reconstruction techniques typically require precise scanning of the target area and recording of a large amount of spatial information. However, existing techniques often require complex equipment and multiple sensors to measure the complete spatial pose parameters of the ultrasound probe. This not only increases the complexity and cost of the equipment but also places high demands on practical operation. Furthermore, the lack of real-time feedback in existing methods may lead to reconstruction results that do not fully cover the target area, affecting diagnostic accuracy.

[0088] According to one or more embodiments of this disclosure, a three-dimensional reconstruction method and apparatus based on an ultrasonic probe are provided to achieve inexpensive and efficient three-dimensional reconstruction of a target area. By introducing real-time processing, such as segmentation algorithms and simplified pose measurement, the three-dimensional reconstruction of the target area can be completed by measuring only three pose parameters (α, β, γ) of the probe. Furthermore, the reconstruction process can be initiated by a trigger button, making operation simpler and more efficient.

[0089] According to one or more embodiments of this disclosure, three-dimensional data reconstruction can be performed only for a target area of ​​interest to the user. When operating the ultrasound probe, the doctor can identify the target area through a simple scan and trigger the reconstruction process.

[0090] According to one or more embodiments of this disclosure, real-time segmentation and stitching can be achieved by introducing real-time processing, such as segmentation algorithms. For example, the target region can be segmented in real-time while acquiring ultrasound images, and the segmentation results can be stitched into three-dimensional image data by combining the probe's pose information. Real-time segmentation can also be used to correct pose data, further improving the accuracy of reconstruction.

[0091] Figure 3A illustrates a schematic diagram of an exemplary embodiment according to the present disclosure, showing the alignment of an image using feature signals to obtain a position at a specified angle. Since gyro provides the pose, i.e., the spatial angle plane angle of each video frame, the image information can be used to determine the appropriate position within a slice.

[0092] As an example, alignment can be based on a certain frame as a reference. Adjacent frames are aligned and positioned using image segmentation, image feature matching, and other methods. Segmentation determines boundary alignment. Image features determine content alignment. After alignment, adjacent frames obtain the appropriate position parameters for their cross-sections. Other methods follow the same logic.

[0093] The core of segmentation and image feature matching lies in processing the feature signals in the image to achieve spatial alignment between multiple frames, thereby determining the positional relationship at a specific angle. This is a crucial foundational step in the 3D reconstruction process. A gyroscope is used to acquire the attitude information of the ultrasound probe, including the spatial angle and orientation of each frame of the ultrasound image. This attitude information provides a reference orientation for each frame in 3D space, helping to locate the image at an accurate slice position.

[0094] For example, image segmentation algorithms can be used to align the boundary regions of an image (boundary alignment). Similarly, image feature matching algorithms can be used to align the content within an image (content alignment). Both methods can achieve image frame alignment. After alignment, adjacent image frames can obtain their accurate slice position parameters in three-dimensional space. Based on this method, frame-by-frame alignment is performed, and the positional relationships of the remaining frames are derived.

[0095] According to some embodiments, target segmentation is used to identify regions of interest (e.g., lesions or organs) and extract features from those regions. By identifying and analyzing the image features of the target region, appropriate location points of the target region in a specific pose can be determined, thereby providing accurate localization basis for subsequent 3D reconstruction.

[0096] According to one or more embodiments of this disclosure, segmentation and image feature matching, by combining pose information provided by a gyroscope, can accurately align and locate image frames. This method can efficiently acquire key parameters for 3D reconstruction, significantly simplify the reconstruction process, and improve reconstruction accuracy and reliability.

[0097] According to one or more embodiments of this disclosure, the positional parameters of the cross section can be obtained through image alignment, and then reconstructed.

[0098] Regarding image alignment, in ultrasound 3D reconstruction, ultrasound images from different frames are acquired from different angles, and each frame represents a cross-section. If these cross-sections lack accurate spatial location information, they cannot be directly used for 3D reconstruction. Image alignment can involve using feature information in the images (such as boundaries, textures, or other salient points) or using segmentation algorithms to match the positions of different cross-section images, enabling them to be arranged in the correct relative positions in 3D space.

[0099] The position of each ultrasound image frame in three-dimensional space is represented by the position parameters of the slice. These parameters may include, for example, spatial coordinates (X, Y, Z), representing the position of the slice in three-dimensional space; and angular information (e.g., alpha, beta, gamma), representing the rotation direction of the slice in space. These parameters can be determined jointly by the attitude information provided by the gyroscope and the results of image alignment.

[0100] Once the accurate location parameters of all the cross-sections are obtained, these cross-sections can be stitched together and superimposed in three-dimensional space to form a three-dimensional reconstruction model of the target object. For example, by stacking the segmented regions (such as target organs or lesions) of each cross-section in three dimensions, the three-dimensional structure of the target can be reproduced.

[0101] According to one or more embodiments of this disclosure, by aligning images (combining image features or segmentation results), the positional parameters (position and angle) of each slice in three-dimensional space can be accurately determined. These positional parameters are the basis for three-dimensional reconstruction, and can be used to accurately combine multiple image slices into a complete three-dimensional ultrasound model.

[0102] The above provides some examples of intermediate processing results. It is understood that intermediate processing results are not limited to these examples. Intermediate processing results can be any geometric, boundary, object features, textures, etc., that can be extracted or processed from an image.

[0103] According to one or more embodiments of this disclosure, a small operating accessory can be provided. According to some exemplary technical solutions, the hardware structure design is simple, requiring only a small accessory to be installed on the ultrasound probe. The user can trigger the reconstruction process by long-pressing, double-clicking, or single-clicking a button. The operating accessory is compatible with various existing ultrasound probes without requiring modification to the probe itself, thus offering strong applicability.

[0104] According to one or more embodiments of this disclosure, real-time prompts and optimized storage can be achieved. When the image quality of the target area is poor or certain angles are not covered, the system can provide real-time visual prompts to guide the doctor to adjust the probe angle to complete the scan. Furthermore, when storing reconstructed data, the system only saves ultrasound image segments relevant to the reconstruction, thereby reducing storage requirements and data processing costs.

[0105] During use, the doctor identifies the target area through a simple scan and presses a button on the probe to initiate the 3D reconstruction process. At this time, the system uses a gyroscope to record the probe's pose parameters (e.g., α, β, γ) and calls a real-time segmentation algorithm to segment the target area. With the support of intermediate processing results and pose information, the system stitches the ultrasound image sequence into 3D data and dynamically updates the reconstruction results on the display device. The doctor can adjust the scanning angle according to real-time prompts to ensure complete coverage of the target area. Finally, the system saves the ultrasound image segments used for reconstruction and generates a complete 3D reconstruction result.

[0106] According to one or more embodiments of this disclosure, three-dimensional reconstruction of a target area can be completed efficiently and at low cost, providing doctors with clear and intuitive three-dimensional ultrasound images, and greatly improving the accuracy and convenience of diagnosis and treatment.

[0107] According to one or more embodiments of this disclosure, by introducing a real-time segmentation algorithm as a reference, 3D reconstruction can be completed by recording only three parameters of the probe. According to one or more embodiments of this disclosure, 3D reconstruction commands are issued to the system via a user button trigger, thereby significantly simplifying the operation steps and improving the user experience.

[0108] According to one or more embodiments of this disclosure, the pose information obtained by the method described above may also be referred to as preliminary pose information, and the method according to embodiments of this disclosure may include fine-tuning the pose information using one or more constraints, including consecutive frame constraints and boundary smoothing constraints. Exemplarily, after obtaining the preliminary pose information, an optimal solution may be searched within a local spatial range (such as a 3×3 or 7×7 range) to ensure that the edges of the stitched image features of adjacent frames—including, for example, but not limited to, features of the target and surrounding tissues—are smooth and free of jagged edges.

[0109] For example, the first constraint can be referred to as a continuous frame constraint, or a spatial / temporal constraint. The first constraint can be based on the consideration that, given the continuity of ultrasound scans, adjacent frames cannot be too far apart spatially or have too drastic abrupt transitions. For example, the second constraint can be referred to as a boundary smoothness constraint or a morphological constraint. The second constraint can be based on the consideration that, based on anatomical common sense, the edges of tissues or organs should be smooth and should not exhibit jagged edges.

[0110] For example, a local search can be performed within a small area in three-dimensional space (called the search window, such as a range of 3×3×3 or 7×7×7 pixels / voxels) to find the optimal position parameters (x, y, z) that satisfy the above constraints. For example, this can be achieved by establishing a cost function to find the optimal solution.

[0111] Exemplarily, the features in the intermediate processing results according to embodiments of this disclosure may include not only the target object itself or the segmentation boundary of the target object, but also the texture features of the surrounding tissue. In such an example, the algorithm may not only be based on simple segmentation, but may also utilize features of the target combined with those of the surrounding tissue for joint anchoring based on a wider range of image features such as texture, gradient, etc. Exemplarily, the algorithm may seek a solution that minimizes spatial jumps.

[0112] Although the various operations are depicted in the accompanying drawings in a specific order, this should not be construed as requiring that these operations must be performed in the specific order shown or in chronological order, nor should it be construed as requiring that all the operations shown must be performed to obtain the desired result. For example, two steps described in order herein may be performed in reverse order or may be performed concurrently. As another example, one or more steps in the various embodiments of this disclosure may be omitted.

[0113] Furthermore, it is understood that the methods for predicting or determining data according to one or more embodiments of this disclosure are not methods for doctors to directly determine diagnostic results, but rather involve data processing or information processing processes during the medical process. The data processing results can be used for doctors' reference, thereby assisting doctors in their medical operations. It is understood that the information processing methods, data prediction methods, determination methods, decision-making methods, etc., according to one or more embodiments of this disclosure are executed by a computer or a device containing a computer.

[0114] Figure 3F is a schematic block diagram illustrating a medical image processing apparatus 300 according to an exemplary embodiment. The medical image processing apparatus 300 may include an image acquisition unit 310, a pose acquisition unit 320, a processing unit 330, and a reconstruction unit 320. The image acquisition unit 310 may be used to acquire an ultrasound image sequence from the ultrasound probe in response to determining that a three-dimensional reconstruction mode has been entered. The pose acquisition unit 320 may be used to acquire pose information of the ultrasound probe related to the ultrasound image sequence. The processing unit 330 may be used to acquire at least one intermediate processing result based on the ultrasound image sequence. The reconstruction unit 320 may be used to acquire three-dimensional reconstruction data of the ultrasound image sequence based on the pose information and the intermediate processing result.

[0115] It should be understood that the various modules of the apparatus 300 shown in FIG. 3F can correspond to the various steps in the method 200 described with reference to FIG. 2. Therefore, the operations, features, and advantages described above for method 200 and its variations also apply to apparatus 300 and its included modules. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0116] According to embodiments of the present disclosure, a computing device is also disclosed, including a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the medical image processing method and variations thereof according to embodiments of the present disclosure.

[0117] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium is also disclosed, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the medical image processing method and its variations according to embodiments of the present disclosure.

[0118] According to embodiments of the present disclosure, a computer program product is also disclosed, including a computer program, wherein when the computer program is executed by a processor, it implements the steps of the medical image processing method and variations thereof according to embodiments of the present disclosure.

[0119] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein may be divided into multiple modules, and / or at least some functions of multiple modules may be combined into a single module. The specific module discussed herein performing an action includes the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action may include the specific module performing the action itself and / or another module that the specific module calls or otherwise accesses to perform the action. For example, the various modules or units described according to one or more embodiments of this disclosure may be combined into a single module or unit in some embodiments. As another example, two or more modules or units may be described in parallel in one or more embodiments of this disclosure, while in other embodiments, these modules and units may have one or more inclusion relationships. As used herein, the phrase "entity A initiates action B" or "entity A causes action B to be performed" may refer to entity A issuing an instruction to perform action B, but entity A itself does not necessarily perform action B. For example, the phrase "display module causes display..." could mean that the display module instructs a display (not shown) or other possible display device to display, without the display module itself needing to perform the "display" action.

[0120] It should also be understood that various techniques can be described herein in the general context of software hardware elements or program modules. The various modules described above with respect to Figure 3F can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, in some embodiments, one or more of the modules or units described according to one or more embodiments of this disclosure can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components of other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0121] According to one aspect of this disclosure, a computing device is provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0122] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0123] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0124] The following description, in conjunction with Figure 4, illustrates illustrative examples of such computer devices, non-transitory computer-readable storage media, and computer program products.

[0125] Figure 4 illustrates an example configuration of computer device 400 that can be used to implement the methods described herein. For example, server 120 and / or client device 110 shown in Figure 1 may include an architecture similar to computer device 400. The aforementioned medical image processing apparatus / device may also be implemented wholly or at least partially by computer device 400 or similar devices or systems.

[0126] Computer device 400 can be a variety of different types of devices, such as a service provider's server, a device associated with a client (e.g., a client device), a system-on-a-chip, and / or any other suitable computer device or computing system. Examples of computer device 400 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablets, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on. Therefore, the range of computer device 400 can be from full-resource devices with large amounts of memory and processor resources (e.g., personal computers, game consoles) to low-resource devices with limited memory and / or processing resources (e.g., traditional set-top boxes, handheld game consoles).

[0127] Computer device 400 may include at least one processor 402, memory 404, multiple communication interfaces 406, display device 408, other input / output (I / O) devices 410, and one or more mass storage devices 412 capable of communicating with each other, such as via system bus 414 or other suitable connections.

[0128] Processor 402 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 402 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 402 may be configured to acquire and execute computer-readable instructions stored in memory 404, mass storage device 412, or other computer-readable media, such as program code of operating system 416, program code of application program 418, program code of other program 420, etc.

[0129] Memory 404 and mass storage device 412 are examples of computer-readable storage media for storing instructions executed by processor 402 to perform the various functions described above. For example, memory 404 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 412 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 404 and mass storage device 412 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 402 as a specific machine configured to perform the operations and functions described in the examples herein.

[0130] Multiple program modules may be stored on mass storage device 412. These programs include operating system 416, one or more application programs 418, other programs 420, and program data 422, and they may be loaded into memory 404 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing components / functions including method 200 (including any suitable steps of method 200) and / or other embodiments described herein.

[0131] Although illustrated in Figure 4 as stored in memory 404 of computer device 400, modules 416, 418, 420, and 422, or portions thereof, may be implemented using any form of computer-readable medium accessible by computer device 400. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer storage media and communication media.

[0132] Computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transfer medium that can be used to store information for access by computer equipment.

[0133] In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data within modulated data signals such as carrier waves or other transmission mechanisms. Computer storage media as defined herein do not include communication media.

[0134] Computer device 400 may also include one or more communication interfaces 406 for exchanging data with other devices, such as via a network, direct connection, etc., as discussed above. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces, near field communication (NFC) interfaces, etc. Communication interface 406 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 406 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0135] In some examples, a display device 408, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 410 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0136] Although this disclosure has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative and suggestive, not restrictive; this disclosure is not limited to the disclosed embodiments. By studying the drawings, the disclosure, and the appended claims, those skilled in the art will be able to understand and implement variations of the disclosed embodiments in practicing the claimed subject matter. In the claims, the word "comprising" does not exclude other elements or steps not listed, and the words "a" or "an" do not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be beneficial.

Claims

1. A medical image processing method, comprising: In response to determining that the system is entering a three-dimensional reconstruction mode, an ultrasound image sequence from the ultrasound probe is obtained; Obtain the orientation information of the ultrasound probe related to the ultrasound image sequence; obtain at least one intermediate processing result based on the ultrasound image sequence; and obtain three-dimensional reconstruction data of the ultrasound image sequence based on the orientation information and the intermediate processing result.

2. The method according to claim 1, wherein, Obtaining attitude information related to the ultrasound probe includes obtaining the attitude information based on a pose detection device associated with the ultrasound probe in response to determining entry into a three-dimensional reconstruction mode.

3. The method according to claim 2, wherein, The pose detection device is a gyroscope.

4. The method according to any one of claims 1-3, wherein, Obtaining at least one intermediate processing result based on the ultrasound image sequence includes enabling an image processing process in response to determining entry into a three-dimensional reconstruction mode to obtain the at least one intermediate processing result.

5. The method according to claim 4, wherein, The image processing process is a real-time segmentation algorithm.

6. The method according to claim 4 or 5, wherein, The image processing procedure can be used to determine the segmentation boundary of at least one object or at least one image feature in the ultrasound image sequence.

7. The method according to any one of claims 1-6, wherein, The three-dimensional reconstruction commands are derived from an operating accessory that is removably attached to the ultrasound probe.

8. The method according to claim 7, wherein the operating accessory includes a button.

9. The method according to claim 7 or 8, wherein, Obtaining at least one intermediate processing result based on the ultrasound image sequence is performed by an image processing device, wherein the operating accessory is communicatively connected to the image processing device.

10. The method according to any one of claims 1-9, wherein, Obtaining the three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing results includes: correcting the posture information based on the intermediate processing results; and obtaining the three-dimensional reconstruction data based on the corrected posture information and the ultrasound image sequence.

11. The method of claim 10, wherein, Correcting the attitude information based on the intermediate processing results includes: determining a cost function, the cost function including at least one of the following: a smoothing constraint term, used to constrain the spatial continuity of attitude or position parameters of adjacent sections in the ultrasound image sequence, and a feature matching term, used to measure the alignment degree of features of adjacent sections in the ultrasound image sequence; and finding correction parameters that minimize the cost function within a predetermined local spatial range to obtain the corrected attitude information.

12. The method according to claim 11, wherein, The correction parameters include translation correction parameters dx and dy for each ultrasound image section, and wherein the correction parameters are obtained through a dual-input deep learning network configured to use the ultrasound image sequence and the angle sequence corresponding to each frame of the ultrasound image sequence as input.

13. The method according to any one of claims 1-12, wherein, Obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing results includes generating prompt information based on at least one of the posture information and the intermediate processing results. The prompt information is used to guide the acquisition of additional ultrasound images, which are used in the three-dimensional reconstruction data.

14. The method according to claim 13, wherein, The intermediate processing result is related to the first medical analysis target, and the prompt information indicates the angle or position where the first medical analysis target is missing in the ultrasound image sequence.

15. The method according to any one of claims 1-14, further comprising stitching together an ultrasound image sequence based on the intermediate processing result and posture information to generate an intermediate reconstruction result, and dynamically updating visual prompts in the intermediate reconstruction result display area to guide the user to complete the remaining scanning actions.

16. The method according to any one of claims 1-15, further comprising, after obtaining the three-dimensional reconstruction data, discarding image portions of the ultrasound image sequence that were not used to generate the three-dimensional reconstruction data.

17. The method according to any one of claims 1-16, wherein, Entering 3D reconstruction mode is determined based on the 3D reconstruction command from the user operating the ultrasound probe.

18. The method according to any one of claims 1-17, wherein, Obtaining the three-dimensional reconstruction data of the ultrasound image sequence based on the posture information and the intermediate processing results further includes: performing non-rigid registration on the ultrasound image sequence or the intermediate processing results to compensate for or correct the elastic deformation of tissue caused by the pressure of the ultrasound probe, and to eliminate the influence of tissue deformation differences on the accuracy of three-dimensional reconstruction.

19. The method according to any one of claims 1-18, wherein, The obtained ultrasound image sequence includes at least one reference ultrasound image scanned along a first direction and at least two scanning ultrasound images scanned along a second direction, wherein the first direction is not parallel to the second direction.

20. The method according to claim 19, wherein, The first direction is perpendicular to the second direction.

21. The method according to claim 19 or 20, wherein, Obtaining at least one intermediate processing result based on the ultrasound image sequence includes identifying at least one region of interest (ROI) related to the target human body, and wherein obtaining three-dimensional reconstruction data of the ultrasound image sequence based on the pose information and the intermediate processing result includes: obtaining spatial position constraints of the at least two scanned ultrasound images based on the position of at least one geometric feature of the ROI in the reference ultrasound image and the position of the corresponding geometric feature of the ROI in the at least two scanned ultrasound images; and obtaining the three-dimensional reconstruction data based on the spatial position constraints and the pose information.

22. The method according to any one of claims 19-21, further comprising: In response to determining that at least a first ultrasound image has been obtained along the first direction and that the first ultrasound image contains the outline of at least one object of interest, the first ultrasound image is determined as the reference ultrasound image; And output visual cues, which are used to indicate scanning along a second direction different from the first direction.

23. A medical image processing device, comprising: An image acquisition unit is configured to acquire a sequence of ultrasound images from the ultrasound probe in response to determining that a three-dimensional reconstruction mode has been entered. A pose acquisition unit is used to acquire pose information of the ultrasound probe related to the ultrasound image sequence; a processing unit is used to acquire at least one intermediate processing result based on the ultrasound image sequence; and a reconstruction unit is used to acquire three-dimensional reconstruction data of the ultrasound image sequence based on the pose information and the intermediate processing result.

24. A computing device, comprising: A memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1-22.

25. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-22.

26. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-22.

Images