Systems and methods for fetal monitoring

Abstract

The present disclosure provides systems and methods for fetal monitoring. The methods can include obtaining ultrasound data related to a fetus collected by an ultrasound imaging device; generating a 4D image of the fetus based on the ultrasound data; instructing a display component of an extended reality device to display the 4D image to an operator; detecting a motion of the fetus based on the ultrasound data; and instructing a haptic component of the extended reality device to provide haptic feedback to the operator regarding the motion.

Description

Technical Field

[0001] This disclosure generally relates to the field of fetal monitoring, and more specifically to systems and methods for fetal monitoring using virtual reality technology. Background Technology

[0002] Users, such as new parents, may experience excitement by observing fetal movement and hearing the fetal heartbeat collected via, for example, ultrasound imaging. In current clinical settings, ultrasound images or videos are projected onto conventional monitors or sent to remote devices such as smartphones or computers. These devices are 2D devices that cannot present true 3D images. Furthermore, users cannot interact with the fetus via such conventional devices. Therefore, there is a need to provide systems and methods for fetal monitoring that allow users to visually understand the fetus's condition, hear the fetal heartbeat, and / or feel fetal movement. Summary of the Invention

[0003] According to one aspect of this disclosure, a system for fetal monitoring is provided. The system may include: at least one storage device storing a set of instructions; and at least one processor configured to communicate with the at least one storage device. When executable instructions are executed, the at least one processor may be configured to instruct the system to perform one or more of the following operations: The system may acquire ultrasound data related to the fetus collected by an ultrasound imaging device. The system may generate at least one 4D image of the fetus based on the ultrasound data and instruct a display component of an extended reality (XR) device, such as a virtual reality (VR) device, to display the at least one 4D image to an operator. The system may also detect fetal movement based on the ultrasound data and instruct a haptic component of the XR device, such as a VR device, to provide tactile feedback regarding the movement to the operator.

[0004] In some embodiments, generating at least one 4D image of the fetus based on ultrasound data may include: generating a plurality of initial 3D images based on ultrasound data related to the fetus; generating a plurality of 3D fetal images by segmenting portions representing the fetus from the respective initial 3D images; and generating at least one 4D image based on the plurality of 3D fetal images.

[0005] In some embodiments, generating at least one 4D image based on multiple 3D fetal images may include: extracting mesh surfaces from the respective 3D fetal images; and rendering at least one 4D mesh surface including multiple mesh surfaces to generate at least one 4D rendered image.

[0006] In some embodiments, at least one 4D image may include a first 3D fetal image and a second 3D fetal image captured prior to the first 3D fetal image. Detecting fetal motion based on ultrasound data may include: determining a vertex correspondence between a plurality of first vertices on a first grid surface and a plurality of second vertices on a second grid surface, the first grid surface representing the first 3D fetal image and the second grid surface representing the second 3D fetal image; and for each of the plurality of first vertices, determining motion information from its corresponding second vertex to the first vertex based on the vertex correspondence.

[0007] In some embodiments, determining the vertex correspondence between a plurality of first vertices on a first mesh surface and a plurality of second vertices on a second mesh surface may include: determining a motion field between a first 3D fetal image and a second 3D fetal image; and determining the vertex correspondence based on the motion field.

[0008] In some embodiments, the sports field can be determined based on optical flow-based techniques or a sports field determination model.

[0009] In some embodiments, determining the vertex correspondence between a plurality of first vertices on a first mesh surface and a plurality of second vertices on a second mesh surface may include: generating a first point cloud corresponding to a first 3D fetal image and a second point cloud corresponding to a second 3D fetal image; and determining the vertex correspondence by registering the first point cloud to the second point cloud.

[0010] In some embodiments, haptic feedback may include a feedback force. The system may determine the magnitude of the feedback force by: obtaining user interaction information about at least one 4D image, the user interaction information including at least one target region of the at least one 4D image to which the user interaction is directed; determining the magnitude of a vertex force based on motion information of each of a plurality of target first vertices in a portion of a first grid surface corresponding to the target region; and determining the magnitude of the feedback force based on the magnitude of the vertex forces of the plurality of target first vertices.

[0011] In some embodiments, the system may instruct the speaker to play sounds related to the fetus.

[0012] In some embodiments, the ultrasound imaging device may be a 4D ultrasound imaging device.

[0013] According to another aspect of this disclosure, a method for fetal monitoring is provided. The method can be implemented on a computing device having at least one processor and at least one storage device. The method may include: acquiring fetal-related ultrasound data collected by an ultrasound imaging device; generating at least one 4D image of the fetus based on the ultrasound data; and instructing a display component of an extended reality (XR) device to display the at least one 4D image to an operator. The method may further include: detecting fetal movement based on the ultrasound data; and instructing a tactile component of the extended reality device to provide tactile feedback regarding the movement to the operator.

[0014] According to another aspect of this disclosure, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include at least one set of instructions for fetal monitoring. When executed by at least one processor of a computing device, the at least one set of instructions may instruct the at least one processor to perform operations including: acquiring fetal-related ultrasound data collected by an ultrasound imaging device; generating at least one 4D image of the fetus based on the ultrasound data; instructing a display component of an extended reality (XR) device to display the at least one 4D image to an operator; detecting fetal movement based on the ultrasound data; and instructing a tactile component of the extended reality device to provide tactile feedback regarding the movement to the operator.

[0015] Additional features will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon review of the following and the accompanying drawings, or may be learned by the generation or operation of examples. The features of this disclosure can be realized and obtained by practice or by using various aspects of the methods, apparatus, and combinations thereof set forth in the detailed examples discussed below. Attached Figure Description

[0016] This disclosure is further described with reference to exemplary embodiments. These exemplary embodiments are described in detail with reference to the accompanying drawings. The drawings are not to scale. These embodiments are non-limiting exemplary embodiments, wherein the same reference numerals denote similar structures in several views of the drawings, and in the drawings:

[0017] Figure 1 This is a schematic diagram illustrating an exemplary fetal monitoring system according to some embodiments of the present disclosure;

[0018] Figure 2 This is a block diagram illustrating an exemplary processing apparatus according to some embodiments of the present disclosure;

[0019] Figure 3 This is a flowchart illustrating an exemplary process for fetal monitoring according to some embodiments of the present disclosure;

[0020] Figure 4This is a flowchart illustrating an exemplary process for determining fetal motion information based on fetal-related ultrasound data according to some embodiments of the present disclosure;

[0021] Figure 5 This is a flowchart illustrating an exemplary process for determining fetal motion information based on fetal-related ultrasound data, according to some embodiments of the present disclosure; and

[0022] Figure 6 This is a flowchart illustrating an exemplary process for determining the magnitude of a feedback force according to some embodiments of the present disclosure. Detailed Implementation

[0023] The following description is provided to enable those skilled in the art to make and use this disclosure, and is given in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this disclosure. Thus, this disclosure is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the claims.

[0024] These and other features and characteristics of this disclosure, as well as the methods of operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, will become more apparent when considered in conjunction with the accompanying drawings, all of which form part of this disclosure. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this disclosure. It should be understood that the drawings are not to scale.

[0025] The flowcharts used in this disclosure illustrate operations performed by the system according to some embodiments of this disclosure. It should be clearly understood that the operations in the flowcharts may not be performed sequentially. Instead, the operations may be performed in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0026] In this disclosure, the term "image" can refer to a two-dimensional (2D) image, a three-dimensional (3D) image, or a four-dimensional (4D) image (e.g., a time series of 3D images). In some embodiments, the term "image" can refer to an image of a region of a subject (e.g., a region of interest (ROI)). In some embodiments, an image can be a medical image, an optical image, etc.

[0027] In this disclosure, for the sake of brevity, a representation of a subject (e.g., a fetus) in an image may be referred to as "the subject". Further, for the sake of brevity, an image including a representation of the subject may be referred to as an image of the subject or an image including the subject. Still further, for the sake of brevity, an operation performed on a representation of the subject in an image may be referred to as an operation performed on the subject. For example, for the sake of brevity, segmentation of a portion of an image including a representation from a region of interest (ROI) in the image may be referred to as segmentation of the ROI.

[0028] One aspect of this disclosure relates to systems and methods for fetal monitoring using virtual reality technology. For example, the system can acquire fetal-related ultrasound data collected by an ultrasound imaging device. The system can generate a 4D image of the fetus based on the ultrasound data. The system can instruct the display component of an extended reality (XR) device, such as a virtual reality (VR) device, to display the 4D image to an operator. The system can also detect fetal movement based on the ultrasound data and instruct the tactile component of the extended reality device to provide tactile feedback regarding the movement to the operator.

[0029] According to some embodiments of this disclosure, a 3D or 4D image of the fetus can be presented to a user (e.g., the fetus's parents) via a display component of an augmented reality device, allowing the user to intuitively perceive the fetus's state. Additionally, the user can interact with the haptic components of the augmented reality device, enabling them to feel the fetus's movements, thereby improving the user experience.

[0030] Figure 1 This is a schematic diagram illustrating an exemplary fetal monitoring system according to some embodiments of the present disclosure. For example... Figure 1 For example, fetal monitoring system 100 may include an ultrasound imaging device 110, a processing device 120, and an extended reality (XR) device 130. Components in fetal monitoring system 100 may be connected in one or more of various ways. By way of example only, ultrasound imaging device 110 can be connected via a network (…). Figure 1 (Not shown) is connected to the processing unit 120. As another example, the ultrasound imaging device 110 can be directly connected to the processing unit 120, such as... Figure 1 For example, as another example, the augmented reality device 130 may be connected via a network to another component of the fetal monitoring system 100 (e.g., processing device 120). As yet another example, the augmented reality device 130 may be directly connected to the ultrasound imaging device 110 and / or the processing device 120, such as... Figure 1 Example.

[0031] The ultrasound imaging device 110 can be configured to collect ultrasound data (or ultrasound imaging data) relating to at least a portion of a subject (e.g., a fetus). The subject can be biological or non-biological. For example, the subject can include a patient, an artificial subject, etc. As another example, the subject can include a specific part, organ, and / or tissue of a patient. For example, the subject can include a fetus or a portion thereof, including, for example, the fetal head, chest, neck, thoracic cavity, heart, stomach, arm, palm, blood vessels, soft tissue, tumor, nodule, etc., or any combination thereof. For illustrative purposes, a fetus may be considered an example of a subject in this disclosure.

[0032] In some embodiments, the ultrasound imaging device 110 may use the physical properties of ultrasound and the differences in acoustic properties of different regions of the fetus to obtain ultrasound data of the fetus. The ultrasound data may be in the form of waveforms, curves, or images to display and / or record fetal-related features. For example, the ultrasound imaging device 110 may include one or more ultrasound probes for transmitting ultrasound waves to the pregnant woman's abdomen. Ultrasound waves may produce different reflections and attenuations after passing through organs and tissues with different acoustic impedances and attenuation characteristics, thereby forming echoes that can be received by one or more ultrasound probes. The ultrasound imaging device 110 may process (e.g., amplify, convert) and / or display the received echoes to generate ultrasound data. In some embodiments, the ultrasound imaging device 110 may include a B-mode ultrasound device, a color Doppler ultrasound device, a three-dimensional color Doppler ultrasound device, a four-dimensional color Doppler ultrasound device, etc., or any combination thereof.

[0033] Processing device 120 can process data and / or information obtained from ultrasound imaging device 110, extended reality device 130, and / or any other components (e.g., storage devices for storing data and / or information obtained from ultrasound imaging device 110). In some embodiments, processing device 120 can host a simulated virtual world or a virtual reality environment of extended reality device 130. For example, processing device 120 can generate a 4D image of the fetus based on ultrasound data related to the fetus collected by ultrasound imaging device 110. Processing device 120 can instruct a display component of extended reality device 130 to display the 4D image (including a series of 3D images) to an operator (e.g., new parents). As another example, processing device 120 can detect fetal movement based on ultrasound data. Processing device 120 can instruct a tactile device (e.g., a component of extended reality device 130) to provide tactile feedback about the movement to the operator.

[0034] In some embodiments, the processing device 120 may be a computer, a user console, a single server, or a group of servers, etc. The server group may be centralized or distributed. For example, a designated area of ​​virtual reality may be simulated by a single server. In some embodiments, the processing device 120 may include multiple simulation servers dedicated to physics simulation to manage interactions and handle collisions between characters and objects in the virtual reality.

[0035] In some embodiments, the processing device 120 may be local to or remote from the fetal monitoring system 100. For example, the processing device 120 may access information and / or data from the ultrasound imaging device 110 via a network. As another example, the processing device 120 may be directly connected to the ultrasound imaging device 110 to access information and / or data. In some embodiments, the processing device 120 may be implemented on a cloud platform. For example, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, multi-cloud, etc., or combinations thereof.

[0036] In some embodiments, the processing device 120 may include a storage device dedicated to storing data relating to objects and characters in the virtual reality world. The data stored in the storage device may include object shapes, avatar shapes and appearances, audio clips, virtual reality-related scripts, and other virtual reality-related objects. In some embodiments, the processing device 120 may be implemented by a computing device having a processor, storage, input / output (I / O), communication ports, etc. In some embodiments, the processing device 120 may be implemented on the processing circuitry (e.g., processor, CPU) of the extended reality device 130.

[0037] The augmented reality device 130 can be a device that allows users to participate in an augmented reality experience. In some embodiments, the augmented reality device 130 may include an XR helmet, XR glasses, XR patches, stereoscopic head-mounted displays, or any combination thereof. For example, the augmented reality device 130 may include Google Glass. TM Oculus Rift TM Gear VR TM Specifically, the extended reality device 130 may include a display component 131 on which virtual content can be rendered and displayed. Users can view the virtual content (e.g., a 3D or 4D image of a fetus) via the display component 131. The extended reality device 130 may also be an immersive system, a type of extended reality system that provides an immersive experience. The visual effects presented by an immersive system do not require eye-aiding devices, such as glasses or helmets; in other words, the immersive system can achieve a naked-eye 3D effect.

[0038] In some embodiments, a user can interact with virtual content via display component 131. For example, when a user wears display component 131, the user's head movements and / or gaze direction can be tracked, allowing virtual content to be rendered in response to changes in the user's position and / or orientation to provide an immersive and compelling extended reality experience that reflects changes in the user's perspective.

[0039] In some embodiments, the extended reality device 130 may further include an input component 132. The input component 132 enables user interaction between the user and virtual content displayed on the display component 131 (e.g., virtual content related to or representing a fetus). For example, the input component 132 may include a touch sensor, microphone, etc., configured to receive user input, which may be provided to the extended reality device 130 and used to control the virtual world by changing the visual content rendered on the display component. In some embodiments, the user input received by the input component may include, for example, touch, voice input, and / or gesture input, and may be sensed via any suitable sensing technology (e.g., capacitive, resistive, acoustic, optical). In some embodiments, the input component 132 may include a handle, glove, stylus, game console, etc.

[0040] In some embodiments, display component 131 (or processing device 120) may track input component 132 and render virtual elements based on the tracking of input component 132. Virtual elements may include a representation of input component 132 (e.g., an image of a user's hand or fingers). Virtual elements may be rendered in a 3D location corresponding to the real-world location of input component 132 within an extended reality experience. For example, one or more sensors may be used to track input component 132. Display component 131 may receive signals collected by one or more sensors from input component 132 via a wired or wireless network. Signals may include any suitable information enabling tracking of input component 132, such as outputs from one or more inertial measurement units (e.g., accelerometers, gyroscopes, magnetometers) in input component 132, a Global Positioning System (GPS) sensor in input component 132, or combinations thereof. Signals may indicate the position (e.g., in three-dimensional coordinates) and / or orientation (e.g., in three-dimensional rotational coordinates) of input component 132. In some embodiments, sensors may include one or more optical sensors for tracking input component 132. For example, the sensor may use a visible light and / or a depth camera to locate the input component 132.

[0041] In some embodiments, input component 132 may include a tactile component capable of providing tactile feedback to a user. For example, a user can feel fetal movement through feedback force provided by the tactile component. The tactile component may include multiple force sensors, motors, and / or actuators. Force sensors may measure the magnitude and direction of the force applied by the user and input these measurements to processing device 120. Processing device 120 may convert the input measurements into movement of one or more virtual elements (e.g., virtual fingers, virtual hands, etc.) that can be displayed on display component 131. Processing device 120 may then calculate one or more interactions between the one or more virtual elements and at least a portion of the fetus and output the interactions as computer signals (i.e., signals representing feedback force). The motors or actuators in the tactile component may apply feedback force to the user based on the computer signals received from processing device 120, allowing the user to feel fetal movement. In some embodiments, the magnitude of the feedback force may be set according to the default settings of the fetal monitoring system 100 or preset by the user or operator via, for example, a terminal device (e.g., extended reality device 130). Alternatively, the magnitude of the feedback force may be determined based on the magnitude of fetal movement. When a user wears the haptic device and detects fetal movement, the motors and / or actuators in the haptic device can apply feedback force to the user.

[0042] In some embodiments, the fetal monitoring system 100 may further include an audio device (not shown) configured to provide audio signals to a user. For example, the audio device (e.g., a speaker) may play sounds related to the fetus (e.g., fetal heartbeat sounds). In some embodiments, the audio device may include an electromagnetic speaker (e.g., a moving-coil speaker, a balanced-iron speaker, etc.), a piezoelectric speaker, an electrostatic speaker (e.g., a condenser speaker), or any combination thereof. In some embodiments, the audio device may be integrated into the extended reality device 130. In some embodiments, the extended reality device 130 may include two audio devices located on the left and right sides of the extended reality device 130, respectively, to provide audio signals to the user's left and right ears.

[0043] It should be noted that the above description of the fetal monitoring system 100 is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Various changes and modifications can be made based on the teachings of this disclosure by those skilled in the art. For example, the assembly and / or functionality of the fetal monitoring system 100 may vary or be altered depending on the specific implementation scenario. In some embodiments, the fetal monitoring system 100 may include one or more additional components (e.g., storage devices, networks, etc.), and / or one or more components of the fetal monitoring system 100 described above may be omitted. Alternatively or additionally, two or more components of the fetal monitoring system 100 may be integrated into a single component. Components of the fetal monitoring system 100 may be implemented on two or more sub-components.

[0044] Figure 2 This is a block diagram illustrating exemplary processing apparatuses according to some embodiments of the present disclosure. For example... Figure 2 For example, the processing device 120 may include an acquisition module 210, an image generation module 220, a motion detection module 230, and a control module 240. A module may be all or part of the hardware circuitry of the processing device 120. A module may also be implemented as an application or instruction set read and executed by the processing device 120. Further, a module may be any combination of hardware circuitry and applications / instructions. For example, when the processing device 120 executes an application / instruction set, the module may be part of the processing device 120.

[0045] The acquisition module 210 can be configured to acquire ultrasound data related to the fetus collected by the ultrasound imaging device.

[0046] The image generation module 220 can be configured to generate at least one 4D image of the fetus based on ultrasound data.

[0047] The motion detection module 230 can be configured to detect fetal motion based on ultrasound data.

[0048] Control module 240 can be configured to instruct the display component of the extended reality (XR) device to display at least one 4D image to the operator. In some embodiments, control module 240 can also be configured to instruct the haptic component of the extended reality device to provide haptic feedback about movement to the operator. Further description of fetal monitoring can be found elsewhere in this disclosure (e.g., Figures 3 to 6 (and its description) were found.

[0049] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Various changes and modifications can be made by those skilled in the art based on the teachings of this disclosure. However, these changes and modifications do not depart from the scope of this disclosure. In some embodiments, the above-described modules may be divided into two or more units. For example, the control module 240 may be divided into two units, one of which may be configured to instruct the display component of the extended reality device 130 to display a 4D image of the fetus to the operator, while the other may be configured to instruct the haptic component of the extended reality device 130 to provide haptic feedback regarding movement to the operator. In some embodiments, the processing device 120 may include one or more additional modules, such as a storage module (not shown) for storing data.

[0050] Figure 3 This is a flowchart illustrating an exemplary process for fetal monitoring according to some embodiments of the present disclosure. In some embodiments, process 300 may be implemented as a set of instructions (e.g., an application) stored in a storage device. Processing device 120 (e.g., in...) Figure 2 The instruction set can be executed (implemented on one or more of the illustrated modules), and when the instructions are executed, the processing device 120 can be configured to execute process 300. The operation of the illustrated process 300 presented below is intended to be illustrative. In some embodiments, process 300 may be accomplished using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 3 The order of operations of the illustrated and described process 300 below is not intended to be limiting.

[0051] In 310, the processing device 120 (e.g., acquisition module 210) can acquire ultrasound data related to the fetus collected by the ultrasound imaging device.

[0052] Ultrasound data can be collected by scanning a pregnant woman carrying a fetus using one or more ultrasound probes of an ultrasound imaging device (e.g., a 3D ultrasound imaging device). In some embodiments, ultrasound data can be obtained directly from an ultrasound imaging device (e.g., ultrasound imaging device 110). In some embodiments, ultrasound data can be obtained from a storage device. For example, the ultrasound imaging device can transfer the acquired ultrasound data to a storage device for storage. The processing device 120 can obtain ultrasound data from the storage device. In some embodiments, ultrasound data may include 2D ultrasound data, 3D ultrasound data, etc.

[0053] In some embodiments, the ultrasound data may be real-time data collected during an examination of the pregnant woman. The collection, transmission, and post-processing of the ultrasound data may be performed substantially simultaneously, enabling the user (e.g., an operator of the extended reality device described below) to understand the real-time status of the fetus. Alternatively, the ultrasound data may be historical data collected from previous examinations of the pregnant woman. For example, process 300 may be performed after the pregnant woman's ultrasound examination is completed.

[0054] In some embodiments, the processing device 120 may perform preprocessing operations on the ultrasound data. Exemplary preprocessing operations may include noise reduction operations, enhancement operations, filtering operations, and any combination thereof.

[0055] In some embodiments, for example, if the processing device 120 is outside the examination room where the ultrasound imaging device is located, the ultrasound data (or pre-processed ultrasound data) can be encrypted during transmission over a network (e.g., a wired network, a wireless network). The processing device 120 can then decrypt the encrypted ultrasound data and perform subsequent processing.

[0056] In some embodiments, the processing device 120 may also acquire fetal heartbeat sounds via a sound collector (e.g., a Doppler fetal monitor). The sound collector may be part of an ultrasound imaging apparatus or a separate device. The heartbeat sounds may be streamed to the processing device 120 along with the ultrasound data.

[0057] In 320, the processing device 120 (e.g., image generation module 220) can generate a 4D image of the fetus based on ultrasound data. The 4D image of the fetus may include multiple 3D images of the fetus corresponding to different time points (also referred to as 3D fetal images). Thus, the 4D image of the fetus can dynamically display the state of the fetus.

[0058] In some embodiments, the processing device 120 can generate multiple 3D fetal images by segmenting the fetus from an initial 3D image. The initial 3D images may be generated based on ultrasound data and include representations of other organs and / or tissues of the fetus and the pregnant woman. In some embodiments, if the ultrasound data includes 3D ultrasound data, the initial 3D images can be generated directly based on the 3D ultrasound data using an image reconstruction algorithm. If the ultrasound data includes 2D ultrasound data, the processing device 120 can generate multiple 2D images based on the 2D ultrasound data. The processing device 120 can also reconstruct the initial 3D images based on the multiple 2D images using a 3D reconstruction algorithm or a 3D reconstruction model. Exemplary 3D reconstruction algorithms may include boundary contour-based algorithms, non-uniform rational B-spline (NURBS)-based algorithms, triangulation model-based algorithms, etc. Exemplary 3D reconstruction models may include convolutional neural network (CNN) models, deep CNN (DCNN) models, fully convolutional network (FCN) models, recurrent neural network (RNN) models, etc., or any combination thereof.

[0059] The processing device 120 can generate multiple 3D fetal images by segmenting portions representing the fetus from each initial 3D image. The processing device 120 can also generate 4D images based on the multiple 3D fetal images. For example, the processing device 120 can sequentially arrange the multiple 3D fetal images according to their respective capture times.

[0060] In some embodiments, a segmentation algorithm can be used to segment the fetus from various initial 3D images. Exemplary segmentation algorithms may include threshold-based segmentation algorithms, compression-based algorithms, edge detection algorithms, machine learning-based segmentation algorithms, and any combination thereof. In some embodiments, a segmentation model can be used to segment the fetus from various initial 3D images. The segmentation model can be trained based on multiple sets of training data. Each set of training data may include sample initial 3D images and corresponding training labels (e.g., 3D fetal images, segmentation masks). The processing device 120 may input multiple initial 3D images into the segmentation model to determine multiple 3D fetal images corresponding to the multiple initial 3D images. In some embodiments, the segmentation model may include a convolutional neural network (CNN) model, a deep CNN (DCNN) model, a fully convolutional network (FCN) model, a recurrent neural network (RNN) model, and any combination thereof, or may be a network based on transformer results.

[0061] In some embodiments, after generating 3D fetal images, processing device 120 may extract mesh surfaces from the individual 3D fetal images. The mesh surface may include a set of vertices, edges, and faces defining the 3D shape of the fetus. Processing device 120 may render a 4D mesh surface including multiple mesh surfaces to generate a 4D rendered image (e.g., by performing one or more visual rendering operations on it). In some embodiments, processing device 120 may use a moving cubes algorithm to extract mesh surfaces from the 3D fetal images. In some embodiments, the mesh surface of the individual 3D fetal images may be a low-resolution mesh surface for faster computation in a real-time setting. The low-resolution mesh surface of the 3D fetal images may use a relatively small number of vertices (e.g., less than a threshold) to represent the fetus. In some embodiments, visual rendering operations may include visual transformations, color operations, lighting operations, texture mapping operations, animation effects operations, etc., or combinations thereof.

[0062] In some embodiments, the processing device 120 may send at least one 3D fetal image to a 3D printing device to print a physical baby shape.

[0063] In 330, the processing device 120 (e.g., control module 240) can instruct the display component of the extended reality (XR) device to display a 4D image to the operator.

[0064] The processing device 120 can transmit the rendering result (i.e., 4D image) to an extended reality device (e.g., the display component of the extended reality device 130) for display.

[0065] In some embodiments, if the extended reality device's display components include a first display component corresponding to the left eye and a second display component corresponding to the right eye, the processing device 120 can render a first image corresponding to a first-eye view and a second image corresponding to a second-eye view based on 4D images. The processing device 120 can instruct the first display component to display the first image to an operator (e.g., the parents of a fetus) and instruct the second display component to display the second image to the operator (e.g., the parents of a fetus). For example, the first image may correspond to the left-eye view and be displayed by the first display component worn on the operator's left eye, and the second image may correspond to the right-eye view and be displayed by the second display component worn on the operator's right eye.

[0066] In some embodiments, the processing device 120 may acquire user interaction information regarding the displayed 4D image. The user interaction information may relate to user instructions that an operator wishes to input. For example, the user interaction information may include motion data corresponding to a part of the user's body (such as a hand, head, eyes, neck, etc.). The processing device 120 may update the display of the 4D image based on the user interaction information. Based on the user interaction information, the processing device 120 may perform operations on the 4D image, such as rotating, moving, zooming in, zooming out, etc. As another example, the user interaction information may relate to user interaction pointing to a target part of the fetus. Based on the user interaction information, the processing device 120 may determine the magnitude of a feedback force to represent the movement of the target part and instruct the haptic components of the extended reality device to provide the user with a feedback force of the determined magnitude. In some embodiments, the user interaction information may be collected via an interactive device such as a handle, touchscreen, or microphone. For example, a user may input user interaction information via an interactive device by typing, speaking, touching, drawing, etc.

[0067] In some embodiments, user interaction information can be collected via a display component using an object tracking algorithm. For example, when a user wears the display component, the user's head movements and / or gaze direction can be tracked, allowing a 4D image of the fetus to be rendered in response to changes in the user's position and / or orientation, providing an immersive and compelling extended reality experience that reflects changes in the user's perspective. In some embodiments, the object tracking algorithm may include Kalman filter tracking, smoothing filter tracking, kernel correlation filter (KCF) tracking, circulant structure of tracking-by-detection with kernel (CSK) tracking, color name (CN) tracking, and any combination thereof.

[0068] In some embodiments, the processing device 120 may instruct a speaker to play the sound of a fetal heartbeat while displaying a 4D image. In some embodiments, the processing device 120 may generate a sound that mimics the sound of a fetus and instruct the speaker to play that sound.

[0069] In 340, the processing device 120 (e.g., motion detection module 230) can detect fetal movement based on ultrasound data.

[0070] In some cases, the fetus may move between two points in time during the examination. Fetal movement can be reflected by motion information between two 3D fetal images generated based on ultrasound data collected at the two time points. For example, fetal movement can be represented by fetal movement information from a second time point (e.g., a historical time point) to a first time point (e.g., the current time point), which can be determined based on a first 3D fetal image corresponding to the first time point and a second 3D fetal image corresponding to the second time point. In some embodiments, motion information may include velocity, acceleration, displacement, etc.

[0071] In some embodiments, if the 4D image includes a first 3D fetal image and a second 3D fetal image captured prior to the first 3D fetal image, the processing device 120 may determine motion information from the second 3D fetal image to the first 3D fetal image based on a first mesh surface representing the first 3D fetal image and a second mesh surface representing the second 3D fetal image. The mesh surface of the 3D fetal image may include a plurality of vertices. Specifically, the processing device 120 may determine the motion information based on vertex correspondences between a plurality of first vertices of the first mesh surface and a plurality of second vertices of the second mesh surface. Further description of determining motion information based on vertex correspondences can be found elsewhere in this disclosure (e.g., Figure 4 and Figure 5 (and its description) were found.

[0072] In some embodiments, the processing device 120 may perform post-processing operations on the detected motion. For example, fetal motion may be filtered by a filter (e.g., a Kalman filter) to smooth the motion. As another example, fetal motion may be amplified so that the operator can perceive the fetal motion more clearly.

[0073] In 350, the processing device 120 (e.g., control module 240) can instruct the tactile components of the extended reality device to provide tactile feedback about motion to the operator.

[0074] In some embodiments, tactile feedback may include a feedback force. If fetal movement is detected, the tactile component may provide a feedback force to the operator, allowing the operator to feel the fetal movement. For example, the processing device 120 may determine the magnitude of the feedback force based on the fetal movement and transmit a signal representing the magnitude of the feedback force to the tactile component. A motor or actuator in the tactile component may apply the feedback force to the user based on the signal received from the processing device 120. Thus, the operator can feel the fetal movement.

[0075] In some embodiments, the magnitude of the feedback force can be set according to the default settings of the fetal monitoring system 100, or preset by the user or operator via a terminal device (e.g., the extended reality device 130). Alternatively, the processing device 120 may determine the magnitude of the feedback force based on motion information from the second 3D fetal image to the first 3D fetal image. By way of example only, if the motion information includes the displacement of the fetus from the second 3D fetal image to the first 3D fetal image, the processing device 120 may determine the magnitude of the feedback force based on the displacement. The greater the displacement, the greater the magnitude of the feedback force.

[0076] In some embodiments, the processing device 120 may determine the magnitude of the feedback force based on the degree of fetal movement. For example, if the displacement of the fetus from the second 3D fetal image to the first 3D fetal image is less than a displacement threshold, the processing device 120 may determine that the magnitude of the feedback force is zero. In other words, the tactile component may not apply any force to the operator. In some embodiments, the displacement of the fetus from the second 3D fetal image to the first 3D fetal image may be the average displacement of a plurality of first vertices, the displacement of the first vertex corresponding to the fetal center of mass, the maximum or minimum displacement among the plurality of first vertices, etc. It should be noted that the magnitude of the feedback force may also be determined in a manner similar to displacement based on other fetal motion information (such as acceleration, velocity, etc., or any combination thereof). For example, if the motion information includes the acceleration of the fetus from the second 3D fetal image to the first 3D fetal image, the processing device 120 may determine the magnitude of the feedback force based on the acceleration. The greater the acceleration, the greater the magnitude of the feedback force.

[0077] In some embodiments, the processing device 120 may determine the magnitude of the feedback force based on motion information and a target region of a 4D image. Further description of determining the magnitude of the feedback force based on motion information and a target region can be found elsewhere in this disclosure (e.g., Figure 6 (and its description) were found.

[0078] In some embodiments, the processing device 120 may instruct the speaker to play the fetal heartbeat sound while providing feedback force. In some embodiments, the processing device 120 may generate a sound that mimics the fetal sound and instruct the speaker to play the sound while providing feedback force.

[0079] According to some embodiments of this disclosure, the operator (new parents) can simultaneously see a stereoscopic image of the fetus (e.g., a 3D or 4D image of the fetus), feel the fetus's movements, and hear sounds related to the fetus. This effectively simulates the real-life interaction between parents and the fetus, thereby satisfying the operator's curiosity and further improving the user experience.

[0080] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Various changes and modifications can be made by those skilled in the art based on the teachings of this disclosure. However, these changes and modifications do not depart from the scope of this disclosure. In some embodiments, one or more operations may be omitted and / or one or more additional operations may be added. For example, operations 330 and 350 may be combined into a single operation. As another example, one or more other optional operations (e.g., user interaction information acquisition operations) may be added before operation 340. In some embodiments, the models used in this disclosure (e.g., 3D reconstruction models, segmentation models) may be obtained via a network from one or more components of the fetal monitoring system 100 or from an external source. For example, the segmentation model may be pre-trained by a computing device and stored in a storage device. The processing device 120 may access the storage device and retrieve the segmentation model.

[0081] Figure 4 This is a flowchart illustrating an exemplary process for determining fetal motion information based on fetal-related ultrasound data, according to some embodiments of the present disclosure. In some embodiments, one or more operations of process 400 may be performed to achieve, as in combination with... Figure 3 At least a portion of the described operation 340. For example, fetal movements detected in 340 can be determined according to process 400.

[0082] In 410, the processing device 120 (e.g., acquisition module 210) can acquire a first 3D fetal image.

[0083] In 420, the processing device 120 (e.g., acquisition module 210) can acquire a second 3D fetal image captured prior to the first 3D fetal image.

[0084] As described in conjunction with operation 320, a 4D image comprising multiple 3D fetal images can be generated based on fetal ultrasound data. The first 3D fetal image and the second 3D fetal image can be two of the multiple 3D fetal images corresponding to different time points. For example, the first 3D fetal image can be generated based on first ultrasound data related to the fetus collected by the ultrasound imaging device at a first time point, and the second 3D fetal image can be generated based on second ultrasound data related to the fetus collected by the ultrasound imaging device at a second time point prior to the first time point. In some embodiments, the first 3D fetal image can be represented by a first mesh surface comprising multiple first vertices, and the second 3D fetal image can be represented by a second mesh surface comprising multiple second vertices.

[0085] In 430, the processing device 120 (e.g., motion detection module 230) can determine the motion field between the first 3D fetal image and the second 3D fetal image.

[0086] The motion field may include multiple motion vectors. Motion vectors can be used to describe the motion of a spatial point (or physical point) of the fetus from a second time point to a first time point. In some embodiments, motion vectors can be determined by registering two 3D fetal images using an image registration algorithm. For example, after registering two 3D fetal images, the positions of two voxels corresponding to the same spatial point of the fetus in the two 3D fetal images can be determined. Then, the motion vector of the spatial point can be determined based on the positions of the two corresponding voxels. In some embodiments, the image registration algorithm may include grayscale-based algorithms, transform domain-based algorithms, feature-based algorithms, etc., or any combination thereof. In some embodiments, the motion field may include a portion or all of the motion vectors between the two 3D fetal images.

[0087] In some embodiments, the motion field can be determined based on optical flow-based techniques or a motion field determination model. As used herein, a motion field determination model can refer to a neural network model configured to receive a pair of 3D fetal images and output the motion field between (or relative to) the pair of 3D fetal images. In some embodiments, the motion field determination model can include a convolutional neural network (CNN) model, a deep CNN (DCNN) model, a fully convolutional network (FCN) model, a recurrent neural network (RNN) model, a transformer, etc., or any combination thereof.

[0088] In some embodiments, the motion field determination model may be trained by processing device 120 or another computing device (e.g., processing device of a supplier of the motion field determination model) based on multiple training samples. Each training sample may include a sample first 3D fetal image and a sample second 3D fetal image. The motion field determination model can be trained by performing an iterative operation including multiple iterations. For illustrative purposes, an implementation of the current iteration is described. For example, for each training sample, processing device 120 may determine the predicted motion field by inputting the sample first 3D fetal image and the sample second 3D fetal image of the training sample into an intermediate model. If the current iteration is the first iteration in a series of iterations, the intermediate model may be a preliminary model to be trained. If the current iteration is an iteration different from the first iteration, the intermediate model may be an updated model generated in a previous iteration. For each training sample, processing device 120 may then use the predicted motion field to deform the corresponding sample first 3D fetal image (or sample second 3D fetal image) to generate a deformed 3D fetal image. Furthermore, the processing device 120 can determine the value of the loss function based on the deformed 3D fetal image of each training sample and the sample second 3D fetal image (or sample first 3D fetal image). The processing device 120 can update at least one parameter of the intermediate model based on the value of the loss function, or designate the intermediate model as the motion field determination model when a termination condition is met. Exemplary termination conditions may include the loss function value being minimum or less than a threshold (e.g., a constant), the loss function value converging, or a specified number (or counted) of iterations having been performed during training.

[0089] According to some embodiments of this disclosure, a motion field determination model is used to determine the motion field between a first 3D fetal image and a second 3D fetal image. The motion field determination model, trained using machine learning techniques, learns the optimal mechanism for motion field determination from large datasets. Applying the motion field determination model can improve the accuracy and efficiency of motion field determination, which in turn improves the accuracy of subsequent analysis.

[0090] In 440, the processing device 120 (e.g., motion detection module 230) can determine the vertex correspondence between a plurality of first vertices on the first mesh surface and a plurality of second vertices on the second mesh surface.

[0091] Vertex correspondence can indicate the correspondence between a first vertex and a second vertex. In some embodiments, each first vertex of the fetus representing a first time point on a first grid surface can correspond to a second vertex of the fetus representing a second time point on a second grid surface. The correspondence between a first vertex on the first grid surface and a second vertex on the second grid surface can mean that the first vertex and the second vertex represent the same spatial point or portion of the fetus.

[0092] In some embodiments, the processing device 120 can determine vertex correspondence by deforming the first mesh surface or the second mesh surface based on the motion field determined in 430.

[0093] In 450, the processing device 120 (e.g., motion detection module 230) can determine motion information from the second 3D fetal image to the first 3D fetal image.

[0094] For each of a plurality of first vertices, the processing device 120 may determine motion information from its corresponding second vertex to the first vertex based on vertex correspondence. For example, for a first vertex, the processing device 120 may determine the displacement from a second position of its corresponding second vertex to a first position of the first vertex. As another example, the processing device 120 may determine the velocity of the first vertex as it moves from the second position to the first position based on the displacement and the time length between the first and second time points. As yet another example, the processing device 120 may determine the acceleration of the first vertex based on the velocity and the time length.

[0095] The processing device 120 can determine motion information from a second 3D fetal image to a first 3D fetal image based on motion information from a plurality of first vertices. For example, the motion information of the first vertices can be designated as motion information between the first 3D fetal image and the second 3D fetal image. As another example, the processing device 120 can determine the motion information from the second 3D fetal image to the first 3D fetal image by averaging the motion information associated with the plurality of first vertices. For example, if the motion information includes velocity, the processing device 120 can determine the velocity from the second 3D fetal image to the first 3D fetal image by averaging the velocities associated with the plurality of first vertices. As another example, the processing device 120 can determine the maximum or minimum value of the motion information associated with the plurality of first vertices as the motion information from the second 3D fetal image to the first 3D fetal image.

[0096] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Various changes and modifications can be made by those skilled in the art based on the teachings of this disclosure. However, these changes and modifications do not depart from the scope of this disclosure. In some embodiments, one or more operations may be omitted and / or one or more additional operations may be added. For example, operations 410 and 420 may be combined into a single operation.

[0097] Figure 5 This is a flowchart illustrating an exemplary process for determining fetal motion information based on fetal-related ultrasound data, according to some embodiments of the present disclosure. In some embodiments, one or more operations of process 500 may be performed to achieve, as in combination with... Figure 3At least a portion of the described operation 340. For example, fetal movements detected in 340 can be determined according to process 500.

[0098] In 510, the processing device 120 (e.g., acquisition module 210) can acquire a first 3D fetal image.

[0099] In 520, the processing device 120 (e.g., acquisition module 210) can acquire a second 3D fetal image captured prior to the first 3D fetal image.

[0100] Operations 510 and 520 can be performed in a similar manner to operations 410 and 420, and their descriptions will not be repeated here.

[0101] In 530, the processing device 120 (e.g., motion detection module 230) can generate a first point cloud corresponding to the first 3D fetal image.

[0102] In 540, the processing device 120 (e.g., motion detection module 230) can generate a second point cloud corresponding to the second 3D fetal image.

[0103] A point cloud may include multiple points, each representing a spatial point on the surface of the fetus's body, and may be described using one or more feature values ​​of the spatial points (e.g., feature values ​​related to the location (e.g., 3D coordinates) and / or composition of the spatial points). A first point cloud may include multiple first points, and a second point cloud may include multiple second points.

[0104] A first 3D fetal image and / or a corresponding first mesh surface may be represented using a first point cloud, and a second 3D fetal image and / or a corresponding second mesh surface may be represented using a second point cloud. For example, a first vertex in the first mesh surface may be designated as a first point in the first point cloud, and a second vertex in the second mesh surface may be designated as a second point in the second point cloud. Similar to the first vertex in the first mesh surface described elsewhere in this disclosure, each first point in the first point cloud may correspond to a second point in the second point cloud. The correspondence between a first point in the first point cloud and a second point in the second point cloud may mean that the first point and the second point represent the same spatial point or portion of the fetus.

[0105] In 550, the processing device 120 (e.g., motion detection module 230) can determine the vertex correspondence between a plurality of first vertices on the first mesh surface and a plurality of second vertices on the second mesh surface based on the first point cloud and the second point cloud.

[0106] In some embodiments, the processing device 120 can determine vertex correspondence by registering a first point cloud and a second point cloud. For example, the second point cloud can be used as a reference point cloud. Then, a point cloud registration algorithm can be used to register the reference point cloud to the first point cloud. Exemplary point cloud registration algorithms may include the Iterative Closest Point (ICP) algorithm, the Kernel Correlation (KC) algorithm, the Robust Point Matching (RPM) algorithm, the Unscented Particle Filter (UPF) algorithm, the Unscented Kalman Filter (UKF) algorithm, etc. Based on the registration result between the first point cloud and the second point cloud, the correspondence between first points of the first point cloud and second points of the second point cloud can be determined. Since each first point corresponds to a first vertex in the first mesh surface, and each second point corresponds to a second vertex in the second mesh surface, vertex correspondence can be determined based on the correspondence between the first points and the second points.

[0107] In 560, the processing device 120 (e.g., motion detection module 230) can determine motion information from the second 3D fetal image to the first 3D fetal image.

[0108] Operation 560 can be performed in a similar manner to operation 450, and its description will not be repeated here.

[0109] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Various changes and modifications can be made by those skilled in the art based on the teachings of this disclosure. However, these changes and modifications do not depart from the scope of this disclosure. In some embodiments, one or more operations may be omitted and / or one or more additional operations may be added. For example, operations 510 and 520 may be combined into a single operation.

[0110] Figure 6 This is a flowchart illustrating an exemplary process for determining the magnitude of a feedback force according to some embodiments of the present disclosure. In some embodiments, process 600 may be implemented as a set of instructions (e.g., an application) stored in a storage device. Processing device 120 (e.g., in...) Figure 2 The instruction set can be executed (implemented on one or more of the illustrated modules), and when the instructions are executed, the processing device 120 can be configured to execute process 600. The operation of the illustrated process 600 presented below is intended to be illustrative. In some embodiments, process 600 may be accomplished using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 6 The order of operations of the illustrated and described process 600 below is not intended to be limiting.

[0111] In 610, the processing device 120 (e.g., acquisition module 210) can determine the target area of ​​the 4D image.

[0112] The target area of ​​the 4D image can be a specific area of ​​the fetus that the operator is interested in. In some embodiments, the target area of ​​the 4D image can be set according to the default settings of the fetal monitoring system 100, or preset by the user or operator via a terminal device (e.g., extended reality device 130). In some embodiments, the target area can be part or all of the fetus. For example, the target area can be the fetal heart region.

[0113] In some embodiments, the target area can be determined based on user interaction information about the 4D image. The user interaction information may include at least the target area of ​​the 4D image to which the user interaction is directed. For example, the display component of the extended reality device can display virtual elements and a 4D image of a fetus, and the operator can control the virtual elements via the haptic component of the extended reality device to interact with a specific area of ​​the 4D image. The processing device 120 can identify the specific area as the target area. Different operators can feel different feedback forces via the haptic component. In some embodiments, the virtual element can be a virtual finger, virtual hand, virtual arm, virtual leg, virtual foot, etc., or any combination thereof. In some embodiments, the virtual element can be controlled by, for example, the operator's finger, hand, arm, etc., via the haptic component.

[0114] In 620, for each of the plurality of target first vertices in the portion of the first grid surface corresponding to the target region, the processing device 120 (e.g., motion detection module 230) can determine the magnitude of the vertex force based on the motion information of the target first vertex.

[0115] In some embodiments, the motion information of the target first vertex includes displacement (or acceleration, velocity), and the processing device 120 can determine the magnitude of the vertex force based on the displacement (or acceleration, velocity) of the target first vertex. The greater the displacement (or acceleration, velocity), the greater the magnitude of the vertex force. In some embodiments, the processing device 120 can determine the magnitude of the vertex force based on the degree of fetal movement. For example, if the displacement of the target first vertex is less than a displacement threshold, the processing device 120 can determine that the magnitude of the vertex force is zero. As another example, if the velocity of the target first vertex is less than a velocity threshold, the processing device 120 can determine that the magnitude of the vertex force is zero.

[0116] In some embodiments, for each target first vertex, the magnitude of the vertex force at the target first vertex can be determined based on Newton's second law. Specifically, the magnitude of the feedback force can be determined according to the following equation (1):

[0117] F = ma, (1)

[0118] Where F represents the vertex force, m represents the vertex mass, and a represents the acceleration of the first vertex of the target. In some embodiments, the vertex mass may be determined based on the size or volume of the fetus. In some embodiments, the vertex mass may be determined according to the default settings of the fetal monitoring system 100, or preset by the user or operator via a terminal device (e.g., the extended reality device 130).

[0119] In 630, the processing device 120 (e.g., motion detection module 230) can determine the magnitude of the feedback force based on the magnitudes of the vertex forces of the multiple target first vertices. For example, the processing device 120 can determine the magnitude of the feedback force by integrating the magnitudes of the vertex forces of the multiple target first vertices. As another example, the processing device 120 can determine the magnitude of the feedback force by summing the magnitudes of the vertex forces of the multiple target first vertices.

[0120] Furthermore, the processing device 120 can instruct the tactile components of the extended reality device to provide feedback force of a defined magnitude to the operator. This allows the operator (new parents) to feel the fetus's movements, thereby improving the user experience. In some embodiments, once a virtual element touches a 3D or 4D image of the fetus via the tactile components, the processing device 120 can instruct a speaker to play fetal heartbeat sounds collected by a sound collector (e.g., a Doppler fetal monitor) or generate sounds mimicking fetal sounds, and instruct the speaker to play the sounds, thereby further improving the user experience.

[0121] It should be noted that the above description of process 600 is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Various changes and modifications can be made by those skilled in the art based on the teachings of this disclosure. However, these changes and modifications do not depart from the scope of this disclosure.

[0122] Having described the basic concepts herein, it will be quite apparent to those skilled in the art, upon reading this detailed disclosure, that the foregoing detailed disclosure is intended to be illustrative only and not restrictive. Although not expressly stated herein, various changes, modifications, and alterations may be made, and such changes, modifications, and alterations are expected by those skilled in the art. These changes, modifications, and alterations are intended to be suggested by this disclosure and are within the spirit and scope of the exemplary embodiments of this disclosure.

[0123] Furthermore, certain terms have been used to describe embodiments of this disclosure. For example, the terms "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Therefore, it is to be emphasized and understood that references to "an embodiment" or "an embodiment" or "alternative embodiment" two or more times in various parts of this specification do not necessarily all refer to the same embodiment. Additionally, specific features, structures, or characteristics may be appropriately combined in one or more embodiments of this disclosure.

[0124] Furthermore, those skilled in the art will understand that aspects of this disclosure can be exemplified and described in any of the many patentable classes or backgrounds, including any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Therefore, aspects of this disclosure can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of software and hardware implementations, all of which may be generally referred to herein as “units,” “modules,” or “systems.” Additionally, aspects of this disclosure can take the form of a computer program product embodied in one or more computer-readable media on which computer-readable program code is embodied.

[0125] Non-transient computer-readable signal media may include propagated data signals (e.g., in baseband or as part of a carrier wave) on which computer-readable program code is internally implemented. Such propagated signals may take any of a variety of forms, including electromagnetic, optical, etc., or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and can transmit, propagate, or deliver a program used by or relating to an instruction execution system, device, or apparatus. Any suitable medium may be used to transmit program code implemented on a computer-readable signal medium, including wireless, wired, fiber optic cable, RF, etc., or any suitable combination thereof.

[0126] Computer program code used to perform the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc.), general programming languages ​​(such as the "C" programming language, Visual Basic, Fortran, Perl, COBOL, PHP, ABAP), dynamic programming languages ​​(such as Python, Ruby, and Groovy), or other programming languages. The program code can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (e.g., via the Internet using an Internet service provider) or in a cloud computing environment, or provided as a service such as Software as a Service (SaaS).

[0127] Furthermore, the order of description of processing elements or sequences, or the use of numbers, letters, or other markings, is therefore not intended to limit the claimed processes and methods to any order unless specified in the claims. Although the foregoing disclosure has discussed various useful embodiments currently considered to be part of this disclosure by way of various examples, it should be understood that such details are for that purpose only, and the appended claims are not limited to the disclosed embodiments, but rather are intended to cover modifications and equivalent arrangements within the spirit and scope of the disclosed embodiments. For example, while embodiments of the various components described above may be implemented in hardware devices, they may also be implemented as software-only solutions, such as installations on existing servers or mobile devices.

[0128] Similarly, it should be understood that in the foregoing description of embodiments of this disclosure, various features are sometimes grouped together in a single embodiment, drawing, or description thereof in order to simplify the disclosure and aid in understanding one or more of the various inventive embodiments. However, the disclosed approach is not to be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in the claims. Rather, the inventive embodiments consist of fewer features than those in a single foregoing disclosure.

[0129] In some embodiments, the numbers used to describe quantities, properties, etc., of certain embodiments of this application should be understood to be modified in some cases by the terms “approximately,” “about,” or “roughly.” For example, “approximately,” “about,” or “roughly” can indicate a variation of ±20% of the value it describes, unless otherwise stated. Thus, in some embodiments, the numerical parameters set forth in the written description and appended claims are approximate values ​​that may vary depending on the desired properties sought to be obtained by a particular embodiment. In some embodiments, numerical parameters should be interpreted based on the number of significant digits reported and by applying common rounding techniques. Although the wide range of numerical ranges and parameters described in some embodiments of this application are approximate values, the numerical values ​​set forth in the specific examples are reported as precisely as possible.

[0130] All patents, patent applications, patent application gazettes, and other materials, such as papers, books, specifications, publications, documents, and things, cited herein are incorporated in their entirety for all purposes, except for any examination document history associated with them, any foregoing content that is inconsistent with or conflicts with this document, or any foregoing content that may limit the widest scope of the claims now or hereafter associated with this document. By way of example, if there is any inconsistency or conflict between the description, definition, and / or use of terms associated with any incorporated material and the description, definition, and / or use of terms associated with this document, the description, definition, and / or use of terms in this document shall prevail.

[0131] Finally, it should be understood that the embodiments of this application disclosed herein are illustrative of the principles of embodiments of this application. Other modifications may be made within the scope of this application. Thus, alternative configurations of the embodiments of this application can be utilized based on the teachings herein by way of example rather than limitation. Therefore, the embodiments of this application are not limited to those precisely shown and described.

Claims

1. A method for fetal monitoring, implemented on a computing device having at least one processor and at least one storage device, the method comprising: To obtain ultrasound data related to the fetus collected by an ultrasound imaging device; At least one 4D image of the fetus is generated based on the ultrasound data; The display component of the extended reality device is instructed to display the at least one 4D image to the operator; Based on the ultrasound data, the movement of at least a portion of the fetus is detected; as well as The tactile component of the extended reality device is instructed to provide feedback force to the operator regarding the movement, wherein the at least one 4D image includes a first 3D fetal image and a second 3D fetal image captured prior to the first 3D fetal image, and the detection of fetal movement based on the ultrasound data includes: Determine the vertex correspondence between a plurality of first vertices on a first mesh surface and a plurality of second vertices on a second mesh surface, wherein the first mesh surface represents the first 3D fetal image and the second mesh surface represents the second 3D fetal image; and For each of the plurality of first vertices, motion information from its corresponding second vertex to the first vertex is determined based on the vertex correspondence, wherein the at least one processor is configured to instruct the fetal monitoring system to determine the magnitude of the feedback force in the following manner: Determine the target region of the at least one 4D image; Among the plurality of first vertices, a plurality of target first vertices are determined in the portion of the first mesh surface corresponding to the target region; For each of the plurality of target first vertices, the magnitude of the vertex force is determined based on the motion information of the target first vertex, wherein the motion information of the target first vertex is related to the motion of at least a portion of the fetus; and The magnitude of the feedback force is determined by integrating the magnitude of the vertex force of the first vertex of the plurality of targets.

2. The method according to claim 1, wherein, Generating at least one 4D image of the fetus based on the ultrasound data includes: Multiple initial 3D images are generated based on the ultrasound data related to the fetus; Multiple 3D fetal images are generated by segmenting portions representing the fetus from various initial 3D images; and At least one 4D image is generated based on the plurality of 3D fetal images.

3. The method according to claim 2, wherein, The process of generating the at least one 4D image based on the plurality of 3D fetal images includes: Extracting mesh surfaces from various 3D fetal images; and The rendering includes at least one 4D mesh surface of the plurality of mesh surfaces to generate the at least one 4D rendered image.

4. The method according to claim 1, wherein, Determining the vertex correspondence between multiple first vertices of the first mesh surface and multiple second vertices of the second mesh surface includes: Determine the motion field between the first 3D fetal image and the second 3D fetal image; and The vertex correspondence is determined based on the described motion field.

5. The method according to claim 4, wherein, The sports field is determined based on optical flow-based techniques or a sports field determination model.

6. The method according to claim 1, wherein, Determining the vertex correspondence between multiple first vertices of the first mesh surface and multiple second vertices of the second mesh surface includes: Generate a first point cloud corresponding to the first 3D fetal image and a second point cloud corresponding to the second 3D fetal image; and The vertex correspondence is determined by registering the first point cloud to the second point cloud.

7. The method according to claim 1, wherein, The at least one processor is further configured to instruct the fetal monitoring system to perform operations, including: Instruct the speaker to play sounds related to the fetus.

8. A non-transitory computer-readable medium comprising at least one instruction set for fetal monitoring, wherein, When executed by at least one processor of a computing device, the at least one instruction set instructs the at least one processor to perform operations, including: To obtain ultrasound data related to the fetus collected by an ultrasound imaging device; At least one 4D image of the fetus is generated based on the ultrasound data; The display component of the extended reality device is instructed to display the at least one 4D image to the operator; Based on the ultrasound data, the movement of at least a portion of the fetus is detected; and The tactile component of the extended reality device is instructed to provide feedback force to the operator regarding the movement, wherein the at least one 4D image includes a first 3D fetal image and a second 3D fetal image captured prior to the first 3D fetal image, and the detection of fetal movement based on the ultrasound data includes: Determine the vertex correspondence between a plurality of first vertices on a first mesh surface and a plurality of second vertices on a second mesh surface, wherein the first mesh surface represents the first 3D fetal image and the second mesh surface represents the second 3D fetal image; and For each of the plurality of first vertices, motion information from its corresponding second vertex to the first vertex is determined based on the vertex correspondence, wherein the at least one processor is configured to instruct the fetal monitoring system to determine the magnitude of the feedback force in the following manner: Determine the target region of the at least one 4D image; Among the plurality of first vertices, a plurality of target first vertices are determined in the portion of the first mesh surface corresponding to the target region; For each of the plurality of target first vertices, the magnitude of the vertex force is determined based on the motion information of the target first vertex, wherein the motion information of the target first vertex is related to the motion of at least a portion of the fetus; and The magnitude of the feedback force is determined by integrating the magnitude of the vertex force of the first vertex of the plurality of targets.

Images