Computer-implemented method and apparatus for determining a puncture path based on a body surface sticker

Abstract

The embodiment of the disclosure discloses a computer-implemented method and device for determining a puncture path based on a body surface film, wherein the method comprises: based on a current breathing state of a target object and a current pressure vector of a target body surface region corresponding to the body surface film of the target object, determining a first motion vector of a target organ of the target object in a scanning image coordinate system corresponding to a reference scanning image sequence by using a target organ motion prediction model obtained by pre-training; based on the current breathing state and the current pressure vector, determining a second motion vector of the target body surface region in the scanning image coordinate system by using a target body surface motion prediction model obtained by pre-training; and determining a current puncture path based on the reference scanning image sequence, the first motion vector and the second motion vector. The embodiment of the disclosure can effectively reduce the harm to the target object and improve the processing efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of medical device technology, and in particular to a computer-based method and apparatus for determining puncture paths based on a skin patch. Background Technology

[0002] With advancements in medical technology, minimally invasive diagnosis and treatment guided by CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) are increasingly used in clinical practice. Examples include CT-guided biopsies of lung nodules, liver nodules, and vertebral bodies, as well as ablation therapy for lesions in the thoracic and abdominal cavities. However, due to the deep location and small size of lesions, and the fact that CT and MRI imaging are not real-time, CT- and MRI-guided puncture procedures often require multiple scans to continuously determine the location of the puncture point and lesion, correcting the direction of the puncture needle (which may include a biopsy needle, ablation needle, or drainage needle). These repeated scans expose patients to significant radiation exposure, potentially leading to adverse consequences. Summary of the Invention

[0003] To address the technical problems mentioned above, such as the adverse consequences for patients caused by repeated scanning, this disclosure is proposed. Embodiments of this disclosure provide a computer-based method and apparatus for determining the puncture path based on a skin dressing.

[0004] According to one aspect of the present disclosure, a computer-based method for determining a puncture path based on a skin patch is provided, applied to a medical assistive device, comprising:

[0005] Obtain the current respiratory state of the target object, the reference scan image sequence, and the current pressure vector of the target body surface region corresponding to the body surface patch of the target object;

[0006] Based on the current respiratory state and the current pressure vector, the first motion vector of the target organ of the target object in the scanning image coordinate system corresponding to the reference scanning image sequence is determined using a pre-trained target organ motion prediction model.

[0007] Based on the current respiratory state and the current pressure vector, the second motion vector of the target body surface region in the coordinate system of the scanned image is determined using a pre-trained target body surface motion prediction model.

[0008] Based on the reference scan image sequence, the first motion vector, and the second motion vector, the current puncture path between the target puncture point and the target target point of the target organ is determined, wherein the target puncture point is any point within the target body surface area.

[0009] According to another aspect of the present disclosure, a computer-based device for determining a puncture path based on a skin patch is provided, comprising:

[0010] The first acquisition module is used to acquire the current respiratory state of the target object, the reference scan image sequence, and the current pressure vector of the target body surface region corresponding to the body surface patch of the target object;

[0011] The first processing module is used to determine the first motion vector of the target organ of the target object in the scanning image coordinate system corresponding to the reference scanning image sequence, based on the current respiratory state and the current pressure vector and using a pre-trained target organ motion prediction model.

[0012] The second processing module is used to determine the second motion vector of the target body surface region in the coordinate system of the scanned image based on the current breathing state and the current pressure vector and using a pre-trained target body surface motion prediction model.

[0013] The third processing module is used to determine the current puncture path between the target puncture point and the target target point of the target organ based on the reference scan image sequence, the first motion vector and the second motion vector, wherein the target puncture point is any point within the target body surface area.

[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the puncture guidance method based on a skin patch as described in any of the above embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the computer implementation method for determining the puncture path based on a skin patch as described in any of the above embodiments of the present disclosure.

[0016] Based on the computer-implemented method and apparatus for determining puncture paths based on body surface patches provided in the above embodiments of this disclosure, a mapping relationship between the respiratory state of the subject, the pressure vector of the target body surface area, and the movement of the target organ is established through a pre-trained target organ motion prediction model. This allows for the real-time determination of the target organ's location and the location of the target body surface puncture point based on the subject's real-time respiratory state and pressure vector. This enables real-time determination of the current puncture path, effectively reducing the number of repeated scans and significantly mitigating adverse consequences for the subject. Furthermore, the real-time determination of the current puncture path effectively reduces repeated puncture errors caused by the non-real-time nature of CT and MRI scans, further reducing harm to the subject. Additionally, based on body surface patches, when predicting body surface movement, only the motion vector of the patch area needs to be predicted, rather than the motion vector of the entire body surface, effectively reducing the computational load and improving processing efficiency.

[0017] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 This is a flowchart illustrating a computer-implemented method for determining a puncture path based on a skin patch, provided in an exemplary embodiment of this disclosure.

[0020] Figure 2 This is a flowchart illustrating a computer-implemented method for determining a puncture path based on a skin patch, provided in another exemplary embodiment of this disclosure.

[0021] Figure 3 This is a flowchart illustrating a computer-implemented method for determining a puncture path based on a skin patch, provided in yet another exemplary embodiment of this disclosure.

[0022] Figure 4 This is a flowchart illustrating step 250 provided in an exemplary embodiment of this disclosure;

[0023] Figure 5This is a schematic diagram of the structure of a computer-implemented device for determining puncture paths based on a body surface patch, provided in an exemplary embodiment of this disclosure.

[0024] Figure 6 This is a schematic diagram of the structure of a computer-implemented device for determining puncture paths based on a body surface patch, provided in another exemplary embodiment of this disclosure;

[0025] Figure 7 This is a schematic diagram of the structure of a respiratory volume measurement device provided in an exemplary embodiment of this disclosure;

[0026] Figure 8 This is a schematic diagram of the structure of a puncture guidance system based on a skin patch provided in an exemplary embodiment of the present disclosure;

[0027] Figure 9 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0028] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0029] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0030] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0031] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0032] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0033] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0034] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0035] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0036] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0037] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0038] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0039] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0040] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0041] This disclosure outlines

[0042] In developing this disclosure, the inventors discovered that with advancements in medical technology, minimally invasive diagnosis and treatment guided by CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) are increasingly used in clinical practice. Examples include CT-guided biopsies of lung nodules, liver nodules, and vertebral bodies, as well as ablation therapy for lesions in the thoracic and abdominal cavities. However, because lesions are often deep and small, and CT and MRI imaging are not real-time, CT and MRI-guided puncture procedures often require multiple scans to continuously determine the location of the puncture point and lesion, correcting the direction of the puncture needle (which may include a biopsy needle, ablation needle, or drainage needle). These repeated scans expose patients to significant radiation exposure, potentially causing adverse effects.

[0043] Exemplary methods

[0044] Figure 1 This is a flowchart illustrating a computer-implemented method for determining a puncture path based on a skin patch, provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically medical assistive devices, such as… Figure 1 As shown, it includes the following steps:

[0045] Step 210: Obtain the current respiratory state of the target object, the reference scan image sequence, and the current pressure vector of the target body surface region corresponding to the target object's body surface patch.

[0046] The target object can be any patient or subject for whom the location of the target point needs to be determined. The target point can be a lesion in any thoracic or abdominal organ, such as a pulmonary nodule, a pulmonary tumor, a liver nodule, a liver tumor, etc. The reference scan image sequence can include a scan image sequence obtained under a preset breathing state, and the scan image sequence can include at least one of CT image sequences, MRI image sequences, and any other possible image sequences.

[0047] In some alternative embodiments, the reference scan image sequence is a CT image sequence.

[0048] In some alternative embodiments, the reference scan image sequence is an MRI image sequence.

[0049] In some alternative embodiments, the reference scan image sequence includes CT image sequences and MRI image sequences.

[0050] In some optional embodiments, the preset respiratory state and the current respiratory state can be determined by a respiratory state quantification device, which may include at least one of a respiratory volume quantification device and a chest and abdominal surface pressure acquisition device. Correspondingly, the current respiratory state may include at least one of the current respiratory volume and the current chest and abdominal surface pressure. The preset respiratory state may include at least one of the preset respiratory volume and the preset chest and abdominal surface pressure. The respiratory volume quantification device is used to determine the respiratory volume of the target object, such as the current respiratory volume and the preset respiratory volume, while the chest and abdominal surface pressure acquisition device is used to determine the chest and abdominal surface pressure of the target object, such as the aforementioned current chest and abdominal surface pressure and the preset chest and abdominal surface pressure.

[0051] In some optional embodiments, a skin patch refers to a patch applied to the target body surface area of ​​the target object. It can be used for registration (coordinate transformation) between the scanned image coordinate system and the target spatial coordinate system. The target spatial coordinate system can be the world coordinate system or a reference coordinate system rigidly connected to the world coordinate system, which can be set according to actual needs. The skin patch can also characterize the area where the puncture point is located, facilitating the determination of the puncture point. The current pressure vector of the target body surface area refers to the pressure vector of the target body surface area acquired when the target object is in its current respiratory state. The pressure vector of the target body surface area has a certain linkage relationship with the movement of the target body surface area and the movement of the target organs. For example, as the target object's breathing changes, the target body surface area moves, causing a corresponding change in the pressure vector of the target body surface area.

[0052] In some alternative embodiments, the current pressure vector of the target body surface region can be obtained based on anisotropic piezoelectric materials. For example, an anisotropic piezoelectric flexible pressure sensor can be set on the body surface film according to certain rules. Through the anisotropic sensing characteristics of the flexible pressure sensor, the magnitude and direction of stress at any position of the target body surface region can be detected, and a corresponding piezoelectric signal can be generated. By fitting the piezoelectric signal, the movement of the target body surface region can be identified.

[0053] Step 220: Based on the current respiratory state and current pressure vector, the first motion vector of the target organ of the target object in the scanning image coordinate system corresponding to the reference scanning image sequence is determined using a pre-trained target organ motion prediction model.

[0054] The target organ motion prediction model is a pre-trained neural network model used to predict the motion of a target organ relative to a reference scan image sequence based on the respiratory state of the target object. The network structure of the target organ motion prediction model can be any feasible network structure, such as a convolutional neural network, a Transformer-based network, etc. The first motion vector can include the motion distance and direction of each voxel belonging to the target organ in the reference scan image sequence. By using the voxels belonging to the target organ in the reference scan image sequence and their corresponding motion distances and directions, the position of the target organ after motion can be determined.

[0055] In some optional embodiments, the input to the target organ motion prediction model may include the current pressure vector of the target body surface region in addition to the current respiratory state. Specifically, the model can be trained according to actual needs to obtain a target organ motion prediction model with corresponding functions.

[0056] In some optional embodiments, the training process of the target organ motion prediction model may include: acquiring scan image sequences corresponding to at least two respiratory states of the target object and pressure vectors of the target body surface region corresponding to each respiratory state; using any scan image sequence as a reference scan image sequence, determining the motion vector of the target organ in the scan image sequence of each respiratory state relative to the target organ in the reference scan image sequence; and training the initial organ motion prediction model based on each respiratory state, the pressure vector corresponding to each respiratory state, and the motion vector corresponding to each respiratory state to obtain a trained target organ motion prediction model. This target organ motion prediction model can then predict the first motion vector of the target organ based on the current respiratory state and the current pressure vector of the target body surface region.

[0057] In some optional embodiments, the first motion vector of the target organ can be determined using a target organ motion prediction model based on the target object's current respiratory state and a reference scan image sequence. The training process of the corresponding target organ motion prediction model may include: acquiring scan image sequences corresponding to at least two respiratory states of the target object; using any scan image sequence as a reference scan image sequence, determining the motion vector of the target organ in each respiratory state's scan image sequence relative to the target organ in the reference scan image sequence; and training the initial organ motion prediction model based on each respiratory state, the reference scan image sequence, and the motion vector corresponding to each respiratory state to obtain a trained target organ motion prediction model. This target organ motion prediction model can then predict the first motion vector of the target organ based on the current respiratory state and the reference scan image sequence.

[0058] In some optional embodiments, the first motion vector of the target organ can be determined using a target organ motion prediction model based on the target object's current respiratory state, a reference scan image sequence, and the current pressure vector of the target body surface region. Accordingly, the training process of the target organ motion prediction model may include: acquiring scan image sequences corresponding to at least two respiratory states of the target object and the pressure vector of the target body surface region corresponding to each respiratory state; using any scan image sequence as a reference scan image sequence, determining the motion vector of the target organ relative to the target organ in the reference scan image sequence for each respiratory state; and training the initial organ motion prediction model based on each respiratory state, the pressure vector corresponding to each respiratory state, the reference scan image sequence, and the motion vector corresponding to each respiratory state to obtain a trained target organ motion prediction model. This target organ motion prediction model can then predict the first motion vector of the target organ based on the current respiratory state, the reference scan image sequence, and the current pressure vector of the target body surface region.

[0059] Specifically, the initial organ motion prediction model can be a preliminary model obtained through initial training based on scanned image sequences of other objects. For training a target organ motion prediction model that predicts the first motion vector of the target organ based on the current respiratory state and the current pressure vector of the target body surface region, the training can be performed on the target object. Each respiratory state of the target object and the corresponding pressure vector of the target body surface region can be used as model input to obtain the training prediction results for each respiratory state. The motion vector corresponding to each respiratory state is used as a label and compared with the training prediction results. A preset loss function is used to determine the network loss, which is then used to update the model network parameters. Training continues until the network loss meets a preset condition (e.g., convergence) or the number of iterations reaches a preset threshold, at which point the training ends, and the trained target organ motion prediction model is obtained. For example, based on scan image sequences corresponding to two preset respiratory states—end-expiration and end-inspiration—and the pressure vector of the target body surface region, the scan image sequence corresponding to end-expiration is used as the reference scan image sequence, and the scan image sequence corresponding to end-inspiration is used as the motion scan image sequence. By registering the scan image sequences of the two preset respiratory states, the target motion vector of the target organ in the motion scan image sequence at end-inspiration is obtained relative to the target organ in the reference scan image sequence. The motion vector of the reference scan image sequence relative to itself is 0. Based on the two preset respiratory states, the pressure vectors corresponding to the two respiratory states, and the motion vectors of the target organs corresponding to the two respiratory states, the initial organ motion prediction model is further trained to establish the mapping relationship between the respiratory state and pressure vector of the target object and the motion vector of the target organ. Alternatively, the initial organ motion prediction model can be trained by combining at least one respiratory state between end-expiration and end-inspiration, the pressure vector corresponding to that respiratory state, and the motion vector of the target organ in the scan image sequence triggered by that respiratory state relative to the reference scan image sequence, to further improve the accuracy of the model's prediction results. Based on this, an initial organ motion prediction model can be trained using a small number of scanned image sequences of the target object to obtain the mapping relationship between the target object's respiratory state, pressure vector, and motion vector of the target organ. This mapping relationship can be used for real-time localization of the target object's target points, effectively reducing the number of scans, scanning time, and absorbed dose. Registration of the scanned image sequences for two respiratory states involves finding a transformation that allows the reference scanned image sequence to spatially align with the motion scanned image sequence after the transformation. This transformation represents the motion deformation field (motion vector) of the motion scanned image sequence relative to the reference scanned image sequence. The alignment quality can be determined by at least one of distance and similarity metrics, which can be set according to actual needs.

[0060] In some optional embodiments, the pose of each voxel of the target organ in the current respiratory state under the scanning image coordinate system can be determined based on the pose of each voxel of the target organ in the reference scan image sequence and the first motion vector of the target organ under the current respiratory state. Alternatively, the position of the target point in the current respiratory state under the scanning image coordinate system can be determined based on the position of the target point in the target organ in the reference scan image sequence and the motion of the target organ under the current respiratory state.

[0061] Step 230: Based on the current respiratory state and current pressure vector, the second motion vector of the target body surface region in the scanned image coordinate system is determined using a pre-trained target body surface motion prediction model.

[0062] Among them, the target surface motion prediction model is a pre-trained neural network model used to predict the motion of the target surface region of the target object relative to the reference scan image sequence based on the respiratory state of the target object.

[0063] In some optional embodiments, the network structure of the target body surface motion prediction model can adopt any feasible network structure, such as a convolutional neural network-based network structure, a Transformer-based network structure, etc. The second motion vector can include the motion distance and direction of each voxel belonging to the target body surface region in the reference scan image sequence. By using each voxel belonging to the target body surface region in the reference scan image sequence and the motion distance and direction of each voxel, the position of the target body surface region after motion can be determined.

[0064] In some optional embodiments, similar to the target organ motion prediction model, the input to the target body surface motion prediction model may include, in addition to the current respiratory state, the current pressure vector of the target body surface region. Based on the target object's current respiratory state and the current pressure vector of the target body surface region, the target body surface motion prediction model determines a second motion vector of the target body surface region. The specific training process of the corresponding target body surface motion prediction model is similar to that of the corresponding target organ motion prediction model, except that the label used for model training here is the motion vector of the target body surface region relative to the target body surface region in each scanned image sequence, so that the obtained target organ motion prediction model can predict the motion vector of the target body surface region based on the target object's current respiratory state and current pressure vector. The specific process can be found in the aforementioned target organ motion prediction model, and will not be repeated here.

[0065] In some optional embodiments of this disclosure, for the target object, the training of the target body surface motion prediction model and the target organ motion prediction model can be carried out using the same set of respiratory states and the pressure vectors and scan image sequences of the target body surface regions corresponding to each respiratory state. In this way, based on a small number of scan image sequences of the target object, the mapping relationship between the respiratory state and pressure vector of the target object and the motion of the target organ and the motion of the target body surface regions can be established.

[0066] In an optional embodiment, a second motion vector for the target surface region can be determined using a target surface motion prediction model based on the target object's current respiratory state and a reference scan image sequence. The specific training process of the corresponding target surface motion prediction model is similar to that of the corresponding target organ motion prediction model, and will not be described in detail here.

[0067] In an optional embodiment, a second motion vector for the target body surface region can be determined using a target body surface motion prediction model based on the target object's current respiratory state, a reference scan image sequence, and the current pressure vector of the target body surface region. The specific training process of the corresponding target body surface motion prediction model is similar to that of the corresponding target organ motion prediction model, and will not be described in detail here.

[0068] In some optional embodiments, the pose of each voxel of the target body surface region in the current respiratory state under the scanning image coordinate system can be determined based on the pose of each voxel of the target body surface region in the reference scan image sequence and the second motion vector of the target body surface region under the current respiratory state. The position of the puncture point in the current respiratory state under the scanning image coordinate system can also be determined based on the position of the puncture point in the target body surface region in the reference scan image sequence and the motion of the target body surface region under the current respiratory state.

[0069] In some optional embodiments, the training of the target body surface motion prediction model can be further combined with the target object's body state to further improve the model's prediction accuracy. This establishes a mapping relationship between the target object's respiratory state, body state, pressure vector of the target body surface region, and motion of the target body surface region. This allows for accurate and effective determination of the target puncture point's location on the target object's body surface based on any combination of the target object's respiratory state, the pressure vector of the target body surface region corresponding to that respiratory state, and any body state. The target object's body state can include body shape and posture. Correspondingly, the method of this disclosure can also include obtaining the target object's current body state. Step 230 specifically includes: based on the current respiratory state, current body state, and current pressure vector, using the pre-trained target body surface motion prediction model, determining the second motion vector of the target body surface region in the scanned image coordinate system.

[0070] In some alternative embodiments, the body state of the target object can be represented based on any implementable 3D human body model, such as an SMPL (Skinned Multi-Person Linear Model). An SMPL model is a skinned, vertex-based 3D human body model that can accurately represent different shapes and poses of the human body.

[0071] In some optional embodiments, during the training and application of the target body surface motion prediction model, key points on the target object's body surface can be extracted based on scanned image sequences under various breathing states to form corresponding 3D point cloud data. This converts the scanned image sequences into 3D point cloud data. During training, the initial body surface motion prediction model is trained based on the 3D point cloud data to obtain a trained target body surface motion prediction model. During application, based on the reference 3D point cloud data corresponding to the reference scanned image sequence and the target object's current breathing state, the trained target body surface motion prediction model is used to predict the second motion vector of the target object's body surface region under the current breathing state.

[0072] In some alternative embodiments, the network structure of the target body surface motion prediction model may include an encoding network and a decoding network.

[0073] For example, the mapping relationship corresponding to the encoding network can be represented as a learning-based encoding function E(v,θ,β,M)=α, where v represents the breathing state of the target object, θ represents the body posture of the target object, β represents the body shape of the target object, M represents the 3D point cloud data of the reference scan image sequence corresponding to the target body surface region of the target object, such as the 3D point cloud data of the contour of the target body surface region of the chest and abdomen of the target object, and α represents the encoding result. The encoding function is used to encode the body surface features of the target object under specific breathing states (such as the breathing states corresponding to the end of exhalation and the end of inhalation, respectively) and specific body states (such as the body states corresponding to the end of exhalation and the body states corresponding to the end of inhalation). The mapping relationship corresponding to the decoding network can be represented as a decoding function D(v',θ',β',α)=Mpred, where v' represents any breathing state of the target object, such as the current breathing state, θ' represents any body shape of the target object, such as the current body shape, β' represents any body posture of the target object, such as the current body posture, and Mpred represents the motion vector of the target body surface region of the target object obtained by decoding under v', θ', and β'. For the various input scenarios of the target body surface motion prediction model, such as respiratory state and pressure vector, respiratory state and reference scan image sequence, respiratory state, pressure vector and reference scan image sequence, respiratory state, pressure vector and body state, respiratory state, body state, pressure vector and reference scan image sequence, etc., the relevant parameters of the encoding and decoding functions can be set according to actual needs. Here, only one of them is described, and others will not be elaborated.

[0074] In some optional embodiments, the encoding network may include a first encoding function for encoding surface features of the target object under a specific body state and a second encoding function for encoding surface features of the target object under a specific body shape and respiratory state. For example, the first encoding function may be represented as E(θ,β,M)=α1, and the second encoding function may be represented as E(v,θ,M)=α2, where α1 and α2 represent the encoding results of the two encoding functions, respectively, and other symbols are as described above. Correspondingly, the decoding network may be represented as the decoding function D(v',θ',α1,α2)=Mpred.

[0075] In some optional embodiments, after obtaining the motion vectors of the target organ and the body surface during the target object's current respiratory state, a scan image sequence during the current respiratory state can be reconstructed by combining a reference scan image sequence. Specifically, a scan image sequence after the target organ's motion can be generated based on the motion vectors of the target organ and the reference scan image sequence; a scan image sequence after the body surface's motion can be generated based on the motion vectors of the body surface and the reference scan image sequence (or 3D point cloud data); the scan image sequence after the body surface's motion and the scan image sequence after the target organ's motion are fused to obtain the scan image sequence of the target object during the current respiratory state. In practical applications, motion prediction can also be performed on other parts of the target object's reference scan image sequence besides the target organ and the body surface to reconstruct a more effective scan image sequence during the current respiratory state. Based on this, the reconstruction of the scan image sequence of the target object under any respiratory state can be achieved, providing an accurate and effective real-time scan image sequence for determining the puncture path.

[0076] Steps 220 and 230 are not in any particular order.

[0077] Step 240: Based on the reference scan image sequence, the first motion vector and the second motion vector, determine the current puncture path between the target puncture point and the target target point of the target organ. The target puncture point is any point within the target body surface area.

[0078] Specifically, based on the poses of each voxel of the target organ in the reference scan image sequence and the first motion vector of the target organ under the current respiratory state, the poses of each voxel of the target organ in the scan image coordinate system under the current respiratory state can be determined. Combining the position of the target point within the target organ in the reference scan image sequence and the motion of the voxel corresponding to that position, the position of the target point in the scan image coordinate system under the current respiratory state can be determined. Similarly, the position of the target puncture point in the scan image coordinate system can be determined based on the reference scan image sequence and the second motion vector. Furthermore, based on the positions of the target point and the target puncture point, the current puncture path can be determined.

[0079] In some optional embodiments, the current puncture path can be determined by combining the current pose of the puncture needle in the scanned image coordinate system. The current pose of the puncture needle in the scanned image coordinate system can be achieved through registration (coordinate transformation) between the scanned image coordinate system and the target space coordinate system where the puncture needle is located. The target space coordinate system where the puncture needle is located can be the world coordinate system or a reference coordinate system rigidly connected to the world coordinate system, which can be set according to actual needs. The current pose can include the position and orientation of the puncture needle; the position represents the coordinates of the puncture needle, and the orientation represents the direction of the puncture needle. By combining the current position of the target point, the current position of the target puncture point, and the current pose of the puncture needle, the previous puncture path can be adjusted to obtain the adjusted puncture path as the current puncture path.

[0080] In some alternative embodiments, the registration of the target spatial coordinate system of the puncture needle with the scanned image coordinate system can be achieved by the aforementioned body surface film. For example, the registration between the two coordinate systems can be achieved by the position of the body surface film in the scanned image coordinate system and the actual position of the body surface film in the target spatial coordinate system. The specific details will not be elaborated further.

[0081] In some optional embodiments, after the current puncture path is determined, the puncture needle can be guided based on the current puncture path. For example, the direction of travel of the puncture needle can be adjusted according to the current puncture path and the travel distance of the puncture needle can be determined, and the puncture needle can be controlled to travel the corresponding travel distance in the direction of travel.

[0082] In some optional embodiments, the respiratory status of the target object and the pressure vector of the target body surface area can be monitored in real time or at regular intervals during the puncture process. Each time a sample is collected, the respiratory status can be used as the current respiratory status and the pressure vector can be used as the current pressure vector. The puncture path can be continuously adjusted according to the above process to guide the puncture needle until the puncture needle reaches the target point and the puncture is completed.

[0083] In some optional embodiments, a puncture guidance image including the puncture needle and the current puncture path can be generated under the current respiratory state by combining the current pose of the puncture needle and the current puncture path. Displaying the puncture guidance image allows the puncture operator to view the current puncture state and make corresponding judgments or actions based on the current puncture state, such as controlling the movement of the puncture needle, determining whether the puncture needle has reached the target point, etc.

[0084] The computer-based method for determining the puncture path based on a skin surface patch provided in this embodiment establishes a mapping relationship between the respiratory state of the subject, the pressure vector of the target skin surface region, and the movement of the target organ through a pre-trained target organ motion prediction model. This allows for real-time determination of the target organ's location and the location of the puncture point on the skin surface based on the subject's real-time respiratory state and pressure vector. This enables real-time determination of the current puncture path, effectively reducing the number of repeated scans and significantly mitigating adverse consequences for the subject. Furthermore, the real-time determination of the current puncture path effectively reduces repeated puncture errors caused by the non-real-time nature of CT and MRI scans, further reducing harm to the subject. Additionally, based on the skin surface patch, when predicting skin surface movement, only the motion vector of the patch area needs to be predicted, rather than the entire skin surface, effectively reducing the computational load and improving processing efficiency.

[0085] Figure 2 This is a flowchart illustrating a computer-implemented method for determining a puncture path based on a skin patch, provided in another exemplary embodiment of this disclosure.

[0086] In some optional embodiments, the first motion vector includes position change vectors corresponding to each voxel belonging to the target organ in the reference scan image sequence; the second motion vector includes position change vectors corresponding to each voxel belonging to the target body surface region in the reference scan image sequence.

[0087] The position change vector can include the position change distance and the change direction.

[0088] In some optional embodiments, step 240, which determines the current puncture path between the target puncture point and the target target point of the target organ based on the reference scan image sequence, the first motion vector, and the second motion vector, includes:

[0089] Step 2410: Based on the reference scan image sequence and the first motion vector, determine the first position of the target organ after motion and the second position of the target point after motion.

[0090] The specific principles for determining the first and second positions can be found in the aforementioned content, and will not be repeated here.

[0091] Step 2420: Based on the reference scan image sequence and the second motion vector, determine the third position of the target surface region after motion and the fourth position of the target puncture point after motion.

[0092] The specific principles for determining the third and fourth positions can be found in the aforementioned content, and will not be repeated here.

[0093] Step 2430: Determine the current puncture path based on the first position, second position, third position, and fourth position.

[0094] By combining the first position of the target organ, the second position of the target target point, the third position of the target body surface region, and the fourth position of the target puncture point, the positional relationship of the puncture path from the target puncture point to the target target point relative to the target body surface region and the target organ can be determined, which facilitates the generation of puncture guidance images.

[0095] In some optional embodiments, the body patch includes material lines arranged at preset distances that can absorb scanning responses. The material lines have a preset shape on the scanning cross-section for identifying the body patch in the scanned image sequence. The preset positions on the surface of the body patch also include protrusions or depressions of the target shape for registration of the scanned image coordinate system with VR and / or AR space.

[0096] The preset distance can be set according to actual needs, specifically based on the scan slice thickness (e.g., CT scan slice thickness or MRI scan slice thickness). The preset distance can be less than or equal to the scan slice thickness. The scan response can include at least one of the following: X-ray response from CT scans, magnetic field response from MRI scans, and scan responses from other scanning devices. The preset shape can also be set according to actual needs, such as circular, triangular, square, or rhomboid shapes. It can also be determined by combining solid or hollow states. During scanning, since the material pipeline has a preset shape on the scanned cross-section, a preset shaped image region will be formed on the scanned image of the material pipeline in the scanned image sequence. Based on this, by recognizing each scanned image in the scanned image sequence, the preset shaped body surface patch on the reference scanned image sequence can be identified. Then, by combining the predicted movement of the target body surface region based on the current respiratory state, the position of the moved body surface patch in the scanned image coordinate system can be determined. The preset position on the surface of the body surface patch can be set according to actual needs. The target shape can be set according to actual needs, such as circular, triangular, square, or rhomboid shapes, without specific limitations.

[0097] In some optional embodiments, the VR (Virtual Reality) and / or AR (Augmented Reality) space can be the space corresponding to the world coordinate system, or it can be the space of a reference coordinate system rigidly connected to the world coordinate system, set according to actual needs. The specific setting can be determined based on actual requirements. VR and / or AR-based terminal devices (such as head-mounted display devices, specifically VR glasses, AR glasses, etc.) can capture images of the surface of the body surface patch. By identifying the protrusions or depressions of the target shape in the image, the position of the body surface patch in the VR and / or AR space can be determined. Combined with the position of the body surface patch in the scanned image coordinate system obtained above, registration between the scanned image coordinate system and the VR and / or AR space can be achieved, obtaining the transformation relationship between the scanned image coordinate system and the target space coordinate system of the VR and / or AR space.

[0098] In some optional embodiments, a black-and-white graphic representing a preset shape of the scanned cross-section or a QR code or barcode designed according to the distribution of the material pipeline can be printed on the surface of the skin patch at the position corresponding to the material pipeline. The VR and / or AR terminal device captures an image of the surface of the skin patch, identifies the black-and-white graphic or QR code or barcode, and combines it with the position of the moved skin patch in the scanned image coordinate system obtained above to achieve registration between the scanned image coordinate system and the VR and / or AR space, and obtain the transformation relationship between the scanned image coordinate system and the target space coordinate system of the VR and / or AR space.

[0099] This embodiment can effectively achieve registration of the scanned image coordinate system with VR and / or AR space by applying a film to the body surface, thereby enabling puncture guidance based on at least one of VR technology, AR technology, mixed reality technology, etc., improving the convenience and accuracy of puncture operation for puncture workers.

[0100] Figure 3 This is a flowchart illustrating a computer-implemented method for determining a puncture path based on a skin patch, provided in yet another exemplary embodiment of this disclosure.

[0101] In some optional embodiments, the method of this disclosure further includes:

[0102] Step 250: Based on the skin surface film, register the current puncture path and the current pose of the puncture needle to VR and / or AR space.

[0103] In addition, based on the aforementioned content, the transformation relationship between the scanned image coordinate system and the target space coordinate system of VR and / or AR space is obtained based on the skin film. Based on this transformation relationship, the current puncture path and the current pose of the puncture needle can be registered to VR and / or AR space. This allows for the use of at least one of VR technology, AR technology, mixed reality technology, etc., to guide the puncture and improve the convenience and accuracy of the puncture operator.

[0104] This embodiment uses a skin patch to register the puncture scene to VR and / or AR space. It can combine VR and / or AR technology to scan the environment of the puncture scene and build a 3D model of the environment. By acquiring the pose of the head-mounted display device, the fused scene image of the puncture scene and the 3D model of the environment is displayed on the display screen of the head-mounted display device. This allows the puncture operator to perform the puncture operation more accurately and conveniently in real time under the guidance of VR and / or AR, greatly improving the convenience and accuracy of the operation.

[0105] Figure 4 This is a flowchart illustrating step 250 provided in an exemplary embodiment of this disclosure.

[0106] In some optional embodiments, step 250, based on a skin patch, registers the current puncture path and the current pose of the puncture needle to VR and / or AR space, including:

[0107] Step 2510: A VR and / or AR-based terminal device acquires a first image of the skin film.

[0108] The VR and / or AR terminal devices may be equipped with cameras to capture the first image of the film applied to the body surface of the target object.

[0109] Step 2520: Based on the first image, identify the protrusions or depressions of the target shape on the surface of the body film, and obtain the pixel positions of the protrusions or depressions of the target shape in the first image.

[0110] The pixel positions of the protrusions or depressions of the target shape in the first image can be detected based on a pre-trained target detection model. The target detection model can be any implementable model, and can be set according to actual needs. This disclosure does not limit it.

[0111] Step 2530: Determine the target pose of the body surface film in VR and / or AR space based on the pixel position and the camera parameters of the terminal device.

[0112] The camera parameters can include camera intrinsic parameters and camera extrinsic parameters. Based on the camera parameters, the pixel position can be transformed from the image coordinate system corresponding to the first image to the target space coordinate system of VR and / or AR space to obtain the target pose of the body surface film in VR and / or AR space.

[0113] Step 2540: Based on the target pose and the pose of the body surface film in the scanned image coordinate system, determine the transformation relationship between the scanned image coordinate system and VR and / or AR space.

[0114] Since the different poses of the same body surface film in two coordinate systems are determined, the translation and rotation between the two poses represent the translation and rotation between the two coordinate systems. Based on this, the transformation relationship between the scanned image coordinate system and VR and / or AR space can be determined.

[0115] Step 2550: Based on the transformation relationship, register the current puncture path and the current pose of the puncture needle to VR and / or AR space.

[0116] After determining the transformation relationship between the scanned image coordinate system and VR and / or AR space, the current puncture path and the current pose of the puncture needle can be transformed to the target space coordinate system of VR and / or AR space based on this transformation relationship, thereby achieving the registration of the current puncture path and the current pose of the puncture needle to VR and / or AR space.

[0117] This embodiment acquires a first image of the surface of the body patch using a VR and / or AR terminal device. By identifying the protrusions or depressions of the target shape in the first image, the pose of the body patch in VR and / or AR space is determined. Combined with the pose of the body patch in the coordinate system of the scanned image, accurate and effective registration of the current puncture path and puncture needle to VR and / or AR space is achieved, providing strong technical support for puncture guidance based on VR and / or AR technology.

[0118] In some optional embodiments, the current respiratory state includes the current respiratory volume and / or the current chest and abdominal surface pressure; obtaining the current respiratory state of the target object in step 210 includes:

[0119] The current respiratory status of the target object is obtained based on a respiratory status quantification device; the respiratory status quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal surface pressure acquisition device.

[0120] Among them, the respiratory volume quantification device is used to determine the respiratory volume of the target object, such as the current respiratory volume and the preset respiratory volume, and the chest and abdominal surface pressure acquisition device is used to determine the chest and abdominal surface pressure of the target object, such as the current chest and abdominal surface pressure and the preset chest and abdominal surface pressure mentioned above.

[0121] In some optional embodiments, the respiratory volume quantification device may include a breathing tubing and a controller; a first end of the breathing tubing is placed in the mouth of the target object, a breathing valve is provided in the inner lumen of the breathing tubing, and at least one sensor for detecting respiratory volume is provided in the tubing segment located between the breathing valve and the first end; the controller is connected to each sensor and is used to determine the respiratory volume based on the data collected by each sensor; the controller is also connected to a display and is used to display the data collected by each sensor and / or the determined respiratory volume on the display.

[0122] In some optional embodiments, the respiratory volume of the target object can be controlled by the breathing valve of the respiratory volume quantification device, specifically, by controlling the inspiratory volume and / or expiratory volume. When the respiratory volume of the target object reaches a preset threshold, the breathing valve is closed, and a scan is triggered to obtain the scan image sequence corresponding to the preset threshold. This preset threshold is taken as a respiratory state of the target object. By setting different preset thresholds, multiple scan image sequences corresponding to different respiratory states can be obtained.

[0123] In some optional embodiments, at least one sensor for detecting respiratory volume may include at least one of the following sensors: flow rate sensor, pressure sensor, turbine flow sensor, gas mass flow sensor, etc. The specific sensor can be set according to actual needs, as long as it can detect respiratory volume.

[0124] In some optional embodiments, the chest and abdominal surface pressure acquisition device includes a sensor based on a flexible piezoelectric material for detecting pressure values ​​on the chest and abdominal surface that change with respiration; the chest and abdominal surface pressure acquisition device is connected to a controller for displaying the detected pressure values ​​on a display screen via the controller.

[0125] In some optional embodiments, sensors based on flexible piezoelectric materials (which can be called flexible pressure sensors) can realize the mutual conversion of mechanical energy and electrical energy based on flexible piezoelectric materials. They can detect pressure signals and convert them into electrical signals according to a certain law, and have the characteristics of high sensitivity, fast response, high piezoelectric coefficient, and good environmental adaptability. In practical applications, sensors based on flexible piezoelectric materials can be placed around the surface of a target object, such as around the chest and abdomen. When the target object breathes, it will cause the chest and abdomen to rise and fall, thereby causing the flexible piezoelectric material to deform. The flexible piezoelectric material responds linearly to its deformation, converting the mechanical energy of the deformation into an electrical signal. Different changes in the chest and abdomen will output different electrical signals, which can represent the frequency of the chest and abdomen movement and the magnitude of the force applied to the flexible piezoelectric material, called the chest and abdomen surface pressure.

[0126] In some alternative embodiments, the flexible piezoelectric material can be implemented using any feasible material. For example, barium titanate (BaTiO3, BTO), lead zirconate titanate (Pb(PZT)), lithium niobate (LiNbO3, LNO), and potassium sodium niobate (K... 0.5 Na 0.5 One or more of the following materials can be used: NbO3 (KNN), zinc oxide (ZnO), manganese trimethylchloromethylaminochloride (TMCM-MnCl3), polyvinylidene fluoride (PVDF), and nylon. The specific choice can be determined based on actual requirements. PZT is a solid solution formed from lead zirconate (PbZrO3) and lead titanate (PbTiO3), and when their molar ratio is close to 52:48, a quasi-isomorphic phase boundary (MPB) is formed, thus exhibiting excellent piezoelectric properties.

[0127] This embodiment achieves quantitative acquisition of the respiratory status of the target subject through at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device, providing effective respiratory status data for motion prediction of target organs and target body surface areas based on respiratory status, and can acquire the current respiratory status of the target subject in real time during puncture.

[0128] In some optional embodiments, the current respiratory status includes the current respiratory volume and / or the current chest and abdominal surface pressure; the skin patch is provided with a flexible piezoelectric material;

[0129] Step 210, obtaining the current breathing state of the target object, includes:

[0130] The current respiratory volume of the target object is obtained based on a respiratory volume quantification device; the pressure of the target body surface area is obtained based on the flexible piezoelectric material on the body surface film as the current chest and abdominal body surface pressure.

[0131] The specific structure and function of the respiratory volume quantification device can be found in the preceding content and will not be repeated here. A flexible piezoelectric material is applied to the skin patch to detect the pressure in the target skin area as the current chest and abdominal skin pressure. The specific working principle of the flexible piezoelectric material can be found in the preceding content and will not be repeated here.

[0132] This embodiment achieves the acquisition of chest and abdominal surface pressure by using a skin patch. Thus, based on the skin patch, it can simultaneously achieve multiple purposes such as registration of the scanned image coordinate system to VR and / or AR space, detection of chest and abdominal surface pressure, and reduction of model calculation. It can also avoid the use of an additional chest and abdominal surface pressure acquisition device, reduce costs, and improve user experience.

[0133] In some alternative embodiments, the skin patch includes patches covering multiple sub-regions within a target skin area.

[0134] Step 210, obtaining the current pressure vector of the target body surface region corresponding to the skin patch on the target object, includes:

[0135] Based on the flexible piezoelectric material on the film of each sub-region, the pressure vector corresponding to each sub-region is obtained; the pressure vector corresponding to each sub-region is used as the current pressure vector of the target body surface region.

[0136] The size and shape of the sub-regions, as well as the distribution of multiple sub-regions, can be set according to actual needs. For example, a sub-region array can be set at certain intervals on the target body surface area. This sub-region array can include multiple rows and columns, thereby enabling the detection of pressure vectors in multiple sub-regions on the target body surface area. The film of each sub-region can be provided with the aforementioned anisotropic piezoelectric material, forming a flexible pressure sensor with anisotropic sensing characteristics. This sensor detects the magnitude and direction of stress in each sub-region of the target body surface area, generating corresponding piezoelectric signals. By fitting the piezoelectric signals, the pressure vectors of each sub-region of the target body surface area can be identified. Furthermore, by combining the pressure vectors of each sub-region, the current pressure vector of the target body surface area can be obtained.

[0137] This embodiment uses a surface patch on the target body surface area to set up multiple sub-region patches. Based on anisotropic flexible piezoelectric materials, it realizes the acquisition of pressure vectors in the target body surface area, which is used to predict the motion vectors of the target organ and the target body surface area, thereby further improving the accuracy of the prediction results.

[0138] In some optional embodiments, the target organ motion prediction model is obtained through the following steps:

[0139] The respiratory state quantification device determines different respiratory states of the target object, acquires the pressure vector of the target body surface region corresponding to each respiratory state, and triggers scanning in each respiratory state to obtain a scan image sequence corresponding to each respiratory state. The respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each scan image sequence is labeled to obtain the position of the target organ corresponding to each scan image sequence. Using any scan image sequence as a reference scan image sequence, the third motion vector of the target organ corresponding to each scan image sequence relative to the target organ corresponding to the reference scan image sequence is determined based on the position of the target organ corresponding to each scan image sequence. Based on each respiratory state, the pressure vector corresponding to each respiratory state, and each third motion vector, the initial organ motion prediction model is trained to obtain the target organ motion prediction model.

[0140] The specific training process can be found in the aforementioned embodiments, and will not be repeated here.

[0141] In some optional embodiments, the target body surface motion prediction model is obtained through the following steps:

[0142] The respiratory state quantification device determines different respiratory states of the target object, acquires the pressure vector of the target body surface region corresponding to each respiratory state, and triggers scanning in each respiratory state to obtain a scan image sequence corresponding to each respiratory state. The respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each scan image sequence is labeled to obtain the position of the target body surface region corresponding to each scan image sequence. Using any scan image sequence as a reference scan image sequence, the fourth motion vector of the target body surface region corresponding to each scan image sequence relative to the target body surface region corresponding to the reference scan image sequence is determined based on the position of the target body surface region corresponding to each scan image sequence. Based on each respiratory state, the pressure vector corresponding to each respiratory state, and each fourth motion vector, the initial body surface motion prediction model is trained to obtain the target body surface motion prediction model.

[0143] The specific training process can be found in the aforementioned embodiments, and will not be repeated here.

[0144] In some optional embodiments, step 220, based on the current respiratory state and current pressure vector, uses a pre-trained target organ motion prediction model to determine the first motion vector of the target organ in the scan image coordinate system corresponding to the reference scan image sequence, including:

[0145] Based on the current respiratory state, current pressure vector, and reference scan image sequence, the first motion vector of the target organ in the scan image coordinate system corresponding to the reference scan image sequence is determined using a pre-trained target organ motion prediction model.

[0146] The model incorporates the target object's current respiratory state, current pressure vector, and reference scan image sequence as inputs to predict the motion vector of the target organ, which can further improve the accuracy of the prediction results.

[0147] Step 230, based on the current respiratory state and current pressure vector, uses a pre-trained target surface motion prediction model to determine the second motion vector of the target surface region in the scanned image coordinate system, including:

[0148] Based on the current respiratory state, current pressure vector, and reference scan image sequence, the second motion vector of the target body surface region in the scan image coordinate system is determined using a pre-trained target body surface motion prediction model.

[0149] The model incorporates the target object's current respiratory state, current pressure vector, and reference scan image sequence as inputs to predict the motion vector of the target object's surface region, which can further improve the accuracy of the prediction results.

[0150] For training a target organ motion prediction model that predicts the first motion vector of a target organ based on the current respiratory state, the current pressure vector, and a reference scan image sequence, the training process may include: acquiring scan image sequences corresponding to at least two respiratory states of the target object and the pressure vector of the target body surface region corresponding to each respiratory state; using any scan image sequence as a reference scan image sequence, determining the motion vector of the target organ relative to the target organ in the reference scan image sequence in each respiratory state; and training the initial organ motion prediction model based on each respiratory state, the pressure vector corresponding to each respiratory state, the reference scan image sequence, and the motion vector corresponding to each respiratory state to obtain a trained target organ motion prediction model. This target organ motion prediction model can then predict the first motion vector of the target organ based on the current respiratory state, the reference scan image sequence, and the current pressure vector of the target body surface region. The training of a target body surface motion prediction model that predicts the second motion vector of the target body surface region based on the current respiratory state, the current pressure vector, and the reference scan image sequence is similar to that of the corresponding target organ motion prediction model and will not be elaborated further here.

[0151] Specifically, for the target object, the model input can be the target object's respiratory states, reference scan image sequences, and pressure vectors of the target body surface regions corresponding to each respiratory state. The training prediction results corresponding to each respiratory state are obtained. The motion vectors corresponding to each respiratory state are used as labels and compared with the training prediction results. The network loss is determined based on a preset loss function and used to update the model network parameters until the network loss meets preset conditions (such as convergence) or the number of iterations reaches a preset threshold. The training ends, and a trained target organ motion prediction model is obtained. For example, based on the scan image sequences corresponding to two preset respiratory states—end-expiration and end-inspiration—and the pressure vector of the target body surface region, the scan image sequence corresponding to end-expiration is used as the reference scan image sequence, and the scan image sequence corresponding to end-inspiration is used as the motion scan image sequence. By registering the scan image sequences of the two preset respiratory states, the target motion vector of the target organ in the motion scan image sequence at end-inspiration is obtained relative to the target organ in the reference scan image sequence. The motion vector of the reference scan image sequence relative to itself is 0. Based on the two preset respiratory states, the reference scan image sequence, the pressure vectors corresponding to the two respiratory states, and the motion vectors of the target organs corresponding to the two respiratory states, the initial organ motion prediction model is further trained to establish the mapping relationship between the respiratory state of the target object, the reference scan image sequence, the pressure vector, and the motion vector of the target organ. Alternatively, the initial organ motion prediction model can be trained by combining at least one respiratory state between end-expiration and end-inspiration, the pressure vector corresponding to that respiratory state, and the motion vector of the target organ in the scan image sequence triggered by that respiratory state relative to the reference scan image sequence, to further improve the accuracy of the model's prediction results. Based on this, an initial organ motion prediction model can be trained using a small number of scanned image sequences of the target object to obtain the mapping relationship between the target object's respiratory state, pressure vector, reference scanned image sequence, and motion vector of the target organ. This can be used for real-time localization of the target object's target point, thereby improving the accuracy of the target point localization results while effectively reducing the number of scans, scanning time, and absorbed dose of the target object.

[0152] In some optional embodiments, step 220, based on the current respiratory state and current pressure vector, uses a pre-trained target organ motion prediction model to determine the first motion vector of the target organ in the scan image coordinate system corresponding to the reference scan image sequence, including:

[0153] Based on the current respiratory state and current pressure vector, the motion vector of the target point in the target organ of the target object in the scanning image coordinate system corresponding to the reference scanning image sequence is determined using a pre-trained target organ motion prediction model, and is used as the first motion vector.

[0154] In this embodiment, the target organ motion prediction model is directly used to predict the motion vector of the target point based on the target object's current respiratory state and current pressure vector, which can effectively reduce the computational load of the model. The corresponding model training uses the motion vector of the target point as a label for training, and the specific details will not be elaborated further.

[0155] Step 240, based on the reference scan image sequence, the first motion vector, and the second motion vector, determines the current puncture path between the target puncture point and the target target point of the target organ, including:

[0156] Based on the reference scan image sequence and the first motion vector, the current position of the target point after movement is determined; based on the reference scan image sequence and the second motion vector, the current position of the target puncture point after movement is determined; based on the current position of the target point and the current position of the target puncture point, the current puncture path is determined.

[0157] In some optional embodiments, step 230, which determines the second motion vector of the target body surface region in the scanned image coordinate system based on the current respiratory state and current pressure vector using a pre-trained target body surface motion prediction model, may include: determining the motion vector of the target puncture point in the target body surface region of the target object in the scanned image coordinate system corresponding to the reference scanned image sequence, based on the current respiratory state and current pressure vector and using the pre-trained target body surface motion prediction model, as the second motion vector. This achieves direct prediction of the motion vector of the target puncture point based on the model, further reducing the computational load of the model.

[0158] In some optional embodiments, the target organ motion prediction model is obtained through the following steps:

[0159] The respiratory state quantification device determines different respiratory states of the target object, acquires the pressure vector of the target surface region corresponding to each respiratory state, and triggers scanning in each respiratory state to obtain a scan image sequence corresponding to each respiratory state. The respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal surface pressure acquisition device. Each scan image sequence is labeled to obtain the position of the target point corresponding to each scan image sequence. Any scan image sequence is used as a reference scan image sequence, and the fifth motion vector of the target point corresponding to each scan image sequence relative to the target point corresponding to the reference scan image sequence is determined based on the position of the target point corresponding to each scan image sequence. Based on each respiratory state, the pressure vector corresponding to each respiratory state, and each fifth motion vector, the initial organ motion prediction model is trained to obtain the target organ motion prediction model.

[0160] This training process is similar to the model training process used to predict the motion vectors of target organs. The difference is that the labels used in this training process are the motion vectors of the target points corresponding to each respiratory state. The details will not be elaborated further.

[0161] In this embodiment, the motion vector of the target point is used as a label for training the target organ motion prediction model. The obtained model can directly predict the motion vector of the target point, thereby quickly determining the position of the target point. Compared with predicting the motion vector of the target organ, the computational load of the model can be effectively reduced.

[0162] In some optional embodiments, the target body surface motion prediction model is obtained through the following steps:

[0163] The respiratory state quantification device determines different respiratory states of the target object, acquires the pressure vector of the target body surface region corresponding to each respiratory state, and triggers scanning in each respiratory state to obtain a scan image sequence corresponding to each respiratory state. The respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each scan image sequence is labeled to obtain the position of the target puncture point corresponding to each scan image sequence. Using any scan image sequence as a reference scan image sequence, the sixth motion vector of the target puncture point corresponding to each scan image sequence relative to the target puncture point corresponding to the reference scan image sequence is determined based on the position of the target puncture point corresponding to each scan image sequence. Based on each respiratory state, the pressure vector corresponding to each respiratory state, and each sixth motion vector, the initial body surface motion prediction model is trained to obtain the target body surface motion prediction model.

[0164] This training process is similar to the model training process used to predict the motion vectors of the target body surface region. The difference is that the labels used in this training process are the motion vectors of the target puncture point corresponding to each breathing state. The details will not be elaborated further.

[0165] In this embodiment, the motion vector of the target puncture point is used as a label for training the target body surface motion prediction model. The obtained model can directly predict the motion vector of the target puncture point, thereby quickly determining the position of the target puncture point. Compared with predicting the motion vector of the target body surface area, it can effectively reduce the computational load of the model.

[0166] In some optional embodiments of this disclosure, after determining the current puncture path between the target puncture point and the target target point of the target organ based on the reference scan image sequence, the first motion vector, and the second motion vector in step 240, the method further includes:

[0167] Based on the current puncture path, generate robot operation instructions; based on the robot operation instructions, control the robot to perform puncture operation on the target object.

[0168] The robot in question is capable of performing puncture procedures. Its operational instructions are generated by combining path planning and navigation technology for the puncture route, along with individualized information about the target object displayed in imaging and the specific requirements of the puncture operation. These instructions can be tailored to individual needs. For robot control, the corresponding operational instructions are sent to the robot's drive unit according to its control rules, causing the robot to execute the actions corresponding to those instructions, thus enabling it to complete the puncture procedure.

[0169] Due to the complexity of robot dynamics, the strong coupling between degrees of freedom, and the presence of nonlinear characteristics such as hysteresis, this disclosure addresses this issue by achieving precise operation based on robot control technology. Specifically, it combines offline physical modeling of the robot with online active identification of dynamic disturbances to actively suppress unmodeled dynamic disturbances, including hysteresis effects. This approach is validated on robots with different drive methods to obtain high operational accuracy, and the validated robot is then used to execute the puncture operation described in this disclosure.

[0170] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0171] Any computer-based method for determining a puncture path based on a skin patch provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, any computer-based method for determining a puncture path based on a skin patch provided in this disclosure can be executed by a processor, such as by a processor calling corresponding instructions stored in memory to execute any computer-based method for determining a puncture path based on a skin patch mentioned in this disclosure. Further details will not be elaborated below.

[0172] Exemplary device

[0173] Figure 5 This is a schematic diagram of a computer-implemented apparatus for determining a puncture path based on a skin dressing, provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement corresponding method embodiments of this disclosure, such as... Figure 5 The apparatus shown includes: a first acquisition module 510, a first processing module 520, a second processing module 530, and a third processing module 540.

[0174] The first acquisition module 510 is used to acquire the current respiratory state of the target object, the reference scan image sequence, and the current pressure vector of the target body surface region corresponding to the body surface patch of the target object.

[0175] The first processing module 520 is used to determine the first motion vector of the target organ of the target object in the scanning image coordinate system corresponding to the reference scanning image sequence, based on the current respiratory state and the current pressure vector and using a pre-trained target organ motion prediction model.

[0176] The second processing module 530 is used to determine the second motion vector of the target body surface region in the scanned image coordinate system based on the current breathing state and the current pressure vector and using a pre-trained target body surface motion prediction model.

[0177] The third processing module 540 is used to determine the current puncture path between the target puncture point and the target target point of the target organ based on the reference scan image sequence, the first motion vector and the second motion vector. The target puncture point is any point within the target body surface area.

[0178] In some optional embodiments, the first motion vector includes position change vectors corresponding to each voxel belonging to the target organ in the reference scan image sequence; the second motion vector includes position change vectors corresponding to each voxel belonging to the target body surface region in the reference scan image sequence.

[0179] The third processing module 540 is specifically used for:

[0180] Based on the reference scan image sequence and the first motion vector, the first position of the target organ after motion and the second position of the target point after motion are determined; based on the reference scan image sequence and the second motion vector, the third position of the target surface region after motion and the fourth position of the target puncture point after motion are determined; based on the first position, the second position, the third position and the fourth position, the current puncture path is determined.

[0181] In some optional embodiments, the body patch includes material lines arranged at preset distances that can absorb scanning responses. The material lines have a preset shape on the scanning cross-section for identifying the body patch in the scanned image sequence. The preset positions on the surface of the body patch also include protrusions or depressions of the target shape for registration of the scanned image coordinate system with VR and / or AR space.

[0182] Figure 6 This is a schematic diagram of the structure of a computer-implemented device for determining a puncture path based on a skin patch, provided in another exemplary embodiment of this disclosure.

[0183] In some optional embodiments, the apparatus of this disclosure further includes:

[0184] The fourth processing module 550 is used to register the current puncture path and the current pose of the puncture needle to VR and / or AR space based on the body surface film.

[0185] In some optional embodiments, the fourth processing module 550 is specifically used for:

[0186] A VR and / or AR-based terminal device acquires a first image of a skin patch; based on the first image, it identifies protrusions or depressions of a target shape on the surface of the skin patch, and obtains the pixel positions of the protrusions or depressions of the target shape in the first image; based on the pixel positions and the camera parameters of the terminal device, it determines the target pose of the skin patch in VR and / or AR space; based on the target pose and the pose of the skin patch in the scanned image coordinate system, it determines the transformation relationship between the scanned image coordinate system and VR and / or AR space; based on the transformation relationship, it registers the current puncture path and the current pose of the puncture needle to VR and / or AR space.

[0187] In some optional embodiments, the current respiratory state includes the current respiratory volume and / or the current chest and abdominal surface pressure; the first acquisition module 510 is specifically used for:

[0188] The current respiratory status of the target object is obtained based on a respiratory status quantification device; the respiratory status quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal surface pressure acquisition device.

[0189] In some optional embodiments, the current respiratory state includes the current respiratory volume and / or the current chest and abdominal surface pressure; a flexible piezoelectric material is disposed on the surface patch; the first acquisition module 510 is specifically used for:

[0190] The current respiratory volume of the target object is obtained based on a respiratory volume quantification device; the pressure of the target body surface area is obtained based on the flexible piezoelectric material on the body surface film as the current chest and abdominal body surface pressure.

[0191] In some optional embodiments, the skin patch includes patches covering multiple sub-regions within the target skin area; the first acquisition module 510 is specifically used for:

[0192] Based on the flexible piezoelectric material on the film of each sub-region, the pressure vector corresponding to each sub-region is obtained; the pressure vector corresponding to each sub-region is used as the pressure vector of the target body surface region.

[0193] In some optional embodiments, the apparatus of this disclosure further includes: a respiratory state quantification device for acquiring the current respiratory state of the target object.

[0194] In some alternative embodiments, the respiratory status quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal surface pressure acquisition device.

[0195] Figure 7 This is a schematic diagram of the structure of a respiratory volume measurement device provided in an exemplary embodiment of this disclosure.

[0196] The breathing volume measurement device 560 includes a breathing tubing 5610 and a controller 5620. The breathing tubing can be a flexible hose or a straight tube, and there is no specific limitation.

[0197] The first end of the breathing tube 5610 is used to be placed in the mouth of the target object. The inner cavity of the breathing tube is provided with a breathing valve 5630. At least one sensor 5640 for detecting breathing volume is provided in the section of the breathing tube 5610 between the breathing valve 5630 and the first end.

[0198] The breathing valve 5630 can be controlled to rotate in at least one of the two directions indicated by the arrows, so that the target object can inhale and / or exhale gas when breathing. For example, when it is necessary to detect the inspiratory volume, the breathing valve 5630 can be controlled to open only in the first direction to allow gas inhalation. When the inspiratory volume reaches a preset threshold, the breathing valve is controlled to close, thereby limiting the inspiratory volume of the target object and achieving control of a specific breathing state. In turn, a scan can be triggered in this breathing state to obtain a scan image sequence in this breathing state.

[0199] In some alternative embodiments, the scan can be triggered automatically, for example, by configuring a controller to communicate with the scanning device. When the controller detects or controls the breathing valve to close, it sends a trigger signal to the scanning device to trigger the scan. The scan can also be triggered by sending a trigger signal to the operator's terminal device, causing the operator to trigger the scan, and so on. The specific triggering method is not limited.

[0200] The controller 5620 is connected to each sensor 5640 to determine the breathing volume based on the data collected by each sensor 5640. The controller 5620 is also connected to the breathing valve 5630 to control the opening and closing of the breathing valve 5630.

[0201] The connection method between the controller 5620, the breathing valve 5630, and the sensor 5640 can be set according to actual needs. For example, it can be connected wirelessly, as long as communication between the sensor and the controller can be achieved.

[0202] The controller 5620 can also be connected to a display to show the data collected by each sensor 5640 and / or the determined respiratory volume.

[0203] In some optional embodiments, the chest and abdominal surface pressure acquisition device includes a sensor based on a flexible piezoelectric material for detecting pressure values ​​on the chest and abdominal surface that change with respiration; the chest and abdominal surface pressure acquisition device is connected to a controller for displaying the detected pressure values ​​on a display screen via the controller.

[0204] The specific working principle of the sensor based on flexible piezoelectric material is described in the aforementioned embodiments and will not be repeated here.

[0205] In some optional embodiments, the current respiratory state includes the current respiratory volume and / or the current chest and abdominal surface pressure; a flexible piezoelectric material is disposed on the skin patch; the device disclosed herein further includes:

[0206] A respiratory volume measurement device is used to obtain the current respiratory volume of a target subject. The specific structure and working principle of a respiratory volume measurement device can be found in the preceding content and will not be repeated here.

[0207] The skin patch is also used to obtain the pressure of the target skin area as the current chest and abdominal skin pressure.

[0208] The specific working principle of obtaining chest and abdominal surface pressure by applying a skin patch is described in the corresponding method implementation, and will not be repeated here.

[0209] In some optional embodiments, the target organ motion prediction model is obtained through the following steps:

[0210] The respiratory state quantification device determines different respiratory states of the target object, acquires the pressure vector of the target body surface region corresponding to each respiratory state, and triggers scanning in each respiratory state to obtain a scan image sequence corresponding to each respiratory state. The respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each scan image sequence is labeled to obtain the position of the target organ corresponding to each scan image sequence. Using any scan image sequence as a reference scan image sequence, the third motion vector of the target organ corresponding to each scan image sequence relative to the target organ corresponding to the reference scan image sequence is determined based on the position of the target organ corresponding to each scan image sequence. Based on each respiratory state, the pressure vector corresponding to each respiratory state, and each third motion vector, the initial organ motion prediction model is trained to obtain the target organ motion prediction model.

[0211] In some optional embodiments, the target body surface motion prediction model is obtained through the following steps:

[0212] The respiratory state quantification device determines different respiratory states of the target object, acquires the pressure vector of the target body surface region corresponding to each respiratory state, and triggers scanning in each respiratory state to obtain a scan image sequence corresponding to each respiratory state. The respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each scan image sequence is labeled to obtain the position of the target body surface region corresponding to each scan image sequence. Using any scan image sequence as a reference scan image sequence, the fourth motion vector of the target body surface region corresponding to each scan image sequence relative to the target body surface region corresponding to the reference scan image sequence is determined based on the position of the target body surface region corresponding to each scan image sequence. Based on each respiratory state, the pressure vector corresponding to each respiratory state, and each fourth motion vector, the initial body surface motion prediction model is trained to obtain the target body surface motion prediction model.

[0213] In some alternative embodiments, the first processing module 520 is specifically used for:

[0214] Based on the current respiratory state, current pressure vector, and reference scan image sequence, the first motion vector of the target organ in the scan image coordinate system corresponding to the reference scan image sequence is determined using a pre-trained target organ motion prediction model.

[0215] The second processing module 530 is specifically used for:

[0216] Based on the current respiratory state, current pressure vector, and reference scan image sequence, the second motion vector of the target body surface region in the scan image coordinate system is determined using a pre-trained target body surface motion prediction model.

[0217] In some optional embodiments, the first processing module 520 is specifically used for:

[0218] Based on the current respiratory state and current pressure vector, the motion vector of the target point in the target organ of the target object in the scanning image coordinate system corresponding to the reference scanning image sequence is determined using a pre-trained target organ motion prediction model, and is used as the first motion vector.

[0219] The third processing module 540 is specifically used for:

[0220] Based on the reference scan image sequence and the first motion vector, the current position of the target point after movement is determined; based on the reference scan image sequence and the second motion vector, the current position of the target puncture point after movement is determined; based on the current position of the target point and the current position of the target puncture point, the current puncture path is determined.

[0221] In some optional embodiments, the target organ motion prediction model is obtained through the following steps:

[0222] The respiratory state quantification device determines different respiratory states of the target object, acquires the pressure vector of the target surface region corresponding to each respiratory state, and triggers scanning in each respiratory state to obtain a scan image sequence corresponding to each respiratory state. The respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal surface pressure acquisition device. Each scan image sequence is labeled to obtain the position of the target point corresponding to each scan image sequence. Any scan image sequence is used as a reference scan image sequence, and the fifth motion vector of the target point corresponding to each scan image sequence relative to the target point corresponding to the reference scan image sequence is determined based on the position of the target point corresponding to each scan image sequence. Based on each respiratory state, the pressure vector corresponding to each respiratory state, and each fifth motion vector, the initial organ motion prediction model is trained to obtain the target organ motion prediction model.

[0223] In some optional embodiments, the target body surface motion prediction model is obtained through the following steps:

[0224] The respiratory state quantification device determines different respiratory states of the target object, acquires the pressure vector of the target body surface region corresponding to each respiratory state, and triggers scanning in each respiratory state to obtain a scan image sequence corresponding to each respiratory state. The respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each scan image sequence is labeled to obtain the position of the target puncture point corresponding to each scan image sequence. Using any scan image sequence as a reference scan image sequence, the sixth motion vector of the target puncture point corresponding to each scan image sequence relative to the target puncture point corresponding to the reference scan image sequence is determined based on the position of the target puncture point corresponding to each scan image sequence. Based on each respiratory state, the pressure vector corresponding to each respiratory state, and each sixth motion vector, the initial body surface motion prediction model is trained to obtain the target body surface motion prediction model.

[0225] In some optional embodiments, the third processing module 540 can also be used to: generate robot operation instructions based on the current puncture path; and control the robot to perform puncture operations on the target object based on the robot operation instructions.

[0226] The embodiments described above can be implemented individually or in any combination without conflict, and can be set according to actual needs.

[0227] The specific operation and beneficial effects of the various embodiments of the device disclosed herein are similar to those of the corresponding method embodiments, and will not be repeated here.

[0228] Figure 8This is a schematic diagram of a puncture guidance system based on a skin patch provided in an exemplary embodiment of this disclosure. The puncture guidance system may include a computer-implemented device for determining the puncture path based on the skin patch as provided in any of the above embodiments, as well as a VR device and / or an AR device.

[0229] Exemplary electronic devices

[0230] This disclosure also provides an electronic device, including: a memory for storing computer programs;

[0231] A processor is configured to execute a computer program stored in the memory, and when the computer program is executed, to implement the computer implementation method for determining the puncture path based on a skin patch as described in any of the above embodiments of this disclosure.

[0232] Figure 9 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0233] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0234] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0235] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0236] For example, the input device 13 can be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0237] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0238] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0239] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0240] Exemplary computer program products and computer-readable storage media

[0241] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0242] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0243] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0244] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0245] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0246] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0247] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0248] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0249] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0250] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0251] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A computer-based method for determining puncture paths based on skin surface patches, applied to medical auxiliary equipment, characterized in that, include: Obtain the current respiratory state of the target object, the reference scan image sequence, and the current pressure vector of the target body surface region corresponding to the body surface patch of the target object; The reference scan image sequence includes a scan image sequence obtained under a preset respiratory state; Based on the current respiratory state and the current pressure vector, the first motion vector of the target organ of the target object in the scanning image coordinate system corresponding to the reference scanning image sequence is determined using a pre-trained target organ motion prediction model; the target organ motion prediction model is a neural network model used to predict the motion of the target organ of the target object relative to the reference scanning image sequence. Based on the current respiratory state and the current pressure vector, a second motion vector of the target body surface region in the coordinate system of the scanned image is determined using a pre-trained target body surface motion prediction model; the target body surface motion prediction model is a neural network model used to predict the motion of the target body surface region of the target object relative to the reference scanned image sequence. Based on the reference scan image sequence, the first motion vector, and the second motion vector, the current puncture path between the target puncture point and the target target point of the target organ is determined, wherein the target puncture point is any point within the target body surface area; the target puncture point is determined based on the reference scan image sequence and the second motion vector.

2. The method according to claim 1, characterized in that, The step of determining the first motion vector of the target organ of the target object in the scan image coordinate system corresponding to the reference scan image sequence, based on the current respiratory state and the current pressure vector and using a pre-trained target organ motion prediction model, includes: Based on the current respiratory state and the current pressure vector, using a pre-trained target organ motion prediction model, the motion vector of the target target point in the target organ of the target object in the scan image coordinate system corresponding to the reference scan image sequence is determined as the first motion vector. The step of determining the current puncture path between the target puncture point and the target target point of the target organ based on the reference scan image sequence, the first motion vector, and the second motion vector includes: Based on the reference scan image sequence and the first motion vector, determine the current position of the target point after its motion; Based on the reference scan image sequence and the second motion vector, determine the current position of the target puncture point after its motion; The current puncture path is determined based on the current position of the target point and the current position of the puncture point.

3. The method according to claim 1, characterized in that, The first motion vector includes the position change vectors corresponding to each voxel belonging to the target organ in the reference scan image sequence; the second motion vector includes the position change vectors corresponding to each voxel belonging to the target body surface region in the reference scan image sequence. The step of determining the current puncture path between the target puncture point and the target target point of the target organ based on the reference scan image sequence, the first motion vector, and the second motion vector includes: Based on the reference scan image sequence and the first motion vector, determine the first position of the target organ after motion and the second position of the target point after motion; Based on the reference scan image sequence and the second motion vector, the third position of the target body surface region after motion and the fourth position of the target puncture point after motion are determined. The current puncture path is determined based on the first position, the second position, the third position, and the fourth position.

4. The method according to claim 1, characterized in that, The body surface patch includes absorbable scanning response material lines arranged at a preset distance. The material lines have a preset shape on the scanning cross-section and are used to identify the body surface patch in the scanned image sequence. The preset position on the surface of the body film also includes protrusions or depressions of the target shape, used for registration of the scan image coordinate system with VR and / or AR space.

5. The method according to claim 1, characterized in that, Also includes: Based on the skin patch, the current puncture path and the current pose of the puncture needle are registered to VR and / or AR space.

6. The method according to claim 5, characterized in that, The step of registering the current puncture path and the current pose of the puncture needle to VR and / or AR space based on the skin patch includes: A VR and / or AR-based terminal device captures the first image of the body surface film; Based on the first image, identify the protrusions or depressions of the target shape on the surface of the body film, and obtain the pixel position of the protrusions or depressions of the target shape in the first image. Based on the pixel position and the camera parameters of the terminal device, the target pose of the body surface film in the VR and / or AR space is determined; Based on the target pose and the pose of the body surface film in the scanned image coordinate system, determine the transformation relationship between the scanned image coordinate system and the VR and / or AR space; Based on the transformation relationship, the current puncture path and the current pose of the puncture needle are registered to the VR and / or AR space.

7. The method according to claim 1, characterized in that, The current respiratory status includes the current respiratory volume and / or the current chest and abdominal surface pressure; The step of obtaining the current breathing state of the target object includes: The current respiratory state of the target object is obtained based on the respiratory state quantification device; The respiratory status quantitative device includes at least one of a respiratory volume quantitative device and a chest and abdominal surface pressure acquisition device.

8. The method according to claim 1, characterized in that, The current respiratory status includes the current respiratory volume and / or the current chest and abdominal surface pressure; the surface patch is provided with flexible piezoelectric material; The step of obtaining the current breathing state of the target object includes: The current respiratory volume of the target object is obtained based on the respiratory volume quantification device; The pressure of the target body surface region is obtained based on the flexible piezoelectric material on the body surface patch and is used as the current chest and abdominal body surface pressure.

9. The method according to claim 8, characterized in that, The body surface patch includes patches covering multiple sub-regions within the target body surface area; Obtaining the pressure vector of the target body surface region corresponding to the skin patch on the target object includes: Based on the flexible piezoelectric material on the film of each sub-region, the pressure vector corresponding to each sub-region is obtained; The pressure vector corresponding to each of the sub-regions is taken as the pressure vector of the target body surface region.

10. The method according to claim 1, characterized in that, The target organ motion prediction model is obtained through the following steps: The different respiratory states of the target object are determined based on the respiratory state quantification device, the pressure vector of the target body surface area corresponding to each respiratory state is obtained, and scanning is triggered in each respiratory state to obtain the scanning image sequence corresponding to each respiratory state; the respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each of the scanned image sequences is labeled to obtain the location of the target organ corresponding to each scanned image sequence; Using any one of the scanned image sequences as the reference scanned image sequence, a third motion vector of the target organ corresponding to each of the scanned image sequences relative to the target organ corresponding to the reference scanned image sequence is determined based on the position of the target organ corresponding to each of the scanned image sequences. Based on each of the aforementioned respiratory states, the pressure vector corresponding to each of the aforementioned respiratory states, and each of the aforementioned third motion vectors, the initial organ motion prediction model is trained to obtain the target organ motion prediction model; and / or, The target body surface motion prediction model is obtained through the following steps: The different respiratory states of the target object are determined based on the respiratory state quantification device, the pressure vector of the target body surface area corresponding to each respiratory state is obtained, and scanning is triggered in each respiratory state to obtain the scanning image sequence corresponding to each respiratory state; the respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each of the scanned image sequences is labeled to obtain the location of the target body surface region corresponding to each of the scanned image sequences; Using any of the scanned image sequences as the reference scanned image sequence, a fourth motion vector is determined relative to the target body surface region corresponding to each of the scanned image sequences and the target body surface region corresponding to the reference scanned image sequence, based on the position of the target body surface region corresponding to each of the scanned image sequences. Based on each of the breathing states, the pressure vectors corresponding to each of the breathing states, and each of the fourth motion vectors, the initial body surface motion prediction model is trained to obtain the target body surface motion prediction model.

11. The method according to claim 1, characterized in that, The step of determining the first motion vector of the target organ of the target object in the scan image coordinate system corresponding to the reference scan image sequence, based on the current respiratory state and the current pressure vector and using a pre-trained target organ motion prediction model, includes: Based on the current respiratory state, the current pressure vector, and the reference scan image sequence, the first motion vector of the target organ of the target object in the scan image coordinate system corresponding to the reference scan image sequence is determined using a pre-trained target organ motion prediction model. The step of determining the second motion vector of the target body surface region in the scanned image coordinate system based on the current respiratory state and the current pressure vector, using a pre-trained target body surface motion prediction model, includes: Based on the current respiratory state, the current pressure vector, and the reference scan image sequence, the second motion vector of the target body surface region in the scan image coordinate system is determined using a pre-trained target body surface motion prediction model.

12. The method according to claim 2, characterized in that, The target organ motion prediction model is obtained through the following steps: The different respiratory states of the target object are determined based on the respiratory state quantification device, the pressure vector of the target body surface area corresponding to each respiratory state is obtained, and scanning is triggered in each respiratory state to obtain the scanning image sequence corresponding to each respiratory state; the respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each of the scanned image sequences is labeled to obtain the position of the target point corresponding to each of the scanned image sequences; Using any of the scanned image sequences as the reference scanned image sequence, a fifth motion vector of the target point corresponding to each of the scanned image sequences relative to the target point corresponding to the reference scanned image sequence is determined based on the position of the target point corresponding to each of the scanned image sequences. Based on each of the aforementioned respiratory states, the pressure vector corresponding to each of the aforementioned respiratory states, and each of the aforementioned fifth motion vectors, the initial organ motion prediction model is trained to obtain the target organ motion prediction model; and / or, The target body surface motion prediction model is obtained through the following steps: The different respiratory states of the target object are determined based on the respiratory state quantification device, the pressure vector of the target body surface area corresponding to each respiratory state is obtained, and scanning is triggered in each respiratory state to obtain the scanning image sequence corresponding to each respiratory state; the respiratory state quantification device includes at least one of a respiratory volume quantification device and a chest and abdominal body surface pressure acquisition device. Each of the scanned image sequences is labeled to obtain the location of the target puncture point corresponding to each of the scanned image sequences; Using any of the scanned image sequences as the reference scanned image sequence, a sixth motion vector of the target puncture point corresponding to each of the scanned image sequences relative to the target puncture point corresponding to the reference scanned image sequence is determined based on the position of the target puncture point corresponding to each of the scanned image sequences. Based on each of the breathing states, the pressure vectors corresponding to each of the breathing states, and each of the sixth motion vectors, the initial body surface motion prediction model is trained to obtain the target body surface motion prediction model.

13. The method according to any one of claims 1-12, characterized in that, Also includes: Generate robot operation instructions based on the current puncture path; The robot operation instructions are sent to the robot's drive unit.

14. A computer-based device for determining puncture paths based on a skin surface dressing, characterized in that, include: The first acquisition module is used to acquire the current respiratory state of the target object, the reference scan image sequence, and the current pressure vector of the target body surface region corresponding to the body surface patch of the target object; The reference scan image sequence includes a scan image sequence obtained under a preset respiratory state; The first processing module is used to determine the first motion vector of the target organ of the target object in the scanning image coordinate system corresponding to the reference scanning image sequence, based on the current respiratory state and the current pressure vector and using a pre-trained target organ motion prediction model; the target organ motion prediction model is a neural network model used to predict the motion of the target organ of the target object relative to the reference scanning image sequence. The second processing module is used to determine the second motion vector of the target body surface region in the coordinate system of the scanned image based on the current breathing state and the current pressure vector and using a pre-trained target body surface motion prediction model; the target body surface motion prediction model is a neural network model used to predict the motion of the target body surface region of the target object relative to the reference scanned image sequence. The third processing module is used to determine the current puncture path between the target puncture point and the target target point of the target organ based on the reference scan image sequence, the first motion vector and the second motion vector, wherein the target puncture point is any point within the target body surface area; the target puncture point is determined based on the reference scan image sequence and the second motion vector.

15. A computer storage medium, characterized in that, The storage medium stores a computer program for executing the computer implementation method of the puncture path determined based on a skin patch as described in any one of claims 1-13.

16. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the computer implementation method of the puncture path determined based on the body surface patch as described in any one of claims 1-13.

Images