3D Vision-Based Human Carotid Artery Localization Method and Device

Abstract

The present disclosure provides a method and device for human carotid artery localization based on 3D vision, which relates to the field of intelligent medical technology, and includes obtaining ultrasonic image data; using deep learning to detect the ultrasonic image data to generate human specific position feature point parameters; according to the human specific position feature point parameters, using a radial basis function to determine the projection position of the carotid artery; projecting the position of the carotid artery in the ultrasonic image onto the neck surface to guide autonomous ultrasonic scanning. Features are recognized from the facial and neck images captured by the 3D camera, and the projection position of the carotid artery is modeled based on these features. This projection position can be used as an initial planning path for robotic autonomous ultrasonic scanning to guide automated carotid artery autonomous ultrasonic scanning.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent medical technologies, and in particular, to a method and apparatus for human carotid artery localization based on 3D vision. Background Art

[0002] In medical imaging systems such as magnetic resonance imaging (MR) systems or computed tomography imaging (CT) systems, it is sometimes necessary to use a 3D camera in cooperation to collect auxiliary information such as the patient's body position information. The camera of the 3D camera usually consists of a two-dimensional color (RGB) camera and a depth camera.

[0003] When installing the 3D camera, it is usually necessary to install the 3D camera directly above the hospital bed to detect the hospital bed and the patient on the hospital bed, so as to obtain the relevant status and information of the patient to the greatest extent. Summary of the Invention

[0004] The purpose of the present disclosure is to propose a method and apparatus for human carotid artery localization based on 3D vision. The method for human carotid artery localization based on 3D vision is used for robotic ultrasound scanning of the carotid artery.

[0005] The first aspect of the present disclosure provides a method for human carotid artery localization based on 3D vision, including:

[0006] Obtaining ultrasonic image data;

[0007] Using deep learning to detect the ultrasonic image data and generating parameter of characteristic points at specific positions of the human body;

[0008] According to the parameter of characteristic points at specific positions of the human body, using a radial basis function to determine the projection position of the carotid artery;

[0009] Projecting the position of the carotid artery in the ultrasonic image onto the neck surface to guide autonomous ultrasound scanning.

[0010] The second aspect of the present disclosure provides an apparatus for human carotid artery localization based on 3D vision, including:

[0011] An obtaining module, which obtains ultrasonic image data;

[0012] A generating module, which uses deep learning to detect the ultrasonic image data and generates parameter of characteristic points at specific positions of the human body;

[0013] A determining module, which according to the parameter of characteristic points at specific positions of the human body, uses a radial basis function to determine the projection position of the carotid artery;

[0014] A guiding module, which projects the position of the carotid artery in the ultrasonic image onto the neck surface to guide autonomous ultrasound scanning.

[0015] A third aspect of the present disclosure provides an electronic device, including: a memory and one or more processors;

[0016] The memory is used for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for positioning a human carotid artery based on 3D vision provided in any embodiment.

[0018] A fourth aspect of the present disclosure provides a storage medium containing computer-executable instructions, and the computer-executable instructions implement the method for positioning a human carotid artery based on 3D vision provided in any embodiment when implemented by a computer processor.

[0019] The present disclosure provides a method and device for positioning a human carotid artery based on 3D vision, which identify features from facial and neck images captured by a 3D camera, and model the projection position of the carotid artery according to these features. This projection position can be used as an initial planning path for autonomous ultrasonic scanning by a robot to guide automated autonomous ultrasonic scanning of the carotid artery. Description of the Drawings

[0020] Figure 1 It is a flowchart of the method for positioning a human carotid artery based on 3D vision in an embodiment of the present disclosure;

[0021] Figure 2 It is another flowchart of the method for positioning a human carotid artery based on 3D vision in an embodiment of the present disclosure;

[0022] Figure 3 It is another flowchart of the method for positioning a human carotid artery based on 3D vision in an embodiment of the present disclosure;

[0023] Figure 4 For Figure 1 It is a schematic diagram of feature points;

[0024] Figure 5 It is a schematic diagram of the device for positioning a human carotid artery based on 3D vision in an embodiment of the present disclosure;

[0025] Figure 6 It is a schematic diagram of the device for positioning a human carotid artery based on 3D vision in an embodiment of the present disclosure;

[0026] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Embodiments

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the present invention, rather than limiting the present invention. In addition, it should be noted that for the sake of description, only the parts related to the present invention are shown in the drawings, rather than all the structures.

[0028] Since a 3D camera cannot directly sense arterial blood vessels, it is necessary to identify features from the facial and neck images captured by the 3D camera, and model the projection position of the carotid artery based on these features. This projection position can be used as the initial planning path for the robot's autonomous ultrasound scanning to guide the automated autonomous ultrasound scanning of the carotid artery. This method includes two stages: feature recognition and projection position modeling.

[0029] As Figure 1 shown, an embodiment of the present disclosure provides a method for positioning the carotid artery of the human body based on 3D vision, including:

[0030] S101. Obtain ultrasonic image data;

[0031] The ultrasonic image data includes: aligned color map data and depth map data;

[0032] The obtaining of the ultrasonic image data includes: using a 3D camera to take pictures to obtain aligned color map data and depth map data.

[0033] In the embodiment of the present disclosure, a fixedly installed 3D camera is used to capture RGB-D color maps and depth maps of the complete human face and neck regions, and calculate the projection position of the carotid artery on the neck surface therefrom.

[0034] S102. Use deep learning to detect the ultrasonic image data and generate feature point parameters of specific positions of the human body. In the embodiment of the present disclosure, the specific positions of the human body include at least one of the left lower jaw contour, the right lower jaw contour, the left neck contour, and the right neck contour.

[0035] As Figure 2 and 4 shown, using deep learning to detect the ultrasonic image data and generate feature point parameters of specific positions of the human body includes:

[0036] S201. Use deep learning to detect the feature points of the left lower jaw contour, the right lower jaw contour, the left neck contour, and the right neck contour, and generate corresponding fitting curves;

[0037] S202. Uniformly sample each fitted curve according to the set number of samples;

[0038] S203. Calculate the feature point parameters according to the number of samplings. For i from 0 to N - 1, calculate the corresponding sampling point coordinates according to the following formula to complete the sampling; the feature point parameters include: sampling step length and sampling point coordinates;

[0039]

[0040] x i = x0 + i·t

[0041] y i = L(x i )

[0042] S204. According to the feature point parameters, use the Lagrange interpolation method to perform curve fitting on the extracted feature points to generate a corresponding fitting curve; or,

[0043] S205. If a certain one of the four curves of the left lower jaw contour, right lower jaw contour, left neck contour, and right neck contour fails to be successfully extracted in the above S201 step, then record all the sampling point coordinates on the curve where the extraction fails as a preset value. For example, if there is a large angular deflection of the head, it will cause the feature points on some curves to fail to be extracted. All the sampling point coordinates on the curve where the extraction fails are recorded as (-1, -1).

[0044] For the curve fitting of the extracted feature points using the Lagrange interpolation method to generate a corresponding fitting curve, the following formula is used:

[0045] where L(x) represents the analytical expression of the fitting curve, l i (x) is a term in the expression L(x), (x i , y i ) are the coordinates of the feature point, x i is the abscissa, y i is the ordinate, and k is the number of feature

[0046]

[0047]

[0048] points.

[0049] S103. According to the feature point parameters of specific positions of the human body, use the radial basis function to determine the projection position of the carotid artery;

[0050] S104. Project the position of the carotid artery in the ultrasonic image onto the neck surface to guide the autonomous ultrasonic scanning.

[0051] For the projection of the carotid artery onto the neck surface to guide the autonomous ultrasonic scanning, the following formula is used:

[0052]

[0053] Among them, u and v are the horizontal and vertical coordinates of the image projection position points, D is the depth map data, where f x , f y are the focal lengths of the 3D camera in the x or y direction respectively, c x , c y is the offset of the optical axis from the coordinate center of the projection plane.

[0054] As Figure 3 shown, the method for determining the projection position of the carotid artery using the radial basis function according to the parameter of the specific position feature point of the human body includes:

[0055] S301. Calculate the projection position probability distribution of the carotid artery on the color image according to the parameter of the specific position feature point of the human body;

[0056]

[0057] Among them, F represents the set of feature points, n is the number of feature points, μ i is the center of the radial basis function, determined by the coordinates of the feature points. and λ i are the variance and coefficient of the radial basis function, which are parameters to be trained.

[0058] S302. According to the projection position probability distribution, (or pre-calculate and store the parameters of the model using the maximum likelihood estimation model). Specifically, pre-collect and label the training data {X i , F i}, including different deflection postures, F i represents the set of feature points extracted, and X i is the set of image projection position points of the corresponding carotid artery. Estimate the model parameters λ and σ;

[0059]

[0060] S303. Extract the region where the probability reaches the set threshold and the preset region according to the parameter of the specific position feature point of the human body and the parameter of the pre-constructed maximum likelihood estimation model, so as to obtain the projection position of the carotid artery on the color image.

[0061] As can be seen from the above, the present disclosure provides a method for positioning the carotid artery of the human body based on 3D vision, which models the position of the carotid artery based on the features of the face and neck, overcomes the defect that the 3D camera cannot directly perceive the carotid artery, avoids the process of manual path planning and searching before autonomous ultrasound scanning, and realizes the automation of the whole process.

[0062] As Figure 5 and 6As shown in the figure, the 3D vision-based human carotid artery positioning device 600 in the embodiments of the present disclosure includes:

[0063] An acquisition module 601, configured to acquire ultrasonic image data;

[0064] A generation module 602, configured to detect the ultrasonic image data by using deep learning and generate human body specific position feature point parameters;

[0065] A determination module 603, configured to determine the projection position of the carotid artery by using a radial basis function according to the human body specific position feature point parameters;

[0066] A guidance module 604, configured to project the position of the carotid artery in the ultrasonic image onto the neck surface to guide autonomous ultrasonic scanning.

[0067] The ultrasonic image data includes: aligned color map data and depth map data;

[0068] The acquisition module 601 is configured to acquire the aligned color map data and depth map data by using a 3D camera for photographing.

[0069] The human body specific position includes at least one of a left lower jaw contour, a right lower jaw contour, a left neck contour, and a right neck contour;

[0070] The generation module 602 is configured to detect feature points of the left lower jaw contour, the right lower jaw contour, the left neck contour, and the right neck contour by using deep learning and generate corresponding fitting curves;

[0071] Uniformly sample each fitted curve according to a set number of samples;

[0072] According to the number of samples, calculate the feature point parameters to complete the sampling; the feature point parameters include: a sampling step size and sampling point coordinates;

[0073] According to the feature point parameters, use the Lagrange interpolation method to perform curve fitting on the extracted feature points to generate corresponding fitting curves.

[0074] When using the Lagrange interpolation method to perform curve fitting on the extracted feature points to generate corresponding fitting curves, the following formula is used:

[0075]

[0076]

[0077] Among them, L(x) represents the analytical expression of the fitting curve, li(x) is a term in the expression L(x), (x i , y i ) are the coordinates of the feature points, x is the abscissa, y = L(x) is the ordinate, and k is the number of feature points.

[0078] The determination module 603 is configured to calculate the projection position probability distribution of the carotid artery on the color map according to the characteristic point parameters of the specific positions of the human body;

[0079] Calculate the parameters of the pre-constructed maximum likelihood estimation model according to the projection position probability distribution;

[0080] Extract the regions and preset regions where the probability reaches the set threshold according to the characteristic point parameters of the specific positions of the human body and the parameters of the pre-constructed maximum likelihood estimation model, so as to obtain the projection position of the carotid artery on the color map.

[0081] The determination module 603 is configured to, if the regions where the extracted probability does not reach the set threshold and / or are not in the preset region, record all the sampling point coordinates on the curve where the extraction fails as preset values.

[0082] The projection of the carotid artery onto the neck surface is used to guide the autonomous ultrasound scan by the following formula:

[0083]

[0084] Where u and v are the horizontal and vertical coordinates of the picture projection position point, D is the depth map data, where f x f y are the focal lengths of the 3D camera in the x or y direction respectively, c x c y is the offset of the optical axis from the coordinate center of the projection plane.

[0085] The 3D vision-based human carotid artery positioning device provided by the embodiments of the present disclosure can execute the 3D vision-based human carotid artery positioning method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0086] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. As Figure 7 shown, the electronic device of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0087] Exemplarily, the computer program 703 can be divided into one or more modules / units, which are stored in the memory 702 and executed by the processor 701 to complete the present disclosure. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program 703 in the electronic device.

[0088] The electronic device can be a desktop computer, a notebook, a palm computer, a cloud server, and other electronic devices. The electronic device can include, but is not limited to, the processor 701 and the memory 702. Those skilled in the art can understand that Figure 7 merely examples of the electronic device, which do not constitute a limitation to the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device may further include input / output devices, network access devices, a bus, etc.

[0089] The processor 701 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0090] The memory 702 can be an internal storage unit of the electronic device, for example, the hard disk or memory of the electronic device. The memory 702 can also be an external storage device of the electronic device, for example, a plug-in hard disk equipped on the electronic device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 702 can also include both the internal storage unit and the external storage device of the electronic device. The memory 702 is used to store the computer program and other programs and data required by the electronic device. The memory 702 can also be used to temporarily store the data that has been output or will be output.

[0091] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example for illustration. In actual applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0092] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0093] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in the form of hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this disclosure.

[0094] In the embodiments provided by this disclosure, it should be understood that the disclosed device / electronic device and method can be implemented in other ways. For example, the device / electronic device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division, and there can be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the device or unit can be in electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0096] In addition, in each embodiment of the present disclosure, each functional unit may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0097] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present disclosure, it may also be completed by instructing relevant hardware through a computer program. The computer program may be stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments may be implemented. The computer program may include computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0098] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A method for positioning the human carotid artery based on 3D vision, characterized in that, Including: Obtain ultrasonic image data; Use deep learning to detect ultrasonic image data and generate feature point parameters at specific positions of the human body; According to the feature point parameters at specific positions of the human body, use a radial basis function to determine the projection position of the carotid artery; Project the position of the carotid artery in the ultrasonic image onto the neck surface to guide autonomous ultrasonic scanning; Among them, the step of using a radial basis function to determine the projection position of the carotid artery according to the feature point parameters at specific positions of the human body includes: According to the feature point parameters at specific positions of the human body, calculate the projection position probability distribution of the carotid artery on the color map; According to the projection position probability distribution, calculate the parameters of a pre-constructed maximum likelihood estimation model; According to the feature point parameters at specific positions of the human body and the parameters of the pre-constructed maximum likelihood estimation model, extract the regions and preset regions where the probability reaches the set threshold to obtain the projection position of the carotid artery on the color map.

2. The method according to claim 1, wherein The ultrasonic image data includes: aligned color map data and depth map data; The step of obtaining ultrasonic image data includes: using a 3D camera to take pictures to obtain aligned color map data and depth map data.

3. The method according to claim 1, characterized in that, The specific positions of the human body include at least one of the left lower jaw contour, right lower jaw contour, left neck contour, and right neck contour; Using deep learning to detect ultrasonic image data and generate feature point parameters at specific positions of the human body includes: Use deep learning to detect the feature points of the left lower jaw contour, right lower jaw contour, left neck contour, and right neck contour, and generate corresponding fitting curves; Perform uniform sampling on each fitted curve according to the set number of samples; According to the number of samples, calculate the feature point parameters to complete the sampling; the feature point parameters include: sampling step length and sampling point coordinates; According to the feature point parameters, use the Lagrange interpolation method to perform curve fitting on the extracted feature points to generate corresponding fitting curves; or, If a certain one of the four curves of the left lower jaw contour, right lower jaw contour, left neck contour, and right neck contour fails to be successfully extracted, all the sampling point coordinates on the curve with extraction failure are recorded as preset values.

4. The method according to claim 3, characterized in that, The formula for using the Lagrange interpolation method to perform curve fitting on the extracted feature points to generate corresponding fitting curves is as follows: Among them, L(x) represents the analytical expression of the fitting curve, and l i (x) is a term in the expression L(x), (x i , y i ) are the coordinates of the feature points, x i is the abscissa, y i is the ordinate, and k is the number of feature points.

5. The method according to claim 3, wherein The formula for guiding autonomous ultrasonic scanning according to the projection of the carotid artery onto the neck surface is as follows: where u and v are the horizontal and vertical coordinates of the image projection position points, D is the depth map data, and f x , f y are the focal lengths of the 3D camera in the x or y directions respectively, c x , c y is the offset of the optical axis from the coordinate center of the projection plane.

6. A human carotid artery positioning device based on 3D vision, characterized in that, Including: An acquisition module for obtaining ultrasonic image data; A generation module for using deep learning to detect ultrasonic image data and generate feature point parameters at specific positions of the human body; A determination module for using a radial basis function to determine the projection position of the carotid artery according to the feature point parameters at specific positions of the human body; A guidance module for projecting the position of the carotid artery in the ultrasonic image onto the neck surface to guide autonomous ultrasonic scanning; Among them, the determination module is used to calculate the projection position probability distribution of the carotid artery on the color map according to the feature point parameters at specific positions of the human body; According to the projection position probability distribution, calculate the parameters of a pre-constructed maximum likelihood estimation model; According to the feature point parameters at specific positions of the human body and the parameters of the pre-constructed maximum likelihood estimation model, extract the regions and preset regions where the probability reaches the set threshold to obtain the projection position of the carotid artery on the color map.

7. An electronic device, characterized in that, Including: a memory and one or more processors; the memory for storing one or more programs; when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the method according to any one of claims 1-5.

8. A storage medium containing computer-executable instructions, characterized in that, the computer-executable instructions implement the method according to any one of claims 1-5 when implemented by a computer processor.

Images