An ultrasonic scanning system with 3D vision and contact force fusion sensing optimization

Abstract

The present invention discloses an ultrasound scanning system with 3D vision and contact force fusion sensing optimization, the system comprising: an ultrasound probe, the ultrasound probe is configured to perform continuous ultrasound scanning on a patient to form multiple ultrasound images; one or more three-dimensional vision acquisition modules, the one or more three-dimensional vision acquisition modules are used to continuously acquire the patient and the ultrasound probe to form three-dimensional visual information; a processing device, the processing device is used to generate a three-dimensional ultrasound model matching the multiple ultrasound images in real time according to the first position information of the three-dimensional vision acquisition module, the multiple ultrasound images and the three-dimensional visual information, the three-dimensional ultrasound model includes the position and orientation of the three-dimensional patient model and the ultrasound probe. The present invention provides a three-dimensional ultrasound model to enhance the perception of ultrasound doctors, and standardizes, digitizes and visualizes the process of ultrasound scanning. At the same time, since the entire scanning process can be recorded, it can be used to guide other ultrasound doctors to read ultrasound images.

Description

Technical Field

[0001] The present invention relates to the field of ultrasonic scanning technology, and in particular to an ultrasonic scanning system with 3D vision and contact force fusion sensing optimization. Background Art

[0002] Ultrasound scanning is a commonly used method in medical treatment, and the experience and level of ultrasound doctors directly determine the quality of ultrasound scanning. The quality assessment of ultrasound scanning needs to be judged with the help of additional information in the ultrasound scan, such as the relative position of the ultrasound probe and the patient's body. Because different ultrasound doctors use ultrasound probes to perform ultrasound scans on human parts, there will be different ultrasound probe orientations, which will result in different ultrasound images when different doctors operate the same part of the human body. The disadvantage of this situation is that when the doctor judges the ultrasound image, if the ultrasound image is not scanned by himself, he will not be able to read the ultrasound image. This will hinder the treatment of patients. Summary of the invention

[0003] In view of the above technical problems, the present invention provides an ultrasonic scanning system with 3D vision and contact force fusion sensing optimization to solve the above-mentioned problems.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0005] The present invention discloses an ultrasound scanning system with 3D vision and contact force fusion sensing optimization, the system comprising: an ultrasound probe, the ultrasound probe being configured to perform continuous ultrasound scanning on a patient to form a plurality of ultrasound images; one or more three-dimensional vision acquisition modules, the one or more three-dimensional vision acquisition modules being used to continuously acquire images of the patient and the ultrasound probe to form three-dimensional visual information; a processing device, the processing device being used to generate a three-dimensional ultrasound model matching the plurality of ultrasound images in real time according to first position information of the three-dimensional vision acquisition module, the plurality of ultrasound images and the three-dimensional visual information, the three-dimensional ultrasound model comprising a three-dimensional patient model and the position and orientation of the ultrasound probe.

[0006] Furthermore, the processing device includes a position measurement module, a three-dimensional reconstruction module, and a fusion module, wherein: the position measurement module is used to calculate the first position information and the three-dimensional visual information to obtain second position information of the ultrasound probe relative to the patient, and the second position information includes the spatial coordinates and orientation of the ultrasound probe; the three-dimensional reconstruction module is used to obtain the three-dimensional patient model based on the first position information and the three-dimensional visual information; and the fusion module is used to synthesize the second position information and the three-dimensional patient model to obtain the three-dimensional ultrasound.

[0007] Furthermore, in the process of obtaining the second position information, the position measurement module specifically performs one or more of the following situations:

[0008] Acquire a three-dimensional point cloud model of the ultrasound probe in the three-dimensional visual information, and obtain the second position information according to a spatial position transfer matrix of the three-dimensional point cloud model of the ultrasound probe;

[0009] Identify a two-dimensional image contained in the three-dimensional visual information, wherein the two-dimensional image has a first marking image set on the housing of the ultrasound probe, calculate a relative spatial position relationship between the first marking image and the three-dimensional visual acquisition module, and obtain second position information of the ultrasound probe;

[0010] Identify a plurality of second marker images of the data cable disposed on the ultrasound probe in the three-dimensional visual information, calculate a relative spatial position relationship between the second marker image and the three-dimensional visual acquisition module, and obtain second position information of the ultrasound probe;

[0011] A three-dimensional point cloud model of a physical marker structure arranged on the ultrasound probe is obtained in the three-dimensional visual information, and the second position information is obtained according to a spatial position transfer matrix of the three-dimensional point cloud model of the physical marker structure, wherein the physical marker structure has multiple structures which are regular or irregular in shape and present different colors.

[0012] Furthermore, the position measurement module is also used to obtain a two-dimensional image containing the physical marking structure, and obtain the second position information of the ultrasound probe by calculating the relative spatial position relationship between the physical marking structure and the three-dimensional vision acquisition module in the two-dimensional image.

[0013] Furthermore, one or more of the three-dimensional vision acquisition modules are connected to the same robotic arm as the ultrasound probe; or one or more of the three-dimensional vision acquisition modules are arranged on one or more multi-axis robots that can move freely and independently, and the multi-axis robots are arranged at any position that can acquire one or more of the ultrasound probes and the patient's posture, and the multi-axis robot has multiple degrees of freedom of movement.

[0014] Furthermore, when acquiring the ultrasound image, the processing device also obtains contact force information generated by the ultrasound probe contacting the patient, and the processing device matches the contact force information into the three-dimensional ultrasound model.

[0015] Furthermore, the ultrasonic wave generating area of ​​the ultrasonic probe is covered with an elastic sensing material, and the ultrasonic image includes the shape of the elastic sensing material; the processing device calculates the deformation amount of the elastic sensing material based on the ultrasonic image, and obtains the contact force information exerted on the elastic sensing material based on the deformation amount.

[0016] Furthermore, the ultrasonic probe is connected to a force sensing module, and the force sensing module is used to sense the contact force information when an external force is applied to the ultrasonic probe.

[0017] Furthermore, the three-dimensional vision acquisition module includes one or both of a three-dimensional camera and a three-dimensional line scan laser scanner.

[0018] Furthermore, the processing device is also used to display the three-dimensional ultrasound model including the ultrasound image and the three-dimensional visual information in real time, and store it in the form of streaming media or pictures.

[0019] The technical solution disclosed in this disclosure has the following beneficial effects:

[0020] Provided is an ultrasound scanning system with 3D vision and contact force fusion sensing optimization, which can perform standardized processing on the ultrasound scanning process after optimization. The experience and level of the ultrasound doctor directly determine the quality of the ultrasound scan. The quality assessment of the ultrasound scan needs to be judged with the help of additional information in the ultrasound scan, such as the relative position of the ultrasound probe and the patient's body. The system provides a three-dimensional ultrasound model to enhance the perception of the ultrasound doctor, uses the relative position of the ultrasound probe to confirm the acquisition point of the three-dimensional ultrasound model and the two-dimensional ultrasound image, and uses force sensing technology to sense the contact force between the ultrasound probe and the patient to judge the doctor's technique during the scanning process. Therefore, based on the present disclosure, the behavior of the ultrasound doctor during the scanning process can be recorded, supplemented by the corresponding three-dimensional ultrasound model, two-dimensional ultrasound image and contact force information. With the help of the present disclosure, standardized information can be collected for any ultrasound scanning process, and a standardized model can be established at one time.

[0021] The required information will not only be displayed in real time during the ultrasound scan, but will also be recorded and used as a cross-verification after the scan is completed. During the film reading and diagnosis process, the system can provide the doctor with all the information during the ultrasound scan, thereby assisting the doctor in making a better judgment on the ultrasound results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the structure of the ultrasonic scanning system in the embodiment of this specification;

[0023] Figure 2 This is an exemplary structural diagram of a three-dimensional vision acquisition module in an embodiment of this specification;

[0024] Figure 3 Another exemplary structural diagram of the 3D visual acquisition module in the embodiment of this specification

[0025] Figure 4 This is a schematic diagram of the force of the elastic sensing material of the ultrasonic probe in the embodiment of this specification;

[0026] Figure 5 This is a flow chart of a method for calculating contact force information from deformation variables in an embodiment of this specification;

[0027] Figure 6 is one exemplary structural diagram of an ultrasonic probe in an embodiment of this specification;

[0028] Figure 7 This is a structural block diagram of a processing device in an embodiment of this specification;

[0029] Figure 8 This is another exemplary structural diagram of the ultrasound probe in the embodiments of this specification.

[0030] Explanation of the reference numerals in the accompanying drawings: 100, ultrasonic probe; 101, elastic sensing material; 102, object to be detected; 103, first shell; 104, second shell; 105, first marking image; 106, second marking image; 107, physical marking structure; 200, three-dimensional vision device; 201, three-dimensional vision acquisition module; 202, multi-axis robot; 203, action wheel; 300, processing device; 301, position measurement module; 302, three-dimensional reconstruction module; 303, fusion module; 304, contact force calculation module; 400, display device. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In order to provide an easy-to-understand 3D vision-guided surgical navigation system framework, many specific details are involved in the following system description, such as specific examples of design, components, methods, etc. However, the rights and interests protected by this patent include but are not limited to the specific examples described in the text, and any specific examples that can be deployed or replaced to implement this technical route should be protected.

[0033] The "3D visual acquisition module" in this patent may be a device that can provide a 3D depth image, point cloud or other digital output within a scene. The "ultrasound probe" may refer to an ultrasound probe with a "+" shape, a "T" shape or a matrix shape, which can scan multiple planes or provide real-time 3D volumetric ultrasound scanning.

[0034] like Figure 1 As shown, an embodiment of the present specification provides an ultrasound scanning system with 3D vision and contact force fusion sensing optimization, including: an ultrasound probe 100, which is configured to perform continuous ultrasound scanning on a patient to form a plurality of ultrasound images; a three-dimensional vision device 200, which includes one or more three-dimensional vision acquisition modules 201, and the one or more three-dimensional vision acquisition modules 201 are used to continuously acquire the patient and the ultrasound probe 100 to form three-dimensional visual information; a processing device 300, and the processing device 300 is used to generate a three-dimensional ultrasound model matching the plurality of ultrasound images in real time according to the first position information of the three-dimensional vision acquisition module 201, the plurality of ultrasound images and the three-dimensional visual information, wherein the three-dimensional ultrasound model includes the position and orientation of the three-dimensional patient model and the ultrasound probe 100.

[0035] Among them, the processing device 300 can be a device with computing and storage functions such as a computer, a controller, a server, etc., which calculates the first position information, multiple ultrasound images and three-dimensional visual information to obtain a three-dimensional ultrasound model, which can be implemented by one or more modules / units running thereof. The module / unit stores an ultrasound scanning algorithm, an AI model, etc. The AI ​​model can be divided into functional areas such as visual three-dimensional model reconstruction and ultrasound three-dimensional model reconstruction.

[0036] The 3D visual device 200 scans and records the contents within its working range, and can obtain the initial information of the 3D model of the patient and the ultrasound probe 100, that is, the 3D visual information. The 3D visual information is recognized and processed in the processing device 300, and a 3D patient model and a model of the ultrasound probe 100 are generated based on the actual scanned data, that is, the patient's body and the ultrasound probe 100 are matched to the 3D model.

[0037] The ultrasonic image is generated by the ultrasonic probe 100. Each time it is generated, the position and orientation of the corresponding ultrasonic probe 100 will be different, and these differences will be recorded by the three-dimensional visual device 200 and reflected in the three-dimensional ultrasonic model. In other words, the processing device 300 can use the relevant features of different ultrasonic images, three-dimensional visual information and the posture information of the ultrasonic probe 100 to obtain a three-dimensional ultrasonic model by fusing the obtained ultrasonic image with the three-dimensional visual information. According to the three-dimensional ultrasonic model, any ultrasound physician knows the ultrasonic image currently being read, the corresponding orientation of the ultrasonic probe 100 when it is generated, the relative position with the patient and where the patient's body is scanned, so that the ultrasound physician can make a judgment and diagnosis of the ultrasonic image.

[0038] In addition, in order to improve the practicality of the present invention, the ultrasonic scanning system may further include a display device 400, which may play ultrasonic images in real time so that the user may know the ultrasonic scanning situation. The display device 400 may also display a three-dimensional ultrasonic model in real time to display it in a more intuitive way.

[0039] In one embodiment, if Figure 2 As shown, one or more three-dimensional vision acquisition modules 201 are connected to the same mechanical arm as the ultrasound probe 100; or Figure 3 As shown, one or more three-dimensional vision acquisition modules 201 are arranged on one or more multi-axis robots 202 that can move freely and independently. The multi-axis robot 202 is arranged at any position that can acquire one or more ultrasound probes 100 and the patient's posture, and the multi-axis robot 202 has multiple degrees of freedom of movement.

[0040] In addition, the multi-axis robot 202 can be actively or manually controlled by the robot controller so that a more ideal visual acquisition field of view can be obtained. In addition, the existence of the robot controller makes the spatial position of the three-dimensional visual acquisition module 201 known.

[0041] At the same time, in order to ensure that the three-dimensional vision acquisition module 201 can acquire one or more ultrasound probes 100 and the patient's posture, the multi-axis robot 202 can be provided with motion wheels 203 at the bottom to facilitate the movement of the multi-axis robot 202 according to the usage scenario.

[0042] As a supplement, the three-dimensional vision acquisition module 201 includes one or both of a three-dimensional camera and a three-dimensional line scan laser scanner.

[0043] In one embodiment, when acquiring the ultrasound image, the processing device 300 also acquires the contact force information generated by the ultrasound probe 100 contacting the patient, and the processing device 300 matches the contact force information into the three-dimensional ultrasound model.

[0044] Since contact force is also one of the criteria for judging the quality of ultrasound imaging, obtaining contact force information can not only provide a basis for judgment for ultrasound physicians in the future, but also be used as an operating standard for ultrasound physicians when performing ultrasound scanning.

[0045] like Figure 4 As shown, the ultrasonic wave generating area of ​​the ultrasonic probe 100 is covered with an elastic sensing material 101, and the ultrasonic image includes the shape of the elastic sensing material 101; the processing device 300 calculates the deformation amount of the elastic sensing material 101 based on the ultrasonic image, and obtains the contact force information exerted on the elastic sensing material 101 based on the deformation amount.

[0046] In order to understand the positional relationship and application scenarios of the elastic sensing material 101 and the ultrasonic probe 100, please continue to refer to Figure 4 , the elastic sensing material 101 covered on the surface of the ultrasonic probe 100 contacts the surface of the detected object 102, and a distributed force is applied on the surface of the elastic sensing material 101. When the elastic sensing material 101 is any deformable material, the applied force will cause the elastic sensing material 101 to deform. The elastic sensing material 101 is nested on the ultrasonic generating component of the ultrasonic probe 100 in the form of a protective cover. The exemplary first housing 103 can protect the ultrasonic probe 100 and fix the elastic sensing material 101.

[0047] When ultrasonic imaging of the patient is performed, ultrasonic imaging of the elastic sensing material 101 can also be performed. Generally speaking, the distance and position between each point in the elastic sensing material 101 and the ultrasonic detection are fixed. Specifically, the elastic sensing material 101 is covered / attached to the ultrasonic probe 100, and in order not to affect the emission of ultrasonic waves, there is no air between the elastic sensing material 101 and the ultrasonic probe 100. Specifically, a coupling agent can be provided between the elastic sensing material 101 and the ultrasonic probe 100 to ensure good imaging quality. In addition, the elastic sensing material 101 should be a material compatible with ultrasonic detection, that is, it can be understood that the ultrasonic energy emitted by the ultrasonic probe 100 can easily pass through the elastic sensing material 101 and has no effect on the ultrasonic imaging effect.

[0048] The elastic sensing material 101 can be displayed in ultrasound images, such as Figure 5As shown, region 6 is the acquired ultrasound image, region 7 is the image of the object to be detected 102, and the region of the elastic sensing material 101 in the ultrasound image is extracted as region 8. By calculating region 8 through a regular algorithm / method, the deformation amount of the elastic sensing material 101 can be extracted, such as region 9. Since the force on each surface / point in the elastic sensing material 101 is different, the deformation amount generated is also different. By calculating the deformation amount of each point / surface, that is, calculating its displacement, the force on each point / surface can be obtained, such as region 10. Exemplarily, the calculation method can be as follows:

[0049] Δd=d i -d e , Δd is the deformation, the initial position d i and deformation position d e The difference between.

[0050] The deformation variable Δd of each point / surface needs to be calculated, and the force at each point / surface can be calculated using Δd as follows: F=k×Δd.

[0051] Where k is the elastic coefficient k that satisfies Hooke's law, which can be known from the known elastic sensing material, and the contact force information can be obtained.

[0052] In another embodiment, in order to obtain the contact force information, a force sensing module may be used. The ultrasonic probe 100 is connected to the force sensing module, and the force sensing module is used to sense the contact force information when an external force is applied to the ultrasonic probe 100.

[0053] Among them, Figure 6 As shown, the force sensing module can be a multi-axis force sensor, which is distributed in the second shell 104 of the ultrasonic probe 100 and connected between the second shell 104 and the ultrasonic probe 100. When the ultrasonic probe 100 is touched, the force generated is directly fed back to the multi-axis force sensor, so the contact force information can be recorded.

[0054] In one embodiment, if Figure 7 As shown, the processing device 300 includes a position measurement module 301, a three-dimensional reconstruction module 302, a fusion module 303 and a contact force module, wherein: the position measurement module 301 is used to calculate the first position information and the three-dimensional visual information to obtain the second position information of the ultrasound probe 100 relative to the patient, and the second position information includes the spatial coordinates and orientation of the ultrasound probe 100; the three-dimensional reconstruction module 302 is used to obtain a three-dimensional patient model according to the first position information and the three-dimensional visual information; the fusion module 303 is used to synthesize the second position information and the three-dimensional patient model to obtain three-dimensional ultrasound, and the contact force calculation module 304 is used to calculate the contact force information based on the deformation of the elastic sensing material 101 or the force of the force sensor module.

[0055] The 3D visual acquisition module 201 inputs the 3D visual information of the ultrasound probe 100 and the body of the scanned person and the 3D image, the first position information of the 3D visual acquisition module 201, and the 2D ultrasound image of the ultrasound probe 100 and the body of the scanned person to the position measurement module 301, the 3D reconstruction module 302, and the fusion module 303. The contact force information obtained by the contact force calculation module 304 is input to the fusion module 303.

[0056] At the beginning of the ultrasonic scanning, the multi-axis robot 202 moves the 3D vision acquisition module 201 to a suitable position to acquire the body of the scanned person. The first position information and the 3D visual information of the 3D vision acquisition module 201 are sent to the 3D reconstruction module 302, and the 3D reconstruction module 302 can reconstruct the human body surface using the first position information and the 3D visual information. Therefore, the 3D reconstruction module 302 can output a 3D model of the body of the scanned person. The ultrasound probe 100 part in the 3D visual information and the first position information of the 3D vision acquisition module 201 will be sent to the position measurement module 301, and the position measurement module 301 obtains the second position information accordingly.

[0057] The position measurement module 301 performs different processes according to different settings of the ultrasound probe 100, such as Figure 8 As shown in the middle area AD, the area AD includes four forms of the ultrasound probe 100. According to the different forms of the ultrasound probe 100, the position measurement module 301 specifically performs one of the following steps S701-S704.

[0058] In step S701 , a three-dimensional point cloud model of the ultrasound probe 100 is acquired in the three-dimensional visual information, and second position information is obtained according to a spatial position transfer matrix of the three-dimensional point cloud model of the ultrasound probe 100 .

[0059] Specific as Figure 8 As shown in area A, the surface of the ultrasonic probe 100 may not be marked with any mark, and the ultrasonic probe 100 is identified by the position measurement module 301 only by its own shape. The position measurement module 301 obtains the three-dimensional data of the ultrasonic probe 100 in the three-dimensional visual information, analyzes each point in the three-dimensional data, and determines the relative distance between the ultrasonic probe 100 and the three-dimensional visual acquisition module 201 through spatial position transfer matrix calculation. Since the position of the three-dimensional visual acquisition module 201 is known, after determining the relative position of the ultrasonic probe 100 and the three-dimensional visual acquisition module 201, the relative position of the ultrasonic probe 100 and the scanned person can also be determined. As for the relative position of the scanned person and the three-dimensional visual acquisition module 201, it can be pre-set or obtained using the same three-dimensional point cloud method.

[0060] In step S702, if Figure 8 As shown in area B, the two-dimensional image contained in the three-dimensional visual information is identified, the two-dimensional image has a first marked image 105 set on the shell of the ultrasound probe 100, the relative spatial position relationship between the first marked image 105 and the three-dimensional visual acquisition module 201 is calculated, and the second position information of the ultrasound probe 100 is obtained;

[0061] Among them, the first marked image 105 is set on the ultrasound probe 100, and the recognition method and recognition efficiency are relatively efficient and simple. Multiple two-dimensional images can be intercepted from the three-dimensional visual information, or they can be obtained from an additional camera. After obtaining the two-dimensional image, it is only necessary to analyze the first marked image 105. The principle of the analysis is similar to step S701. When the position of the three-dimensional visual acquisition module 201 is known, the position of the ultrasound probe 100 can be obtained through the conventional calibration algorithm, that is, the second position information is obtained.

[0062] In step S703 , multiple second marked images 106 of the data cable set on the ultrasound probe 100 in the three-dimensional visual information are identified, the relative spatial position relationship between the second marked image 106 and the three-dimensional visual acquisition module 201 is calculated, and the second position information of the ultrasound probe 100 is obtained.

[0063] Among them, Figure 8 As shown in area C of FIG. 702 , a method for obtaining the ultrasound probe 100 by using the first marked image 105 is provided in step S702. However, in actual applications, the ultrasound probe 100 is often held by an ultrasound physician, and the first marked image 105 may be blocked. Therefore, a second marked image 106 is provided on the connection cable of the ultrasound probe 100 to supplement step S702.

[0064] In step S704 , a three-dimensional point cloud model of the physical marker structure 107 disposed on the ultrasound probe 100 is acquired in the three-dimensional visual information, and second position information is obtained according to a spatial position transfer matrix of the three-dimensional point cloud model of the physical marker structure 107 .

[0065] Among them, Figure 8As shown in area D, different from steps S701-S703, some identifiable physical marking structures 107 can also be set on the ultrasound probe 100. The physical marking structure 107 can be a spherical structure, a square structure or other structures. The physical marking structure 107 must exist on the upper surface of the ultrasound probe 100 to avoid affecting the holding and operation of the doctor scanning the image. By setting the physical marking structure 107, since it has a different physical touch from the first marking image 105 or the second marking image 106, the ultrasound physician can avoid blocking the physical marking structure 107 when holding the ultrasound probe 100. Therefore, the acquisition of the second position information can be carried out in the same method as step S701, but the object of detection becomes the physical marking structure 107.

[0066] As a supplement to step S704, there are multiple physical marker structures 107, which are regular or irregular in shape and present different colors. Furthermore, the position measurement module 301 can obtain a two-dimensional image containing the physical marker structure 107, and obtain the second position information of the ultrasound probe 100 by calculating the relative spatial position relationship between the physical marker structure 107 and the three-dimensional visual acquisition module 201 in the two-dimensional image. That is, a two-dimensional calibration method similar to steps S702-S703 is adopted.

[0067] In one embodiment, the processing device 300 should have a storage unit so that the processing device 300 can display the three-dimensional ultrasound model including the ultrasound image and three-dimensional visual information in real time, and can also store the above data in the form of streaming media or pictures through the storage unit.

[0068] Through the above-mentioned implementation, it is known that the present invention provides an ultrasound scanning system with 3D vision and contact force fusion sensing optimization, which can perform standardized processing on the ultrasound scanning process after optimization. The experience and level of the ultrasound doctor directly determine the quality of the ultrasound scan. The quality assessment of the ultrasound scan needs to be judged with the help of additional information in the ultrasound scan, such as the relative position of the ultrasound probe 100 and the patient's body. The system provides a three-dimensional ultrasound model to enhance the perception of the ultrasound doctor, uses the relative position of the ultrasound probe 100 to confirm the acquisition point of the three-dimensional ultrasound model and the two-dimensional ultrasound image, and uses force sensing technology to sense the contact force between the ultrasound probe 100 and the patient to judge the doctor's technique during the scanning process. Therefore, based on the present disclosure, the behavior of the ultrasound doctor during the scanning process can be recorded, supplemented by the corresponding three-dimensional ultrasound model, two-dimensional ultrasound image and contact force information. With the help of the present disclosure, standardized information can be collected for any ultrasound scanning process, and a standardized model can be established at one time.

[0069] The required information will not only be displayed in real time during the ultrasound scan, but will also be recorded and used as a cross-verification after the scan is completed. During the film reading and diagnosis process, the system can provide the doctor with all the information during the ultrasound scan, thereby assisting the doctor to better judge the ultrasound results.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention. In addition, those skilled in the art can understand that although some embodiments described herein include certain features included in other embodiments instead of other features, the combination of features of different embodiments means that they are within the scope of the present invention and form different embodiments. For example, in the above claims, any one of the embodiments claimed for protection can be used in any combination. The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as admitting or suggesting in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. An ultrasonic scanning system with 3D vision and contact force fusion sensing optimization, characterized in that: The system comprises: An ultrasound probe, wherein the ultrasound probe is configured to perform continuous ultrasound scanning on a patient to form a plurality of ultrasound images; One or more three-dimensional visual acquisition modules, wherein the one or more three-dimensional visual acquisition modules are used to continuously acquire the patient and the ultrasound probe to form three-dimensional visual information; A processing device, the processing device is used to generate a three-dimensional ultrasound model matching the multiple ultrasound images in real time according to the first position information of the three-dimensional vision acquisition module, the multiple ultrasound images and the three-dimensional vision information, wherein the three-dimensional ultrasound model includes a three-dimensional patient model and the position and orientation of the ultrasound probe; the processing device includes a position measurement module, a three-dimensional reconstruction module, and a fusion module, wherein: the position measurement module is used to calculate the first position information and the three-dimensional vision information to obtain the second position information of the ultrasound probe relative to the patient, and the second position information includes the spatial coordinates and orientation of the ultrasound probe; the three-dimensional reconstruction module is used to obtain the three-dimensional patient model according to the first position information and the three-dimensional vision information; the fusion module is used to synthesize the second position information and the three-dimensional patient model to obtain the three-dimensional ultrasound; when obtaining the ultrasound image, the processing device also obtains the contact force information generated by the ultrasound probe contacting the patient, and the processing device matches the contact force information into the three-dimensional ultrasound model; In the process of obtaining the second position information, the position measurement module specifically performs one or more of the following situations: Acquire a three-dimensional point cloud model of the ultrasound probe in the three-dimensional visual information, and obtain the second position information according to a spatial position transfer matrix of the three-dimensional point cloud model of the ultrasound probe; Identify a two-dimensional image contained in the three-dimensional visual information, wherein the two-dimensional image has a first marking image set on the housing of the ultrasound probe, calculate a relative spatial position relationship between the first marking image and the three-dimensional visual acquisition module, and obtain second position information of the ultrasound probe; Identify a plurality of second marker images of the data cable disposed on the ultrasound probe in the three-dimensional visual information, calculate a relative spatial position relationship between the second marker image and the three-dimensional visual acquisition module, and obtain second position information of the ultrasound probe; Acquire a three-dimensional point cloud model of a physical marker structure disposed on the ultrasound probe in the three-dimensional visual information, and obtain the second position information according to a spatial position transfer matrix of the three-dimensional point cloud model of the physical marker structure, wherein the physical marker structure has a plurality of regular or irregular shapes and presents different colors; The position measurement module is also used to obtain a two-dimensional image containing the physical marking structure, and obtain the second position information of the ultrasound probe by calculating the relative spatial position relationship between the physical marking structure and the three-dimensional vision acquisition module in the two-dimensional image.

2. The ultrasonic scanning system with 3D vision and contact force fusion sensing optimization according to claim 1, characterized in that: One or more of the three-dimensional vision acquisition modules and the ultrasound probe are connected to the same mechanical arm; Or one or more of the three-dimensional vision acquisition modules are arranged on one or more multi-axis robots that can move freely and independently, and the multi-axis robot is arranged at any position that can acquire one or more of the ultrasound probes and the patient's posture, and the multi-axis robot has multiple degrees of freedom of movement.

3. The ultrasonic scanning system with 3D vision and contact force fusion sensing optimization according to claim 1, characterized in that: The ultrasonic wave generating area of ​​the ultrasonic probe is covered with an elastic sensing material, and the ultrasonic image includes the shape of the elastic sensing material; the processing device calculates the deformation amount of the elastic sensing material based on the ultrasonic image, and obtains the contact force information exerted on the elastic sensing material based on the deformation amount.

4. The ultrasonic scanning system with 3D vision and contact force fusion sensing optimization according to claim 1, characterized in that: The ultrasonic probe is connected to a force sensing module, and the force sensing module is used to sense the contact force information when an external force is applied to the ultrasonic probe.

5. The ultrasonic scanning system with 3D vision and contact force fusion sensing optimization according to claim 1, characterized in that: The three-dimensional vision acquisition module includes one or both of a three-dimensional camera and a three-dimensional line scan laser scanner.

6. The ultrasonic scanning system with 3D vision and contact force fusion sensing optimization according to claim 1, characterized in that: The processing device is also used to display the three-dimensional ultrasound model including the ultrasound image and the three-dimensional visual information in real time, and store it in the form of streaming media or pictures.

Images