Ultrasonic automatic detection method and system based on human body posture
By recognizing human posture and adjusting the posture of the ultrasound probe, the problem of poor image quality caused by improper human posture in ultrasound detection was solved, and high-quality ultrasound detection images were acquired.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2023-03-23
- Publication Date
- 2026-07-03
AI Technical Summary
Current ultrasound detection technology cannot detect improper human posture in a timely manner without the supervision of a professional physician, resulting in poor ultrasound image quality or failure.
By acquiring the three-dimensional coordinate data of the joints of the human skeleton and the positioning points of its rotation direction, a three-dimensional human skeleton model is established, the posture of the human body detection area is identified, and the detection posture of the ultrasound probe is adjusted by a robotic arm. Combined with a depth camera and a region segmentation algorithm, the ultrasound probe is ensured to fit closely to the skin.
It enables accurate identification of human posture under unsupervised conditions, improves the quality of ultrasound detection images, solves the problem of image discontinuity caused by human movement during the detection process, and improves the fit between the probe and the skin.
Smart Images
Figure CN116407273B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to automatic control technology, and more particularly to an automatic ultrasonic detection method and system based on human posture. Background Technology
[0002] Ultrasound, as an important method of modern medical diagnosis, is widely used in the examination and diagnosis of internal organs such as the liver, heart, and thyroid, as well as superficial structures, due to its advantages such as being painless, non-invasive, radiation-free, and capable of real-time dynamic imaging. However, ultrasound examinations require a high level of technical skill from the examiners, and physicians often need to undergo extensive training and accumulate experience to perform the diagnostic process effectively.
[0003] To address this, existing technologies have proposed a series of solutions for autonomous ultrasound detection. For example, the current autonomous ultrasound positioning methods and ultrasound detection path planning generally use preoperative medical image data such as CT and MRI, or real-time sensing by external sensors such as RGB-D, to plan the detection path of the robotic arm driving the ultrasound probe.
[0004] However, a practical problem is that, currently, without the supervision of a professional physician (human intervention for identification), if the human body deviates or moves before or during the detection process, resulting in an unsuitable posture for detection, existing automatic ultrasound detection technology has difficulty detecting this in time. This leads to poor ultrasound image quality or even detection failure. Summary of the Invention
[0005] Therefore, the main objective of this invention is to provide an automatic ultrasound detection method and system based on human posture, so as to accurately identify human posture and adjust the detection posture of the ultrasound probe accordingly to obtain ultrasound detection images of better quality.
[0006] To achieve the above objectives, according to one aspect of the present invention, an automatic ultrasound detection method based on human posture is provided, the steps of which include:
[0007] Step S100: Obtain the three-dimensional coordinate data of the joints of the human skeleton and their rotation direction positioning points to establish a three-dimensional human skeleton model, and obtain the rotation angle of each joint through rotation registration, thereby identifying the pose of the human body detection parts.
[0008] Step S200: When the detection unit meets the detection pose, use it as coarse positioning data so that the robotic arm can move above the detection unit to complete the coarse positioning.
[0009] Step S300: Acquire an RGB image of the detection area for the physician to delineate the detection region and draw the detection path; acquire a depth map of the detection area, align it with the RGB image, generate seed points on the corresponding detection area of the depth map, segment the detection area into multiple test areas using a region segmentation algorithm and map them onto the point cloud; use PCA principal component analysis to estimate the point cloud normal vector of the test area corresponding to the detection path.
[0010] Step S400: Adjust the robotic arm to fit the ultrasound probe along the normal vector to the human skin until the first detection feedback is within the preset range.
[0011] In a possible preferred embodiment, the first detection feedback includes either force sensing feedback from the ultrasonic probe or ultrasonic detection image quality feedback.
[0012] In a possible preferred embodiment, step S100, the step of obtaining the rotation angle of each joint, includes:
[0013] Let the coordinate information of the skeletal joint point p be as follows: The length of the surrounding bones is The rotation matrix for converting the skeleton keyframe coordinate system to the world coordinate system is: If T is the matrix transpose, then the three rotational directions around it are the positioning points. , , The coordinate information is as follows:
[0014] ;
[0015] ;
[0016] ;
[0017] The initial three-dimensional spatial coordinate system { } and the three-dimensional spatial coordinate system during actual detection { The rotation registration process includes the following steps:
[0018] ;
[0019] Where the matrix T is the transformation matrix, Let R be the rotation matrix and t be the translation matrix. Then, the Euler angles are calculated using the rotation matrix R. );
[0020] The length of each bone segment is calculated based on the 3D coordinate information of the skeletal joints and rotation direction positioning points around the detection unit. and using the formula The rotation matrix is obtained by reverse calculation. .
[0021] In a possible preferred embodiment, step S200 further includes: when the detection unit does not conform to the detection posture, adjusting the human body posture until the rotation angle of each joint of the detection unit is within a preset range, and then determining that the detection posture conforms.
[0022] In a possible preferred embodiment, step S300, the step of the region segmentation algorithm to segment the detection region into multiple test regions includes: calculating the difference between the depth value of each pixel and the depth value of the seed point; when the difference meets a preset threshold, it is segmented into a region, and finally the detection region is segmented into multiple regions.
[0023] To achieve the above objectives, in accordance with the above method, according to another aspect of the present invention, an automatic ultrasonic detection system based on human posture is also provided, comprising: a robotic arm, an ultrasonic probe, a first depth camera, a second depth camera, a feedback unit, a control unit, a processing unit, and a storage unit, wherein the ultrasonic probe and the second depth camera are disposed at the end of the robotic arm, and the robotic arm is connected to the control unit.
[0024] The storage unit is used to store a program including the steps of the ultrasound automatic detection method based on human posture as described above, so that the control unit and the processing unit can retrieve and execute it as needed.
[0025] The control unit instructs the first depth camera, positioned above the detection platform, to acquire initial image data of the human body.
[0026] The processing unit is used to input the initial image data into the neural network to obtain the three-dimensional coordinate data of the human skeletal joints and their rotation direction positioning points, so as to establish a three-dimensional human skeleton model; and to perform rotation registration between the initial three-dimensional spatial coordinate system and the actual three-dimensional spatial coordinate system during detection, to obtain the rotation angle of each joint, and then identify the pose of the human body detection part.
[0027] The processing unit is also used to evaluate whether the current human body pose conforms to the detection pose by taking the human skeletal joints and the rotation angle of the corresponding joints of the detection unit as a reference. If it does, it is used as coarse positioning data and transformed into the end pose of the robotic arm through coordinate system transformation so that the robotic arm can move above the detection unit to complete the coarse positioning.
[0028] The control unit is also used to control the robotic arm to drive the second depth camera to acquire RGB images of the detection unit for the physician to delineate the detection area, draw the detection path, and acquire the depth map of the detection unit.
[0029] The processing unit aligns the depth map with the RGB image, generates seed points on the corresponding detection area of the depth map, divides the detection area into multiple regions using a region segmentation algorithm and maps them onto the point cloud; and uses PCA principal component analysis to estimate the point cloud normal vectors of each region involved in the detection path.
[0030] The control unit is also used to adjust the robotic arm to fit the ultrasound probe along the normal vector to the human skin. At the same time, the processing unit obtains the first detection feedback from the feedback unit so as to instruct the control unit to adjust the robotic arm in a timely manner until the first detection feedback is within a preset range.
[0031] In a possible preferred embodiment, the feedback unit is an image feedback unit used to evaluate the quality of ultrasound images, and the first detection feedback is ultrasound image quality feedback.
[0032] In a possible preferred embodiment, the feedback unit is a force sensing feedback unit, which is used to acquire the pressure applied by the ultrasound probe to the human body detection part, and the first detection feedback is pressure sensing feedback.
[0033] To achieve the above objectives, in accordance with another aspect of the present invention, a computer device is also provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.
[0034] To achieve the above objectives, in accordance with another aspect of the present invention, a computer-readable storage medium is also provided, having stored thereon a computer program that, when executed by a processor, implements the steps of the method described in any one of the above embodiments.
[0035] The automatic ultrasound detection method and system based on human posture provided by this invention can utilize a three-dimensional spatial coordinate system composed of skeletal joints and rotation direction positioning points to perform dynamic tracking through registration, thereby solving the problem that when the patient moves during the detection process, the obtained 2D ultrasound images are discontinuous, resulting in poor quality of the final synthesized 3D ultrasound image.
[0036] Furthermore, this invention divides the area to be detected into regions and determines the axial direction of the ultrasound probe by solving the point cloud normal vector of the local region. Combined with medical ultrasound scanning techniques, this improves the phenomenon of poor adhesion between the ultrasound probe and the skin of the area to be detected. Attached Figure Description
[0037] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:
[0038] Figure 1 This is a schematic diagram of the steps of the automatic ultrasound detection method based on human posture according to the present invention;
[0039] Figure 2 This is a schematic diagram of the neural network structure of the automatic ultrasound detection method based on human posture according to the present invention;
[0040] Figure 3 This is a schematic diagram of the three-dimensional human skeleton model structure of the ultrasound automatic detection method based on human pose according to the present invention.
[0041] Figure 4 This is a schematic diagram of the rotation direction positioning point structure of the skeletal joints in the ultrasonic automatic detection method based on human posture according to the present invention.
[0042] Figure 5 This is a schematic diagram of a physician drawing a detection path within the detection area in the ultrasound automatic detection method based on human posture according to the present invention;
[0043] Figure 6 This is a schematic diagram of the area to be tested on the detection path after the depth map has been segmented in the ultrasonic automatic detection method based on human pose of the present invention.
[0044] Figure 7 This is a schematic diagram of the ultrasonic automatic detection system based on human posture according to the present invention. Detailed Implementation
[0045] To enable those skilled in the art to better understand the technical solutions of the present invention, the specific technical solutions of the present invention will be clearly and completely described below in conjunction with embodiments, so as to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this application are merely some embodiments of the present invention, and not all embodiments. It should be noted that, for those skilled in the art, the embodiments and features in the embodiments of this application can be combined with each other without departing from the concept of the present invention and without conflict. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the disclosure and protection scope of the present invention.
[0046] Furthermore, the terms "first," "second," "S1," "S2," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those described herein. At the same time, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. Unless otherwise expressly specified and limited, the terms "set," "arranged," "installed," "connected," and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this case based on the specific circumstances and in conjunction with existing technology.
[0047] The present invention aims to replace the doctor's handheld ultrasound probe in the detection of the human body by manipulating a robotic arm with existing technology. At the same time, in order to accurately obtain the human body's pose in space so that the robotic arm can locate the detection area, the present invention uses a depth camera to obtain an image acquisition tool for calculating the human body's pose.
[0048] Please see Figures 1 to 6 As shown, in order to accurately identify human body pose and adjust the detection posture of the ultrasound probe accordingly to obtain high-quality ultrasound detection images, Embodiment 1 of the present invention provides an automatic ultrasound detection method based on human body pose, the steps of which include:
[0049] Step S100: Obtain the three-dimensional coordinate data of the joints of the human skeleton and their rotation direction positioning points to establish a three-dimensional human skeleton model, and obtain the rotation angle of each joint through rotation registration, thereby identifying the pose of the human body detection parts.
[0050] Specifically, initially, the patient lies flat on the testing platform as required, and the robotic arm is in its initial set position, such as above the testing platform. At this time, the first depth camera, positioned above the testing platform, can capture complete RGB and depth images of both the testing platform and the patient. Thus, initial image data is acquired through the first depth camera after the testing begins.
[0051] Next, the initial RGB image data is cropped to 256*256 pixel size RGB image data and input into a pre-set neural network to obtain the three-dimensional coordinate data of the patient's bone joints and rotation direction positioning points.
[0052] For example: if the coordinate information of the skeletal joint point p is The length of the surrounding bones is The rotation matrix for converting the skeleton keyframe coordinate system to the world coordinate system is: Then the three rotational positioning points around it , , The coordinate information is as follows:
[0053] ;
[0054] ;
[0055] ;
[0056] like Figure 2 As shown, the neural network structure includes the following parts: First, the input RGB image is scaled to 256*256 pixels to fix the feature map size. ResNet50 is used to extract its features. Then, the regression prediction head uses a 1*1 convolutional kernel to reduce the number of feature channels. The number of output channels is set according to N*8, where N is the total number of bone joints and rotation direction positioning points output by the algorithm, and 8 is the number of channels that is more suitable for each output point.
[0057] Next, a 5x5 convolution kernel with 2x2 padding is used to perform grouped convolutions on the channels, with N groups and 8 output channels per group. Then, a 1x1 convolution is performed on each group. Finally, an 8x8 convolution kernel of the same size as the feature map is used for convolution, resulting in 3 output channels. This yields N sets of 3x1x1 outputs, representing the 3D information of N keypoints.
[0058] Then, the error in predicting the positions of the N key points is used as the loss function L, where For the predicted coordinate information of N key points, We have the true coordinates of N key points. The model is trained by minimizing the loss function to improve its robustness. The loss function is as follows:
[0059] ,
[0060] During training, a ResNet50 network was used as the backbone, with its classification prediction head pruned and a regression prediction head integrated. The regression prediction head underwent warm-up training with an appropriate number of iterations. Subsequent iterative training was conducted using a dynamically adjusted learning rate, with the batch size adjustable based on the available GPU memory. The MPII Dataset was used as the training dataset.
[0061] Once the coordinate data of the patient's skeletal joints and rotation direction positioning points are successfully obtained, such as Figure 3As shown, a 3D human skeleton model is built based on the coordinate data of skeletal joints. The method is as follows: PAF (Part Affinity Fields) is used to determine whether the coordinates of adjacent parts are successfully connected.
[0062] ;
[0063] ;
[0064] Where dj1 and dj2 refer to the direction vectors corresponding to candidate skeletal joints j1 and j2; p(u) refers to the insertion position between body parts j1 and j2; Lc represents the confidence score for measuring the connection between positions along the line segment. If the direction of Lc is parallel to the direction vectors of dj1 and dj2, then j1 and j2 can be considered to be connected. If adjacent joints can be successfully connected, a three-dimensional human skeleton model is established.
[0065] Furthermore, such as Figure 4 As shown, for the skeletal joints of the neck, left (right) elbow, left (right) wrist, left (right) knee, and left (right) foot, three rotational direction positioning points are generated respectively. Therefore, the number of keypoint coordinates generated by the neural network in this example is N=45. For the nine skeletal joints mentioned above, with the skeletal joint as the origin, the lines connecting the three rotational direction positioning points around it and the skeletal joint form the x, y, and z axes respectively, forming a three-dimensional spatial coordinate system.
[0066] In addition, this example sets up an initial 3D human skeleton model (e.g., containing 45 key points), such as Figure 4 As shown, the initial three-dimensional spatial coordinate system is located at the neck, left elbow, left wrist, left knee, and left foot; for example, the three-dimensional spatial coordinate system at the left axis. These are skeletal joints. , , These are the rotation direction positioning points on the x, y, and z axes, respectively.
[0067] The initial three-dimensional spatial coordinate system { } and the three-dimensional spatial coordinate system during actual detection { By performing rotational registration, the rotation angles of each joint can be obtained. The methods for rotational registration and calculating joint rotation angles are as follows:
[0068] ;
[0069] Where matrix T is the transformation matrix. Let t be the rotation matrix and t be the translation matrix.
[0070] Rotation matrix It is expressed as follows:
[0071] ;
[0072] By rotation matrix Find Euler angles:
[0073] ;
[0074] ;
[0075] ;
[0076] The formula is as follows:
[0077] ;
[0078] When the patient's desired area for testing is input, the network outputs and calculates the 3D coordinates of the skeletal joints and rotation direction positioning points around the tested area, thus calculating the length of each bone segment. and using the formula The rotation matrix is obtained by reverse calculation. .
[0079] Step S200 When the detection unit meets the detection pose, it is used as coarse positioning data so that the robotic arm can move above the detection unit to complete the coarse positioning.
[0080] Specifically, after processing the acquired 3D coordinate information of the skeletal joints and rotation direction positioning points to obtain the rotation angle (i.e., Euler angle) of the part to be detected, For specific testing sites, there is an initial suitable rotation angle range set by the physician. When the Euler angle... All three fall within their respective ranges, meaning it's known whether the patient's position and orientation at the site to be tested meet the requirements for permissible testing. If testing is permitted, this data is sent to the computer as coarse localization data.
[0081] The coarse positioning coordinate data is converted into the end effector posture of the robotic arm by transforming the coordinate system. Then, the robotic arm moves above the detection site to complete the coarse positioning. If it is not allowed, the patient is prompted to make the corresponding posture adjustment so that the rotation angle of the detection site is within the range of the initial appropriate rotation angle set by the physician, so as to meet the basic requirements for allowing detection.
[0082] In addition, the part of the human body to be tested is a curved surface. Therefore, it is necessary to select a suitable detection axis direction for the ultrasound probe to ensure better contact between the probe and the part to be tested during the test. Good contact is a prerequisite for ensuring appropriate contact force between the ultrasound probe and the part to be tested.
[0083] In order to set the appropriate detection axis direction for the ultrasound probe, the first step is for a professional physician to pre-segment the human body according to the set standard, thereby avoiding the situation where the growth algorithm of this patent segments some detection areas into rings, that is, the segmentation effect is poor.
[0084] Step S300: Acquire an RGB image of the detection area for the physician to delineate the detection region and draw the detection path; acquire a depth map of the detection area, align it with the RGB image, generate seed points on the corresponding detection area of the depth map, segment the detection area into multiple test areas using a region segmentation algorithm and map them onto the point cloud; use PCA principal component analysis to estimate the point cloud normal vector of the test area corresponding to the detection path.
[0085] Specifically, firstly, a second depth camera located at the end of the robotic arm captures RGB and depth images of the area to be detected. Then, on the acquired RGB images, a professional physician selects and draws the detection region S and the detection path. Finally, image processing is used to obtain the two-dimensional pixel coordinate information of this region.
[0086] First, the RGB image and depth image are aligned. A series of seed points (pixels) are generated in the detection region S of the depth image. A region segmentation algorithm is then used to segment region S. The segmentation rule is as follows: calculate the difference between the depth value of each pixel and the depth value of the seed point; pixels whose depth difference meets a threshold are segmented into a single region. Finally, the detection region S is segmented into multiple regions. The segmented regions on the depth image are then mapped onto a point cloud, with different colors used to display the point clouds of different regions.
[0087] like Figure 5 The diagram shown is an RGB representation: the black box represents the detection area S drawn by a professional physician, and the curve represents the detection path.
[0088] Further as Figure 6 The diagram shows a depth map, where the three small black boxes represent three segmented regions. The numbers in the squares represent depth values. The depth difference threshold is 5 in the example. The squares with depth values of 80, 74, and 68 (i.e., the circled values) serve as seed points for the segmentation algorithm.
[0089] Subsequently, PCA principal component analysis was used to estimate the point cloud normal vectors for different regions. Determine which region a point on the detection path belongs to, and select the corresponding normal vector for that region. Therefore, adjusting the posture of the robotic arm's end effector so that the axial direction of the ultrasonic probe is... This provides a basis for properly fitting human skin.
[0090] Step S400: Adjust the robotic arm to fit the ultrasound probe along the normal vector to the human skin until the first detection feedback is within the preset range.
[0091] Specifically, after obtaining the normal vector, the ultrasound probe also needs to perform sagittal and transverse plane scans respectively:
[0092] (1) Sagittal plane scanning: along the long axis of the ultrasound probe It is aligned with the direction of the central axis that divides the human body into left and right parts, that is, it is aligned with the x-axis direction of the world coordinate system of the robotic arm base;
[0093] (2) Cross-sectional scanning: along the long axis of the ultrasound probe It is aligned with the central axis that divides the human body into upper and lower parts, which is also aligned with the y-axis of the world coordinate system of the robotic arm base.
[0094] The direction of the minor axis of the ultrasound probe is determined by the right-hand rule. The coordinates from the camera coordinate system to the detection area (target object) coordinate system are transformed as follows:
[0095] ;
[0096] Where P is the coordinate vector of a series of points on the detection path in the camera coordinate system.
[0097] .
[0098] The second depth camera is mounted at the end of the robotic arm, therefore the coordinate transformation from the camera coordinate system to the robotic arm base coordinate system is as follows:
[0099] ,
[0100] , , These are the three axial direction vectors at the end of the robotic arm. Let the rotation matrix be... Convert to quaternion , , , The quaternion is sent to the computer. After receiving the information, the computer performs a corresponding rotation operation at the end of the robotic arm, which drives the ultrasonic probe to rotate to a suitable detection posture.
[0101] , , This refers to the position of the detection point in the world coordinate system of the robotic arm base. , , The coordinate data is sent to the computer, and after receiving the information, the robotic arm performs the corresponding movement operation.
[0102] After the robotic arm completes the aforementioned movement, the ultrasonic probe can then be placed against the skin. At this point, if the first detection feedback is pressure feedback, and the force sensor does not detect any force, the ultrasonic probe continues along... The axial height is lowered so that the force sensor's detection value is within a preset stable range.
[0103] At this point, the axial height of the ultrasound probe is adjusted in real time based on the pressure values received from the mechanical sensors, ensuring that the pressure between the ultrasound probe and the skin in the detection area remains within a suitable range. The axial direction is updated in real time according to the segmentation region to which the current position P of the ultrasound probe belongs. , Update coordinate transformation matrix ; then Substitute to obtain .
[0104] Will Updated rotation matrix in the matrix Convert to quaternion , , , The quaternion is sent to the computer. After receiving the information, the computer's end effector performs a corresponding rotation operation, which drives the ultrasonic probe to rotate to a suitable detection posture. The updated... , , The information is sent to the computer, and after the computer receives the information, the robotic arm performs the corresponding movement operation, so that the ultrasonic probe can achieve the effect of real-time contact with the human body surface.
[0105] If the patient moves during the examination, the ultrasound robot will stop the examination. The coordinate information of the robotic arm end effector and the posture information of the ultrasound probe at the moment the patient begins to move are saved and sent to the computer, and marked as a breakpoint (the breakpoint serves as a marker point). The robotic arm end effector moves upward and away from the examination area, so that the second depth camera can complete the image information and point cloud information of the examination area.
[0106] Once the patient is stationary, the second depth camera takes a new picture, registering the three-dimensional coordinate system formed by the skeletal joints and rotation direction positioning points before and after the patient's movement. The breakpoint positions in the new three-dimensional coordinate system are then calculated. After coordinate transformation, the required position information is obtained and sent to the robotic arm's end effector.
[0107] The robotic arm's end effector moves above the breakpoint and re-completes the coarse positioning. Then, the attitude of the robotic arm's end effector is readjusted so that the ultrasonic probe is oriented... The ultrasound probe is then rotated until it aligns with the probe's orientation before the patient's movement. The breakpoint is used as the starting point for a new detection path, and detection continues. After the detection is complete, the obtained two-dimensional ultrasound images are filtered and synthesized to obtain a three-dimensional ultrasound image of the detected area.
[0108] The definition of whether a patient has exercised is as follows: Calculate the key skeletal points near the detection site. and rotation direction positioning point The coordinate data changes of these points are weighted and summed. If the sum exceeds a set distance threshold for determining if movement has occurred, the result is considered positive. If the result exceeds the set distance threshold for determining movement, then stop detecting; otherwise, continue detecting until the result exceeds the set distance threshold for determining movement. .
[0109] ,
[0110] in and Key points of the skeleton and rotation direction positioning point The total number; and Each skeletal key point and rotation direction positioning point Distance weight parameters; , , Key points of the skeleton The changes in distance in the x, y, and z directions; , , Positioning points for rotation direction The changes in distance in the x, y, and z directions.
[0111] In case of an emergency, the patient can press the emergency stop device, which will stop the ultrasound robot and return it to its initial position to ensure patient safety.
[0112] On the other hand, such as Figure 7 As shown, in another aspect of the present invention, corresponding to the above method, an automatic ultrasonic detection system based on human posture is also provided, which includes: a robotic arm, an ultrasonic probe, a first depth camera, a second depth camera, a feedback unit, a control unit, a processing unit, and a storage unit, wherein the ultrasonic probe and the second depth camera are disposed at the end of the robotic arm, and the robotic arm is connected to the control unit.
[0113] The storage unit is used to store a program including the steps of the ultrasound automatic detection method based on human posture as described above, so that the control unit and the processing unit can retrieve and execute it as needed.
[0114] The control unit instructs the first depth camera, positioned above the detection platform, to acquire initial image data of the human body.
[0115] The processing unit is used to input the initial image data into the neural network to obtain the three-dimensional coordinate data of the human skeletal joints and their rotation direction positioning points, so as to establish a three-dimensional human skeleton model; and to perform rotation registration between the initial three-dimensional spatial coordinate system and the actual three-dimensional spatial coordinate system during detection, to obtain the rotation angle of each joint, and then identify the pose of the human body detection part.
[0116] The processing unit is also used to evaluate whether the current human body pose conforms to the detection pose by taking the human skeletal joints and the rotation angle of the corresponding joints of the detection unit as a reference. If it does, it is used as coarse positioning data and transformed into the end pose of the robotic arm through coordinate system transformation so that the robotic arm can move above the detection unit to complete the coarse positioning.
[0117] The control unit is also used to control the robotic arm to drive the second depth camera to acquire RGB images of the detection unit for the physician to delineate the detection area, draw the detection path, and acquire the depth map of the detection unit.
[0118] The processing unit aligns the depth map with the RGB image, generates seed points on the corresponding detection area of the depth map, divides the detection area into multiple regions using a region segmentation algorithm and maps them onto the point cloud; and uses PCA principal component analysis to estimate the point cloud normal vectors of each region involved in the detection path.
[0119] The control unit is also used to adjust the robotic arm to fit the ultrasound probe along the normal vector to the human skin. At the same time, the processing unit obtains the first detection feedback from the feedback unit so as to instruct the control unit to adjust the robotic arm in a timely manner until the first detection feedback is within a preset range.
[0120] In a possible preferred embodiment, the feedback unit is an image feedback unit used to evaluate the quality of ultrasound images, and the first detection feedback is ultrasound image quality feedback.
[0121] In a possible preferred embodiment, the feedback unit is a force sensing feedback unit, which is used to obtain the pressure applied by the ultrasound probe to the human body detection part, and the first detection feedback is pressure sensing feedback.
[0122] Furthermore, in accordance with the above method, according to another aspect of the present invention, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.
[0123] Furthermore, in accordance with the above method, according to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the above embodiments.
[0124] In summary, the ultrasound automatic detection method and system based on human posture provided by this invention can utilize a three-dimensional spatial coordinate system composed of skeletal joints and rotation direction positioning points to perform dynamic tracking through registration, thereby solving the problem that when the patient moves during the detection process, the obtained 2D ultrasound images are discontinuous, resulting in poor quality of the final synthesized 3D ultrasound image.
[0125] Furthermore, this invention divides the area to be detected into regions and determines the axial direction of the ultrasound probe by solving the point cloud normal vector of the local region. Combined with medical ultrasound scanning techniques, this effectively improves the phenomenon of poor adhesion between the ultrasound probe and the skin of the area to be detected.
[0126] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The present invention is limited only by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.
[0127] Those skilled in the art will understand that, besides implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.
[0128] Furthermore, all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0129] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.
Claims
1. An automatic ultrasound detection method based on human posture, characterized by the following steps: include: Step S100: Obtain the three-dimensional coordinate data of the joints of the human skeleton and their rotation direction positioning points to establish a three-dimensional human skeleton model, and obtain the rotation angle of each joint through rotation registration, thereby identifying the pose of the human body detection parts. The steps include: Step S101: Acquire initial image data of the patient on the detection platform, process it through a pre-set neural network, and obtain the coordinate information of the skeletal joint point p. Let the length of the surrounding bones be... The rotation matrix for converting the skeleton keyframe coordinate system to the world coordinate system is: If T is the matrix transpose, then the three rotational directions around it are the positioning points. , , The coordinate information is as follows: ； ； ； Step S102 determines whether the coordinates of adjacent parts are successfully connected: ； ； Where dj1 and dj2 refer to the direction vectors corresponding to candidate skeletal joints j1 and j2; p(u) refers to the position inserted between body parts j1 and j2; Lc represents the confidence of the connection between positions along the line segment. If the direction of Lc is parallel to the direction vectors of dj1 and dj2, then j1 and j2 are considered to be connected. If adjacent joints can be connected, a three-dimensional human skeleton model is established. Step S103 establishes the initial three-dimensional spatial coordinate system { } and the three-dimensional spatial coordinate system during actual detection { Perform rotational registration to obtain the rotation angles of each joint. The steps include calculation: ； Where the matrix T is the transformation matrix, Let t be the rotation matrix and t be the translation matrix. Rotation matrix It is expressed as follows: ； By rotation matrix Find Euler angles: ； ； ； ； Step S104 calculates the length of each bone segment based on the 3D coordinate information of the skeletal joints and rotation direction positioning points around the detection unit. and using the formula ； The rotation matrix is obtained by reverse calculation. To identify the posture of human body detection parts; Step S200: When the detection unit conforms to the detection pose, it is used as coarse positioning data to move the robotic arm above the detection unit to complete coarse positioning; when the detection unit does not conform to the detection pose, the human body posture is adjusted until the rotation angle of each joint of the detection unit is within a preset range, at which point the detection pose is considered to be conforming. The steps include: Step S201: Calculate key points of the skeleton near the detection unit. and rotation direction positioning point The coordinate data changes of these points are calculated, and the weighted sum of these coordinate data changes is performed. Step S202: If the sum of weights is greater than the set distance threshold... If the movement is detected, the detection stops, indicating that movement has occurred; otherwise, the detection continues until the result exceeds the set distance threshold. ; ； in and Key points of the skeleton and rotation direction positioning point The total number; and Each skeletal key point and rotation direction positioning point Distance weight parameters; , , Key points of the skeleton The changes in distance in the x, y, and z directions; , , Positioning points for rotation direction The changes in distance in the x, y, and z directions; Step S300: Acquire an RGB image of the detection area for the physician to delineate the detection region and draw the detection path; acquire a depth map of the detection area, align it with the RGB image, generate seed points on the corresponding detection area of the depth map, segment the detection area into multiple test areas using a region segmentation algorithm and map them onto the point cloud; use PCA principal component analysis to estimate the point cloud normal vector of the test area corresponding to the detection path. Step S400: Adjust the robotic arm to fit the ultrasound probe along the normal vector to the human skin until the first detection feedback is within the preset range.
2. The automatic ultrasound detection method based on human posture according to claim 1, characterized in that, The first detection feedback includes either force sensing feedback from the ultrasonic probe or ultrasonic detection image quality feedback.
3. The automatic ultrasound detection method based on human posture according to claim 1, characterized in that, In step S300, the step of the region segmentation algorithm to segment the detection region into multiple test regions includes: calculating the difference between the depth value of each pixel and the depth value of the seed point; when the difference meets a preset threshold, it is segmented into a region, and finally the detection region is segmented into multiple regions.
4. An automatic ultrasonic detection system based on human posture, characterized in that... include: The system includes a robotic arm, an ultrasonic probe, a first depth camera, a second depth camera, a feedback unit, a control unit, a processing unit, and a storage unit, wherein the ultrasonic probe and the second depth camera are located at the end of the robotic arm, and the robotic arm is connected to the control unit. The storage unit is used to store a program including the steps of the ultrasound automatic detection method based on human posture as described in any one of claims 1 to 3, so that the control unit and the processing unit can retrieve and execute it as needed. The control unit instructs the first depth camera, positioned above the detection platform, to acquire initial image data of the human body. The processing unit is used to input the initial image data into the neural network to obtain the three-dimensional coordinate data of the human skeletal joints and their rotation direction positioning points, so as to establish a three-dimensional human skeleton model; and to perform rotation registration between the initial three-dimensional spatial coordinate system and the actual three-dimensional spatial coordinate system during detection, to obtain the rotation angle of each joint, and then identify the pose of the human body detection part. The processing unit is also used to evaluate whether the current human body pose conforms to the detection pose by taking the human skeletal joints and the rotation angle of the corresponding joints of the detection unit as a reference. If it does, it is used as coarse positioning data and transformed into the end pose of the robotic arm through coordinate system transformation so that the robotic arm can move above the detection unit to complete the coarse positioning. The control unit is also used to control the robotic arm to drive the second depth camera to acquire RGB images of the detection unit for the physician to delineate the detection area, draw the detection path, and acquire the depth map of the detection unit. The processing unit aligns the depth map with the RGB image, generates seed points on the corresponding detection area of the depth map, divides the detection area into multiple regions using a region segmentation algorithm and maps them onto the point cloud; and uses PCA principal component analysis to estimate the point cloud normal vectors of each region involved in the detection path. The control unit is also used to adjust the robotic arm to fit the ultrasound probe along the normal vector to the human skin. At the same time, the processing unit obtains the first detection feedback from the feedback unit so as to instruct the control unit to adjust the robotic arm in a timely manner until the first detection feedback is within a preset range.
5. The automatic ultrasound detection system based on human posture according to claim 4, characterized in that, The feedback unit is an image feedback unit, which is used to evaluate the quality of ultrasound images, and the first detection feedback is ultrasound image quality feedback.
6. The automatic ultrasound detection system based on human posture according to claim 4, characterized in that, The feedback unit is a force sensing feedback unit, which is used to obtain the pressure applied by the ultrasound probe to the human body detection part, and the first detection feedback is pressure sensing feedback.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.