Method for intelligently identifying chest organs and autonomously positioning and scanning chest organs

Abstract

The invention discloses a method for intelligently identifying chest organs and autonomously positioning and scanning the chest organs. An artificial intelligence robot technology is utilized, and a method for autonomously positioning and identifying human organ feature positions, autonomously positioning the chest organs and collecting ultrasonic images and a method for remotely and adaptively collecting images and videos of the chest, nipples, breasts and heart organs are provided. Autonomous and far-end control acquisition and sharing realize medical image and video sharing, and alleviate the problems of high medical acquisition work pressure, many night shift and the like. Remote and autonomous positioning scanning of chest organs by experts and doctors, acquisition and sharing of medical images, remote inquiry and ward round are improved, common opinions of the experts are provided, clinical cases are efficiently and professionally solved, and the system is widely applied to outpatient service, wards and overseas medical institutions.

Description

Method for intelligently identifying chest organs and autonomously positioning and scanning chest organs Technical Field

[0001] The present invention belongs to the field of medical artificial intelligence technology, involving data analysis technology, robot motion planning technology, image intelligent recognition method, artificial intelligence and medical data analysis and recognition technology and method. Background Art

[0002] Currently used in the medical field, various human factors contribute to poor quality of chest, breast, and heart medical images and videos during examinations. The complex structures of the heart and breast contribute to low standardization of ultrasound medical images and videos, resulting in poor accuracy in identifying medical conditions. Doctors often have widely varying expertise, leading to incorrect diagnoses based on ultrasound medical images. This technology utilizes robotic arms, vision devices, deep vision devices, and various neural network methods and their improved approaches to intelligently recognize faces, organs, and bones, assisting in the acquisition of ultrasound medical images and videos.

[0003] Ultrasound medical image and video acquisition is inefficient, manual acquisition is inaccurate, and manual acquisition can lead to problems such as the spread of plague. By using robotic arms to remotely and autonomously acquire medical images and videos, we can achieve remote and autonomous acquisition, intelligently identify and analyze ultrasound data, images, and videos, autonomously locate, and scan the heart, effectively preventing the spread of infectious diseases, plague, and other major diseases.

[0004] Intelligent recognition of the fusion of visual images and medical images, autonomous positioning scanning methods, adaptive adjustment of robotic arms, intelligent recognition of the human body, organs, bones, identification, remote and autonomous mobile robotic arms, scanning probes, scanning devices, autonomous positioning, scanning parts, organs, according to intelligent scanning methods, autonomous scanning, collection of medical images, videos, greatly improving the efficiency of intelligent recognition, collection of medical data, and medical images and videos. Technical issues

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the above-mentioned prior art and provide a method for utilizing a remote, autonomous robot to identify the chest, heart, and breasts, autonomously locate, plan motions, and move to collect medical images of the chest, heart, and breasts. The intelligent recognition method of the present invention that integrates visual images with medical images and the autonomous positioning and scanning method of the robotic arm achieves high-precision positioning and solves the problem of errors in manual scanning, inspection, collection, diagnosis, and treatment.

[0006] A method for intelligently identifying the facial features, bones, shoulder joints, joints, ribs, breasts, nipples, and other human features and their locations by fusing multiple medical images; a method for autonomously locating organ positions by robots, robotic arms, and medical devices; and a method for scanning and collecting medical images of the heart and breasts have become key technical issues for autonomously identifying, locating, and scanning the heart, breast, and chest organs.

[0007] Effective and complete scanning and acquisition of the heart, breast, and chest organs, based on intelligent recognition of these organs, has become another important technical challenge in ultrasound scanning. This invention provides a method for remote and adaptive scanning of the heart, breast, and chest to capture images and videos.

[0008] The technical solution adopted by the present invention

[0009] A method for intelligently identifying chest organs and autonomously positioning and scanning chest organs is characterized by integrating the human body's five senses, organs, bones, joints, blood vessels, human features and their locations. The method is a method for intelligently identifying human organs, bones, joints, blood vessels, human feature points and their locations by integrating planar visual images, depth images, ultrasound images, videos, and endoscopic images, including the following steps:

[0010] S1. Establish a general visual image model, a human organ model, a blood vessel model, and a feature model. The general image model includes: face, nipple, breast, navel, and characteristic parts and their locations as feature items. Use a neural network algorithm and its improved method to intelligently identify human features and their locations.

[0011] S2. Establish a depth vision device and a depth vision image model, and a chest bone model, and combine the depth information and the position information of the sternum, shoulder joint, second rib, third rib, fourth rib, fifth rib, xiphoid process, and spine of the human body where the bones are located, and the neighboring feature organ information identified by the general image model described in S1 as feature items of the skeleton intelligent recognition model as input items;

[0012] S3. Apply neural network algorithms and their improved methods to intelligently identify each bone and its location, including the second rib, third rib, fourth rib, fifth rib, rib base, xiphoid process, vertebrae, vertebral position, lower limit of bones, shoulder joint and its location, and the location of each joint.

[0013] S4, using the information of the nipple and the heart, a characteristic organ adjacent to the breast, identified by the general image model described in S1, as external scan feature information, and using the external scan bone information identified by S2, as well as the ultrasound image, video, blood vessel color information under the video, blood vessel position information, and the contour shape features, structural features, color features, and position features of the organs in the ultrasound image and video as input items;

[0014] S5. The feature items are input into the neural network and its improved method and weight optimizer as input items, and the output value is obtained through image training;

[0015] S6. Identify organs, blood vessels, bones, and their location information on the human body, and scan and autonomously locate organs, blood vessels, bones, and their location on the human body;

[0016] A method for intelligently identifying chest organs and autonomously positioning and scanning chest organs is characterized by a robotic arm, an ultrasound device, and a method for the ultrasound scanning device to autonomously identify the heart, breast, and chest organs and position and scan the organs, and the steps are as follows:

[0017] S1. Obtain the coordinates of the organ and its external location area of ​​the human body image corresponding to the acquisition task, set it as the target, set the target name, target parameters, location information, and set the communication target;

[0018] S2, robotic arm, ultrasound device, ultrasound scanning device, the robot subscribes to the corresponding organ external scanning area position information,

[0019] Subscribe to target, parameters, target pose, pose tag, set target head id, target pose, direction value, set timestamp;

[0020] S3. The remote main control system and the ultrasound scanning device mounted on the autonomous robotic arm move and scan the human body collection area according to the subscribed collection area location and the image collection action plan of the robotic arm. The ultrasound scanning device and the ultrasound device release the collected image information, blood vessel color information, blood vessel location information and ultrasound image, and the contour shape characteristics, structural characteristics, and color characteristics of the video organ;

[0021] S4. The robot main system determines all target organ harvesting tasks according to the returned target organ harvesting completion information;

[0022] S5. Based on S2, according to the robot's autonomous positioning, the coordinates of the external position area of ​​the human body, the method of scanning and collecting medical images, the robot arm, the ultrasound device, the ultrasound scanning device, the robot main system subscribes to the breast, the position information of the external scanning area of ​​the heart, the scanning method, the probe angle, the subscription target, the parameters, the target pose, the pose tag, the target for the head ID, the target pose, the direction value, and the timestamp;

[0023] S6. Based on S1, using a general visual device and a multi-medical image fusion intelligent recognition method, intelligently identify the human nipple, breast, heart and their locations; using a deep vision acquisition device and a multi-medical image fusion intelligent recognition method, intelligently identify the bones, sternum, second rib, third rib, fourth rib, fifth rib, and the scanning locations of the corresponding structural parts of the heart organ;

[0024] S7. The robot main system and the robotic arm remotely and adaptively adjust the image parameters of the medical device, gain parameters, color gain parameters, sensitivity time control adjustment parameters, time gain control parameters, focus parameters, depth parameters, frame size, blood flow velocity scale parameters, video parameters, image acquisition method, robotic arm, ultrasound scanning device, remote control and adaptive adjustment of probe rotation angle, tilt angle, scanning mode of the ultrasound scanning device, probe angle, parameters to the following organs, tissues for effective and complete acquisition;

[0025] S8, remote control and adaptive movement of the robotic arm. Based on S2, the deep vision device uses images collected to intelligently identify the chest ribs. Based on S1, the planar vision device intelligently identifies the breast and its position. The robotic arm scans the heart along the second rib, third rib, fourth rib, and intercostal space along the left edge of the sternum.

[0026] A method for intelligently identifying chest organs and autonomously positioning and scanning chest organs is characterized by remotely controlling and adaptively moving a robotic arm to autonomously scan the heart between the ribs, and the steps are as follows:

[0027] S1. The ultrasound probe scans the image, intelligently identifying the dynamic pulsating organs. The robotic arm moves along the right shoulder to scan the long-axis image of the left ventricle. The probe scans inward and tilts outward to display the anterior leaflet of the mitral valve and the left ventricle. The probe is rotated to scan along the ascending aorta.

[0028] S2. The image collected by the visual device is used to intelligently identify the papilla and its location. The probe rotates counterclockwise to scan, and the robotic arm scans the anterior papillary muscle, chordae tendineae, and anterior leaflet of the mitral valve to examine the long axis and image of the anterior leaflet.

[0029] S3: The image collected by the visual device is used to intelligently identify the nipple and its position. The probe rotates counterclockwise to scan, and the robotic arm scans the long axis and image of the posterior mitral leaflet along the posterior papillary muscle, chordae tendineae, and posterior leaflet of the mitral valve.

[0030] S4: The ultrasound probe is lowered one intercostal space and scanned. The probe is tilted medially or laterally to scan the papillary muscles, chordae tendineae, posterior leaflet of the mitral valve, midline long axis and image, anterior leaflet long axis and image, and posterior leaflet long axis and image of the next rib.

[0031] S5. Intelligently identify the chest ribs and breasts and their positions using images captured by the deep vision device. The robotic arm scans the high intercostal long axis and images along the second and third ribs and between the ribs along the left edge of the sternum.

[0032] S6. Using the positional and structural features of each part of the ultrasound image, the probe scans the image to intelligently identify its circular, inverted triangle, and Y-shaped features during diastole and systole. The probe is moved slightly outward and downward, and the robotic arm moves along the left edge of the sternum, rotating 90 degrees clockwise to scan the four heart chambers and the mitral valve, tricuspid valve, four-chamber heart on the left chest wall, pulmonary artery, aortic valve, pulmonary valve, and images.

[0033] S7. Using ultrasound images and the location and structural features of various parts of the body as shown in the video, the robotic arm moves along the left sternal border to scan. Starting from the left ventricular long-axis image, the probe rotates counterclockwise to obtain the right ventricular inflow tract and right ventricular outflow tract long-axis images. The probe scans the right ventricular infundibulum, pulmonary valve, and truncus arteriosus in a counterclockwise direction.

[0034] S8. Through the images collected by the probe scanning, the dynamic pulsating organs are intelligently identified. Based on the positional characteristics and structural characteristics of each part in the ultrasound image and video, the probe is placed at the apex of the heart at the pulsating position, tilted from the chest wall side to the back side, and rotated to scan the four heart chambers and the mitral valve, tricuspid valve, and the apex of the four-chamber heart image;

[0035] S9. Rotate the ultrasound probe counterclockwise to scan the apical two-chamber image and apical long-axis image, left ventricle, and right ventricle images;

[0036] S10: The vision device intelligently identifies the nipple and its location. Using images captured by the depth vision device, the rib base bones and xiphoid process are intelligently identified, as well as the apex of the heart. The robotic arm, positioned at the xiphoid process, uses a pressure device to apply pressure, tilts the probe headward, rotates it, and adaptively adjusts the tilt angle to scan the four cardiac chambers and the mitral valve, tricuspid valve, and the four-chamber heart image below the xiphoid process.

[0037] S11. Rotate the probe clockwise to record the three-chamber view (left ventricle, left atrium, and aorta) and the two-chamber view (left ventricle and left atrium) from the four-chamber heart image.

[0038] S12. The image acquired by the probe is intelligently identified as a circular shape and marked as the left ventricle. The scan is adaptively adjusted to the center of the image, the short-axis image of the left ventricle under the xiphoid process, the long-axis image of the inferior vena cava, the short-axis image, and the image of the abdominal aorta. The tilt angle is adaptively adjusted in the right direction to scan the long-axis image of the inferior vena cava, the short-axis image, and the image of the abdominal aorta.

[0039] S13. Using images captured by the depth vision device, intelligently identify the suprasternal fossa, the first intercostal space on the left side of the sternum, tilt the probe toward the head, rotate the probe, adaptively adjust the probe rotation angle and tilt angle, and scan the ascending aortic arch, distal end, aortic arch, descending aorta, and suprasternal fossa aortic arch images;

[0040] S14. Using the images captured by the deep vision device, intelligently identify the third and fourth ribs at the right sternal border. Move the probe to the third and fourth intercostal spaces, adaptively adjust the probe rotation and tilt angles, and scan the left atrium, right atrium, atrial septum, and right sternal border at the cardiac level. Adjust the probe to rotate 90 degrees counterclockwise and scan the cardiac sagittal image at the right sternal border.

[0041] S15. Using the images captured by the depth vision device, intelligently identify the second and third ribs on the right side of the sternum, move the probe to the second or third intercostal space, adaptively adjust the probe rotation angle and tilt angle, and scan the long-axis image of the ascending artery on the right side of the sternum;

[0042] A method for intelligently identifying chest organs and autonomously positioning and scanning chest organs is characterized by remotely controlling and adaptively moving a robotic arm to autonomously scan the breast between ribs, and the steps are as follows:

[0043] S1. Intelligently identify nipples, breasts, and their positions, ranges, and contours through images collected by a visual device;

[0044] S2. Establish a depth vision device and a depth vision image model, a chest bone model, and a shoulder joint axillary model. Utilize the depth information and the position of the bones in the human body as input into a neural network and its improved method to intelligently identify the sternum, shoulder joints, shoulder joint axillary, and their position information.

[0045] S3. Using the images collected by the depth vision device to intelligently identify the sternum, both shoulder joints, and the axillary sides of both shoulder joints, as well as their positional information, the probe is moved longitudinally from the mid-axillary line, moving from the outside to the inside of the breast and then from the inside to the outside. The probe is placed in the mid-axillary line and scanned through the nipple to the lower edge of the breast until the gland disappears, ending the longitudinal scan.

[0046] S4. Using the images collected by the depth vision device to intelligently identify the sternum, shoulder joints, and axillary sides of the shoulder joints, as well as their positional information, the probe is placed on the clavicle and scanned in a transverse direction from the clavicle to the mid-axillary line. The probe is moved from the cranial side of the upper breast to the crural side of the lower breast, and then from the cranial side to the cranial side of the upper breast until the gland disappears, ending the longitudinal scan.

[0047] S5. Using the image captured by the visual device, the nipple and its location are intelligently identified. The probe is placed on the nipple. With the nipple as the center, the ultrasound probe is rotated to scan from the nipple toward the outer edge of the breast until the gland disappears from the opposite ribs. This ends the rotational scanning.

[0048] In summary, the beneficial effects of the present invention are

[0049] The present invention can autonomously complete various medical care tasks in the ward through medical robotic devices, robotic arms, and remotely isolated and autonomous collection of medical devices, isolated collection, autonomous positioning and identification of characteristic positions of human organs, and collection of ultrasound images and videos, and improve problems such as multiple collection tasks, high work pressure, and frequent night shifts.

[0050] At the same time, the present invention provides a method for remote and adaptive acquisition of heart, breast images, and videos; it realizes remote and autonomous effective acquisition, complete acquisition of medical images and videos, sharing of medical pictures and videos, multi-expert remote joint consultation, and real-time acquisition of data and images and videos collected by robots, greatly improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] FIG1 is a flow chart of a method for intelligently identifying human organs, bones, joints, blood vessels, human feature points and their locations, and a method for autonomous positioning in this application specification;

[0052] FIG2 is a flowchart of a method for a robot, a medical device, a robotic arm, an ultrasound device, and an ultrasound scanning device to autonomously identify a heart and a breast and locate and scan organs in this application specification;

[0053] Figure 3 is a flow chart of the method for autonomously scanning the heart using a robot, a medical device, a robotic arm, an ultrasound device, an ultrasound scanning device, and the like in this application specification. DETAILED DESCRIPTION

[0054] The purpose of this invention is to design a remotely controlled robot that can replace human work, realize remote control of robotic arm acquisition, and effectively solve the problem of autonomous image and video acquisition. It uses artificial intelligence robotics technology, autonomous acquisition in the field of automation, robotic arm motion planning, and autonomous recognition of human faces, facial features, arms, human external features, and depth camera skeleton and joint images.

[0055] The fusion of visual images and medical images to intelligently identify the human body's facial features, organs, bones, joints, blood vessels, human features and their positions, as well as the method for autonomous positioning of robots, robotic arms, and medical devices, and the method for scanning and collecting medical images in the external position area of ​​the human body is called autonomous positioning and scanning.

[0056] The present invention provides a method for remote and adaptive intelligent recognition and scanning of the heart and breast, enabling remote robotic control and autonomous acquisition of medical images and videos, remote control of medical image and video acquisition devices, and image sharing. This eliminates the potential for non-standard acquisition and diagnostic and treatment errors caused by human intervention, and improves the accuracy of intelligent acquisition, intelligent disease recognition, and identification of medical data anomalies. To better understand the above technical solution, the present invention is further described in detail below in conjunction with examples and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0057] The technical solution implemented in this application is the following general idea for solving the above technical problems:

[0058] The present invention provides a method for intelligently identifying the human body's facial features, organs, bones, joints, blood vessels, and their locations by fusing medical and visual images. A general visual image model, a human organ model, a blood vessel model, and a feature model are established. The general image model includes the face, facial features, breasts, nipples, navels, characteristic genitalia, characteristic parts, and their locations as feature items. A neural network algorithm and its improved method are used to intelligently identify human features and their locations. A depth vision device, a depth vision image model, and a bone model are established. Depth information and the positional information of bones within the human body are used to autonomously locate and identify the characteristic locations of human organs and to capture ultrasound images. Furthermore, a method for remotely and adaptively capturing images and videos of the heart, breasts, and chest is also provided.

[0059] Example 1:

[0060] The present invention is further described in detail below in conjunction with the embodiments and Figures 1, 2, and 3, but the embodiments of the present invention are not limited thereto. General visual images, human organs, blood vessels, and features are input, including: face, facial features, ears, lips, mandible, neck, navel, nipple, characteristic genitals, characteristic parts and their positions; depth visual images, bone information, mandible, rib bottom bones, xiphoid process, spine, spine position, bone lower limit position, shoulder joint, knee joint, foot, bone, foot joint, waist joint, and various joints and their positions are input.

[0061] Ultrasound images use vascular color information, vascular location information, and the contour, shape, structural, and color features of organs. This information is combined with the organ information and features in the ultrasound image to create a comprehensive information combination item, which serves as the feature item and input item of the ultrasound image model. External scan feature information and external scan bone information are used as external scan information. An improved deep neural network method and weight optimizer are applied to image training to obtain output values ​​and scanned organ and location information of organs, blood vessels, bones, and the human body in which they are located. The result of autonomously locating the organs, blood vessels, bones, and the human body in which they are located is output.

[0062] According to the administrator, the doctor publishes the collection task and medical order message, obtains the organ and its external position area coordinates of the human body collection image corresponding to the collection task, sets it as the target, sets the target name, target parameters, location information, and sets the communication target.

[0063] The robot's visual acquisition device and visual recognition module publish the external features corresponding to each organ, the external scanning feature information of the neighboring feature organs identified by the general image model, and the coordinates of the external position area of ​​the human body; the depth information released by the depth camera, the external scanning bone information of the neighboring bone information.

[0064] The robotic arm, ultrasound device, ultrasound probe, and robot subscribe to the corresponding organ's external scanning area location information, subscribe to the target, parameters, target pose, pose tag, set the target head ID, target pose, orientation value, and set a timestamp. The remote main control system and the ultrasound probe mounted on the autonomous robotic arm move and scan the human body collection area according to the subscribed collection area location and the actions of the robotic arm's image acquisition motion planning module. The ultrasound probe and ultrasound device publish the collected image information, including vascular color information, vascular position information, and the contour, shape, structural, and color features of the organ in the ultrasound image. The robot, medical device, and visual recognition module subscribe to this image information. The vascular color information, vascular position information, and the contour, shape, structural, and color features of the organ in the ultrasound image are extracted from the ultrasound image and input into the computational model to intelligently identify whether the image is the target organ tissue. If so, the target organ is to be scanned.

[0065] Set acquisition target parameters (pose tag, timestamp, target head ID, COG target pose, and orientation), set position and pose tolerances, allow re-planning if motion planning fails, set the target position reference coordinate system, and set a time limit for each motion planning. The robot, medical device, and robotic arm remotely and adaptively adjust the image and video parameters of the medical device, image acquisition method, and whether the image and video recognition standards are met and whether the acquisition is valid. The robot, medical device, and robotic arm remotely and adaptively adjust the ultrasound probe scanning mode, probe angle, and parameters. Check the acquisition location of the target organ and verify that all images and videos of the acquired organ and tissue are complete under all scanning modes. Return target organ acquisition completion information. The robot and robotic arm subscribe to task information. The robot, medical device, and robotic arm remotely and adaptively move the ultrasound probe to the external scanning position of the next target organ. Based on the returned target organ acquisition completion information, the robot and medical device determine that all target organ acquisition tasks are complete.

[0066] The acquisition method is to scan the image through the ultrasound probe, intelligently identify the dynamic pulsating organ, and move the robotic arm along the right shoulder direction to scan the long axis image of the left ventricle. The probe scans inwards and tilts outwards to display the anterior leaflet of the mitral valve and the left ventricle, and rotates the probe to scan along the ascending aorta; the image acquired by the visual device intelligently identifies the papillary and its position, the probe rotates counterclockwise to scan, and the robotic arm scans the anterior papillary muscle, chordae tendineae, and anterior leaflet of the mitral valve to check the long axis and image of the anterior leaflet; the image acquired by the visual device intelligently identifies the papillary and its position, the probe rotates counterclockwise to scan, and the robotic arm scans the posterior papillary muscle, chordae tendineae, and posterior leaflet of the mitral valve to check the long axis and image of the posterior leaflet; the ultrasound probe descends one intercostal space to scan, and the probe tilts inwards or outwards to scan the papillary muscle, chordae tendineae, posterior leaflet of the mitral valve, the midline long axis and image, the anterior leaflet long axis and image, and the posterior leaflet long axis and image of the next rib;

[0067] The images collected by the deep vision device are used to intelligently identify the chest ribs and the breasts and their positions. The robotic arm scans the high intercostal long axis and images along the second rib, third rib, and between the ribs on the left side of the sternum.

[0068] Through the positional and structural features of each part of the ultrasound image, the probe scans the image to intelligently identify its circular, inverted triangle, and Y-shaped features during diastole and systole. The probe then moves slightly outward and downward, and the robotic arm moves along the left edge of the sternum, rotating 90 degrees clockwise to scan the four heart chambers and the mitral valve, tricuspid valve, four-chamber heart on the left chest wall, pulmonary artery, aortic valve, pulmonary valve, and images.

[0069] Through ultrasound images and the location and structural features of various parts in the video, the robotic arm moves along the left edge of the sternum to scan. Starting from the left ventricular long-axis image, the probe rotates counterclockwise to obtain the right ventricular inflow tract long-axis image and the right ventricular outflow tract long-axis image. The probe scans the right ventricular infundibulum, pulmonary valve, and truncus arteriosus in the counterclockwise direction.

[0070] Through the images collected by the probe scanning, the dynamic pulsating organs are intelligently identified. Through the positional features and structural features of each part in the ultrasound image and video, the probe is placed at the apex of the heart at the pulsating position, tilted from the chest wall side to the dorsal side, and rotated to scan the four heart chambers and the mitral valve, tricuspid valve, and the apical four-chamber heart image; the ultrasound probe rotates counterclockwise to scan the apical two-chamber heart image and the apical long-axis image, the left ventricle, and the right ventricle image; the visual device intelligently identifies the nipple and its position, and uses the images collected by the depth vision device to intelligently identify the bones at the bottom of the ribs and the position of the xiphoid process, identify the position of the apex, and the robot arm uses the pressure device to apply pressure from the xiphoid process position, tilt toward the head, rotate the probe, and adaptively adjust the tilt angle to scan the four heart chambers and the mitral valve, tricuspid valve, and the four-chamber heart image under the xiphoid process; the probe rotates clockwise to record the three-chamber section (left ventricle, left atrium, aorta) and the two-chamber section (left ventricle, left atrium) in sequence from the four-chamber heart image;

[0071] The image captured by the probe is used to intelligently identify a circular shape and mark it as the left ventricle. The scan is adaptively adjusted to the image center, the short-axis image of the left ventricle under the xiphoid process, the long-axis image of the inferior vena cava, the short-axis image, and the abdominal aorta image. The tilt angle is adaptively adjusted in the right direction to scan the long-axis image, short-axis image, and abdominal aorta image of the inferior vena cava. The image captured by the depth vision device is used to intelligently identify the suprasternal fossa, the first intercostal space on the left side of the sternum, tilt toward the head, rotate the probe, adaptively adjust the probe rotation angle and tilt angle, and scan the ascending aortic arch, distal end, arterial arch, descending aorta, and suprasternal fossa aortic arch images.

[0072] Using images captured by the deep vision device, the system intelligently identifies the third and fourth ribs on the right side of the sternum, moves the probe to the third and fourth intercostal spaces, and adaptively adjusts the probe's rotation and tilt angles to scan the left atrium, right atrium, atrial septum, and right side of the sternum. The probe is rotated 90 degrees counterclockwise to scan the sagittal heart image on the right side of the sternum.

[0073] The image collected by the deep vision device is used to intelligently identify the second and third ribs on the right side of the sternum, move the probe to the second or third intercostal space, and adaptively adjust the probe rotation angle and tilt angle to scan the long-axis image of the ascending artery on the right side of the sternum;

[0074] Example 2:

[0075] The present invention is further described in detail below in conjunction with the embodiments and Figures 1, 2, and 3, but the embodiments of the present invention are not limited thereto. General visual images, human faces, nipples, characteristic genitals, characteristic parts, human organs, blood vessels and features are input, and depth visual images are input to identify bone information, the bottom end bones of the ribs, the shoulder joint, various joints, the lower limit of the bones and their positions.

[0076] Based on the vascular color information, vascular position information and the contour shape characteristics, structural characteristics and color characteristics of the organs in the ultrasound image, the vascular information, vascular position information and the organ information in the ultrasound image are used as input items, and the improved deep neural network method and weight optimizer are applied. Through image training, the output values, organs, blood vessels, bones, and the organs and position information of the human body in which they are located are obtained, and the results of autonomous positioning of organs, blood vessels, bones, and the positions of the human body in which they are located are output.

[0077] According to the administrator, the doctor publishes the collection task and medical order message, obtains the organ and its external position area coordinates of the human body collection image corresponding to the collection task, sets it as the target, sets the target name, target parameters, location information, and sets the communication target.

[0078] The robot's visual acquisition device and visual recognition module publish the external features corresponding to each organ, the external scanning feature information of the neighboring feature organs identified by the general image model, and the coordinates of the external position area of ​​the human body; the depth information released by the depth camera, the external scanning bone information of the neighboring bone information.

[0079] The robotic arm, ultrasound device, ultrasound probe, and robot subscribe to the corresponding organ external scanning area position information, subscribe to the target, parameters, target pose, pose tag, set the target for the head ID, target pose, direction value, set the timestamp, the remote main control system and the ultrasound probe carried by the autonomous robotic arm move according to the subscribed collection area position and the actions of the robotic arm image acquisition action planning module, and scan the human body collection area.

[0080] The ultrasound probe and device publish captured image information, including vascular color and location information, as well as the contour, shape, structural, and color features of the organs in the ultrasound image. Robots, medical devices, and visual recognition modules consume this image information. These extract vascular color and location information, as well as the contour, shape, structural, and color features of the organs in the ultrasound image, and input them into a computational model to intelligently identify whether the image represents the target organ or tissue. If so, the scanned target organ is detected.

[0081] Set acquisition target parameters, set position and posture tolerances, allow re-planning when motion planning fails, set the reference coordinate system for the target position, and set a time limit for each motion planning. Robots, medical devices, and robotic arms can also adjust and adaptively capture image parameters and video parameters for medical devices, as well as image acquisition methods, to determine if they meet image and video recognition standards and ensure effective acquisition.

[0082] The robot, medical device, and robotic arm's distal end adaptively adjust the ultrasound probe's scanning mode, probe angle, and parameters, check the target organ acquisition location, and verify that all images and videos of the acquired organ and tissue are complete under all scanning modes, including those of the nipple and breast. The robot and robotic arm return target organ acquisition completion information. The robot, medical device, and robotic arm subscribe to the task information. The robot, medical device, and robotic arm's distal end adaptively move the ultrasound probe to the external scanning location of the next target organ. Based on the returned nipple and breast organ acquisition completion information, the robot and medical device determine that the nipple and breast acquisition task is complete.

[0083] Intelligently identify nipples and breasts and their location, range, and contours through images collected by a visual device; establish a depth vision device and a depth vision image model, a chest bone model, and a shoulder joint axillary model; use depth information and the location of the bones in the human body as input into a neural network and its improved method to intelligently identify the sternum, shoulder joints, shoulder joint axillary, and their location information;

[0084] Using images collected by the depth vision device to intelligently identify the sternum, shoulder joints, and axillary sides of the shoulder joints, as well as their positional information, the probe is scanned longitudinally from the mid-axillary line. The probe is moved from the outside to the inside of the breast, and then from the inside to the outside. The probe is placed in the mid-axillary line and scanned through the nipple to the lower edge of the breast until the gland disappears, ending the longitudinal scan.

[0085] Using images collected by the depth vision device to intelligently identify the sternum, shoulder joints, and axillary sides of the shoulder joints, as well as their positional information, the probe is placed on the clavicle and scanned in a transverse direction from the clavicle to the mid-axillary line. The probe is moved from the cranial side of the upper breast to the crural side of the lower breast, and then from the cranial side to the cranial side of the upper breast until the gland disappears, ending the longitudinal scan.

[0086] The nipple and its position are intelligently identified using images collected by the visual device. The probe is placed on the nipple and the ultrasound probe is rotated to scan from the nipple toward the outer edge of the breast until it reaches the opposite rib and the gland disappears, ending the rotational scanning.

Claims

1. A method for intelligently identifying chest organs and autonomously positioning and scanning chest organs, characterized in that: An intelligent recognition method for integrating the five senses, organs, bones, joints, blood vessels, human features and their locations of the human body is provided. The method is a method for integrating planar visual images, depth images, ultrasound images, videos, and endoscopic images to intelligently recognize human organs, bones, joints, blood vessels, human feature points and their locations, including the following steps: S1. Establish a general visual image model, a human organ model, a blood vessel model, and a feature model. The general image model includes: face, nipple, breast, navel, and characteristic parts and their locations as feature items. Use a neural network algorithm and its improved method to intelligently identify human features and their locations. S2. Establish a depth vision device and a depth vision image model, and a chest bone model, and combine the depth information and the position information of the sternum, shoulder joint, second rib, third rib, fourth rib, fifth rib, xiphoid process, and spine of the human body where the bones are located, and the neighboring feature organ information identified by the general image model described in S1 as feature items of the skeleton intelligent recognition model as input items; S3. Apply neural network algorithms and their improved methods to intelligently identify each bone and its location, including the second rib, third rib, fourth rib, fifth rib, rib base, xiphoid process, vertebrae, vertebral position, lower limit of bones, shoulder joint and its location, and the location of each joint. S4, using the information of the nipple and the heart, a characteristic organ adjacent to the breast, identified by the general image model described in S1, as external scan feature information, and using the external scan bone information identified by S2, as well as the ultrasound image, video, blood vessel color information under the video, blood vessel position information, and the contour shape features, structural features, color features, and position features of the organs in the ultrasound image and video as input items; S5. The feature items are input into the neural network and its improved method and weight optimizer as input items, and the output value is obtained through image training; S6. Identify organs, blood vessels, bones, and their location information on the human body, and scan and autonomously locate organs, blood vessels, bones, and their location on the human body; 2. A method for intelligently identifying chest organs and autonomously positioning and scanning chest organs, characterized by a robotic arm, an ultrasound device, and a method for autonomously identifying the heart, breast, and chest organs and positioning and scanning the organs, comprising the following steps: S1. Obtain the coordinates of the organ and its external location area of ​​the human body image corresponding to the acquisition task, set it as the target, set the target name, target parameters, location information, and set the communication target; S2, robotic arm, ultrasound device, ultrasound scanning device, the robot subscribes to the corresponding organ external scanning area position information, subscribes to the target, parameters, target pose, pose tag, sets the target for the head ID, target pose, direction value, and sets the timestamp; S3. The remote main control system and the ultrasound scanning device mounted on the autonomous robotic arm move and scan the human body collection area according to the subscribed collection area location and the image collection action plan of the robotic arm. The ultrasound scanning device and the ultrasound device release the collected image information, blood vessel color information, blood vessel location information and ultrasound image, and the contour shape characteristics, structural characteristics, and color characteristics of the video organ; S4. The robot main system determines all target organ harvesting tasks according to the returned target organ harvesting completion information; S5. Based on S2, according to the robot's autonomous positioning, the coordinates of the external position area of ​​the human body, the method of scanning and collecting medical images, the robot arm, the ultrasound device, the ultrasound scanning device, the robot main system subscribes to the breast, the position information of the external scanning area of ​​the heart, the scanning method, the probe angle, the subscription target, the parameters, the target pose, the pose tag, the target for the head ID, the target pose, the direction value, and the timestamp; S6. Based on S1, using a general visual device and a multi-medical image fusion intelligent recognition method, intelligently identify the human nipple, breast, heart and their locations; using a deep vision acquisition device and a multi-medical image fusion intelligent recognition method, intelligently identify the bones, sternum, second rib, third rib, fourth rib, fifth rib, and the scanning locations of the corresponding structural parts of the heart organ; S7. The robot main system and the robotic arm remotely and adaptively adjust the image parameters of the medical device, gain parameters, color gain parameters, sensitivity time control adjustment parameters, time gain control parameters, focus parameters, depth parameters, frame size, blood flow velocity scale parameters, video parameters, image acquisition method, robotic arm, ultrasound scanning device, remote control and adaptive adjustment of probe rotation angle, tilt angle, scanning mode of the ultrasound scanning device, probe angle, parameters to the following organs, tissues for effective and complete acquisition; S8, remote control and adaptive movement of the robotic arm. Based on S2, the deep vision device uses images collected to intelligently identify the chest ribs. Based on S1, the planar vision device intelligently identifies the breast and its position. The robotic arm scans the heart along the second rib, third rib, fourth rib, and intercostal space along the left edge of the sternum.

3. A method for intelligently identifying chest organs and autonomously positioning and scanning chest organs, characterized by remotely controlling and adaptively moving a robotic arm to autonomously scan the heart between the ribs, comprising the following steps: S1. According to claim 1 and claim 2, an ultrasound probe is used to scan an image and intelligently identify a dynamic pulsating organ. The robotic arm moves along the right shoulder to scan the long axis image of the left ventricle. The probe scans inward and tilts outward to display the anterior leaflet of the mitral valve and the left ventricle. The probe is rotated to scan along the ascending aorta. S2. According to claim 1 and claim 2, the nipple and its position are intelligently identified through images collected by the visual device, the probe rotates counterclockwise to scan, and the robotic arm scans the long axis and image of the anterior leaflet along the anterior papillary muscle, chordae tendineae, and anterior leaflet of the mitral valve; S3. According to claim 1 and claim 2, the nipple and its position are intelligently identified through images collected by the visual device, the probe rotates counterclockwise for scanning, and the robotic arm scans the long axis and image of the posterior leaflet along the posterior papillary muscle, chordae tendineae, and posterior leaflet of the mitral valve; S4: The ultrasound probe is lowered one intercostal space and scanned. The probe is tilted medially or laterally to scan the papillary muscles, chordae tendineae, posterior leaflet of the mitral valve, midline long axis and image, anterior leaflet long axis and image, and posterior leaflet long axis and image of the next rib. S5. According to claim 1 and claim 2, the chest ribs are intelligently identified through images collected by the depth vision device, and the breast and its position are intelligently identified through images collected by the vision device. The robotic arm scans the high intercostal long axis and image along the second rib, the third rib, and between the ribs on the left side of the sternum; S6. According to claim 1 and claim 2, the positional features and structural features of each part of the image collected by ultrasound are used to intelligently identify the circular shape, inverted triangle shape, and Y-shaped features in diastole and systole by scanning the image with the probe. The part, position, and structure are intelligently identified. The probe is moved slightly outward and downward, and the robotic arm moves along the left edge of the sternum and rotates 90 degrees clockwise to scan the four heart chambers and the mitral valve, tricuspid valve, four-chamber heart of the left chest wall, pulmonary artery, aortic valve, pulmonary valve, and images; S7. According to claim 1 and claim 2, through ultrasound images, the positional features and structural features of each part under the video, the robot arm moves along the left edge of the sternum to scan, starting from the left ventricular long axis image, the probe rotates counterclockwise, the right ventricular inflow tract long axis image and the right ventricular outflow tract long axis image, the probe scans the right ventricular infundibulum, pulmonary valve, and pulmoidal artery trunk counterclockwise; S8. According to claim 1 and claim 2, the probe is used to scan and collect images, and intelligently identify dynamic pulsating organs. Through ultrasound images and the positional characteristics and structural characteristics of each part under the video, the probe is placed at the apex of the heart at the pulsating position, tilted from the chest wall side to the back side, and rotated to scan the four heart chambers and the mitral valve, tricuspid valve, and the apex of the four-chamber heart image; S9. Rotate the ultrasound probe counterclockwise to scan the apical two-chamber image and apical long-axis image, left ventricle, and right ventricle images; S10. According to claim 1 and claim 2, the visual device intelligently identifies the nipple and its position, uses the image collected by the depth vision device to intelligently identify the bones at the bottom of the ribs and the position of the xiphoid process, identifies the position of the apex of the heart, and the robotic arm uses the pressure device to apply pressure from the xiphoid process position, tilts toward the head, rotates the probe, and adaptively adjusts the tilt angle to scan the four heart chambers and the mitral valve, tricuspid valve, and the four-chamber heart image below the xiphoid process; S11. According to claim 1 and claim 2, the probe rotates clockwise to record the three-chamber view (left ventricle, left atrium, aorta) and the two-chamber view (left ventricle, left atrium) in sequence from the four-chamber heart image; S12. According to claim 1 and claim 2, the image collected by the probe is intelligently identified as a circular shape and marked as the left ventricle, and the scan is adaptively adjusted to the center of the image, the short-axis image of the left ventricle under the xiphoid process, the long-axis image of the inferior vena cava, the short-axis image, the image of the abdominal aorta, and the tilt angle is adaptively adjusted in the right direction to scan the long-axis image of the inferior vena cava, the short-axis image, and the image of the abdominal aorta; S13. Based on claim 1 and claim 2, the image captured by the depth vision device is used to intelligently identify the suprasternal fossa, the first intercostal space on the left side of the sternum, tilt the probe toward the head, rotate the probe, adaptively adjust the probe rotation angle and tilt angle, and scan the ascending aortic arch, distal end, aortic arch, descending aorta, and suprasternal fossa aortic arch images; S14. Based on claims 1 and 2, use the images captured by the depth vision device to intelligently identify the third and fourth ribs at the right sternal border, move the probe to the third and fourth intercostal spaces, and adaptively adjust the probe rotation and tilt angles to scan the left atrium, right atrium, atrial septum, and right sternal border cardiac horizontal images. Adjust the probe to rotate 90 degrees counterclockwise to scan the right sternal border cardiac sagittal images; S15. According to claim 1 and claim 2, the image captured by the depth vision device is used to intelligently identify the second rib and the third rib on the right side of the sternum, the probe is moved to the second rib or the third intercostal space, the rotation angle and the tilt angle of the probe are adaptively adjusted, and the long-axis image of the ascending artery on the right side of the sternum is scanned; 4. A method for intelligently identifying chest organs and autonomously positioning and scanning chest organs, characterized by remotely controlling and adaptively moving a robotic arm to autonomously scan the breast between the ribs, comprising the following steps: S1. According to claim 1 and claim 2, the nipple, breast and its position, range and outline are intelligently identified through images collected by a visual device; S2. Based on claim 1 and claim 2, a depth vision device and a depth vision image model, a chest bone model, and a shoulder joint axillary model are established. The depth information and the position of the bones on the human body are input into a neural network and an improved method thereof to intelligently identify the sternum, shoulder joints, shoulder joint axillary and their position information. S3. Based on claim 1 and claim 2, the sternum, shoulder joints, axillary sides of the shoulder joints and their position information are intelligently identified using images collected by a depth vision device. The probe is scanned longitudinally from the mid-axillary line, and the probe is moved from the outside to the inside of the breast, and then from the inside to the outside. The probe is placed in the mid-axillary line, and the probe is scanned through the nipple to the lower edge of the breast until the gland disappears, and the longitudinal scanning is terminated. S4. Based on claim 1 and claim 2, the sternum, shoulder joints, axillary sides of the shoulder joints and their position information are intelligently identified using images collected by a depth vision device. The probe is placed on the clavicle and scanned in the transverse direction from the clavicle and the mid-axillary line. The probe is moved from the head side of the upper breast to the foot side of the lower breast, and then from the foot side to the head side of the upper breast until the gland disappears, ending the longitudinal scan. S5. Based on claims 1 and 2, the nipple and its position are intelligently identified using images collected by a visual device. The probe is placed on the nipple. With the nipple as the center, the ultrasound probe is rotated and scanned from the nipple toward the outer edge of the breast until the ribs in the opposite direction disappear and the gland disappears, ending the rotational scanning.

Images