Human mammary gland automatic examination system based on large language model combined with six-axis robot

Abstract

The invention discloses an automatic human mammary gland examination system based on a large language model combined with a six-axis robot, which can accurately understand a natural language instruction of a clinician by deeply fusing the natural language processing capability of the large language model, the high-precision motion control of the robot and an advanced medical image processing algorithm. A scanning path is automatically planned and executed, a two-dimensional ultrasonic image sequence is collected, a three-dimensional mammary gland model is reconstructed, the model is analyzed through an intelligent algorithm, and a structured clinical diagnosis report is automatically generated, so that full-chain automation of mammary gland ultrasonic scanning from intention understanding to report generation is realized; the standardization degree, efficiency and diagnosis consistency of mammary gland ultrasonic examination are remarkably improved, a reliable automatic solution is provided for early screening and precise diagnosis of mammary gland diseases, and the technical difficulties that in the prior art, due to manual operation, efficiency is low, repeatability is poor, and results depend on experience of operators are solved.

Description

Technical Field

[0001] This invention belongs to the technical field, specifically relating to an automated human breast examination system based on a large language model combined with a six-axis robot. Background Technology

[0002] Traditional breast ultrasound scans rely on the operator's experience and skill level, leading to inconsistent results and low efficiency.

[0003] Existing technologies lack a fully automated system that can integrate intelligent demand analysis, automatic path planning, precise motion execution, and intelligent report generation, making it difficult to achieve standardized and highly repeatable breast ultrasound examinations. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention provides an automated human breast examination system based on a large language model and a six-axis robot, comprising: The central control unit receives ultrasound scanning requests from general surgeons' workstations via a switch and the hospital's local area network. It then uses a large language model to perform intelligent semantic parsing and intent recognition on the natural language descriptions of the scanning needs in the ultrasound scanning requests, generating a structured task description. The system determines whether the structured task description falls within the preset breast scan area. If so, it generates a key point description containing anatomical locations and scan focus areas, as well as instructions for patient positioning and triggering of adjustable bed height. Based on the key point description, it generates 3D camera control instructions. 3D point cloud images of the breast region are acquired and coarsely registered with a standard human breast model. A vision-based robot path planning algorithm, combined with structured task description and key point description, is used to mark key scanning areas on the point cloud. This generates the trajectory of a six-axis robot end effector as a high-precision scanning path command and scanning trigger signal. The system receives spatiotemporally synchronized 2D scanning image sequences from a clinical ultrasound scanner and robot pose data from a six-axis robot. Each pixel is mapped to the world coordinate system through coordinate transformation, and a voxel fusion algorithm is used for reconstruction to generate a 3D breast model. A post-processing algorithm integrating morphological analysis and machine learning classifiers is used to segment the 3D breast model, distinguishing between glandular tissue, fat, and suspected lesion areas. Key morphological features of suspected lesion areas are extracted, and the proportion of nodule volume to glandular volume, spatial distribution, and key morphological features in the breast are calculated. For suspected nodule areas with key morphological features, texture and depth features are extracted and used as a multi-dimensional feature vector. This vector is input into a deep learning classifier or support vector machine to calculate elasticity score, benign / malignant probability, and BI-RADS classification as key clinical indicators. Key clinical indicators are combined with a medical knowledge base to generate detailed ultrasound diagnostic reports that conform to clinical standards. These reports are then sent to the clinical diagnostic report display and printing system via the system's local area network for ultrasound physicians to review. After review, the detailed ultrasound diagnostic reports are fed back to general surgeons via the hospital's local area network.

[0005] Preferably, after receiving an ultrasound scan request, the central control unit registers the patient information and the doctor's ultrasound scan request in a MySQL database.

[0006] Preferably, if the structured task description is not within the preset breast scan range, a reason for refusal and a suggestion are generated and fed back to the requesting doctor via the original route.

[0007] Preferably, the trajectory of the six-axis robot end effector includes position and orientation information, and the trajectory is fitted using fifth-order polynomial interpolation with spline interpolation technology to generate a continuous and smooth robot motion trajectory.

[0008] Preferably, while obtaining the trajectory of the six-axis robot end effector, the feedback contact pressure from the robot force sensor is processed, and the motion is dynamically adjusted through impedance control to ensure that the contact force is stable at a preset safety threshold.

[0009] Preferably, the vision-based robot path planning algorithm aims to cover all key areas while minimizing path length and movement time, and avoiding obstacles.

[0010] Preferably, the detailed ultrasound diagnostic report includes examination findings, measurement data, diagnostic opinions, and recommendations.

[0011] This invention provides an automated breast examination system based on a large language model combined with a six-axis robot. By deeply integrating the natural language processing capabilities of the large language model, the high-precision motion control of the robot, and advanced medical image processing algorithms, it can accurately understand the natural language instructions of clinicians, automatically plan and execute scanning paths, acquire two-dimensional ultrasound image sequences, reconstruct a three-dimensional breast model, and automatically generate a structured clinical diagnostic report through intelligent algorithm analysis. This achieves fully automated operation of breast ultrasound examination from task reception to report generation, solving the problems of low efficiency and inconsistent results in traditional breast ultrasound examinations. Attached Figure Description

[0012] Figure 1 A schematic diagram of an automated human breast examination system based on a large language model and a six-axis robot. Figure 2 This is a schematic diagram illustrating the implementation process of an automated human breast examination system based on a large language model and a six-axis robot. Detailed Implementation

[0013] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0014] like Figure 1 and Figure 2 As shown, the automatic human breast examination system based on a large language model and a six-axis robot provided in this embodiment of the invention includes: General surgeons send structured ultrasound scan requests, including patient identification information and scan requirements described in natural language, to the central control unit via the hospital information system and the hospital's local area network.

[0015] The central control unit is responsible for receiving ultrasound scan requests sent by general surgeons' workstations via switches and the hospital's local area network. It registers patient information and doctors' ultrasound scan requests in a MySQL database, and uses a large language model to perform intelligent semantic parsing and intent recognition (named entity recognition and intent classification) on the natural language descriptions of the scan requirements in the ultrasound scan requests, generating a structured task description T={region, pattern, key points}.

[0016] The system determines whether the structured task description falls within the preset breast scan area. If not, it generates a rejection reason and suggestion, which is then fed back to the requesting doctor via the original route. If it does fall within the area, it generates a key point description including anatomical location and scan focus, as well as instructions for patient positioning and triggering of a height-adjustable bed. Based on the key point description, it generates 3D camera control instructions, acquires a 3D point cloud image of the breast region P={pi∈R3}, performs coarse registration with a standard human breast model, and uses a vision-based robot path planning algorithm, combined with the structured task description and key point description, to mark the key scan area on the point cloud. This generates the trajectory of the six-axis robot end effector as a high-precision scan path instruction and scan trigger signal, which consists of a series of discrete path points Q={qj∈R6}. The definition includes position and attitude information. To ensure smooth motion, a fifth-order polynomial interpolation technique is used for trajectory fitting to generate a continuous and smooth robot motion trajectory s(t). Its position component is expressed as: s(t)=a0+a1t+a2t2+a3t3+a4t4+a5t5, where t is the normalized time, and the coefficient ak is determined by the position, velocity, and acceleration boundary conditions of the starting point and the target point. At the same time, the feedback contact pressure F∈R6 from the robot force sensor is processed, and the motion is dynamically adjusted through algorithms such as impedance control to ensure that the contact force fz is stable at the preset safety threshold: Fsafe.

[0017] Vision-based robot path planning algorithms aim to cover all critical areas while minimizing path length and movement time, and avoiding obstacles.

[0018] The system receives a 2D scanning image sequence I(x,y,t) from a clinical ultrasound scanner and a robot pose Tworldprobe(t) from a six-axis robot. Each pixel is mapped to the world coordinate system through coordinate transformation, and a voxel fusion algorithm is used for reconstruction, or a 3D reconstruction algorithm based on surface reconstruction is used to generate a three-dimensional breast model.

[0019] For each voxel v, its intensity value V(v) is obtained by a weighted average of all pixel values ​​projected onto that voxel: V(v)=∑iwi∑iwi I(pi) Here, wi is a weighting factor, which is usually related to the measurement distance and confidence level, and finally a high-fidelity three-dimensional breast model is obtained.

[0020] A post-processing algorithm integrating morphological analysis and machine learning classifiers was used to segment a 3D breast model, distinguishing between glandular tissue, fat, and suspected lesion regions (Rlesion). Suspected lesion regions (Rlesion) were extracted, and key morphological features such as the target region's volume (Vol = ∑v∈R1 surface area) and sphericity (describing shape regularity) were calculated. Ψ=π1 / 3(6Vol)2 / 3SurfaceAreaΨ=SurfaceAreaπ1 / 3(6Vol)2 / 3 The proportion of nodule volume (Vollesion) to glandular volume, spatial distribution, and key morphological features in the breast are calculated. For suspected nodule regions with key morphological features, texture and depth features are extracted and used as a multidimensional feature vector x. This vector is then input into a deep learning classifier or support vector machine (SVM) to calculate elasticity score, benign / malignant probability, and BI-RADS classification as key clinical indicators D={Vol,Ψ,Pmalignant,...}. The decision function of the SVM is: f(x)=sign(∑i=1nαiyiK(xi,x)+b) Where x is the feature vector and K is the kernel function, which is used to calculate the benign or malignant probability of the nodule and the suggested BI-RADS classification.

[0021] Key clinical indicators are combined with a medical knowledge base to generate detailed ultrasound diagnostic reports that conform to clinical standards. These reports are then sent to the clinical diagnostic report display and printing system via the system's local area network for ultrasound physicians to review.

[0022] A detailed ultrasound diagnostic report includes examination findings, measurement data, diagnostic opinions, and recommendations.

[0023] After the review is completed, a detailed ultrasound diagnostic report is sent to the general surgeon via the hospital's local area network.

[0024] The patient call and management system is responsible for informing patients of their designated positions via a waiting screen. Patients are sequentially called into the waiting area and examination room. Voice prompts guide each patient to lie in a standard lateral decubitus position on an adjustable bed, allowing for minor adjustments to their posture and breathing coordination. After the examination, the patient is guided to leave. This completes the fully automated examination process for a single patient, the system resets, and prepares for the next patient, achieving seamless workflow within the examination room. The entire process, coordinated by the patient call and management system, ensures the orderly and efficient operation of the examination process.

[0025] The height-adjustable bed is responsible for receiving trigger commands from the central control unit via the local area network, adjusting it to the preset initial height and angle, so that the patient's breast area is in the most suitable spatial position for six-axis robot scanning and 3D camera shooting, thus improving the adaptability of the entire system.

[0026] The 3D camera is responsible for capturing images of the exposed breast area according to the 3D camera control instructions, obtaining 3D point cloud images of the breast area, using structured light or binocular vision principles, and transmitting the 3D point cloud images to the central control unit in real time via USB or gigabit Ethernet, providing key spatial geometric information for initial registration and path planning.

[0027] The six-axis robot is responsible for receiving and executing scanning path instructions sent by the central control unit via a local area network. As a mechanical motion component for performing breast scans, it has sub-millimeter-level positioning accuracy and can accurately execute complex three-dimensional trajectories. Its end effector integrates a force sensing module, which can monitor and provide feedback on the contact pressure between the scanning probe and the body surface in real time. This allows the central control unit to dynamically adjust the path or pressure settings based on the feedback, ensuring patient safety and comfort while acquiring high-quality images.

[0028] The clinical ultrasound scanner is responsible for receiving trigger signals sent by the central control unit via a local area network, executing specific ultrasound imaging tasks, acquiring 2D scanning image sequences when the six-axis robot moves to a predetermined position, and has temporary buffering capabilities.

[0029] The clinical diagnostic report display and printing system is responsible for obtaining detailed ultrasound diagnostic reports through the system's local area network, and for visualizing, printing, and managing the review process. It provides an interactive interface for ultrasound physicians to review detailed ultrasound diagnostic reports, make modifications and confirmations, and record review logs, thus achieving a closed loop in report generation and management.

[0030] The network switch is the hub of internal system communication, ensuring stable and high-speed data exchange between the central control unit and equipment such as the hospital LAN, six-axis robot, clinical ultrasound scanner, adjustable bed, patient call and management system, and clinical diagnostic report display and printing system.

[0031] The system's local area network is an internal high-speed communication network that ensures real-time control and low-latency data transmission between the central control unit and various execution components.

[0032] The hospital's local area network (LAN) serves as a bridge connecting the system to the hospital's information network. It is responsible for receiving scan requests from the hospital's information system, providing access to cloud-based or local large language models, and sending the final approved diagnostic report back to the doctor's workstation.

[0033] This invention provides an automated breast examination system based on a large language model combined with a six-axis robot. By deeply integrating the natural language processing capabilities of the large language model, the high-precision motion control of the robot, and advanced medical image processing algorithms, it can accurately understand the natural language instructions of clinicians, automatically plan and execute scanning paths, acquire two-dimensional ultrasound image sequences, reconstruct a three-dimensional breast model, and automatically generate a structured clinical diagnostic report through intelligent algorithm analysis. This achieves full-chain automation of breast ultrasound scanning from intent understanding to report generation, significantly improving the standardization, efficiency, and diagnostic consistency of breast ultrasound examinations. It provides a reliable automated solution for early screening and accurate diagnosis of breast diseases, with broad clinical application prospects. It solves the technical difficulties of low efficiency, poor repeatability, and reliance on operator experience caused by manual operation in existing technologies. It provides an innovative technical platform for achieving high-quality, reproducible breast disease screening and diagnosis, and has the potential for widespread deployment in medical institutions at all levels.

Claims

1. An automated human breast examination system based on a large language model and a six-axis robot, characterized in that, include: The central control unit receives ultrasound scanning requests from general surgeons' workstations via a switch and the hospital's local area network. It then uses a large language model to perform intelligent semantic parsing and intent recognition on the natural language descriptions of the scanning needs in the ultrasound scanning requests, generating a structured task description. The system determines whether the structured task description falls within the preset breast scan area. If so, it generates a key point description containing anatomical locations and scan focus areas, as well as instructions for patient positioning and triggering of adjustable bed height. Based on the key point description, it generates 3D camera control instructions. 3D point cloud images of the breast region are acquired and coarsely registered with a standard human breast model. A vision-based robot path planning algorithm, combined with structured task description and key point description, is used to mark key scanning areas on the point cloud. This generates the trajectory of a six-axis robot end effector as a high-precision scanning path command and scanning trigger signal. The system receives spatiotemporally synchronized 2D scanning image sequences from a clinical ultrasound scanner and robot pose data from a six-axis robot. Each pixel is mapped to the world coordinate system through coordinate transformation, and a voxel fusion algorithm is used for reconstruction to generate a 3D breast model. A post-processing algorithm integrating morphological analysis and machine learning classifiers is used to segment the 3D breast model, distinguishing between glandular tissue, fat, and suspected lesion areas. Key morphological features of suspected lesion areas are extracted, and the proportion of nodule volume to glandular volume, spatial distribution, and key morphological features in the breast are calculated. For suspected nodule areas with key morphological features, texture and depth features are extracted and used as a multi-dimensional feature vector. This vector is input into a deep learning classifier or support vector machine to calculate elasticity score, benign / malignant probability, and BI-RADS classification as key clinical indicators. Key clinical indicators are combined with a medical knowledge base to generate detailed ultrasound diagnostic reports that conform to clinical standards. These reports are then sent to the clinical diagnostic report display and printing system via the system's local area network for ultrasound physicians to review. After review, the detailed ultrasound diagnostic reports are fed back to general surgeons via the hospital's local area network.

2. The automated human breast examination system based on a large language model and a six-axis robot as described in claim 1, characterized in that, After receiving an ultrasound scan request, the central control unit registers the patient information and the doctor's ultrasound scan request in a MySQL database.

3. The automated human breast examination system based on a large language model and a six-axis robot as described in claim 1, characterized in that, If the structured task description is not within the preset breast scan range, a reason for refusal and a suggestion will be generated and fed back to the requesting doctor via the original route.

4. The automated human breast examination system based on a large language model and a six-axis robot as described in claim 1, characterized in that, The trajectory of the six-axis robot end effector includes position and orientation information, and the trajectory is fitted using fifth-order polynomial interpolation with spline interpolation technology to generate a continuous and smooth robot motion trajectory.

5. The automated human breast examination system based on a large language model and a six-axis robot as described in claim 1, characterized in that, While obtaining the trajectory of the six-axis robot end effector, the feedback contact pressure from the robot force sensor is processed, and the motion is dynamically adjusted through impedance control to ensure that the contact force is stable at a preset safety threshold.

6. The automated human breast examination system based on a large language model and a six-axis robot as described in claim 1, characterized in that, The vision-based robot path planning algorithm aims to cover all critical areas while minimizing path length and movement time, and avoiding obstacles.

7. The automated human breast examination system based on a large language model and a six-axis robot as described in claim 1, characterized in that, The detailed ultrasound diagnostic report includes examination findings, measurement data, diagnostic opinions, and recommendations.

Images