Multimodal ultrasound scanning method, apparatus and related systems

Abstract

The application relates to the technical field of medical intelligent diagnosis, and provides a multi-modal ultrasonic scanning method, device and related system. The method collects depth image data of a skin area of a user through a camera module, controls a mechanical arm to adjust an ultrasonic probe to a first target pose according to scanning intention information of the user and the depth image data, wherein the ultrasonic probe and a force feedback sensor are integrated at the tail end of the mechanical arm, calculates an ultrasonic quality evaluation parameter according to first ultrasonic image data of the ultrasonic probe and first contact pressure data of the force feedback sensor, obtains a first ultrasonic quality evaluation parameter, adjusts the ultrasonic probe to a second target pose through the mechanical arm according to the first ultrasonic quality evaluation parameter, and scans the user according to the depth image data, second ultrasonic image data of the ultrasonic probe and second contact pressure data of the force feedback sensor to obtain a scanning result. In this way, the stability and safety of the ultrasonic probe during scanning are improved.

Description

Technical Field

[0001] This application relates to the field of medical intelligent diagnostic technology, and in particular to a multimodal ultrasound scanning method, device and related system. Background Technology

[0002] Currently, ultrasound diagnosis, as one of the means of clinical imaging examination, is widely used for the early screening and accurate diagnosis of lesions in various parts of the body, including the abdomen, cardiovascular system, and superficial organs. With the development of ultrasound robots and intelligent diagnostic and treatment technologies, automated and precise ultrasound diagnosis has become an important direction for improving diagnostic and treatment efficiency.

[0003] However, existing technologies have significant limitations in dealing with interference from human physiological movements. During ultrasound diagnosis, the patient's spontaneous breathing causes the skin position in the scanning area to fluctuate periodically. This physiological movement directly causes fluctuations in the contact state between the probe and the body surface. On the other hand, respiratory displacement can lead to false contact of the probe, resulting in blurred ultrasound images, distorted cross-sectional structures, and reduced accuracy of diagnostic results.

[0004] Therefore, how to effectively counteract the displacement deviation caused by the patient's respiratory movements and improve the stability and operational safety of ultrasound scanning is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a multimodal ultrasound scanning method, device, and related system to solve the problem of poor contact stability and safety of ultrasound probes when scanning skin areas in the existing field of ultrasound diagnostics.

[0006] To achieve the objectives of this application, the following technical solution is provided:

[0007] In a first aspect, this application provides a multimodal ultrasound scanning method, applied to the host device of an ultrasound scanning system, the method comprising:

[0008] The camera module collects depth image data of the user's target skin area;

[0009] Based on the user's scanning intent and the depth image data, the robotic arm is controlled to adjust the ultrasonic probe to the first target pose. The ultrasonic probe and force feedback sensor are integrated at the end of the robotic arm.

[0010] The ultrasonic quality assessment parameters are calculated based on the first ultrasonic image data of the ultrasonic probe and the first contact pressure data of the force feedback sensor to obtain the first ultrasonic quality assessment parameters.

[0011] Based on the first ultrasonic quality assessment parameters, the robotic arm adjusts the ultrasonic probe to the second target pose.

[0012] The user is scanned based on the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area.

[0013] Secondly, this application provides a multimodal ultrasound scanning device, applied to the host device of an ultrasound scanning system, the device comprising:

[0014] The acquisition unit is used to acquire depth image data of the user's target skin area through the camera module;

[0015] The control unit is used to control the robotic arm to adjust the ultrasonic probe to the first target pose according to the user's scanning intention information and the depth image data. The ultrasonic probe and the force feedback sensor are integrated at the end of the robotic arm.

[0016] The calculation unit is used to calculate the ultrasonic quality assessment parameters based on the first ultrasonic image data of the ultrasonic probe and the first contact pressure data of the force feedback sensor, and obtain the first ultrasonic quality assessment parameters.

[0017] The control unit is further configured to adjust the ultrasound probe to the second target pose via the robotic arm according to the first ultrasound quality assessment parameters; and to scan the user according to the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area.

[0018] Thirdly, this application provides an electronic device including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing steps in any of the methods of the first aspect of the embodiments of this application.

[0019] Fourthly, embodiments of this application provide a multimodal ultrasound scanning system, wherein the multimodal ultrasound scanning system performs some or all of the steps described in any method of the first aspect of embodiments of this application.

[0020] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in any method of the first aspect of this application. The computer program product may be a software installation package.

[0021] By implementing the embodiments of this application, the following beneficial effects are achieved:

[0022] This application provides a multimodal ultrasound scanning method, apparatus, and related system, applied to a host device in a multimodal ultrasound scanning system. The multimodal ultrasound scanning system further includes: an ultrasound robot communicatively connected to the host device, and a terminal device communicatively connected to the host device. The ultrasound robot includes a robotic arm, a camera module, an ultrasound probe, and a force feedback sensor mounted on the robotic arm. The method includes: acquiring depth image data of a user's target skin area through the camera module; controlling the robotic arm to adjust the ultrasound probe to a first target pose based on the user's scanning intention information and the depth image data; wherein the ultrasound probe and the force feedback sensor are integrated at the end of the robotic arm; calculating ultrasound quality assessment parameters based on the first ultrasound image data of the ultrasound probe and the first contact pressure data of the force feedback sensor to obtain a first ultrasound quality assessment parameter; adjusting the ultrasound probe to a second target pose via the robotic arm based on the first ultrasound quality assessment parameter; and scanning the user based on the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain an ultrasound scan result of the target skin area. Since ultrasound quality assessment parameters can characterize the contact status between the ultrasound probe at the end of the robotic arm and the target skin area at the current moment, the host device can adjust the posture of the robotic arm in real time according to this quantitative index to optimize the contact status between the ultrasound probe and the target skin area. This allows for adaptive optimization and adjustment of abnormal interaction states such as virtual contact between the ultrasound probe and the target skin area caused by user shaking, breathing, and other body movements, thereby improving the contact stability and safety of the ultrasound probe when scanning the target skin area. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a system architecture diagram of a multimodal ultrasound scanning method provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the structure of an ultrasonic robot provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0027] Figure 4 This is a schematic flowchart of a multimodal ultrasound scanning method provided in an embodiment of this application;

[0028] Figure 5This is a schematic diagram of a scenario where an ultrasound robot performs ultrasound diagnosis, as provided in an embodiment of this application.

[0029] Figure 6 This is a schematic diagram of a remote ultrasound diagnostic scenario provided in an embodiment of this application;

[0030] Figure 7 This is a block diagram of the functional modules of a multimodal ultrasound scanning device provided in an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0032] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0033] It should be understood that the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document indicates that the preceding and following related objects are in an "or" relationship. In the embodiments of this application, "multiple" refers to two or more.

[0034] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0035] In this application, the term "connection" refers to various connection methods, such as direct connection or indirect connection, to achieve communication between devices. This application does not impose any limitations on this.

[0036] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0037] The following is an explanation of the relevant terms used in this application:

[0038] 6DoF (Six Degrees of Freedom) refers to the complete motion capability of an object in a three-dimensional coordinate system, which allows for translational motion along the three orthogonal axes (X, Y, and Z) and rotational motion around the same three orthogonal axes. It accurately describes the spatial position and orientation of an object in three-dimensional space. In the ultrasonic robot technology scenario of this application, 6DoF is used to characterize the full-dimensional pose adjustment capability of the robotic arm's end effector and the ultrasonic probe, enabling precise spatial positioning of the diagnostic area and multi-angle scanning posture adaptation.

[0039] The following is combined with Figure 1 The system architecture of a multimodal ultrasound scanning method according to an embodiment of this application will be described. Figure 1 This is a system architecture diagram of a multimodal ultrasound scanning method provided in an embodiment of this application. The multimodal ultrasound scanning system 100 includes a smart device layer 110, an edge computing layer 120, a cloud platform service layer 130, and an application layer 140.

[0040] The intelligent device layer 110 serves as the physical sensing and execution terminal of the multimodal ultrasound scanning system 100. It collects multimodal clinical sensing data and executes control commands issued from the edge / cloud. Through short-range communication via StarFlash service and 5.5G industrial-grade communication technology, it establishes a low-latency, highly reliable bidirectional communication link with the edge computing layer 120, providing a physical interaction foundation for ultrasound diagnosis. The intelligent device layer 110 includes an ultrasound robot 111, a 6-DOF collaborative robotic arm platform adapted to clinical ultrasound diagnostic scenarios. Internally, it integrates a camera module, robotic arm, ultrasound probe, and force feedback sensor. These components work together to complete multimodal data acquisition and precise operation execution. The camera module is a 3D depth vision acquisition unit that collects surface depth image data of the patient's target diagnostic area. After preprocessing, this data is converted into 3D point cloud data, characterizing the surface morphology and positional features of the patient's target skin area, providing spatial geometric constraints for the subsequent personalized construction of a digital twin model. The robotic arm is a 6DoF collaborative actuator used to carry the ultrasound probe and achieve high-precision adjustment of its end effector posture. Each joint has a built-in photoelectric encoder and position sensor, which can provide real-time feedback on motion parameters such as joint angle and angular velocity. It communicates with the edge computing layer 120 through the HarmonyOS underlying adapter framework to receive and execute motion control commands generated by the robotic arm's operating agent. The ultrasound probe is the unit for ultrasound image acquisition. It emits ultrasound waves to the target tissue of the patient and receives echo signals to generate two-dimensional ultrasound image data, providing an image data source for tissue feature extraction and lesion analysis. It is also rigidly connected to the end effector of the robotic arm, enabling synchronous spatial posture adjustment. A force feedback sensor is set at the connection between the ultrasound probe and the end effector of the robotic arm. It collects real-time contact pressure data between the ultrasound probe and the patient's body surface, monitors the force applied by the probe, and avoids excessive pressure causing patient discomfort or insufficient pressure leading to decreased image quality. This provides feedback for the force control strategy of the robotic arm.

[0041] For a better understanding of the ultrasonic robot 111, please refer to [link / reference needed]. Figure 2 , Figure 2This is a schematic diagram of the structure of an ultrasonic robot provided in an embodiment of this application. Specifically, the ultrasonic robot uses a multi-degree-of-freedom robotic arm as the actuator and integrates an ultrasonic probe, a force feedback sensor, and a camera module assembly. The robotic arm employs a multi-joint serial structure, possessing multi-dimensional posture adjustment and spatial positioning capabilities. It can precisely deliver the end effector to the target scanning area according to the clinical needs of ultrasound scanning, adapting to ultrasound detection scenarios of different body parts and angles. The ultrasound probe directly contacts the human body surface, transmitting ultrasound signals to the target tissue and receiving echo data, providing the raw data source for subsequent ultrasound image generation and diagnostic analysis. A force feedback sensor is integrated at the connection interface between the ultrasound probe and the robotic arm's end effector, collecting physical parameters such as contact force, pressure distribution, and posture deviation between the probe and human tissue in real time during scanning. This provides sensory input for the robotic arm's force control closed-loop adjustment, preventing tissue damage due to overload or image quality degradation due to insufficient pressure. The camera module is deployed on the side of the robotic arm's base, acquiring visual images of the scanning scene to achieve visual recognition and positioning of the scanning area, probe posture, and human body surface features, assisting the robotic arm in completing precise path planning and dynamic posture correction.

[0042] It is evident that this ultrasound robot, through multimodal modular integration, has formed an automated scanning architecture that is flexible, safe, and accurate. On one hand, the multi-degree-of-freedom robotic arm provides the motion basis for full-coverage scanning of complex body surface areas. On the other hand, the dual-sensing mechanism of the force feedback sensor and camera module provides real-time feedback from the multimodal scanning process of physical interaction and visual environment, ensuring both the safety and stability of the scanning operation and improving the consistency and reliability of ultrasound imaging.

[0043] The edge computing layer 120 is deployed at edge nodes close to the intelligent device layer 110. It is used for real-time preprocessing, local inference, and instruction generation of multimodal data. By scheduling NPU computing power through a heterogeneous computing architecture for Neural Networks (CANN), it achieves low-latency data processing and decision response, alleviating the computing pressure on the cloud platform service layer 130. The edge computing layer 120 includes two functional modules: an AI large model 121 and a robotic arm operating agent 122. The AI ​​large model 121 is a multimodal medical large model, pre-trained based on massive ultrasound clinical data and an anatomical knowledge base. It is used to perform multimodal fusion analysis on ultrasound image data, depth image data, and contact pressure data, extracting diagnostic parameters such as tissue texture and lesion morphology, providing inference basis for subsequent diagnostic results generation, and simultaneously enabling automatic segmentation of ultrasound images and anatomical structure localization. The robotic arm operating agent 122 serves as the control unit, integrating a reinforcement learning policy network and a 6DoF robotic arm inverse kinematics solution algorithm. Based on the diagnostic requirements and multimodal perception data output by the AI ​​large model 121, combined with the pose solution results in the world coordinate system, it transforms high-level diagnostic intentions into executable joint-level drive commands, which are then sent to the robotic arm via the Star Flash communication module to achieve precise positioning and stable scanning of the ultrasound probe.

[0044] The cloud platform service layer 130 serves as the computing power and data service center for the multimodal ultrasound scanning system 100. It establishes an edge-cloud collaborative communication link with the edge computing layer 120 via a 5.5G medical-grade dedicated slice, responsible for storing, managing, and deeply processing clinical data, while also providing digital twin modeling and global decision-making services. The cloud platform service layer 130 includes a host device 131 and a real-time digital twin model 132. The host device 131 is the scheduling and management unit of the cloud platform, coordinating data interaction between the edge computing layer 120 and the application layer 140, scheduling the cloud-based AI computing cluster to complete large-scale data processing and model training, and simultaneously using blockchain technology to hash and store diagnostic data on the blockchain, ensuring the immutability and traceability of the data. The real-time digital twin model 132 is a personalized virtual anatomical model for patients. It is constructed based on the three-dimensional point cloud data of the body surface, ultrasound image data and contact pressure data collected by the intelligent device layer 110. Through non-rigid deformation matching and voxel dynamic correction technology, it achieves accurate mapping with the patient's real-time anatomical state, position and tissue physical characteristics. It provides a virtual simulation environment for the robotic arm operating intelligent body 122 for rehearsing operation procedures and verifying control strategies.

[0045] The application layer 140 serves as the human-computer interaction and clinical application entry point for the multimodal ultrasound scanning system 100. Through the terminal device 141, it provides medical staff with diagnostic result display, operation monitoring, and remote intervention functions, enabling the clinical application of ultrasound diagnosis. The terminal device 141 can be deployed as a desktop computer, tablet, or AR / VR terminal. It communicates with the host device 131 via a 5.5G network, receiving diagnostic results, digital twin model visualization data, and robotic arm operation status data. It also supports medical staff inputting diagnostic intentions and initiating remote operation commands.

[0046] As can be seen, through a layered system architecture, the multimodal ultrasound scanning system 100 achieves collaboration between "physical sensing, edge computing, cloud services, and clinical applications." Efficient data links are established between each layer via short-range inter-satellite communication and 5.5G industrial-grade communication technology. Combined with the CANN and HarmonyOS underlying adaptation framework, real-time acquisition, transmission, processing, and application of multimodal data are ensured. This enables automation, precision, and intelligence in ultrasound diagnosis, improving clinical diagnostic efficiency and quality, and providing a reliable technical solution for ultrasound diagnosis in primary healthcare settings.

[0047] The following is combined with Figure 3 The electronic devices in the embodiments of this application will be described. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 3 As shown, the electronic device 300 includes a processor 310, a memory 320, a communication interface 330, and one or more programs 321. The processor 310 is communicatively connected to the memory 320 and the communication interface 330 via an internal communication bus.

[0048] The one or more programs 321 are stored in the memory 320 and configured to be executed by the processor 310. The one or more programs 321 include instructions for performing any step in the above method embodiments.

[0049] The processor 310 can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, cells, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication unit can be a communication interface, transceiver, transceiver circuitry, etc., and the storage unit can be a memory.

[0050] The memory 320 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0051] It is understood that the electronic device 300 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, a Wi-Fi module, a speaker, a Bluetooth module, sensors, a display module, etc., without limitation. It is understood that the electronic device 300 may be equipped with... Figure 1 The architecture of the multimodal ultrasound scanning system.

[0052] After understanding the software and hardware architecture of this application, the following will be combined with... Figure 4 This application describes a multimodal ultrasound scanning method as described in its embodiments. Figure 4 This is a flowchart illustrating a multimodal ultrasound scanning method provided in an embodiment of this application. The method is applied to the host device in an ultrasound scanning system, and specifically includes the following steps:

[0053] Step S410: Acquire depth image data of the user's target skin area using the camera module.

[0054] The ultrasound scanning system also includes an ultrasound robot and a terminal device that communicate with the host device. The ultrasound robot includes a robotic arm, a camera module, an ultrasound probe, and a force feedback sensor mounted on the robotic arm. The camera module is located near the ultrasound robot body or the end effector of the robotic arm and is used to perform three-dimensional spatial perception of the area to be diagnosed. The depth image data can include two-dimensional grayscale or color image information, and can also include distance information corresponding to each pixel, thereby forming a three-dimensional point cloud distribution structure of the target skin area. The depth image data is used to provide the spatial geometric contour of the user's body surface and current body posture information. In addition, the robotic arm may also include a flexible end effector, which is installed at the end of the robotic arm and connected to the ultrasound probe. This flexible actuator is used to achieve flexible contact between the ultrasound probe and the human body surface, offsetting positional deviations caused by the curvature of the human body surface and breathing, avoiding pressure on the human body from hard contact, and ensuring good contact between the ultrasound probe and the body surface to ensure image clarity and adapt to the body contours of users of different body types.

[0055] Specifically, the camera module continuously scans the target skin area at a preset frame rate, generating a real-time updated depth image sequence. The depth image data is preprocessed by edge computing nodes, including spatial filtering, outlier removal, and multi-frame fusion processing, to eliminate the impact of ambient light interference and measurement noise on the accuracy of 3D reconstruction. Through the above preprocessing of the depth images, a stable and continuous set of 3D spatial coordinate data of the body surface can be obtained. Based on this coordinate set, the spatial bounding box parameters, curvature distribution features, and body surface normal vector information of the target skin area can be extracted.

[0056] It should be noted that in remote diagnostic or interventional scenarios, depth image data is transmitted via a deterministic network and processed uniformly on the host device to ensure synchronization and consistency in the time dimension. To avoid spatial positioning errors caused by network latency or data jitter, timestamp alignment and state caching management are performed after receiving the depth image data to match the current spatial state.

[0057] Step S420: Based on the user's scanning intention information and the depth image data, control the robotic arm to adjust the ultrasonic probe to the first target pose. The ultrasonic probe and force feedback sensor are integrated at the end of the robotic arm.

[0058] The scanning intent information is determined by the doctor's diagnostic request information, which may take the form of voice commands, text input, or preset examination item selections. This request information characterizes the diagnostic site, standard section type, and examination purpose that the doctor or operator expects to obtain. The scanning intent information can be obtained by analyzing the diagnostic request. The first target pose is the current position and orientation of the robotic arm.

[0059] Specifically, upon receiving a diagnostic request, the system first uses a large AI model to standardize the input information, including speech-to-text conversion, terminology normalization, and medical vocabulary alignment. Then, based on a medical knowledge graph or a pre-defined diagnostic rule base, semantic matching is performed on the text to identify target anatomical structure identifiers, such as a thyroid transverse section. For the identified anatomical structure identifiers, a standard section database is further used to determine corresponding section feature constraint parameters, including probe tilt range, rotation angle range, and spatial translation direction constraints. Simultaneously, based on the diagnostic scenario type (e.g., routine screening or remote intervention), matching contact pressure control range parameters are generated to limit the force feedback range between the ultrasound probe and the body surface. Furthermore, the scanning intent information includes spatial location-related parameters, scanning priority, and image quality target parameters. For example, for screening scenarios requiring high-resolution observation of small lesions, image clarity weight parameters can be increased; for interventional positioning scenarios, real-time performance and section stability parameters are strengthened. Further, to ensure that the scanning intent information matches the current user's body surface spatial state, the structured intent descriptor is correlated and verified with depth image data.

[0060] In one possible embodiment, controlling the robotic arm to adjust the ultrasonic probe to the first target pose based on the user's scanning intent information and the depth image data specifically includes the following steps:

[0061] 421. Determine the three-dimensional spatial coordinates of the target skin region based on the depth image data;

[0062] 422. Construct a world coordinate system based on the preset hand-eye-instrument calibration matrix, the three-dimensional spatial coordinates, and the first pose parameter of the ultrasonic probe;

[0063] 423. Perform structured parsing on the scanning intent information to obtain a structured intent descriptor, the structured intent descriptor including: the second pose parameter of the ultrasonic probe, the standard sectional constraint parameter and the contact force range parameter of the ultrasonic probe;

[0064] 424. Determine the motion trajectory parameters of the robotic arm based on the standard sectional constraint parameters, the first pose parameters, and the second pose parameters;

[0065] 425. Under the constraints of the contact force range parameters and the standard sectional constraint parameters, the ultrasonic probe is adjusted to the first target pose by the robotic arm according to the motion trajectory parameters.

[0066] The first attitude parameter describes the position and orientation of the ultrasound probe in three-dimensional space, including spatial position parameters and attitude angle parameters. The spatial position parameters can be represented in three-dimensional coordinates, for example... , , Coordinate values ​​and attitude parameters can be represented using Euler angles or quaternions to describe the rotational direction of the target object relative to the reference coordinate system. The "hand-eye-instrument" calibration matrix describes the spatial coordinate transformation relationship between the visual sensing device, the robotic arm, and the ultrasound probe. Here, "hand" represents the coordinate system of the robotic arm's end effector, "eye" represents the visual coordinate system corresponding to the camera module, and "instrument" represents the instrument coordinate system where the ultrasound probe is located. By performing calibration operations beforehand, spatial transformation matrices between different coordinate systems can be obtained, thereby achieving spatial alignment between the visual system, the mechanical system, and the medical device, enabling spatial data from different sources to be calculated and fused under a unified coordinate system. The structured intent descriptor transforms the diagnostic request information input by the user or doctor into a set of structured parameters that can be recognized and calculated by the system. The structured intent descriptor can include information such as the target ultrasound scan position coordinates, standard anatomical section constraint parameters, and probe contact pressure range parameters. Standard section feature constraint parameters are used to limit the tilt angle range and rotational degrees of freedom of the ultrasound probe in space; contact pressure range parameters are used to provide a safe threshold range for subsequent force feedback control. Furthermore, after acquiring both the depth image data and the structured diagnostic intent descriptor, the target skin region is spatially located and cropped based on the 3D spatial model constructed from the depth image data, combined with the target anatomical region identifier indicated in the diagnostic intent, to determine the candidate scanning region range. Simultaneously, based on standard section feature constraint parameters, an initial pose estimation set is generated within the candidate scanning region, providing an initial solution space for subsequent robotic arm motion adjustments.

[0067] Specifically, the process begins by acquiring the current first pose parameter of the robotic arm and the current second pose parameter of the ultrasound probe to determine the probe's position and orientation in physical space. Real-time acquisition of position and orientation information establishes an initial spatial reference for the current state. Next, based on depth image data acquired by the camera module, a 3D spatial reconstruction of the target skin region is performed, determining its 3D spatial coordinates. Each pixel in the depth image includes corresponding distance information. By combining this with the camera module's intrinsic parameters, the 2D pixel coordinates are converted into 3D point cloud data, thus obtaining the actual position of the target skin region in space. Then, based on a pre-set hand-eye-device calibration matrix, the 3D spatial coordinates, the robotic arm's current pose, and the probe's pose are mapped to the same physical space coordinate system, constructing a world coordinate system. Establishing a unified coordinate system eliminates differences between different device coordinate systems, enabling unified calculation of visual data, robot motion data, and medical device data. Simultaneously, the host device performs semantic parsing of the scanning intent information, converting it into a structured intent descriptor. For example, when a user requests an ultrasound examination of a specific organ region, the location of the target scanning area, the required standard anatomical cross-sectional direction, and the reasonable contact pressure range between the probe and the body surface can be extracted from the request. This forms calculable structured parameters, and a robotic arm pose optimization model is constructed based on the standard cross-sectional constraint parameters, contact force range parameters, and the current pose parameters of the robotic arm. In this model, the scanning position error, cross-sectional direction error, and contact pressure constraint are used as optimization objectives or constraints. The optimal pose of the robotic arm that satisfies these conditions is determined through multi-objective optimization. Finally, the pose optimization model is solved based on the target pose deviation parameters, generating motion trajectory parameters for the robotic arm's end effector. These trajectory parameters are typically represented in time series form, describing the motion paths of each joint of the robotic arm during movement and the spatial trajectory of the end effector. The first target pose for driving the ultrasound probe is then determined based on these motion trajectory parameters.

[0068] In one possible embodiment, adjusting the ultrasonic probe to the first target pose via the robotic arm according to the motion trajectory parameters, under the constraints of the contact force range parameters and the standard sectional constraint parameters, specifically includes the following steps:

[0069] 4251. Determine the pose control command of the robotic arm within a preset time period based on the motion trajectory parameters to obtain a control command sequence;

[0070] 4252. Based on a preset robotic arm operating intelligent agent, the control command sequence is deconstructed to obtain multiple control parameters of the robotic arm, wherein the control parameters include at least speed parameters, acceleration parameters, and rotation encoding parameters.

[0071] 4253. Determine multiple pressure parameters based on the contact force range parameters;

[0072] 4254. Generate a first motion control command to drive the robotic arm based on the speed parameters, the acceleration parameters, and the rotation encoding parameters;

[0073] 4255. Generate force control commands for the robotic arm based on the plurality of pressure parameters;

[0074] 4256. Control the robotic arm to perform a first operation according to the first motion control command, and control the robotic arm to perform a second operation according to the force control command, so as to adjust the ultrasonic probe to the first target pose.

[0075] The motion trajectory parameters describe the robotic arm's motion path in three-dimensional space and its pose state over time. This includes a set of spatial position parameters and attitude angle parameters corresponding to discrete time points, representing the trajectory nodes the robotic arm must traverse during the scanning task. The control command sequence includes the target spatial position of the robotic arm at a specific time point and its corresponding orientation. The robotic arm operating agent is a control model used to parse and execute robot control commands. Based on a preset robotic arm kinematics model and control strategy, it transforms high-level trajectory planning results into a set of parameters suitable for low-level drive control. By parsing the control command sequence, the abstract pose control commands can be decomposed into executable control parameters for each joint or execution unit of the robotic arm. Velocity parameters describe the changes in motion velocity of each joint during movement; acceleration parameters describe the acceleration or deceleration process of joint velocity changes; and rotational encoding parameters characterize the angular changes or encoder feedback values ​​of each joint, ensuring that the robotic arm can complete spatial motion according to a predetermined trajectory. In addition, based on the contact force range parameters contained in the structured intent descriptor, multiple pressure parameters can be determined to constrain the contact pressure range of the ultrasound probe when it comes into contact with the human body surface, so as to avoid excessive pressure causing patient discomfort or insufficient pressure leading to a decrease in imaging quality.

[0076] Specifically, firstly, based on the motion trajectory parameters, the continuous motion trajectory of the robotic arm in space is discretized according to a preset time period, and a corresponding pose control command is determined at each discrete time node, thus forming a control command sequence. Subsequently, a preset robotic arm operation agent performs parameter decomposition processing on the control command sequence. This agent, combining the kinematic model of the robotic arm and the control interface specification, further decomposes each pose control command into multiple low-level control parameters. For example, the spatial pose changes of the robotic arm are converted into velocity, acceleration, and rotational encoding parameters corresponding to each joint. Simultaneously, based on the contact force range parameters contained in the structured intent descriptor, the contact pressure that the robotic arm may generate during scanning is analyzed, and multiple pressure parameters are determined as reference values ​​for subsequent force control. Next, based on the velocity, acceleration, and rotational encoding parameters, a first motion control command is generated to drive the robotic arm. This control command is used to control the movement of the drive units of each joint of the robotic arm, enabling the robotic arm to adjust its spatial position and posture according to the planned trajectory. Simultaneously, force control commands are generated based on multiple pressure parameters and applied to the end effector of the ultrasound probe. These commands control the force feedback control module of the robotic arm, ensuring the ultrasound probe remains within a preset pressure range when in contact with the human body surface. Thus, during ultrasound examination, slight movements of the patient or movements caused by breathing can cause the probe to automatically adjust the contact pressure according to changes in the curvature of the body surface during scanning, thereby improving the stability of ultrasound image acquisition. Finally, the first motion control command and the force control command are adjusted to move the ultrasound probe to the first target pose.

[0077] In one possible embodiment, generating a first motion control command to drive the robotic arm based on the velocity parameter, the acceleration parameter, and the rotation encoding parameter specifically includes the following steps:

[0078] A1. Obtain the joint angle parameters and joint angular velocity parameters of multiple joints of the robotic arm;

[0079] A2. Based on the motion trajectory parameters, determine the pose of the end effector of the robotic arm in the world coordinate system to obtain the second target pose;

[0080] A3. The robotic arm operates the intelligent agent to generate joint motion parameters of the multiple joints based on the second target pose;

[0081] A4. Generate joint motion trajectories corresponding to the plurality of joints of the robotic arm based on the acceleration parameters and the joint angle parameters;

[0082] A5. Determine the driving control parameters corresponding to the multiple joints based on the rotational encoding parameters, the joint angular velocity parameters, and the joint motion trajectory to obtain multiple driving control parameters;

[0083] A6. Determine the first motion control command to drive the robotic arm based on the plurality of drive control parameters.

[0084] Among them, the joint angle parameters and joint angular velocity parameters are state parameters for the motion control of the robotic arm, used to characterize the real-time motion state of each joint of the 6DoF robotic arm. The joint angle parameters can be expressed as follows: , Let be the actual rotation angle of the i-th joint, and let the joint angular velocity parameter be... , represents the real-time rotational angular velocity of the i-th joint. The world coordinate system is a unified global coordinate system constructed based on the hand-eye-instrument calibration matrix, serving as the spatial reference for calculating the pose of the robotic arm's end effector. The second target pose characterizes the desired spatial position and orientation of the robotic arm's end effector within this coordinate system. The robotic arm operating agent is a control model integrating a reinforcement learning policy network, with a built-in 6DoF robotic arm inverse kinematics operator, capable of converting the high-level pose requirements of the robotic arm's end effector into executable motion parameters for each joint. The joint motion trajectory is the motion path of each joint over time, generated through collaborative planning of acceleration parameters and current joint state parameters. The rotation encoding parameters are the actual rotation amounts collected by the encoders of each joint of the robotic arm, serving as feedback parameters to compensate for joint motion errors. The drive control parameters are the low-level control quantities adapted to the robotic arm joint drive unit, directly serving as the basis for generating motion control commands.

[0085] Specifically, the joint angle and angular velocity parameters of the six joints are collected in real time by photoelectric encoders built into each joint of the robotic arm. After preprocessing by the edge computing unit, the collected data is transmitted to the control module with low latency via a StarFlash short-range communication module, ensuring the real-time performance and accuracy of the parameters. Subsequently, based on the preset motion trajectory parameters and combined with the homogeneous transformation rules of the world coordinate system, the target pose parameters of the robotic arm's end effector are calculated using a spatial linear interpolation algorithm. ,in, The rotation matrix represents the end-effector attitude. The position vector represents the spatial position of the end effector, achieving precise alignment between the motion trajectory and the physical space of ultrasound diagnosis and treatment. O is a zero matrix. Then, the robotic arm is invoked to operate the intelligent agent, in order to... Taking the target pose parameters as input, and combining the mechanical constraints of ultrasound diagnosis, the operator is calculated through inverse kinematics. Complete the calculation and generate the joint motion parameters for each joint. , For the joint target angle parameters, the system realizes the parameter conversion from the robot arm's end-effector pose to joint motion. The calculation process is based on the CANN architecture to schedule NPU computing power, improving calculation efficiency. Simultaneously, acceleration parameters are used... Based on the current joint angle parameter θ, a trapezoidal velocity planning algorithm is used to generate continuous joint motion trajectories θ(t) for each joint, thereby constraining the joint motion process and avoiding motion shock and jitter. Furthermore, rotational encoding parameters ξ and joint angular velocity parameters are used... Multi-parameter fusion with the joint motion trajectory θ(t) is performed, and the driving control parameters are solved through a proportional-derivative-rotational coding closed-loop feedback algorithm to achieve precise compensation for motion errors. Finally, the multiple driving control parameters are standardized according to the robotic arm joint driving protocol, and combined with the kernel-level task identification rules of the HarmonyOS FFRT task concurrency scheduling framework, a highest priority scheduling identifier is added to the parameters, ultimately generating the first motion control command that can directly drive the robotic arm movement.

[0086] As can be seen, this embodiment achieves accurate conversion from motion parameters such as speed and acceleration to the underlying drive control commands of the robotic arm through a full-link process including basic state parameter acquisition, spatial target pose calculation, joint motion parameter inference, motion trajectory planning, drive parameter correction, and control command encapsulation. Furthermore, the scenario-based constraints and low-latency control requirements of ultrasound diagnosis are incorporated into the calculation and planning process. Based on the robotic arm operating agent and closed-loop feedback algorithm, the accuracy and rationality of motion control command generation are effectively improved.

[0087] Step S430: Calculate the ultrasonic quality assessment parameters based on the first ultrasonic image data of the ultrasonic probe and the first contact pressure data of the force feedback sensor to obtain the first ultrasonic quality assessment parameters.

[0088] The first ultrasound quality assessment parameter reflects the clarity of the image itself, the degree of structural recognizability, and whether the current contact state is within a safe and stable range.

[0089] Specifically, firstly, the first ultrasound image data undergoes image preprocessing, including grayscale normalization, noise suppression, and edge enhancement, to reduce the interference of imaging noise on the quality evaluation results. Subsequently, multiple image feature parameters are extracted from the processed image frames, such as image contrast, signal-to-noise ratio (SNR), edge gradient intensity, target skin texture clarity, and visibility score of key anatomical structures. The SNR is obtained by calculating the ratio of pixel intensity in the target skin region to background noise intensity; the edge gradient intensity is calculated using the Sobel operator; and the visibility of key structures is obtained by similarity matching with a standard cross-sectional template to obtain a matching score. Simultaneously, time-series analysis is performed on the first contact pressure data. The mean pressure, pressure fluctuation amplitude, and pressure stability index are calculated. Pressure stability is represented by the standard deviation or coefficient of variation of pressure change per unit time, reflecting whether the contact process is smooth. Excessive pressure fluctuation may lead to image jitter or local tissue deformation, thus affecting image quality. After obtaining the image feature parameter vector and pressure feature parameter vector, a comprehensive score is performed using a pre-defined weighted fusion model. The weighted fusion model can be constructed based on a linear weighting method or a pre-trained quality assessment network model. For different diagnostic scenarios, the weights of image features and pressure features can be dynamically adjusted. For example, in a detailed lesion screening scenario, the weights of image clarity and structural similarity are increased; in a remote automated operation scenario, the weights of pressure stability and safety are increased. This allows for the calculation of the first ultrasound quality assessment parameter in the form of a single scalar or multi-dimensional vector. Furthermore, to enhance the robustness of the evaluation results, a sliding window averaging process can be applied to several consecutive frames of image data to avoid the bias of occasional noise in a single frame on the overall evaluation results. Simultaneously, low-pass filtering is applied to the pressure data to remove high-frequency disturbance signals. The first ultrasound quality assessment parameter output after filtering and fusion processing can more objectively reflect the comprehensive imaging effect under the current robotic arm posture and contact state.

[0090] It is evident that by introducing a joint evaluation mechanism of image and force data, the accuracy and reliability of imaging quality control during automated ultrasound scanning are effectively improved, thereby enhancing the system's adaptive adjustment capability and diagnostic accuracy in remote scenarios.

[0091] In one possible embodiment, the step of calculating the ultrasonic quality assessment parameters based on the first ultrasonic image data of the ultrasonic probe and the first contact pressure data of the force feedback sensor to obtain the first ultrasonic quality assessment parameters specifically includes the following steps:

[0092] 431. The first ultrasound image data is calculated based on a preset edge detection algorithm to obtain image edge contrast parameters; the image edge contrast parameters include: grayscale contrast parameters and signal-to-noise ratio parameters;

[0093] 432. Input the first ultrasound image data into a preset cross-sectional structure recognition model to obtain the target structure recognition result;

[0094] 433. Determine the confidence score of the cross-sectional structure corresponding to the target structure recognition result to obtain the structure confidence score;

[0095] 434. Calculate the contact pressure stability parameter of the ultrasonic probe based on the first contact pressure data;

[0096] 435. When the signal-to-noise ratio parameter is greater than or equal to a preset signal-to-noise ratio threshold, determine the image quality score corresponding to the first ultrasound image data according to the grayscale contrast parameter.

[0097] 436. The first ultrasound quality assessment parameter is obtained by calculating based on the image quality score, the structure confidence score, and the contact pressure stability parameter.

[0098] The first ultrasound image data characterizes the real-time ultrasound imaging results acquired by the ultrasound probe in its current pose and contact state. It includes two-dimensional grayscale images or continuous image frames reflecting the acoustic structure within the tissue. The image edge contrast parameter characterizes the degree of grayscale variation between different tissue interfaces in the ultrasound image, reflecting the clarity of tissue boundaries. The grayscale contrast parameter describes the magnitude of grayscale value changes within a local area of ​​the image, reflecting the clarity of tissue structure boundaries. The signal-to-noise ratio parameter describes the ratio between the effective signal and noise components in the image, measuring the identifiability of effective structural information in the image. The section structure recognition model is an image analysis model built based on deep learning or pattern recognition methods, used to identify target anatomical structures or standard diagnostic sections in ultrasound images. For example, in abdominal or cardiac ultrasound diagnosis, it is usually necessary to determine whether the current image contains the target organ or meets the structural characteristics of a standard scanning section. By performing structural recognition analysis on the first ultrasound image data, the target structure recognition result can be obtained, and the confidence score of the corresponding structure can be further determined through the probability value or score value output by the model. The structural confidence score characterizes the degree of matching of the target anatomical structure in the current image, thus reflecting whether the scanning section is close to the expected standard section. The contact pressure stability parameter describes the stability of the contact pressure between the ultrasound probe and the human body surface during scanning. Since ultrasound imaging quality largely depends on the contact state between the probe and the body surface, excessive or fluctuating contact pressure may lead to tissue deformation or increased imaging noise; while insufficient pressure may result in insufficient acoustic coupling, thereby reducing image clarity.

[0099] Specifically, the first step involves edge detection processing of the first ultrasound image data. Edge detection algorithms can employ gradient-based image processing methods, such as identifying tissue boundary regions by calculating image grayscale gradient changes, extracting the grayscale change amplitude of each pixel region in the image, and calculating the grayscale contrast parameter accordingly. Simultaneously, by statistically analyzing the image signal intensity and background noise, the signal-to-noise ratio parameter of the image can be obtained, thus yielding the image edge contrast parameter. Next, the first ultrasound image data is input into a preset cross-sectional structure recognition model to automatically identify anatomical structures in the image. After recognition, the model outputs the recognition result and matching probability of the corresponding structure, and determines the confidence score of the target structure based on this probability value. Then, based on the first contact pressure data collected by the force feedback sensor, the pressure changes of the ultrasound probe during the scanning process are analyzed. For example, the fluctuation amplitude or variance of the pressure data can be calculated within a preset time window, and the contact pressure stability parameter can be obtained accordingly. Next, a threshold judgment is performed on the signal-to-noise ratio (SNR) parameter. When the SNR parameter is greater than or equal to a preset SNR threshold, it indicates that the effective structural information in the current image has high reliability. At this point, an image quality score is further calculated based on the grayscale contrast parameter. This avoids directly using the grayscale contrast parameter for scoring when image noise is high, thereby improving the reliability of the evaluation results. Finally, the image quality score, structural confidence score, and contact pressure stability parameter are comprehensively calculated, for example, through weighted calculation or a preset comprehensive evaluation function, to obtain a first ultrasound quality evaluation parameter. This evaluation parameter is used to comprehensively characterize the overall level of image quality under the current ultrasound scanning conditions and serves as an important feedback basis for subsequent optimization of the robotic arm motion strategy.

[0100] It is evident that by conducting multi-dimensional comprehensive analysis of the edge contrast features, structural recognition results, and contact pressure stability of ultrasound images, a comprehensive evaluation mechanism for ultrasound scan quality can be constructed. This mechanism not only reflects the clarity of the image itself but also comprehensively evaluates the scan quality by combining the anatomical structure matching degree and probe contact state. This provides reliable feedback for subsequent robotic arm posture optimization and scan path adjustment, which helps to achieve image quality-driven adaptive optimization of ultrasound scans, improve the standardization of scan sections, and enhance overall diagnostic accuracy.

[0101] Step S440: Adjust the ultrasonic probe to the second target pose using the robotic arm according to the first ultrasonic quality assessment parameters.

[0102] The first ultrasound quality assessment parameter is used to quantify the overall imaging effect and contact stability under the current robotic arm posture. When the assessment parameter does not reach the preset quality threshold, the control variable is fine-tuned to obtain the optimal posture that meets the diagnostic requirements.

[0103] Specifically, the first ultrasound quality assessment parameter is first mapped to a reward value or a cost function value. The reward value increases when image clarity improves and pressure stability remains within a safe range; it decreases when image contrast decreases or pressure fluctuations exceed a threshold. The reward value is then input into a preset reinforcement learning model or policy optimization network. The model uses the current pose parameters of the robotic arm, contact pressure characteristics, and image feature vectors as state inputs, and pose increment adjustment and force control correction as action outputs. A new control increment is calculated through the policy network. In each iteration, based on the pose correction output by the optimization model, the robotic arm's end effector is adjusted slightly, including advancing or retreating along the surface normal, rotating around a specific axis, and tangential translation compensation. After adjustment, new ultrasound image data and contact pressure data are acquired again, and the updated ultrasound quality assessment parameters are recalculated. Through this "acquisition-evaluation-optimization-adjustment" mechanism, the optimal solution of the quality assessment function is gradually approximated. To ensure the stability and safety of the optimization process, step size limits and safety constraints are set during strategy updates to prevent abnormal contact pressure or sudden attitude changes due to excessive adjustments. Simultaneously, a convergence criterion is introduced: the optimization process is considered complete when the increment of the quality assessment parameter is less than a preset threshold or the assessment parameter reaches a preset target value within a certain number of consecutive iterations, and the current robotic arm pose is output as the final result. After optimization, the target position and target attitude of the robotic arm are determined. The target position corresponds to the spatial coordinates when the ultrasound image quality reaches its optimal level or meets the diagnostic threshold; the target attitude corresponds to the combination of attitude angles when the imaging structure has the highest clarity and the contact pressure is within a stable range.

[0104] It is evident that by implementing precise attitude control and force feedback adjustment after the robotic arm reaches the target position, and by achieving synchronous acquisition of image and pressure data, not only is the stability and repeatability of ultrasound images guaranteed, but reliable data support is also provided for subsequent reinforcement learning optimization based on image clarity and pressure constraints, thereby improving the imaging accuracy and operational safety in the automated ultrasound diagnostic process.

[0105] In one possible embodiment, adjusting the ultrasound probe to the second target pose via the robotic arm according to the first ultrasound quality assessment parameters specifically includes the following steps:

[0106] 441. Determine the pose adjustment parameters of the robotic arm based on the first target pose;

[0107] 442. Construct a comprehensive reward function for the reinforcement learning algorithm based on the first ultrasound quality assessment parameters;

[0108] 443. An enhanced state space constructed based on the first target pose and the first ultrasound image data as state variables; and an enhanced action space constructed based on the pose adjustment parameters;

[0109] 444. Construct a reinforcement learning policy model based on the reinforcement action space and the comprehensive reward function, and input the reinforcement state space into the reinforcement learning policy model for solving to obtain the pose control action parameters;

[0110] 445. The posture control action parameters are sent to the robotic arm. After the robotic arm is controlled to adjust its posture according to the posture control action parameters, the third ultrasonic image data and the third contact pressure data are collected through the camera module and the force feedback sensor.

[0111] 446. Determine the second ultrasound quality assessment value based on the third ultrasound image data and the third contact pressure data, and input the second ultrasound quality assessment value into the comprehensive reward function to obtain the reward value corresponding to the posture control action parameters;

[0112] 447. The reinforcement learning policy model is iteratively optimized and solved according to the reward value. When the output value of the comprehensive reward function satisfies the preset convergence condition, the second target pose is obtained.

[0113] The pose adjustment parameters describe the adjustable motion dimensions of the robotic arm's end effector in three-dimensional space, and can include position increment parameters in the translational direction and rotation angle parameters in the orientation direction. For example, in a six-DOF robotic arm system, the pose adjustment parameters can be expressed as follows: The equal spatial adjustment is used to control the minute displacements and posture changes of the robotic arm's end effector in space. Reinforcement learning algorithms are machine learning methods that continuously interact with the environment and optimize decision-making strategies based on reward feedback. They involve constructing a state space, action space, and reward function, and obtaining the optimal strategy that maximizes cumulative rewards through policy optimization algorithms. The comprehensive reward function is used to evaluate the ultrasonic imaging quality obtained from the current robotic arm posture. It can be constructed based on the first ultrasonic quality assessment parameters and is used to quantify the degree of image quality improvement and probe contact stability.

[0114] Specifically, firstly, the relevant pose adjustment parameters that the robotic arm needs to adjust are determined based on the first target pose. These parameters represent the set of fine-tuning actions that the robotic arm's end effector can perform in the current scanning state, used to limit the range of actions that the reinforcement learning algorithm can explore. Next, a reward function for the reinforcement learning algorithm is constructed based on the first ultrasound quality assessment parameters. This reward function measures the degree of improvement in imaging quality obtained after the robotic arm performs a certain pose action. In one implementation, the reward function can be expressed as:

[0115]

[0116] in, These represent the weight parameters for image quality score, anatomical structure matching degree, and contact pressure stability, respectively. Indicates image quality score; Indicates the degree of structural matching revealed; Indicates the stability of contact pressure; This represents the reward value obtained at time t.

[0117] Next, the state space and action space of reinforcement learning are constructed. The state space describes the environmental state of the system at the current moment, and the state variables can be represented as follows: .in, This represents the spatial position parameter of the robotic arm's end effector at time t. Indicates the end effector posture parameters of the robotic arm. This represents the feature parameters of the ultrasound image acquired at time t. This represents the state variable at time t.

[0118] The motion space is composed of pose adjustment parameters, which can be represented as: This motion vector represents the fine-tuning motion of the robotic arm's end effector based on the current posture. Then, a reinforcement learning policy model is constructed based on the reinforced motion space and the comprehensive reward function. Preferably, a policy model based on a deep neural network can be used, such as the Deep Deterministic Policy Gradient (DDPG) algorithm or the Proximal Policy Optimization (PPO) algorithm. This policy model can be represented as a policy function: ,in, This represents the parameterized policy function. By inputting the enhanced state space into the policy model, the pose control action parameters in the current state can be obtained. Next, the pose control action parameters are sent to the control system in the robotic arm of the ultrasonic robot, driving the robotic arm to perform the corresponding posture adjustment. After the robotic arm completes the action, the ultrasonic probe again acquires new ultrasonic image data and contact pressure data, obtaining third ultrasonic image data and third contact pressure data, respectively. Using a similar image quality assessment method, the third ultrasonic image data and third contact pressure data are analyzed to obtain a second ultrasonic quality assessment value. Subsequently, this assessment value is input into the comprehensive reward function to calculate the reward value corresponding to the current action. The parameters of the reinforcement learning policy model are updated based on the obtained reward value. In one implementation, the policy network parameters can be updated based on the policy gradient method, and the update formula can be expressed as: .in, For learning rate, This represents the gradient of the policy network parameters. By continuously executing the "action-feedback-update" training process, the policy model can gradually learn the optimal action policy that maximizes the cumulative reward. When the output value of the comprehensive reward function satisfies the preset convergence condition in several consecutive iterations, it indicates that the policy model has found a stable optimal pose adjustment policy, and the corresponding spatial position and pose of the robotic arm are determined as the target position and target pose.

[0119] It is evident that by constructing a reinforcement learning optimization mechanism based on ultrasound image quality and contact pressure feedback, the robotic arm can continuously explore and optimize the probe posture during scanning, allowing ultrasound image quality to directly drive the dynamic adjustment of the scanning strategy. Compared to traditional methods that rely on manual experience to adjust the probe posture, this method can automatically learn the optimal scanning strategy under the constraints of multi-dimensional quality evaluation indicators, thereby achieving adaptive optimization of the scanning section, reducing errors caused by patient movement, improving the clarity of ultrasound images and the matching degree of anatomical structures, and further enhancing the automation level and diagnostic accuracy of the remote ultrasound scanning system.

[0120] Step S450: Scan the user based on the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area.

[0121] After determining the pose of the second target, it can be loaded as the final execution pose parameter into the robotic arm control system. A position-attitude-force coordinated control mechanism then allows the robotic arm to stably transition to the optimal scanning state. In this state, high-quality ultrasonic image data and contact pressure data are simultaneously acquired via an ultrasonic probe and a force feedback sensor to form the final ultrasonic scan result.

[0122] Specifically, during the execution phase, trajectory smoothing planning is first performed on the second target pose to generate a transition path from the current pose to the second target pose. Next, the robotic arm calculates the corresponding joint control parameters based on an inverse kinematics model and drives the joint motors to coordinate their movements, causing the end effector to gradually approach the second target pose. Upon approaching the contact area on the body surface, a force closed-loop adjustment mode is activated. Contact pressure data is collected in real time by a pressure feedback sensor in the robotic arm, and the propulsion speed and attitude angle are fine-tuned to ensure the contact pressure remains within the target pressure range. Once the robotic arm reaches the target position and meets the target attitude constraints, it enters the stabilization phase. During this phase, an attitude locking control algorithm maintains the spatial angle stability of the end effector, while continuously monitoring the force feedback sensor output to counteract disturbances caused by the user's breathing or slight body movements. A dynamic micro-compensation mechanism ensures that the probe's acoustic beam direction is always aligned with the target standard tangent, thereby maintaining the consistency of the imaging structure. In a stable contact state, the ultrasound probe begins acquiring second ultrasound image data. Compared to the first ultrasound image data, the second ultrasound image data was acquired after iterative optimization, and its image clarity, structural contrast, and target anatomical region integrity all met preset quality thresholds. Finally, the acquired image frames were cached in real time, and the second contact pressure data within the corresponding time period was recorded synchronously. A unified timestamp mechanism ensured precise alignment between the image data and pressure data, guaranteeing that subsequent diagnostic analysis could be based on synchronized data. Furthermore, the stability of the second contact pressure data could be verified. When pressure fluctuations were detected to be less than a preset tolerance range, the current contact state was deemed stable and reliable; if abnormal pressure fluctuations occurred, a fine-tuning compensation or re-optimization mechanism was triggered, and the target skin area was scanned with ultrasound.

[0123] As can be seen, by controlling the robotic arm to precisely reach the target position and posture after optimization, and simultaneously acquiring second ultrasound image data and second contact pressure data in a stable contact state, the transition from initial exploratory scanning to optimal stable scanning is achieved. This process not only ensures that the final imaging quality meets diagnostic standards, but also improves operational safety and data consistency through force feedback constraints, thereby providing a high-quality data foundation for subsequent intelligent diagnostic analysis or remote medical interpretation, significantly improving the accuracy and reliability of the automated ultrasound scanning system.

[0124] In one possible embodiment, after scanning the user based on the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area, the method further includes the following steps:

[0125] 451. Extract the three-dimensional point cloud data of the target skin region from the depth image data;

[0126] 452. Based on the three-dimensional point cloud data, perform non-rigid deformation matching on the preset human digital model to obtain the user's first digital twin model;

[0127] 453. Extract the ultrasound image boundary data corresponding to the target tissue section from the second ultrasound image data to obtain ultrasound boundary data; the target tissue section is any part of the target skin region;

[0128] 454. Determine the contact pressure data of the corresponding area of ​​the target tissue cross-section from the second contact pressure data to obtain the target contact pressure data;

[0129] 455. Based on the ultrasonic image boundary data and the target contact pressure data, the first digital twin model is dynamically corrected using voxels to obtain the target digital twin model;

[0130] 456. Spatial matching is performed between the second ultrasound image data and the target digital twin model to determine the section position parameters corresponding to the second ultrasound image data;

[0131] 457. Determine the tissue feature parameters and lesion feature parameters corresponding to the target tissue section based on the section position parameters;

[0132] 458. Input the tissue feature parameters and the lesion feature parameters into the preset diagnostic model to obtain the user's diagnostic results.

[0133] Among them, the three-dimensional point cloud data of the body surface is a discrete three-dimensional coordinate point set obtained by 3D reconstruction and filtering and denoising of depth image data. ,in, Let K represent the i-th 3D coordinate point, and K represent the total number of valid points in the point cloud dataset. Let P represent the Euclidean space of three-dimensional real numbers, and let P represent the three-dimensional point cloud dataset of the body surface. Discrete three-dimensional coordinate point set. The surface spatial contour and geometric features of the target skin region for user diagnosis are characterized; non-rigid deformation matching is the process of deformation registration of a general human digital model using thin-plate spline algorithm with the three-dimensional point cloud data of the body surface as constraints; ultrasound image boundary data is the set of target tissue anatomical contour points extracted from ultrasound images. ,in, This represents the set of boundary data points of the ultrasound image at time t. Let J represent the j-th three-dimensional coordinate point, and M represent the total number of valid contour points in the boundary point set of the ultrasound image. Provides spatial hard constraints for voxel correction of digital twin models; target contact pressure data are probe-surface contact pressure values ​​matched to the acquisition time of the target tissue section. It is used to correct the elastic physical parameters of human soft tissue; voxel dynamic correction is an operation that drives the dynamic deformation of the digital twin model voxel mesh through a linear elastic finite element model, with ultrasound boundary data and contact pressure data as dual constraints; the section position parameter is the spatial pose parameter of the ultrasound image in the unified world physical space coordinate system. The system is designed to characterize the spatial position and orientation of the ultrasound section; the tissue and lesion feature parameters are quantitative diagnostic indicators extracted from the ultrasound section, including soft tissue elastic modulus, tissue texture, etc., covering lesion size, boundary morphology, echo characteristics, etc.; the diagnostic big model is a multimodal AI model pre-trained based on medical ultrasound knowledge base and clinical diagnostic data, integrating reinforcement learning strategy network and clinical diagnostic rules, which can realize intelligent reasoning from feature parameters to diagnostic results.

[0134] Specifically, firstly, 3D point cloud reconstruction and statistical filtering denoising are performed on the depth image data. Outliers and redundant data are removed to extract effective 3D point cloud data of the target skin region to ensure the accuracy of the point cloud data in representing the body surface contour. Next, using the preprocessed point cloud data as spatial constraints, the TPS non-rigid deformation algorithm is used to register a preset general human digital model. The optimal deformation mapping is solved by minimizing the energy function to obtain a first digital twin model that fits the user's individual anatomical features and body position. Furthermore, based on the CANN architecture and NPU computing power, the Sobel edge detection operator is used to extract edge features from the second ultrasound image data. After threshold segmentation and contour extraction, the ultrasound image boundary data of the target tissue section is obtained. Then, based on the acquisition timestamp of the second ultrasound image data, pressure data from the same time period are selected from the second contact pressure data to obtain target contact pressure data matching the target tissue cross-section. This provides a soft tissue elasticity property reference for subsequent voxel correction. The ultrasound image boundary data is applied as a displacement hard constraint to the voxel mesh nodes of the first digital twin model. Simultaneously, the soft tissue elastic modulus E is corrected in real time based on the target contact pressure data, and the optimal displacement vector U of the voxel nodes is solved using a linear elastic finite element model. The system performs voxel dynamic deformation and parameter updates to obtain the target digital twin model Mt. Finally, based on a unified world physical space coordinate system constructed using hand-eye-instrument calibration, the system achieves precise spatial matching between the second ultrasound image data and the target digital twin model, determining the spatial pose parameters corresponding to the ultrasound sections. After obtaining the spatial pose parameters, the system locates the corresponding anatomical region in the target digital twin model according to the section position parameters, extracts tissue feature parameters such as tissue elasticity and texture, and lesion feature parameters such as morphology, boundary, and echogenicity, and completes the standardized quantification of these parameters. The quantified tissue feature parameters and lesion feature parameters are then input into a large-scale diagnostic model pre-trained with clinical ultrasound diagnostic data. After dual verification using deep inference and clinical diagnostic rules, the system outputs the user's final diagnostic result.

[0135] As can be seen, this embodiment ensures the accuracy of diagnostic feature extraction and the accuracy and reliability of diagnostic inference through precise spatial registration, model correction and parameter quantification, realizing the practical application of digital twin technology in ultrasound diagnosis and treatment.

[0136] For easier understanding, please refer to Figure 5 , Figure 5 This is a schematic diagram of a scenario where an ultrasound robot performs ultrasound diagnosis, as provided in an embodiment of this application. The scenario uses an ultrasound scanning system as the physical execution carrier, and through the collaborative interaction between a multi-degree-of-freedom robotic arm, a camera module, and the patient, an automated and precise ultrasound scanning execution environment is constructed. Specifically, the robotic arm adopts a multi-joint serial structure with 6-dimensional posture adjustment and spatial positioning capabilities. It can precisely deliver the end-effector integrated ultrasound probe to the patient's target diagnostic area based on clinical scanning needs, adapting to ultrasound detection requirements in different body positions and anatomical locations. A camera module is deployed on the side of the robotic arm base to collect visual images and depth data of the diagnostic scene. Through visual recognition and feature extraction technology, it achieves real-time positioning and tracking of the patient's body contour, probe posture, and scanning area, providing visual perception input for the robotic arm's path planning and dynamic posture correction. The patient, as the diagnostic subject, lies on the examination table in a position suitable for clinical scanning, with the target skin area exposed within the robotic arm's scanning range, providing a physical interface for the ultrasound probe. Simultaneously, a force feedback sensor integrated at the interface between the ultrasound probe and the robotic arm's end effector collects real-time contact pressure data between the probe and the body surface, assisting the robotic arm in completing force control closed-loop adjustment. This avoids patient discomfort due to overload or image quality degradation due to insufficient pressure.

[0137] It is evident that a safe and accurate scanning architecture has been constructed through the synergistic integration of multimodal perception and automated execution. On the one hand, the multi-degree-of-freedom robotic arm provides flexible motion execution capabilities for full-coverage scanning of complex body surface areas. On the other hand, the visual perception of the camera module and the physical perception of the force feedback sensor form a dual-modal feedback mechanism, ensuring the stability and accuracy of the scanning operation from both environmental visual and physical interaction perspectives.

[0138] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram of a remote ultrasound diagnosis scenario provided in an embodiment of this application. As can be seen, through multimodal perception, highly reliable communication and remote interaction technologies, doctors can perform non-contact automated ultrasound scanning and diagnostic analysis on patients, providing a complete technical platform for the downward flow of cross-regional medical resources and ultrasound diagnosis and treatment at the grassroots level.

[0139] Specifically, on the patient side, the patient lies in a standardized scanning position on the examination table over the target diagnostic area, providing a physical interface for the ultrasound robot to perform a surface-contact scanning with the ultrasound probe. Simultaneously, force feedback sensors integrated into the probe and the end effector of the robotic arm collect contact pressure data in real time during the scanning process, providing sensory input for the robotic arm's force control closed-loop adjustment. Combined with multiple safety protection mechanisms (such as motion space constraints, speed and acceleration limits), this avoids scanning pressure overload that could cause patient discomfort or tissue damage, while ensuring stable scanning pressure to maintain ultrasound imaging quality. Furthermore, an adaptive flexible contact algorithm can adapt to the patient's surface morphology, achieving a flexible and conformal scanning interaction. On the ultrasound robot side, the ultrasound robot uses a multi-degree-of-freedom collaborative robotic arm as the actuator, integrating multi-modal sensing units such as an ultrasound probe, force feedback sensor, and camera module. The robotic arm employs a multi-joint serial structure, possessing 6D spatial positioning and posture adjustment capabilities. It can precisely deliver the ultrasound probe to the target scanning area of ​​the patient according to clinical scanning needs, adapting to ultrasound examination scenarios of different body parts and positions. Simultaneously, the ultrasound robot establishes a stable connection with the doctor via high-speed communication technologies such as 5.5G. Based on a high-fault-tolerant network transmission mechanism and flexible video coding technology, it achieves low-latency, high-fidelity transmission of ultrasound images, visual perception data, and control commands, ensuring the real-time performance and reliability of remote operation. On the doctor's side, located at a remote control console, the doctor remotely controls the ultrasound robot through a contoured interactive hand and a visual interface. The doctor obtains real-time ultrasound image data and visual information of the scanning scene through the display screen. Combined with remote parameter adjustment functions, the doctor can adjust ultrasound scanning parameters (such as frequency, gain, and depth) in real time to optimize image quality. Furthermore, the doctor can complete lesion identification and diagnostic analysis based on the acquired multimodal data, realizing a closed-loop process for remote ultrasound diagnosis and treatment. It is evident that this remote ultrasound diagnostic scenario, through a distributed architecture design of "patient-ultrasound robot-doctor," integrates multimodal perception, highly reliable communication, and remote interaction technologies. This not only ensures the accuracy and safety of ultrasound scans but also breaks through geographical limitations, enabling the cross-regional dissemination of high-quality medical resources and providing efficient and safe ultrasound diagnostics.

[0140] As can be seen, by implementing the above-described multimodal ultrasound scanning method, spatial misalignment caused by physiological changes is avoided, thus improving diagnostic accuracy. At the same time, by optimizing the robotic arm strategy based on a multidimensional reward model and stabilizing the contact pressure range, both imaging quality and diagnostic stability are ensured, and the safety of diagnosis and treatment is improved.

[0141] As can be seen, this application provides a multimodal ultrasound scanning method. First, a camera module acquires depth image data of the user's target skin area. Then, based on the user's scanning intention information and the depth image data, the robotic arm is controlled to adjust to a first position and maintain a first posture. Next, ultrasound quality assessment parameters are calculated based on the first ultrasound image data of the robotic arm's ultrasound probe and the first contact pressure data of the robotic arm's force feedback sensor, resulting in first ultrasound quality assessment parameters. Then, based on the first ultrasound quality assessment parameters, the robotic arm is controlled to adjust to the target position and target posture. Finally, the user is scanned based on the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor, resulting in an ultrasound scan of the target skin area. Since the ultrasound quality assessment parameters can characterize the contact between the ultrasound probe at the end of the robotic arm and the target skin area at the current moment, the host device can adjust the posture of the robotic arm in real time based on this quantitative index to optimize the contact state between the ultrasound probe and the target skin area. This allows for adaptive optimization and adjustment of abnormal interaction states such as virtual contact between the ultrasound probe and the target skin area caused by user shaking, breathing, and other body movements, improving the contact stability and safety of the ultrasound probe scanning the target skin area.

[0142] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the electronic device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0143] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0144] When dividing each function into modules according to its corresponding function. Figure 7This is a functional block diagram of a multimodal ultrasound scanning device provided in this application embodiment. The multimodal ultrasound scanning device 700 is used as the host device in an ultrasound scanning system. The multimodal ultrasound scanning device 700 includes:

[0145] The acquisition unit 710 is used to acquire depth image data of the user's target skin area through the camera module;

[0146] The control unit 720 is used to control the robotic arm to adjust the ultrasonic probe to the first target pose according to the user's scanning intention information and the depth image data. The ultrasonic probe and the force feedback sensor are integrated at the end of the robotic arm.

[0147] The calculation unit 730 is used to calculate the ultrasonic quality assessment parameters based on the first ultrasonic image data of the ultrasonic probe and the first contact pressure data of the force feedback sensor, and obtain the first ultrasonic quality assessment parameters.

[0148] The control unit 720 is also configured to adjust the ultrasound probe to the second target pose via the robotic arm according to the first ultrasound quality assessment parameters; and to scan the user according to the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area.

[0149] In one possible embodiment, the calculation unit 730, in calculating the ultrasonic quality assessment parameters based on the first ultrasonic image data of the ultrasonic probe and the first contact pressure data of the force feedback sensor, specifically performs the following:

[0150] The first ultrasound image data is calculated based on a preset edge detection algorithm to obtain image edge contrast parameters; the image edge contrast parameters include: grayscale contrast parameters and signal-to-noise ratio parameters;

[0151] The first ultrasound image data is input into a preset cross-sectional structure recognition model to obtain the target structure recognition result;

[0152] Determine the confidence score of the cross-sectional structure corresponding to the target structure recognition result to obtain the structure confidence score;

[0153] The contact pressure stability parameter of the ultrasonic probe is calculated based on the first contact pressure data.

[0154] When the signal-to-noise ratio parameter is greater than or equal to a preset signal-to-noise ratio threshold, the image quality score corresponding to the first ultrasound image data is determined according to the grayscale contrast parameter.

[0155] The first ultrasound quality assessment parameter is obtained by calculating based on the image quality score, the structure confidence score, and the contact pressure stability parameter.

[0156] In one possible embodiment, the control unit 720, in adjusting the ultrasound probe to the second target pose via the robotic arm according to the first ultrasound quality assessment parameters, is specifically configured to:

[0157] The pose adjustment parameters of the robotic arm are determined based on the difference between the first target pose and the second target pose.

[0158] A comprehensive reward function for the reinforcement learning algorithm is constructed based on the first ultrasound quality assessment parameters;

[0159] An enhanced state space is constructed based on the first target pose and the first ultrasound image data as state variables; and an enhanced action space is constructed based on the pose adjustment parameters.

[0160] A reinforcement learning policy model is constructed based on the reinforcement action space and the comprehensive reward function, and the reinforcement state space is input into the reinforcement learning policy model for solving to obtain the pose control action parameters.

[0161] The posture control motion parameters are sent to the robotic arm. After the robotic arm adjusts its posture according to the posture control motion parameters, the third ultrasonic image data and the third contact pressure data are collected through the camera module and the force feedback sensor.

[0162] The second ultrasound quality assessment value is determined based on the third ultrasound image data and the third contact pressure data, and the second ultrasound quality assessment value is input into the comprehensive reward function to obtain the reward value corresponding to the posture control action parameters.

[0163] The reinforcement learning policy model is iteratively optimized and solved based on the reward value. When the output value of the comprehensive reward function satisfies the preset convergence condition, the second target pose is obtained.

[0164] In one possible embodiment, the control unit 720, in controlling the robotic arm to adjust the ultrasonic probe to the first target pose based on the user's scanning intent information and the depth image data, is specifically used for:

[0165] The three-dimensional spatial coordinates of the target skin region are determined based on the depth image data;

[0166] A world coordinate system is constructed based on the preset hand-eye-instrument calibration matrix, the three-dimensional spatial coordinates, and the first pose parameter of the ultrasonic probe;

[0167] The scanning intent information is parsed in a structured manner to obtain a structured intent descriptor, which includes: the second pose parameter of the ultrasonic probe, the standard sectional constraint parameter, and the contact force range parameter of the ultrasonic probe;

[0168] The motion trajectory parameters of the robotic arm are determined based on the standard sectional constraint parameters, the first pose parameters, and the second pose parameters.

[0169] Under the constraints of the contact force range parameters and the standard sectional constraint parameters, the ultrasonic probe is adjusted to the first target pose by the robotic arm according to the motion trajectory parameters.

[0170] In one possible embodiment, the control unit 720, under the constraints of the contact force range parameters and the standard sectional constraint parameters, adjusts the ultrasonic probe to the first target pose via the robotic arm according to the motion trajectory parameters, specifically for:

[0171] Based on the motion trajectory parameters, the pose control commands of the robotic arm within a preset time period are determined to obtain a control command sequence.

[0172] Based on a preset robotic arm operating agent, the control command sequence is deconstructed to obtain multiple control parameters of the robotic arm. The control parameters include at least speed parameters, acceleration parameters, and rotation encoding parameters.

[0173] Multiple pressure parameters are determined based on the contact force range parameters;

[0174] A first motion control command to drive the robotic arm is generated based on the speed parameters, acceleration parameters, and rotation encoding parameters.

[0175] The force control commands for the robotic arm are generated based on the multiple pressure parameters.

[0176] The robotic arm is controlled to perform a first operation according to the first motion control command and a second operation according to the force control command, so as to adjust the ultrasonic probe to the first target pose.

[0177] In one possible embodiment, the control unit 720 is specifically configured to: generate a first motion control command for driving the robotic arm based on the speed parameter, the acceleration parameter, and the rotation encoding parameter;

[0178] Obtain the joint angle parameters and joint angular velocity parameters of multiple joints of the robotic arm;

[0179] Based on the motion trajectory parameters, the pose of the end effector of the robotic arm in the world coordinate system is determined to obtain the second target pose;

[0180] The robotic arm operates the intelligent agent to generate joint motion parameters for the multiple joints based on the second target pose;

[0181] The joint motion trajectories corresponding to the plurality of joints of the robotic arm are generated based on the acceleration parameters and the joint angle parameters.

[0182] Based on the rotational encoding parameters, the joint angular velocity parameters, and the joint motion trajectory, the driving control parameters corresponding to the multiple joints are determined, and multiple driving control parameters are obtained.

[0183] The first motion control command for driving the robotic arm is determined based on the plurality of drive control parameters.

[0184] In one possible embodiment, after the control unit 720 scans the user based on the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area, it is further configured to:

[0185] Extract the three-dimensional point cloud data of the target skin region from the depth image data;

[0186] Based on the three-dimensional point cloud data, a non-rigid deformation matching is performed on the preset human digital model to obtain the user's first digital twin model.

[0187] The ultrasound image boundary data corresponding to the target tissue section is extracted from the second ultrasound image data to obtain ultrasound boundary data; the target tissue section is any part of the target skin region;

[0188] The contact pressure data of the corresponding area of ​​the target tissue cross-section is determined from the second contact pressure data to obtain the target contact pressure data;

[0189] Based on the ultrasonic image boundary data and the target contact pressure data, the first digital twin model is dynamically corrected using voxels to obtain the target digital twin model.

[0190] Spatial matching is performed between the second ultrasound image data and the target digital twin model to determine the cross-sectional position corresponding to the second ultrasound image data;

[0191] The ultrasound probe is controlled to scan the section position to obtain the ultrasound scan result.

[0192] As can be seen, this application provides a multimodal ultrasound scanning device. The host device first acquires depth image data of the user's target skin area through a camera module. Then, based on the user's scanning intention information and the depth image data, it controls a robotic arm to adjust to a first position and maintain a first posture. Next, it calculates ultrasound quality assessment parameters based on the first ultrasound image data from the robotic arm's ultrasound probe and the first contact pressure data from the robotic arm's force feedback sensor, obtaining the first ultrasound quality assessment parameters. Then, based on the first ultrasound quality assessment parameters, it controls the robotic arm to adjust to the target position and target posture. Finally, it scans the user based on the depth image data, the second ultrasound image data from the ultrasound probe, and the second contact pressure data from the force feedback sensor, obtaining the ultrasound scan result of the target skin area. This avoids spatial misalignment caused by physiological changes, improving diagnostic accuracy. Simultaneously, by optimizing the robotic arm strategy based on a multi-dimensional reward model and stabilizing the contact pressure range, it ensures both imaging quality and diagnostic stability, improving treatment safety.

[0193] This application also provides a multimodal ultrasound scanning system, wherein the multimodal ultrasound scanning system can perform some or all of the steps of any of the methods described in the above method embodiments.

[0194] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may include an electronic device.

[0195] It should be noted that, for the sake of simplicity, the above embodiments are all described as a series of actions. Those skilled in the art should understand that this application is not limited to the described order of actions, as some steps in the embodiments of this application can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of this application.

[0196] In the above embodiments, the descriptions of each embodiment in this application have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0197] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0198] The steps of the methods or algorithms described in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (EEPROM), registers, hard disk, portable hard disk, read-only optical disk (CD-ROM), or any other form of storage medium well known in the art. Those skilled in the art should recognize that, in one or more of the above examples, the functions described in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof.

[0199] The various devices and products described in the above embodiments include modules / units that can be software modules / units, hardware modules / units, or a combination of both. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the embodiments of this application. It should be understood that the above descriptions are merely specific implementations of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the embodiments of this application should be included within the scope of protection of the embodiments of this application.

Claims

1. A multimodal ultrasound scanning method, characterized in that, A host device used in an ultrasound scanning system, the method comprising: The camera module collects depth image data of the user's target skin area; Based on the user's scanning intent and the depth image data, the robotic arm is controlled to adjust the ultrasonic probe to the first target pose. The ultrasonic probe and force feedback sensor are integrated at the end of the robotic arm. The ultrasonic quality assessment parameters are calculated based on the first ultrasonic image data from the ultrasonic probe and the first contact pressure data from the force feedback sensor to obtain the first ultrasonic quality assessment parameters. Specifically, this calculation includes: calculating the image edge contrast parameters based on a preset edge detection algorithm on the first ultrasonic image data; the image edge contrast parameters include: grayscale contrast parameters and signal-to-noise ratio parameters; inputting the first ultrasonic image data into a preset cross-sectional structure recognition model to obtain a target structure recognition result; determining the confidence score of the cross-sectional structure corresponding to the target structure recognition result to obtain a structure confidence score; calculating the contact pressure stability parameter of the ultrasonic probe based on the first contact pressure data; when the signal-to-noise ratio parameter is greater than or equal to a preset signal-to-noise ratio threshold, determining the image quality score corresponding to the first ultrasonic image data based on the grayscale contrast parameter; and calculating the first ultrasonic quality assessment parameters based on the image quality score, the structure confidence score, and the contact pressure stability parameter. Based on the first ultrasonic quality assessment parameters, the robotic arm adjusts the ultrasonic probe to the second target pose. The user is scanned based on the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area; the second ultrasound image data is the ultrasound image data collected when the ultrasound probe is in the second target pose, and the second contact pressure data is the contact pressure data collected when the force feedback sensor is in the second target pose; The step of adjusting the ultrasound probe to the second target pose using the robotic arm according to the first ultrasound quality assessment parameters includes: The pose adjustment parameters of the robotic arm are determined based on the first target pose. A comprehensive reward function for the reinforcement learning algorithm is constructed based on the first ultrasound quality assessment parameters; An enhanced state space is constructed based on the first target pose and the first ultrasound image data as state variables; and an enhanced action space is constructed based on the pose adjustment parameters. A reinforcement learning policy model is constructed based on the reinforcement action space and the comprehensive reward function, and the reinforcement state space is input into the reinforcement learning policy model for solving to obtain the pose control action parameters. The posture control motion parameters are sent to the robotic arm. After the robotic arm adjusts its posture according to the posture control motion parameters, the third ultrasonic image data and the third contact pressure data are collected through the ultrasonic probe and the force feedback sensor. The second ultrasound quality assessment value is determined based on the third ultrasound image data and the third contact pressure data, and the second ultrasound quality assessment value is input into the comprehensive reward function to obtain the reward value corresponding to the posture control action parameters; The reinforcement learning policy model is iteratively optimized and solved based on the reward value. When the output value of the comprehensive reward function satisfies the preset convergence condition, the second target pose is obtained.

2. The method as described in claim 1, characterized in that, The step of controlling the robotic arm to adjust the ultrasonic probe to the first target pose based on the user's scanning intention information and the depth image data includes: The three-dimensional spatial coordinates of the target skin region are determined based on the depth image data; A world coordinate system is constructed based on the preset hand-eye-instrument calibration matrix, the three-dimensional spatial coordinates, and the first pose parameter of the ultrasonic probe; The scanning intent information is parsed in a structured manner to obtain a structured intent descriptor, which includes: the second pose parameter of the ultrasonic probe, the standard sectional constraint parameter, and the contact force range parameter of the ultrasonic probe; The motion trajectory parameters of the robotic arm are determined based on the standard sectional constraint parameters, the first pose parameters, and the second pose parameters. Under the constraints of the contact force range parameters and the standard sectional constraint parameters, the ultrasonic probe is adjusted to the first target pose by the robotic arm according to the motion trajectory parameters.

3. The method as described in claim 2, characterized in that, The step of adjusting the ultrasonic probe to the first target pose via the robotic arm according to the motion trajectory parameters, under the constraints of the contact force range parameters and the standard sectional constraint parameters, includes: Based on the motion trajectory parameters, the pose control commands of the robotic arm within a preset time period are determined to obtain a control command sequence. Based on a preset robotic arm operating agent, the control command sequence is deconstructed to obtain multiple control parameters of the robotic arm. The control parameters include at least speed parameters, acceleration parameters, and rotation encoding parameters. Multiple pressure parameters are determined based on the contact force range parameters; A first motion control command to drive the robotic arm is generated based on the speed parameters, acceleration parameters, and rotation encoding parameters. The force control commands for the robotic arm are generated based on the multiple pressure parameters. The robotic arm is controlled to perform a first operation according to the first motion control command and a second operation according to the force control command, so as to adjust the ultrasonic probe to the first target pose.

4. The method as described in claim 3, characterized in that, The step of generating a first motion control command to drive the robotic arm based on the velocity parameter, the acceleration parameter, and the rotation encoding parameter includes: Obtain the joint angle parameters and joint angular velocity parameters of multiple joints of the robotic arm; Based on the motion trajectory parameters, the pose of the end effector of the robotic arm in the world coordinate system is determined to obtain the second target pose; The robotic arm operates the intelligent agent to generate joint motion parameters for the multiple joints based on the second target pose; The joint motion trajectories corresponding to the plurality of joints of the robotic arm are generated based on the acceleration parameters and the joint angle parameters. Based on the rotational encoding parameters, the joint angular velocity parameters, and the joint motion trajectory, the driving control parameters corresponding to the multiple joints are determined, and multiple driving control parameters are obtained. The first motion control command for driving the robotic arm is determined based on the plurality of drive control parameters.

5. The method according to any one of claims 1-4, characterized in that, After scanning the user based on the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area, the method further includes: Extract the three-dimensional point cloud data of the target skin region from the depth image data; Based on the three-dimensional point cloud data, a non-rigid deformation matching is performed on the preset human digital model to obtain the user's first digital twin model. The ultrasound image boundary data corresponding to the target tissue section is extracted from the second ultrasound image data to obtain ultrasound boundary data; the target tissue section is any part of the target skin region; The contact pressure data of the corresponding area of ​​the target tissue cross-section is determined from the second contact pressure data to obtain the target contact pressure data; Based on the ultrasonic image boundary data and the target contact pressure data, the first digital twin model is dynamically corrected using voxels to obtain the target digital twin model. Spatial matching is performed between the second ultrasound image data and the target digital twin model to determine the section position parameters corresponding to the second ultrasound image data; Based on the section position parameters, determine the tissue feature parameters and lesion feature parameters corresponding to the target tissue section; The tissue feature parameters and the lesion feature parameters are input into a preset diagnostic model to obtain the user's diagnostic results.

6. A multimodal ultrasound scanning device, characterized in that, The device is used in the main unit of an ultrasound scanning system, and the device includes: The acquisition unit is used to acquire depth image data of the user's target skin area through the camera module; The control unit is used to control the robotic arm to adjust the ultrasonic probe to the first target pose according to the user's scanning intention information and the depth image data. The ultrasonic probe and the force feedback sensor are integrated at the end of the robotic arm. The calculation unit is configured to calculate ultrasound quality assessment parameters based on the first ultrasound image data of the ultrasound probe and the first contact pressure data of the force feedback sensor, thereby obtaining the first ultrasound quality assessment parameters. The calculation of the ultrasound quality assessment parameters based on the first ultrasound image data of the ultrasound probe and the first contact pressure data of the force feedback sensor includes: calculating the first ultrasound image data based on a preset edge detection algorithm to obtain an image edge contrast parameter; the image edge contrast parameter includes a grayscale contrast parameter and a signal-to-noise ratio parameter; inputting the first ultrasound image data into a preset cross-sectional structure recognition model to obtain a target structure recognition result; determining the confidence score of the cross-sectional structure corresponding to the target structure recognition result to obtain a structure confidence score; calculating the contact pressure stability parameter of the ultrasound probe based on the first contact pressure data; when the signal-to-noise ratio parameter is greater than or equal to a preset signal-to-noise ratio threshold, determining the image quality score corresponding to the first ultrasound image data based on the grayscale contrast parameter; and calculating the first ultrasound quality assessment parameters based on the image quality score, the structure confidence score, and the contact pressure stability parameter. The control unit is further configured to adjust the ultrasound probe to the second target pose via the robotic arm according to the first ultrasound quality assessment parameters; scan the user according to the depth image data, the second ultrasound image data of the ultrasound probe, and the second contact pressure data of the force feedback sensor to obtain the ultrasound scan result of the target skin area; the second ultrasound image data is the ultrasound image data collected when the ultrasound probe is in the second target pose, and the second contact pressure data is the contact pressure data collected when the force feedback sensor is in the second target pose; The step of adjusting the ultrasound probe to the second target pose using the robotic arm according to the first ultrasound quality assessment parameters includes: The pose adjustment parameters of the robotic arm are determined based on the first target pose. A comprehensive reward function for the reinforcement learning algorithm is constructed based on the first ultrasound quality assessment parameters; An enhanced state space is constructed based on the first target pose and the first ultrasound image data as state variables; and an enhanced action space is constructed based on the pose adjustment parameters. A reinforcement learning policy model is constructed based on the reinforcement action space and the comprehensive reward function, and the reinforcement state space is input into the reinforcement learning policy model for solving to obtain the pose control action parameters. The posture control motion parameters are sent to the robotic arm. After the robotic arm adjusts its posture according to the posture control motion parameters, the third ultrasonic image data and the third contact pressure data are collected through the ultrasonic probe and the force feedback sensor. The second ultrasound quality assessment value is determined based on the third ultrasound image data and the third contact pressure data, and the second ultrasound quality assessment value is input into the comprehensive reward function to obtain the reward value corresponding to the posture control action parameters; The reinforcement learning policy model is iteratively optimized and solved based on the reward value. When the output value of the comprehensive reward function satisfies the preset convergence condition, the second target pose is obtained.

7. An electronic device, characterized in that, include: Processor, memory, communication interface, and one or more programs; The one or more programs are stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps of the method as described in any one of claims 1-5.

8. A multimodal ultrasound scanning system, characterized in that, The multimodal ultrasound scanning system performs the method as described in any one of claims 1-5.

Images