Ultrasonic scanning robot control method and device, storage medium and electronic equipment

Abstract

The invention belongs to the technical field of ultrasonic scanning, and particularly relates to an ultrasonic scanning robot control method and device, a computer readable storage medium and electronic equipment. The method comprises the steps that in the ultrasonic scanning process of a robot, a current ultrasonic image is acquired, and a corresponding ultrasonic image template is determined; wherein the ultrasonic image template is an ultrasonic image paradigm of a target object subjected to ultrasonic scanning; performing template matching on the current ultrasonic image and the current template image based on a preset twin neural network to obtain a template matching result; wherein the template matching result comprises a tracking matching window and a corresponding confidence coefficient; and controlling the robot according to the template matching result. According to the method and the device, the target object can be tracked based on template matching of the twin neural network, so that autonomous visual servo motion is realized.

Description

Technical Field

[0001] This application belongs to the technical field of ultrasonic scanning, and particularly relates to a control method, device, computer-readable storage medium and electronic device for an ultrasonic scanning robot. Background Art

[0002] Ultrasonic scanning has the characteristics of non-invasiveness, non-radiation, convenience and speed, and is widely used in the relevant examinations of internal organs and superficial organs. When performing ultrasonic scanning, the ultrasonic technician needs to hold the probe for a long time and maintain the same posture, which may cause muscle fatigue or even injury. At the same time, the accuracy of ultrasonic scanning largely depends on the professional knowledge of the ultrasonic technician, which requires a long learning time and limits the application of ultrasonic scanning technology in underdeveloped areas. To solve this problem, a large number of studies on robot-assisted ultrasonic scanning have been carried out in the past two decades, especially for remote ultrasonic scanning robots, which have been commercialized. However, ultrasonic scanning robots still rely on the operation and experience of ultrasonic technicians and are difficult to achieve autonomous visual servo motion. Summary of the Invention

[0003] In view of this, embodiments of this application provide a control method, device, computer-readable storage medium and electronic device for an ultrasonic scanning robot to solve the problem that the existing control method of ultrasonic scanning robots is difficult to achieve autonomous visual servo motion.

[0004] The first aspect of the embodiments of this application provides a control method for an ultrasonic scanning robot, which may include:

[0005] During the ultrasonic scanning process of the robot, obtain the current ultrasonic image and determine the corresponding ultrasonic image template; wherein, the ultrasonic image template is an ultrasonic image example of the target object of ultrasonic scanning;

[0006] Based on a preset Siamese neural network, perform template matching on the current ultrasonic image and the current template image to obtain a template matching result; wherein, the template matching result includes a tracking matching window and the corresponding confidence level;

[0007] Control the robot according to the template matching result.

[0008] In a specific implementation manner of the first aspect, in the initial search stage, the determining the corresponding ultrasonic image template may include:

[0009] Determine the preset common template as the ultrasonic image template;

[0010] After performing template matching on the current ultrasonic image and the current template image based on the preset Siamese neural network to obtain a template matching result, it further includes:

[0011] When the confidence level of the current ultrasound image is greater than a preset confidence level threshold, determine an individual initial template according to the tracking and matching window of the current ultrasound image.

[0012] In a specific implementation manner of the first aspect, in the horizontal scanning stage and the axial scanning stage, the determining of the corresponding ultrasound image template may include:

[0013] Weight the individual initial template and the dynamic template according to preset weight distribution parameters to obtain the ultrasound image template;

[0014] Wherein, the dynamic template is an image template determined according to the tracking and matching window of the previous frame of ultrasound image.

[0015] In a specific implementation manner of the first aspect, in the axial scanning stage, before weighting the individual initial template and the dynamic template according to preset weight distribution parameters, it may further include:

[0016] Obtain the rotation angle of the ultrasound probe;

[0017] Determine the weight distribution parameters according to the rotation angle; wherein, the weight distribution parameters are negatively correlated with the rotation angle.

[0018] In a specific implementation manner of the first aspect, the controlling of the robot according to the template matching result may include:

[0019] When the confidence level of the current ultrasound image is less than the preset confidence level threshold, control the robot to enter the scanning state recovery control mode;

[0020] When the confidence level of the current ultrasound image is greater than or equal to the confidence level threshold, perform an image quality score on the local image corresponding to the tracking and matching window to obtain a local image quality score;

[0021] Control the robot according to the local image quality score.

[0022] In a specific implementation manner of the first aspect, the controlling of the robot according to the local image quality score may include:

[0023] When the local image quality score is a preset first score, control the robot to enter the scanning state recovery control mode;

[0024] When the local image quality score is a preset second score, control the robot to increase the contact force of the ultrasound scan;

[0025] When the local image quality score is the preset third score, control the robot to continue ultrasonic scanning;

[0026] Wherein, the first score is less than the second score, and the second score is less than the third score.

[0027] In a specific implementation manner of the first aspect, the controlling the robot to enter the scanning state recovery control mode may include:

[0028] Determine the average pixel intensity of the current ultrasonic image;

[0029] When the average pixel intensity is less than a preset pixel intensity threshold, control the robot to increase the contact force of ultrasonic scanning;

[0030] When the average pixel intensity is greater than or equal to the pixel intensity threshold, control the robot to perform target search.

[0031] A second aspect of the embodiments of the present application provides an ultrasonic scanning robot control device, which may include:

[0032] A template determination module, configured to obtain a current ultrasonic image and determine a corresponding ultrasonic image template during the ultrasonic scanning process of the robot; wherein, the ultrasonic image template is an ultrasonic image example of the target object of ultrasonic scanning;

[0033] A target tracking module, configured to perform template matching on the current ultrasonic image and the current template image based on a preset Siamese neural network to obtain a template matching result; wherein, the template matching result includes a tracking matching window and a corresponding confidence level;

[0034] A robot control module, configured to control the robot according to the template matching result.

[0035] In a specific implementation manner of the second aspect, the template determination module may specifically be configured to: in the initial search stage, determine the preset common template as the ultrasonic image template; when the confidence level of the current ultrasonic image is greater than a preset confidence level threshold, determine a personal initial template according to the tracking matching window of the current ultrasonic image.

[0036] In a specific implementation manner of the second aspect, the template determination module may specifically be configured to: in the horizontal scanning stage and the axial scanning stage, weight the personal initial template and the dynamic template according to preset weight distribution parameters to obtain the ultrasonic image template; wherein, the dynamic template is an image template determined according to the tracking matching window of the previous frame of ultrasonic image.

[0037] In a specific implementation manner of the second aspect, the ultrasonic scanning robot control device may further include:

[0038] A weight distribution parameter determination module, configured to obtain the rotation angle of the ultrasonic probe; determine the weight distribution parameter according to the rotation angle; wherein, the weight distribution parameter is negatively correlated with the rotation angle.

[0039] In a specific implementation manner of the second aspect, the robot control module may include:

[0040] A scanning state recovery control unit, configured to control the robot to enter the scanning state recovery control mode when the confidence level of the current ultrasonic image is less than a preset confidence level threshold;

[0041] A quality scoring unit, configured to perform an image quality score on the local image corresponding to the tracking matching window when the confidence level of the current ultrasonic image is greater than or equal to the confidence level threshold, to obtain a local image quality score;

[0042] A robot control unit, configured to control the robot according to the local image quality score.

[0043] In a specific implementation manner of the second aspect, the robot control unit may specifically be configured to: control the robot to enter the scanning state recovery control mode when the local image quality score is a preset first score; control the robot to increase the contact force of ultrasonic scanning when the local image quality score is a preset second score; control the robot to continue ultrasonic scanning when the local image quality score is a preset third score; wherein, the first score is less than the second score, and the second score is less than the third score.

[0044] In a specific implementation manner of the second aspect, the scanning state recovery control unit may specifically be configured to: determine the average pixel intensity of the current ultrasonic image; control the robot to increase the contact force of ultrasonic scanning when the average pixel intensity is less than a preset pixel intensity threshold; control the robot to perform target search when the average pixel intensity is greater than or equal to the pixel intensity threshold.

[0045] A third aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above ultrasonic scanning robot control methods are implemented.

[0046] A fourth aspect of the embodiments of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any one of the above ultrasonic scanning robot control methods are implemented.

[0047] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to execute the steps of any one of the above ultrasonic scanning robot control methods.

[0048] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: During the ultrasonic scanning process of the robot, the current ultrasonic image is acquired, and the corresponding ultrasonic image template is determined; wherein, the ultrasonic image template is an ultrasonic image example of the target object of the ultrasonic scanning; based on a preset siamese neural network, template matching is performed on the current ultrasonic image and the current template image to obtain a template matching result; wherein, the template matching result includes a tracking matching window and the corresponding confidence level; the robot is controlled according to the template matching result. Through the embodiments of the present application, the target object can be tracked based on the template matching of the siamese neural network, so as to achieve autonomous visual servo motion. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0050] Figure 1 It is a flowchart of an embodiment of a method for controlling an ultrasonic scanning robot in an embodiment of the present application;

[0051] Figure 2 It is a schematic diagram of a siamese neural network for target tracking;

[0052] Figure 3 It is a schematic diagram of the phased process of ultrasonic scanning;

[0053] Figure 4 It is a schematic diagram of visual servo during the horizontal scanning stage;

[0054] Figure 5 It is a schematic diagram of a bilinear convolutional neural network for image quality scoring;

[0055] Figure 6 It is a schematic diagram of an overall control framework for ultrasonic scanning in an embodiment of the present application;

[0056] Figure 7 This is a structural diagram of an embodiment of an ultrasonic scanning robot control device in an embodiment of the present application;

[0057] Figure 8 This is a schematic block diagram of an electronic device in an embodiment of the present application. Detailed implementation manners

[0058] To make the objectives, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the embodiments described below are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0059] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0060] It should also be understood that the terms used in this specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification of the present application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0061] It should be further understood that the term "and / or" used in this specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0062] As used in this specification and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" according to the context.

[0063] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0064] The target object in the embodiments of this application can be any internal or superficial organ to be ultrasonically scanned. For the convenience of description, the carotid artery will be used as an example for subsequent explanations.

[0065] Figure 1 The figure shows a schematic diagram of an ultrasonic scanning robot system in the embodiments of this application. The ultrasonic scanning robot system can include, but is not limited to, a robot manipulator, an ultrasonic system with an ultrasonic probe, a force / torque sensor, an external hand camera, an on-hand camera, and a controller with an image acquisition card. As an example, the manipulator can be a 7-degree-of-freedom (DoF) manipulator, and its external force sensing accuracy can be 0.3 Newton (N). The ultrasonic probe can be connected to the end of the manipulator through a customized fixture, and the force / torque sensor is installed between them for contact force measurement. The external hand camera and the on-hand camera can be any RGB-D cameras, including but not limited to Kinect cameras and RealSense cameras, etc. The external hand camera can be set beside the examination bed for detecting key points on the neck. The manipulator can move the probe to a certain distance (such as 300 millimeters) above the key point position and use the on-hand camera fixed at the end of the manipulator to collect the close-up point cloud of the neck.

[0066] When the robot performs ultrasonic scanning, the coordinate space of the patient can be registered to the robot base coordinate system. In the embodiments of this application, {B} represents the robot base coordinate system, {E} represents the robot end coordinate system, {C1} represents the external hand camera coordinate system, {C2} represents the on-hand camera coordinate system, {O} represents the patient coordinate system, and {P} represents the ultrasonic probe coordinate system. The robot base coordinate system {B} is selected as the global reference for robot ultrasonic scanning, and the transformation matrix between the camera coordinate system and the robot base coordinate system can be obtained through calibration, and the transformation matrix from the patient coordinate system to the camera coordinate system can be obtained through the point cloud. Therefore, the transformation matrix from the patient coordinate system to the robot base coordinate system can be obtained from the following formula:

[0067]

[0068] This formula can be used to calculate the spatial information of the key points and the point cloud based on the robot base coordinate system respectively.

[0069] In addition, to achieve visual servo control based on ultrasonic image feedback, it is necessary to establish the mapping relationship between the X-axis direction of the ultrasonic image pixel coordinate system and the X TCP direction of the ultrasonic probe coordinate system. On the neck model, the ultrasonic probe moves along the X-axis direction, and the ultrasonic images before and after the movement are recorded, and the pixel displacement of the center of the carotid artery is measured. Then, the mapping relationship from the pixel coordinate system to the world coordinate system can be calculated using the following formula:

[0070]

[0071] Among them, ΔX TCP is the displacement of the ultrasonic probe coordinate system along the X TCP direction, Δu is the displacement of the ultrasonic image pixel coordinate system along the X-axis direction, and κ is the mapping coefficient.

[0072] To complete fully autonomous ultrasonic scanning, the ultrasonic scanning robot system can first obtain the point cloud of the patient's neck surface. Then, the ultrasonic technician can manually select the approximate position of the carotid artery on the neck point cloud as the initial position of the ultrasonic probe. The Z-axis direction of the ultrasonic probe follows the normal vector of the neck surface calculated by the least squares local plane fitting method. The Y-axis direction of the ultrasonic probe is determined according to the projection of the cephalocaudal direction vector on the normal plane, and then the X-axis direction is obtained according to the right-hand system.

[0073] At the beginning of the ultrasonic scanning process, the ultrasonic scanning robot system can drive the ultrasonic probe to the initial position. Then, continuously adjust the ultrasonic probe until the carotid artery is detected in the ultrasonic image. When the carotid artery is seen in the ultrasonic image, the transverse scanning starts. During the scanning process, monitor the force F TCP of the ultrasonic probe along the Y y , when |F y |≥F ymax , it indicates that the ultrasonic probe reaches the limit position of the neck. Among them, F ymax is the preset force threshold, and its specific value can be flexibly set according to the actual situation. For example, it can be set to 3N or other values, and the embodiments of the present application do not make specific limitations on this. At the same time, keep the ultrasonic probe moving along the normal vector during the scanning process. During the transverse scanning process, the ultrasonic probe keeps a constant speed in the Y direction, and its specific value can be flexibly set according to the actual situation. For example, it can be set to 3 millimeters per second or other values, and the embodiments of the present application do not make specific limitations on this. After completing the transverse scanning, the ultrasonic probe realigns to the geometric center point of the two limit positions and starts to rotate around the Z-axis of the ultrasonic probe to obtain the axial view ultrasonic image of the carotid artery. Keep a constant angular velocity ω during the rotation process, and its specific value can be flexibly set according to the actual situation. For example, it can be set to 0.75° per second or other values, and the embodiments of the present application do not make specific limitations on this.

[0074] To achieve the tracking of the ultrasonic scanning carotid artery, in the embodiments of the present application, a fully convolutional Siamese network (Siamese neural network) can be used as the tracking network for target tracking. Randomly distribute multiple windows in the image, and then use neural network-based template matching to find the window closest to the given template, so as to achieve continuous tracking of the target object.

[0075] Specifically, during the process of the robot performing ultrasonic scanning, the current ultrasonic image can be obtained, and the corresponding ultrasonic image template can be determined; among them, the ultrasonic image template is an example of the ultrasonic image of the target object of ultrasonic scanning, and the ultrasonic image template can include an initial template and a dynamic template, and the dynamic template is an image template determined according to the tracking matching window of the previous frame of ultrasonic image. Then, based on a preset Siamese neural network, template matching can be performed on the current ultrasonic image and the current template image to obtain a template matching result, and the robot can be controlled according to the template matching result. Among them, the template matching result includes a tracking matching window (i.e., the best score window) and the corresponding confidence level.

[0076] Figure 2 The figure shows a schematic diagram of a Siamese neural network for target tracking, where, f θ is a feature extractor composed of ResNet-50, and the three share weights. C is the depthwise cross-correlation coefficient (DW-XCorr), which is used to obtain the correlation between any position in the image and the template. Essentially, the convolution operation is a dot product operation between two sets of deep features. b θ is a confidence regression branch composed of a two-layer neural network (1×1×256 and 1×1×4k), and k is the number of windows. The output of this branch is the center point position of each window and the width and height of the window [u, v, w, h]. is a confidence regression branch composed of a two-layer neural network (1×1×256 and 1×1×2k), and the output is the confidence levels ρ of two categories (with object and without object). Then, it passes through a penalty module, which can include but is not limited to penalty terms such as scale penalty and displacement penalty. Since the windows generally overlap, finally, the non-maximum suppression method is used to select the final object. When the intersection over union (IoU) between two windows is greater than a preset IoU threshold, the window with a lower score will be discarded. Among them, the specific value of the IoU threshold can be flexibly set according to the actual situation. For example, it can be set to 0.6 or other values, and the embodiments of the present application do not make specific limitations on this.

[0077] The initial template and the dynamic template are respectively combined by the feature extractor to obtain the feature map F1, as shown in the following formula:

[0078] F1 = ηf θ (z0) + (1 - η)f θ (z t )

[0079] where, z0 is the initial template, z t is the dynamic template, and η is the weight distribution parameter.

[0080] The output of the Siamese neural network is shown as follows:

[0081] [ρ, u, v, w, h] = TNet(I)

[0082] Where I is the current ultrasound image, and TNet is the processing function of the Siamese neural network.

[0083] When the output confidence ρ is greater than or equal to the preset confidence threshold, this tracking can be considered reliable. On the contrary, when the output confidence ρ is less than the confidence threshold, it can be considered that the target object is lost during the scanning process. At this time, the robot can be controlled to enter the scanning state recovery control mode. Among them, the specific value of the confidence threshold can be flexibly set according to the actual situation. For example, it can be set to 0.95 or other values, and the embodiments of the present application do not make specific limitations on this.

[0084] Such as Figure 3 shown, according to the needs of the clinical process, the process of realizing ultrasound scanning by visual servo navigation can be described in 3 stages:

[0085] (1) In the initial search stage, since there is no separate initial template and dynamic template, a common template z p , η = 1 can be preset, that is, the preset common template is determined as the ultrasound image template. After initial positioning, the robotic arm drives the ultrasound probe to move along the X TCP direction until the carotid artery is traced. When the confidence of the current ultrasound image is greater than the preset confidence threshold, the personal initial template z0 can be determined according to the tracking matching window of the current ultrasound image, and the transverse scanning stage can be entered.

[0086] (2) In the transverse scanning stage, the personal initial template and the dynamic template can be weighted according to the preset weight distribution parameters to obtain the ultrasound image template. Among them, the personal initial template and the dynamic template can have the same weight.

[0087] Figure 4 The following figure shows a schematic diagram of visual servo in the transverse scanning stage. As shown in the figure, in the embodiments of the present application, the deviation ε of the center of the tracking matching window relative to the midline of the ultrasound image can be calculated according to the following formula t :

[0088]

[0089] Where W I is the pixel width of the ultrasound image.

[0090] When the deviation is greater than the preset deviation threshold, the scanning speed of the ultrasound probe can be changed according to the following formula until the deviation is within the allowable range:

[0091]

[0092] Among them, is the speed of the ultrasonic probe scanning along the X TCP direction, is the speed of the ultrasonic probe scanning along the Y TCP direction, and λ ν is a preset error adjustment coefficient, and its specific value can be flexibly set according to the actual situation. The embodiments of the present application do not make specific limitations on this. The specific value of the deviation threshold can be flexibly set according to the actual situation. For example, it can be set to 25% or other values. The embodiments of the present application do not make specific limitations on this.

[0093] Through this adjustment method, the carotid artery can be kept in the middle of the ultrasonic image, and the ultrasonic probe can also cover the entire carotid artery, including two extreme positions in the head direction and the foot direction. During the transverse scanning process, the robot will automatically record the trajectory of the ultrasonic probe.

[0094] (3) In the axial scanning stage, the personal initial template and the dynamic template can have different weights. According to clinical experience and probe size, the midpoint of the entire trajectory can be set to the position rotated to the axial view. During the rotation process, the rotation angle of the ultrasonic probe can be obtained, the weight distribution parameter can be determined according to the rotation angle, and the personal initial template and the dynamic template can be weighted according to the weight distribution parameter to obtain the ultrasonic image template. Among them, the weight distribution parameter is negatively correlated with the rotation angle, that is, the weight distribution parameter decreases as the rotation angle increases, as shown in the following formula:

[0095]

[0096] Among them, Δθ is the rotation angle of the ultrasonic probe. This formula shows that due to the change of the ultrasonic image, the contribution of the initial template to target tracking decreases with rotation. Finally, when Δθ = 90°, η = 0, the carotid artery is shown as a long hypoechoic area, which is quite different from the transverse ultrasonic image.

[0097] The ratio ∈ of the pixel width of the tracking matching window to the pixel width of the ultrasonic image t can be calculated according to the following formula.

[0098]

[0099] Among them, w is the pixel width of the tracking matching window. When ∈ tWhen it is greater than a preset width ratio threshold, it can be considered that the ultrasound probe has reached the axis of the carotid artery. The specific value of the width ratio threshold can be flexibly set according to the actual situation. For example, it can be set to 95% or other values, and the embodiments of the present application do not make specific limitations on this.

[0100] In carotid artery ultrasound scanning, the quality of the local image around the carotid artery is extremely important. When the confidence level of the current ultrasound image is greater than or equal to the confidence level threshold, the local image corresponding to the tracking matching window can be scored for image quality, so as to obtain the local image quality score, and the robot can be controlled according to the local image quality score.

[0101] In the embodiments of the present application, a bilinear convolutional neural network (BCNN) as Figure 5 shown can be used to perform image quality scoring. The input of the neural network is the local image corresponding to the tracking matching window, and the output is the quality score. The controller can provide corresponding control strategies according to different local qualities. As Figure 5 shown, the BCNN can include two parallel feature extractors CNN A and CNN B. The feature extractor can adopt ResNet18 or other networks, and the two feature extractors share the same weights. The features obtained by the two branches pass through the bilinear pooling (BP) layer and the fully connected (FC) layer to obtain the final quality score S.

[0102] When the local image quality score is a preset first score, it means that although the tracking network has successfully identified the target, the edge of the carotid artery is not clear enough and the contrast is not high enough. At this time, the robot can be controlled to enter the scanning state recovery control mode.

[0103] When the local image quality score is a preset second score, it means that the resolution of the carotid artery edge is average and the contrast of the ultrasound image is medium. At this time, the robot can be controlled to increase the contact force of the ultrasound scan until the image quality requirement is met or the preset contact force threshold is reached. Among them, the specific value of the contact force threshold can be flexibly set according to the actual situation. For example, it can be set to 10N or other values, and the embodiments of the present application do not make specific limitations on this.

[0104] When the local image quality score is a preset third score, it means that the edge of the carotid artery is clear and complete, the internal is uniformly hypoechoic, and the contrast of the ultrasound image is obvious. At this time, the robot can be controlled to continue the ultrasound scan.

[0105] Among them, the first score is less than the second score, and the second score is less than the third score. The specific values of the first score, the second score, and the third score can be flexibly set according to the actual situation. For example, the first score, the second score, and the third score can be set to 1, 2, and 3 respectively, and the embodiments of the present application do not make specific limitations on this.

[0106] In the embodiments of the present application, quality control can be used in two stages. The first stage is the horizontal scanning stage to ensure the quality of each cross-sectional carotid artery ultrasound image. The second stage is the axial scanning stage to ensure the quality of the axial cross-sectional ultrasound image. Once the tracking network identifies the object of interest, it is possible that the tracking matching window cannot completely enclose the carotid artery and its edges. In this case, the cropped area is enlarged by 20% relative to the tracking matching window, and the local image quality score is calculated according to the following formula:

[0107] S = QNet(I[u±0.6w, v±0.6h])

[0108] where I[u±0.6w, v±0.6h] is the local image corresponding to the enlarged tracking matching window, and QNet is the processing function of the bilinear convolutional neural network.

[0109] In the scanning state recovery control mode, the average pixel intensity of the current ultrasound image can be determined as shown in the following formula:

[0110]

[0111] where W I and H I respectively represent the pixel width and height of the ultrasound image, I(i, j) represents the pixel intensity, and I mean is the average pixel intensity.

[0112] In the case where the average pixel intensity is less than the preset pixel intensity threshold, the robot can be controlled to increase the contact force of the ultrasound scan. For example, it can be gradually increased by 0.5 N until the average pixel intensity reaches the pixel intensity threshold. At the same time, if the contact force reaches the contact force threshold, the adjustment will stop. The specific value of the pixel intensity threshold can be flexibly set according to the actual situation. For example, it can be set to 70 or other values, and the embodiments of the present application do not make specific limitations on this.

[0113] In the case where the average pixel intensity is greater than or equal to the pixel intensity threshold, the robot can be controlled to perform target search. The initial template is replaced by the last dynamic template before the target is lost. During the horizontal scanning process, the probe moves along the X TCP axis (for example, it can be between [-5, 5] mm); during the axial scanning process, the probe moves along the Y TCP axis (for example, it can be between [-5, 5] mm). Once the target is successfully tracked, it will re-enter the automatic scanning mode. If the target is still not found, the scan is considered to have failed and the scanning process will stop.

[0114] Figure 6The figure shows a schematic diagram of an overall control framework for ultrasonic scanning in an embodiment of the present application, which mainly includes vision servo based on target tracking, image quality control, and recovery control. Target tracking is used to identify the carotid artery and provide ultrasonic probe navigation. Quality control is used to ensure the quality of carotid artery ultrasonic images. Recovery control is used to restore to the normal state when the carotid artery is lost in the ultrasonic image. The specific control method is described in the foregoing content and will not be elaborated here.

[0115] In order to ensure the safety of the contact between the robot and the patient and at the same time control the contact force within a certain range, in a specific implementation manner of the embodiment of the present application, an impedance controller with an in-built joint torque sensor can be used. The Cartesian compliance control law involved is shown as follows:

[0116]

[0117] Where, is the calculated target torque of all joints, is the Jacobian matrix, x e = x - x d is the pose error in the Cartesian space between the current pose and the target pose, F d is the desired contact force between the ultrasonic probe and the patient, M i , B i , K i are the virtual mass matrix, virtual damping matrix, and virtual stiffness matrix. Considering that the acceleration during the robot scanning process is small, M i is ignored, the stiffness can be set to 1000 N / m, 1000 N / m, 200 N / m, 2 Nm / rad, 20 Nm / rad, 20 Nm / rad, and the damping of each degree of freedom can be set to 0.8. Impedance control allows the actuator to deviate from its trajectory to adapt to body contact, thus ensuring safe interaction with the patient. At the same time, F d can maintain stable contact between the ultrasonic probe and the patient's body. To ensure safety, the robot integrates several safety constraints, including collision detection and Cartesian velocity limit. Collision detection is achieved by setting a maximum external force threshold of 12 N and a control frequency of 1000 Hz. When the contact force at the end of the ultrasonic probe or on the robot body exceeds the threshold, the robot will immediately stop moving to ensure safety.

[0118] In summary, in the process of the robot performing ultrasonic scanning in the embodiments of the present application, the current ultrasonic image is acquired, and the corresponding ultrasonic image template is determined; wherein, the ultrasonic image template is an ultrasonic image example of the target object of ultrasonic scanning; based on a preset siamese neural network, template matching is performed on the current ultrasonic image and the current template image to obtain a template matching result; wherein, the template matching result includes a tracking matching window and the corresponding confidence level; the robot is controlled according to the template matching result. Through the embodiments of the present application, the target object can be tracked based on the template matching of the siamese neural network, so as to achieve autonomous visual servo motion.

[0119] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution is prior or subsequent. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0120] Corresponding to the ultrasonic scanning robot control method described in the above embodiments, Figure 7 Fig. shows a structural diagram of an embodiment of an ultrasonic scanning robot control device provided by an embodiment of the present application.

[0121] In this embodiment, an ultrasonic scanning robot control device may include:

[0122] A template determination module 701, configured to acquire the current ultrasonic image and determine the corresponding ultrasonic image template during the process of the robot performing ultrasonic scanning; wherein, the ultrasonic image template is an ultrasonic image example of the target object of ultrasonic scanning;

[0123] A target tracking module 702, configured to perform template matching on the current ultrasonic image and the current template image based on a preset siamese neural network to obtain a template matching result; wherein, the template matching result includes a tracking matching window and the corresponding confidence level;

[0124] A robot control module 703, configured to control the robot according to the template matching result.

[0125] In a specific implementation manner of the embodiments of the present application, the template determination module may be specifically configured to: in the initial search stage, determine a preset common template as the ultrasonic image template; in the case where the confidence level of the current ultrasonic image is greater than a preset confidence level threshold, determine a personal initial template according to the tracking matching window of the current ultrasonic image.

[0126] In a specific implementation manner of the embodiment of the present application, the template determination module may specifically be used for: in the horizontal scanning stage and the axial scanning stage, weighting the personal initial template and the dynamic template according to preset weight distribution parameters to obtain the ultrasonic image template; wherein, the dynamic template is an image template determined according to the tracking matching window of the previous frame of ultrasonic image.

[0127] In a specific implementation manner of the embodiment of the present application, the ultrasonic scanning robot control device may further include:

[0128] A weight distribution parameter determination module, configured to obtain the rotation angle of the ultrasonic probe; determine the weight distribution parameter according to the rotation angle; wherein, the weight distribution parameter is negatively correlated with the rotation angle.

[0129] In a specific implementation manner of the embodiment of the present application, the robot control module may include:

[0130] A scanning state recovery control unit, configured to control the robot to enter the scanning state recovery control mode when the confidence level of the current ultrasonic image is less than a preset confidence level threshold;

[0131] A quality scoring unit, configured to perform an image quality score on the local image corresponding to the tracking matching window when the confidence level of the current ultrasonic image is greater than or equal to the confidence level threshold to obtain a local image quality score;

[0132] A robot control unit, configured to control the robot according to the local image quality score.

[0133] In a specific implementation manner of the embodiment of the present application, the robot control unit may specifically be used for: controlling the robot to enter the scanning state recovery control mode when the local image quality score is a preset first score; controlling the robot to increase the contact force of the ultrasonic scanning when the local image quality score is a preset second score; controlling the robot to continue with the ultrasonic scanning when the local image quality score is a preset third score; wherein, the first score is less than the second score, and the second score is less than the third score.

[0134] In a specific implementation manner of the embodiment of the present application, the scanning state recovery control unit may specifically be used for: determining the average pixel intensity of the current ultrasonic image; controlling the robot to increase the contact force of the ultrasonic scanning when the average pixel intensity is less than a preset pixel intensity threshold; controlling the robot to perform target search when the average pixel intensity is greater than or equal to the pixel intensity threshold.

[0135] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the devices, modules, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0136] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not elaborated or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] Figure 8 The schematic block diagram of an electronic device provided by an embodiment of the present application is shown. For the convenience of description, only the parts related to the embodiment of the present application are shown.

[0138] As Figure 8 shown, the electronic device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps in the above-mentioned various embodiments of the ultrasonic scanning robot control method are implemented, or when the processor 80 executes the computer program 82, the functions of the various modules / units in the above-mentioned device embodiments are implemented.

[0139] Exemplarily, the computer program 82 can be divided into one or more modules / units. The one or more modules / units are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program 82 in the electronic device 8.

[0140] The electronic device 8 can be the controller in the ultrasonic scanning robot system. Those skilled in the art can understand that Figure 8 merely being an example of the electronic device 8 does not constitute a limitation on the electronic device 8. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device 8 may further include input / output devices, network access devices, a bus, etc.

[0141] The processor 80 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0142] The memory 81 may be an internal storage unit of the electronic device 8, such as the hard disk or memory of the electronic device 8. The memory 81 may also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the electronic device 8. Further, the memory 81 may also include both the internal storage unit and the external storage device of the electronic device 8. The memory 81 is used to store the computer program and other programs and data required by the electronic device 8. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In practical applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system may refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0144] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0145] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0146] In the embodiments provided in this application, it should be understood that the disclosed apparatus / electronic device and method can be implemented in other ways. For example, the apparatus / electronic device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0149] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0150] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An ultrasonic scanning robot control method, characterized in that, Including: During the process of the robot performing ultrasonic scanning, obtain the current ultrasonic image and determine the corresponding ultrasonic image template; wherein, the ultrasonic image template is an exemplary ultrasonic image of the target object of ultrasonic scanning; Based on a preset siamese neural network, perform template matching on the current ultrasonic image and the current template image to obtain a template matching result; wherein, the template matching result includes a tracking matching window and the corresponding confidence level; Control the robot according to the template matching result.

2. The ultrasonic scanning robot control method according to claim 1, wherein In the initial search stage, the determining of the corresponding ultrasonic image template includes: Determine a preset common template as the ultrasonic image template; After performing template matching on the current ultrasonic image and the current template image based on the preset siamese neural network to obtain a template matching result, it further includes: In the case where the confidence level of the current ultrasonic image is greater than a preset confidence level threshold, determine a personal initial template according to the tracking matching window of the current ultrasonic image.

3. The ultrasonic scanning robot control method according to claim 2, characterized in that, In the horizontal scanning stage and the axial scanning stage, the determining of the corresponding ultrasonic image template includes: Weight the personal initial template and the dynamic template according to preset weight distribution parameters to obtain the ultrasonic image template; Wherein, the dynamic template is an image template determined according to the tracking matching window of the previous frame of ultrasonic image.

4. The ultrasonic scanning robot control method according to claim 3, characterized in that, In the axial scanning stage, before weighting the personal initial template and the dynamic template according to the preset weight distribution parameters, it further includes: Obtain the rotation angle of the ultrasonic probe; Determine the weight distribution parameters according to the rotation angle; wherein, the weight distribution parameters are negatively correlated with the rotation angle.

5. The ultrasonic scanning robot control method according to any one of claims 1 to 4, characterized in that, The controlling of the robot according to the template matching result includes: In the case where the confidence level of the current ultrasonic image is less than a preset confidence level threshold, control the robot to switch to the scanning state recovery control mode; In the case where the confidence level of the current ultrasonic image is greater than or equal to the confidence level threshold, perform an image quality score on the local image corresponding to the tracking matching window to obtain a local image quality score; Control the robot according to the local image quality score.

6. The ultrasonic scanning robot control method according to claim 5, characterized in that, The controlling of the robot according to the local image quality score includes: In the case where the local image quality score is a preset first score, control the robot to switch to the scanning state recovery control mode; In the case where the local image quality score is a preset second score, control the robot to increase the contact force of ultrasonic scanning; In the case where the local image quality score is a preset third score, control the robot to continue ultrasonic scanning; Wherein, the first score is less than the second score, and the second score is less than the third score.

7. The ultrasonic scanning robot control method according to claim 5, wherein The controlling the robot to switch to the scanning state recovery control mode includes: Determine the average pixel intensity of the current ultrasonic image; In the case where the average pixel intensity is less than a preset pixel intensity threshold, control the robot to increase the contact force of ultrasonic scanning; In the case where the average pixel intensity is greater than or equal to the pixel intensity threshold, control the robot to perform target searching.

8. An ultrasonic scanning robot control device, characterized in that, Including: A template determination module, configured to obtain a current ultrasound image and determine a corresponding ultrasound image template during the process of the robot performing ultrasound scanning; wherein, the ultrasound image template is an ultrasound image example of a target object for ultrasound scanning; A target tracking module, configured to perform template matching on the current ultrasound image and the current template image based on a preset Siamese neural network to obtain a template matching result; wherein, the template matching result includes a tracking matching window and a corresponding confidence level; A robot control module, configured to control the robot according to the template matching result.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the steps of the ultrasound scanning robot control method according to any one of claims 1 to 7 are implemented.

10. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the ultrasound scanning robot control method according to any one of claims 1 to 7 are implemented.

Images