Method for marking space pose of ultrasonic sensor in real time

Abstract

The invention provides a method for marking the spatial pose of an ultrasonic sensor in real time, and relates to the technical field of ultrasonic clinical diagnosis and ultrasonic image body surface marking. The invention provides a method for marking the spatial pose of an ultrasonic sensor in real time, and the method comprises the steps: firstly, building a relative position relation of an ultrasonic image coordinate system under a camera coordinate system based on a pre-built optical positioning system, and obtaining an optical image of a target object; acquiring a position feature map of key points in the target object based on the optical image; and finally, obtaining a mark graph of the target object based on the optical image and the position feature graph. According to the technology for marking the spatial pose of the ultrasonic sensor in real time, the ultrasonic image can be marked more efficiently and accurately, and later analysis and research are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of ultrasonic clinical diagnosis and ultrasonic image body surface marking, and in particular to a method for real-time marking of the spatial position of an ultrasonic sensor. Background Art

[0002] In order to facilitate later analysis and research, during the ultrasound scanning process, the operator usually records the position of the ultrasound image relative to the scanned object, so that the target area can be quickly located according to the image position in the future and subsequent operations can be performed (for example, repeated scanning inspections, auxiliary sampling positioning, etc.). At the same time, the position information of the ultrasound image can provide an intuitive and clear description of the specific parts of the scanned object, which facilitates later communication.

[0003] Currently, during ultrasound examinations, doctors manually record body surface markers when saving ultrasound images. This operation requires manually calling up the body marker library in the ultrasound machine, which contains marker images of various examination parts. The marker image of the corresponding examination part is selected, and then the position and direction of the ultrasound sensor for this scan are manually set and adjusted. In clinical examinations, when faced with numerous ultrasound images of multiple patients, repeatedly manually recording and adjusting body surface markers will add a lot of burden to doctors and reduce the efficiency of clinical examinations. At the same time, different operators will have artificial differences when marking body surface markers, which will affect the accuracy of ultrasound body surface markers.

[0004] Based on this, the present invention proposes a technology for real-time marking of the spatial position of an ultrasonic sensor, in order to reduce the workload of doctors during clinical examinations and improve the accuracy of ultrasonic body surface marking. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies of the prior art, the present invention provides a method for real-time marking of the spatial position of an ultrasonic sensor, which at least solves the problem in the prior art that when saving ultrasonic images, only surface marks can be manually recorded, resulting in a large workload.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present application first proposes a method for real-time marking of the spatial position of an ultrasonic sensor, the method comprising:

[0010] S1. Based on the pre-built optical positioning system, establish the relative position relationship between the ultrasound image coordinate system and the camera coordinate system, and obtain the optical image of the target object;

[0011] S2. Acquire a position feature map of key points in the target object based on the optical image;

[0012] S3. Acquire a marker map of the target object based on the optical image and the position feature map.

[0013] In one embodiment, the S1 includes:

[0014] S11, building an optical positioning system; the optical positioning system includes an optical tracker, a camera and an optical marker;

[0015] S12, calibrating the relative position relationship between the optical marker coordinate system and the ultrasound image coordinate system based on the optical positioning system;

[0016] S13. Based on the relative position relationship between the optical marker coordinate system and the ultrasound image coordinate system, a relative position relationship between the ultrasound image coordinate system and the camera coordinate system is established; at the same time, an optical image of the target object is captured using an optical camera.

[0017] Preferably, the optical tracker includes but is not limited to a multi-eye camera, an optical camera, etc.

[0018] Preferably, the optical markers include but are not limited to metal balls with diffuse reflective surfaces, calibration plates with chessboards or dot arrays, etc.

[0019] More preferably, the combination of the optical tracker, camera and optical marker includes but is not limited to a combination of a multi-eye camera and a metal ball with a diffuse reflection surface, an optical camera and a calibration plate with a chessboard or a dot matrix, etc.

[0020] In one embodiment, the S13 includes calculating according to the following formula to establish the relative position relationship between the ultrasound image coordinate system and the camera coordinate system:

[0021] R I2C =Rt M2C ·Rt I2M

[0022] Among them, Rt I2C Represents the relative position relationship transformation matrix of the ultrasound image coordinate system in the camera coordinate system; Rt M2C Represents the relative position relationship transformation matrix of the optical marker coordinate system in the camera coordinate system; Rt I2M The transformation matrix represents the relative position relationship between the ultrasound image coordinate system and the optical marker coordinate system.

[0023] Preferably, the Zhang Zhengyou calibration method or the small ball calibration method is used to calibrate the internal parameters of the multi-camera, and then the binocular positioning method is used to locate the optical marker; the relative position relationship between the ultrasound image coordinate system and the optical marker coordinate system is calculated by the N-line calibration method, the multi-plane calibration method, etc., that is, the transformation matrix Rt I2M .

[0024] In one embodiment, the S2 includes:

[0025] The optical image is input into a pre-trained first deep learning network to obtain a position feature map of key points in the target object.

[0026] Preferably, the first deep learning network includes an HR-Net backbone network and a classification module.

[0027] In one embodiment, the S3 includes:

[0028] The optical image and the position feature map are input into a pre-trained second deep learning network to obtain a label map of the target object.

[0029] Preferably, the second deep learning network is based on a U-Net network structure, including a feature encoder and a decoder.

[0030] More preferably, the second deep learning network further includes:

[0031] Adding an attention module before the encoder in the U-Net network, wherein the attention module includes multi-scale convolution;

[0032] The convolution of the encoder part uses a dilated convolution, and a dropout layer is set at a specific encoder layer.

[0033] In yet a further embodiment, the method further comprises:

[0034] S4. Generate an indication mark on the marking diagram; the indication mark includes an ultrasonic sensor position indication mark and a scanning direction indication mark.

[0035] Preferably, the indicator mark is a "T"-shaped icon, and the horizontal line portion of the "T"-shape represents the marked side of the ultrasonic sensor.

[0036] In a second aspect, the present application also proposes a system for real-time marking of the spatial position of an ultrasonic sensor, the system comprising:

[0037] The position relationship acquisition module is used to obtain the relative position relationship of the ultrasound image coordinate system in the camera coordinate system based on the pre-built optical positioning system.

[0038] An optical image acquisition module, used to acquire an optical image of a target object based on a pre-built optical positioning system;

[0039] A position feature map acquisition module, used to acquire a position feature map of key points in a target object based on the optical image;

[0040] A label map acquisition module is used to acquire a label map of the target object based on the optical image and the position feature map.

[0041] In a third aspect, the present application further proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for real-time marking of the spatial posture of an ultrasonic sensor as described in any one of the above items are implemented.

[0042] In a fourth aspect, the present application finally proposes a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for real-time marking of the spatial posture of an ultrasonic sensor as described in any one of the above items.

[0043] (III) Beneficial effects

[0044] The present invention provides a method for real-time marking of the spatial position of an ultrasonic sensor. Compared with the prior art, the method has the following beneficial effects:

[0045] The present invention proposes a method for marking the spatial posture of an ultrasonic sensor in real time. First, based on a pre-built optical positioning system, the relative position relationship between the ultrasonic image coordinate system and the camera coordinate system is established, and an optical image of the target object is obtained at the same time; then, a position feature map of key points in the target object is obtained based on the optical image; finally, a marking map of the target object is obtained based on the above optical image and the position feature map. The technology for marking the spatial posture of an ultrasonic sensor in real time proposed in this application solves the problem that the prior art can only manually record body surface marks when saving ultrasonic images, which causes a large workload and is cumbersome for the operator. Compared with the prior art, it can mark ultrasonic images more efficiently and accurately, which facilitates later analysis and research. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is an overall flow chart of a method for real-time marking of the spatial position of an ultrasonic sensor according to the present invention;

[0048] Figure 2 It is a flow chart of a method for real-time marking of the spatial posture of an ultrasonic sensor in an embodiment of the present invention;

[0049] Figure 3 Schematic diagram of an optical positioning system and coordinate system conversion in an embodiment of the present invention;

[0050] Figure 4 This is a diagram of the first deep learning network structure in an embodiment of the present invention;

[0051] Figure 5 This is a diagram of the second deep learning network structure in an embodiment of the present invention;

[0052] Figure 6 Schematic diagram of ultrasonic body surface marking in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] During ultrasound scanning, for later analysis and research, the operator usually records the position of the ultrasound image relative to the scanned object, so that the target area can be quickly located according to the image position in the future and subsequent operations can be performed, such as repeated scanning inspections, auxiliary sampling positioning, etc. At the same time, the position information of the ultrasound image can provide an intuitive and clear description of the specific parts of the scanned object, which facilitates later communication.

[0055] At present, the recording of the orientation of ultrasound images is usually done manually by the operator, and the recording methods include text recording, object marking, etc. Some existing models of ultrasound inspection equipment have a manual recording function for object marking. Ultrasonic object marking includes: marking diagrams and indicator marks. Among them, the marking diagram uses simple lines to depict the structure of the scanned object to provide the operator with general information about the scanned object, and the indicator mark is used to mark the position and direction of the ultrasonic sensor during scanning in the marking diagram.

[0056] When an ultrasonic scanning device with an ultrasonic object marking function adds a mark to the acquired ultrasonic image, it is necessary to first select a suitable basic marking diagram, and then manually move the marking diagram and adjust the direction of the indicator mark to make the added indicator mark match the actual ultrasonic sensor scanning position and direction. Obviously, this will increase the operation steps, prolong the scanning time, and affect the efficiency of the scanning work. Especially in clinical examinations, when faced with numerous ultrasonic images of multiple patients, repeatedly manually recording and adjusting the surface marks will greatly increase the workload and intensity of doctors, and seriously reduce the efficiency of clinical examinations. At the same time, different operators will have artificial differences when marking surface marks, which is not conducive to establishing a unified standard for ultrasonic surface marks, and will affect the accuracy of ultrasonic surface marks, and thus affect subsequent analysis and research work.

[0057] The embodiment of the present application provides a method for real-time marking of the spatial posture of an ultrasonic sensor, thereby solving the problem of the above-mentioned prior art that when saving ultrasonic images, only surface marks can be manually recorded, resulting in a large workload, thereby achieving the purpose of efficiently and accurately marking ultrasonic images.

[0058] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0059] Terminology explanation:

[0060] Optical tracker: A device that uses optical principles to track the position, direction, and motion trajectory of an object in real time, and is widely used in many fields.

[0061] Zhang Zhengyou calibration method: A classic method for camera calibration. By using a flat chessboard template and taking chessboard images at different positions and angles, a simple, flexible and high-precision calibration method is achieved.

[0062] Small ball calibration method: a calibration method used for specific measurement systems (such as laser tracking measurement systems, optical measurement systems, etc.). Usually in some scenarios where high-precision measurement of spatial position is required, a small ball of known size is used as a calibration object to determine the parameters of the measurement system or establish the conversion relationship between the measurement coordinate system and the actual coordinate system.

[0063] Binocular positioning: Simulating the principle of human binocular vision, obtaining image information of the scene through two cameras (binocular cameras) to determine the position of objects in space.

[0064] N-line calibration: refers to using N-shaped straight lines with known spatial positions as calibration objects, scanning the N-line model calibration object with a probe carrying an optical marker, and simultaneously photographing the optical marker with a camera, and calculating the relative transformation relationship between the image coordinate system and the optical marker coordinate system using the geometric relationship between the imaging position of the straight line in the image plane and the actual space. Here, "N" is the shape of the model, and multiple layers of N lines are usually stacked alternately to enrich the constraints.

[0065] Multi-plane calibration: Use multiple planes with known geometric relationships as calibration objects, scan the multi-plane model calibration object with a probe carrying an optical marker, and simultaneously shoot the optical marker with a camera. Based on the imaging position of the plane in the image plane and the geometric relationship in the actual space, calculate the relative transformation relationship between the image coordinate system and the optical marker coordinate system. Compared with using a single plane calibration, multi-plane calibration can provide richer constraint information, thereby improving the accuracy and reliability of calibration.

[0066] Embodiment 1:

[0067] In the first aspect, the present invention first proposes a method for real-time marking of the spatial position of an ultrasonic sensor, see Figure 1 , the method comprising:

[0068] S1. Based on the pre-built optical positioning system, establish the relative position relationship between the ultrasound image coordinate system and the camera coordinate system, and obtain the optical image of the target object;

[0069] S2. Acquire a position feature map of key points in the target object and a posture of the target object based on the optical image;

[0070] S3. Acquire a marker map of the target object based on the optical image and the position feature map.

[0071] The present embodiment proposes a method for real-time marking of the spatial posture of an ultrasonic sensor. First, based on a pre-built optical positioning system, a relative position relationship between the ultrasonic image coordinate system and the camera coordinate system is established, and an optical image of the target object is obtained at the same time; then, a position feature map of key points in the target object is obtained based on the optical image; and finally, a marking map of the target object is obtained based on the above optical image and position feature map. The technology for real-time marking of the spatial posture of an ultrasonic sensor proposed in this embodiment solves the problem that the prior art can only manually record body surface marks when saving ultrasonic images, which causes a large workload and is cumbersome for the operator. Compared with the prior art, ultrasonic images can be marked more efficiently and accurately.

[0072] In principle, the method proposed in this application is applicable to any application scenarios that use ultrasonic sensors to scan target objects and then mark the spatial posture of the target objects in real time, such as recording the probe posture information corresponding to ultrasonic diagnostic images, and recording the probe posture during handheld (freehand) three-dimensional ultrasonic imaging, etc., including but not limited to the use of ultrasonic diagnostic image scanning for disease diagnosis in the medical field, and can also be used for non-destructive testing such as parts inspection and flaw detection in the industrial field, etc. For ease of understanding, the following is a detailed description and explanation using the example of obtaining real-time marking of human ultrasound images in clinical examinations, but this specific implementation does not limit the scope of protection of the technical solution of this application.

[0073] The following is combined with Figure 1-6 , and explanations of specific steps S1-S4 are provided to describe in detail the implementation process of an embodiment of the present invention and its related preferred embodiments.

[0074] S1. Based on the pre-built optical positioning system, the relative position relationship between the ultrasound image coordinate system and the camera coordinate system is established, and the optical image of the target object is obtained.

[0075] S11. Building an optical positioning system, wherein the optical positioning system includes an optical tracker, a camera, and an optical marker.

[0076] The optical positioning system consists of an optical tracker, a camera, and an optical marker. Generally speaking, the optical tracker and the optical marker need to be used together, such as a multi-eye camera and a metal ball with a diffuse reflection surface, an optical camera, and a calibration plate with a chessboard or dot matrix. The camera needs to be able to clearly capture the shape of the scanned object to obtain an optical image of the target scanned object (such as the patient's position). The optical tracker and camera are generally fixed above the ultrasound scanning operation area with a robotic arm or a support frame.

[0077] In this embodiment, preferably, the multi-eye camera is an optical tracker, and a metal ball with a diffuse reflection surface is used as an optical marker, specifically as follows: Figure 3 shown. But Figure 3 The combination shown is not intended to be a limitation on the optical tracker and the optical marker. In other embodiments, the optical tracker and the optical marker may be combined in other forms according to actual needs.

[0078] The optical marker needs to be fixed on the ultrasonic sensor used for scanning, but the specific fixing position is not limited, as long as the camera can clearly capture the image of the optical marker during the scanning process. For example, the optical marker can be fixed at the connection between the tail of the ultrasonic sensor and the data cable, or it can be fixed on the ultrasonic sensor with a bracket in an unobstructed position during the scanning process. Figure 3 FIG. 1 is a schematic diagram of fixing a metal ball on an ultrasonic sensor using a bracket.

[0079] S12. Calibrate the relative position relationship between the optical marker coordinate system and the ultrasound image coordinate system based on the optical positioning system.

[0080] After the optical tracker and the optical marker are fixed, the relative position relationship between the coordinate system of the optical marker and the coordinate system of the image obtained by the ultrasonic sensor (i.e., the ultrasonic image) needs to be calibrated. The usual steps of calibration are to use an ultrasonic sensor with an optical marker to scan a model with known parameters, while the optical tracker locates the optical marker, and calculates the relative position relationship between the image coordinate system obtained by the ultrasonic sensor and the coordinate system of the optical marker through the imaging position of the model in the image and the position in the actual space.

[0081] In this embodiment (i.e., when the multi-camera is used as an optical tracker and the metal ball with a diffuse reflection surface is used as an optical marker), first, the Zhang Zhengyou calibration method or the ball calibration method is used to calibrate the internal parameter Z of the multi-camera. C , and then use binocular positioning and other methods to locate the optical marker. Through N-line calibration, multi-plane calibration and other methods, the relative position relationship between the metal ball optical marker coordinate system and the ultrasound image coordinate system is calculated, that is, the transformation matrix Rt I2M .

[0082] S13. Based on the relative position relationship between the optical marker coordinate system and the ultrasound image coordinate system, a relative position relationship between the ultrasound image coordinate system and the camera coordinate system is established. Meanwhile, the optical camera captures an optical image of the target object.

[0083] After completing the above steps, the operator holds the ultrasonic sensor with the optical marker to perform ultrasonic testing. When scanning the target object (in this embodiment, the body of the examinee being scanned) and completing the ultrasound image to be saved, the multi-eye camera needs to clearly locate the metal ball, and the optical camera can clearly capture the optical image of the examinee's body position. When the operator presses the save option, the multi-eye camera captures and locates the metal ball, and obtains the relative position of the optical marker coordinate system in the camera coordinate system, using the transformation matrix Rt M2C According to the relative position relationship Rt between the optical marker coordinate system and the ultrasound image coordinate system obtained by calibration in S11 I2M , the relative position of the ultrasound image coordinate system in the camera coordinate system is calculated by the following formula, which can be expressed by the transformation matrix as follows:

[0084] R I2C =Rt M2C ·Rt I2M

[0085] Among them, Rt I2CRepresents the relative position relationship transformation matrix of the ultrasound image coordinate system (I) in the camera coordinate system (C); Rt M2C Represents the relative position transformation matrix of the optical marker coordinate system (M) in the camera coordinate system (C); Rt I2M Represents the relative position relationship transformation matrix between the ultrasound image coordinate system (I) and the optical marker coordinate system (M). Among them, "I2C" means image to camera; "M2C" means mark to camera; "I2M" means Image to mark.

[0086] The relative position relationship transformation matrix Rt based on the ultrasound image coordinate system in the camera coordinate system I2C It can be obtained that a specific pixel P in the ultrasound image i The position P in the camera coordinate system c It can be calculated by the following formula:

[0087] P c =Rt I2C ·Z I ·P i

[0088] Among them, P c represents the position of the pixel in the ultrasound image in the camera coordinate system; P i Indicates the position of the pixel in the ultrasound image; Z I It is a parameter related to the ultrasound image, which at least reflects the actual length and width of the pixels in the ultrasound image.

[0089] At the same time, the optical camera takes an optical image of the human body of the person being examined. The computer saves the relative position relationship Rt of the ultrasound image in the camera coordinate system at this time. I2C And the corresponding optical images of human body positions.

[0090] S2. Acquire a position feature map of key points in the target object based on the optical image.

[0091] The specific posture of the scanned target object in the optical image, that is, the specific posture of the scanned person, is determined by a pre-trained first deep learning network. The pre-trained first deep learning network includes but is not limited to an HR-Net backbone network and a classification module.

[0092] In this embodiment, preferably, the pre-trained first deep learning network is composed of an HR-Net backbone network and a classification module, and its structure is as follows: Figure 4 shown.

[0093] The input data of the pre-trained first deep learning network is the optical image of the target object captured in S2. The classification module consists of a convolutional layer, a normalization layer, an activation layer, and a fully connected layer. The HR-Net backbone network outputs a position feature map of the key points of the scanned target object. Figure 1 On the one hand, it is used for the subsequent classification module to further identify the posture information, and on the other hand, it is used to input into the subsequent pre-trained deep learning network to assist in acquisition. In this embodiment, the key points of the above-mentioned parts of the scanned person include but are not limited to the head points, chest points, abdomen points, shoulder joints, elbow joints, hip joints, knee joints, etc.

[0094] In this embodiment, the classification module identifies the specific posture of the target object in the optical image through the feature map and outputs the category of the posture, such as side-lying state, lying flat state, etc. Since the deep learning network has two outputs, the loss function of the training is a weighted superposition of the loss functions of the two. Among them, the loss function of the output feature map is the mean square error between the image and the real point, and the loss function of the output posture information is the cross entropy loss.

[0095] The HR-Net network can perform 2D human posture estimation and obtain the predicted human posture by predicting the heat map of the key points of the human body in the image. The HR-Net network can effectively capture information of different scales through parallel connection of high-resolution to low-resolution convolution flows, making spatial prediction more accurate. The classification module consists of simple convolution, normalization, activation function, and fully connected layers. Since HR-Net has obtained the feature heat map, there is no need to use a complex network structure. Through subsequent prediction of the feature heat map output by HR-Net, the human posture is classified into several postures that often appear in examinations, such as "lying on the side" and "lying flat".

[0096] S3. Acquire a marker map of the target object based on the optical image and the position feature map.

[0097] The specific body position of the patient in the optical image is determined by the pre-trained second deep learning network, and a corresponding human body marker map is generated. The specific structure of the pre-trained second deep learning network is as follows: Figure 5 As shown in the figure. Based on the U-Net network structure, it consists of a feature encoder and a decoder. The input data is the optical image taken in S1 and the position feature map of the key points of the target object output by the HR-Net backbone network in S2. Among them, the backbone network of the feature encoder part can be composed of a full convolution module, a residual module (Res-Block), etc.

[0098] The U-Net network uses a U-shaped network structure. The encoder part gradually extracts the features of the image and reduces the spatial resolution. The decoder part restores the feature map to the size of the input image through upsampling operations and generates segmentation results. Due to the existence of the skip link mechanism, the network can effectively capture feature information at different levels, thereby improving the accuracy of segmentation.

[0099] In order to obtain a higher quality labeled map of the target object, in this embodiment, a preferred method is to add an attention module before the encoder in the U-Net network. The attention module is mainly composed of multi-scale convolutions, and the position feature map obtained in S2 is converted into an attention map of the same size as the feature map output by the encoder of the previous scale, and weighted superposition is performed with it, and then the encoder feature calculation of the scale is performed. Adding an attention module before the encoder in the U-Net network multiplies the feature map by the attention weight, so that the network has a higher weight on the key points.

[0100] Due to the introduction of attention weights, the backbone network of the encoder part may pay too much attention to the parts with high weights in the feature map. In order to expand the convolution field of the encoder, the convolution of the encoder part can be used with dilated convolution (for example, in the shallow layers such as the first and second layers of the encoder module, 5×5 dilated convolution is used instead of ordinary 3×3 convolution to expand the convolution field of view; while the input of the deep encoder module is the result of the convolution of the upper layer, and the convolution will reduce the area of ​​the tensor, so ordinary convolution can be used), and a dropout layer is set in a specific encoder layer. For example, a dropout layer is added after the first layer of the encoder, and the generalization ability of the model is improved by setting random dropout weights for each feature unit. Since the area of ​​the deep feature tensor is small, random dropout will lose a lot of information, avoiding overfitting (especially overfitting of personal data) when the training data set sample size is too small, and improving the generalization ability of the model.

[0101] After the U-Net network outputs the target object's labeling result map, the specific position information of the ultrasonic sensor relative to the detected object is determined based on the semantic segmentation information of the ultrasonic sensor and the target part in the map. Combined with the relative position relationship between the optical tracker and the optical camera known in the optical positioning system in S1, the corresponding labeling map is selected or generated.

[0102] In order to facilitate operators or subsequent researchers to more intuitively see the mark of the target object in the ultrasound image, a more preferred way is that the method for real-time marking of the spatial posture of the ultrasound sensor proposed in this embodiment also includes:

[0103] S4. Generate an indication mark on the marking image, wherein the indication mark includes an ultrasonic sensor position indication mark and a scanning direction indication mark.

[0104] In the marker map (ie, human body marker map) generated by S3, the indication marks of the ultrasonic sensor position and scanning direction are recorded.

[0105] For example, in this embodiment, the transformation matrix Rt of the ultrasound image coordinate system in the camera coordinate system calculated according to S1 is I2C , the position of the left side of the ultrasound image (the side with the ultrasonic sensor mark) in the camera coordinate system can be obtained. Combined with S2 and S3, the posture information of the human body during scanning and the position of the target part of the human body in the camera coordinate system are obtained through the deep learning network, and the indicator mark of the ultrasonic sensor position and the scanning direction is recorded in the human body mark map. This mark can clearly show the position of the left side of the image during scanning (the side with the ultrasonic sensor mark) and the scanning direction. In this embodiment, the indicator mark is represented by a "T"-shaped icon, and the horizontal line part of the "T" shape represents the left side of the ultrasound image, that is, the side with the ultrasonic sensor mark. For details, please refer to Figure 6 .

[0106] At this point, the entire process of the method for real-time marking of the spatial posture of the ultrasonic sensor proposed in this embodiment and its preferred embodiments is completed.

[0107] Embodiment 2:

[0108] In a second aspect, the present application also proposes a system for real-time marking of the spatial position of an ultrasonic sensor, the system comprising:

[0109] The position relationship acquisition module is used to obtain the relative position relationship of the ultrasound image coordinate system in the camera coordinate system based on the pre-built optical positioning system.

[0110] An optical image acquisition module, used to acquire an optical image of a target object based on a pre-built optical positioning system;

[0111] A position feature map acquisition module, used to acquire a position feature map of key points in a target object based on the optical image;

[0112] A label map acquisition module is used to acquire a label map of the target object based on the optical image and the position feature map.

[0113] When the modules of the system perform their respective functions, they implement the steps of the method for real-time marking of the spatial position of the ultrasonic sensor as described above. The method mainly includes:

[0114] S1. Based on the pre-built optical positioning system, establish the relative position relationship between the ultrasound image coordinate system and the camera coordinate system, and obtain the optical image of the target object;

[0115] S2. Acquire a position feature map of key points in the target object based on the optical image;

[0116] S3. Acquire a marker map of the target object based on the optical image and the position feature map.

[0117] It can be understood that the system for real-time marking of the spatial posture of an ultrasonic sensor provided in an embodiment of the present invention corresponds to the method for real-time marking of the spatial posture of an ultrasonic sensor. The explanations, examples, beneficial effects and other parts of the relevant contents can refer to the corresponding contents in the method for real-time marking of the spatial posture of an ultrasonic sensor, and will not be repeated here.

[0118] Embodiment 3:

[0119] In a third aspect, the present application further proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for real-time marking of the spatial position of the ultrasonic sensor as described above are implemented. The method mainly includes:

[0120] S1. Based on the pre-built optical positioning system, establish the relative position relationship between the ultrasound image coordinate system and the camera coordinate system, and obtain the optical image of the target object;

[0121] S2. Acquire a position feature map of key points in the target object based on the optical image;

[0122] S3. Acquire a marker map of the target object based on the optical image and the position feature map.

[0123] It can be understood that the electronic device for real-time marking of the spatial posture of an ultrasonic sensor provided in an embodiment of the present invention corresponds to the method for real-time marking of the spatial posture of an ultrasonic sensor. The explanations, examples, beneficial effects and other parts of the relevant contents can refer to the corresponding contents in the method for real-time marking of the spatial posture of an ultrasonic sensor, and will not be repeated here.

[0124] Embodiment 4:

[0125] In a fourth aspect, the present application finally proposes a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for real-time marking of the spatial posture of an ultrasonic sensor as described above are implemented. The method mainly includes:

[0126] S1. Based on the pre-built optical positioning system, establish the relative position relationship between the ultrasound image coordinate system and the camera coordinate system, and obtain the optical image of the target object;

[0127] S2. Acquire a position feature map of key points in the target object based on the optical image;

[0128] S3. Acquire a marker map of the target object based on the optical image and the position feature map.

[0129] It can be understood that the storage medium for real-time marking of the spatial posture of an ultrasonic sensor provided in an embodiment of the present invention corresponds to the method for real-time marking of the spatial posture of an ultrasonic sensor. The explanations, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the method for real-time marking of the spatial posture of an ultrasonic sensor, and will not be repeated here.

[0130] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0131] 1. The present invention proposes a method for real-time marking of the spatial posture of an ultrasonic sensor. First, based on a pre-built optical positioning system, the relative position relationship between the ultrasonic image coordinate system and the camera coordinate system is established, and an optical image of the target object is obtained at the same time; then, a position feature map of key points in the target object is obtained based on the optical image; finally, a marking map of the target object is obtained based on the above optical image and position feature map. The technology for real-time marking of the spatial posture of an ultrasonic sensor proposed in this application solves the problem that the prior art can only manually record body surface marks when saving ultrasonic images, which causes a large workload and is cumbersome for the operator. Compared with the prior art, it can mark ultrasonic images more efficiently and accurately, which facilitates subsequent analysis and research.

[0132] 2. When acquiring the label map of the target object based on the optical image and the position feature map, the present application utilizes a pre-trained second deep learning network composed of a feature encoder and a decoder based on the U-Net network structure, and adds an attention module before the encoder in the U-Net network, which can not only effectively capture feature information at different levels, thereby improving the accuracy of segmentation, but also enable the network to have a higher weight on key points; at the same time, the convolution of the encoder part uses a dilated convolution to expand the convolution field of view of the encoder. Furthermore, a dropout layer is set at a specific encoder layer to avoid overfitting and improve the generalization ability of the model.

[0133] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0134] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time marking of the spatial position of an ultrasonic sensor, characterized in that: The method comprises: S1. Based on the pre-built optical positioning system, establish the relative position relationship between the ultrasound image coordinate system and the camera coordinate system, and obtain the optical image of the target object; S2. Acquire a position feature map of key points in the target object based on the optical image; S3. Acquire a marker map of the target object based on the optical image and the position feature map.

2. The method according to claim 1, characterized in that The S1 includes: S11, building an optical positioning system; the optical positioning system includes an optical tracker, a camera and an optical marker; S12, calibrating the relative position relationship between the optical marker coordinate system and the ultrasound image coordinate system based on the optical positioning system; S13. Based on the relative position relationship between the optical marker coordinate system and the ultrasound image coordinate system, a relative position relationship between the ultrasound image coordinate system and the camera coordinate system is established; at the same time, an optical image of the target object is captured using an optical camera.

3. The method according to claim 2, characterized in that The S13 includes calculating according to the following formula to establish the relative position relationship of the ultrasound image coordinate system in the camera coordinate system: Rt I2C =Rt M2C ·Rt I2M Among them, Rt I2C Represents the relative position relationship transformation matrix of the ultrasound image coordinate system in the camera coordinate system; Rt M2C Represents the relative position relationship transformation matrix of the optical marker coordinate system in the camera coordinate system; Rt I2M The transformation matrix represents the relative position relationship between the ultrasound image coordinate system and the optical marker coordinate system.

4. The method according to claim 1, characterized in that The S2 includes: Inputting the optical image into a pre-trained first deep learning network to obtain a position feature map of key points in the target object; The first deep learning network includes an HR-Net backbone network and a classification module.

5. The method according to claim 1, characterized in that The S3 includes: Inputting the optical image and the position feature map into a pre-trained second deep learning network to obtain a label map of the target object; The second deep learning network is based on a U-Net network structure and includes a feature encoder and a decoder.

6. The method according to claim 5, characterized in that The second deep learning network also includes: Adding an attention module before the encoder in the U-Net network, wherein the attention module includes multi-scale convolution; The convolution of the encoder part uses a dilated convolution, and a dropout layer is set at a specific encoder layer.

7. The method according to claim 1, characterized in that The method further comprises: S4. Generate an indication mark on the marking diagram; the indication mark includes an ultrasonic sensor position indication mark and a scanning direction indication mark.

8. A system for real-time marking of the spatial position of an ultrasonic sensor, characterized in that: The system comprises: The position relationship acquisition module is used to obtain the relative position relationship of the ultrasound image coordinate system in the camera coordinate system based on the pre-built optical positioning system. An optical image acquisition module, used to acquire an optical image of a target object based on a pre-built optical positioning system; A position feature map acquisition module, used to acquire a position feature map of key points in a target object based on the optical image; A label map acquisition module is used to acquire a label map of the target object based on the optical image and the position feature map.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for real-time marking of the spatial posture of an ultrasonic sensor as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for real-time marking of the spatial pose of an ultrasonic sensor as claimed in any one of claims 1 to 7 are implemented.

Images