Method and system for identifying position information change of surgical body surface positioning piece

Abstract

The method for recognizing the position information change of the surgical body surface positioning piece comprises the following steps that a micro camera is embedded in the tail end of a needle inserting device at the tail end of a mechanical arm of a puncture surgical robot, and the micro camera continuously shoots the positioning piece located on the body surface of a patient within the time period of the puncture needle inserting process; obtaining all to-be-processed images, performing initialization processing on the images, and outputting the images as first to-be-processed images in sequence; all the first to-be-processed images are sequentially sent to a pre-trained first sample training model, X and Y position information of the center points of the positioning pieces in the image plane in all the first to-be-processed images is inferred, and a second to-be-processed image is formed; sending all the second to-be-processed images to a second sample training model in sequence, and reasoning longitudinal Z position information of the center point of the positioning piece; and outputting the X, Y and Z position information of the central points of the positioning pieces in all the second to-be-processed images in sequence to form a real-time output data stream.

Description

Technical Field

[0001] The present invention relates to the field of puncture surgical robots, and in particular to a method and system for identifying changes in position information of a body surface positioning component used for surgery. Background Art

[0002] With the rapid development of medical technology, the working process of the puncture surgical robot is to plan and determine the lesion location, puncture target point, surface needle entry point and puncture path based on the patient's CT image, combined with the doctor's diagnosis and the auxiliary diagnosis and treatment functions provided by the surgical robot, and then send data and instructions to the needle insertion device installed at the end of the puncture surgical robot's mechanical arm. The needle insertion device drives the puncture needle to move accurately to the specified position, and then enters from the surface needle entry point to the puncture target point at the lesion location.

[0003] At present, observation of the surgical needle entry point (locating piece) on the patient's body surface is mostly achieved through a camera (optical tracking system or infrared camera) installed at a distance, that is, the needle entry point is found by locating the center point of the locating piece with the help of the camera. However, during the needle insertion process of the puncture surgery after positioning, the locating piece cannot be further positioned (usually referred to as microscopic observation) using the camera (optical tracking system or infrared camera) installed at a distance, that is, the insertion process of the puncture needle is in an unmonitored state. If the position of the locating piece is not further tracked and located, although the needle insertion process only takes 2-4 seconds, if the patient moves and / or moves abnormally within the 2-4 seconds, it will have a significant impact on the puncture surgery. In addition, there is no monitoring during the moment of needle insertion, which is a blind spot for the surgery and cannot ensure the safety of the surgery. At the same time, no records are made of the surgery process.

[0004] Movement refers to the movement of organs caused by breathing during deep breathing control or deep breath holding control, thereby causing the positioning parts located on the patient's body surface to move.

[0005] Dysfunction refers to any unexpected movement during surgery, including that of the patient or other interfering factors, which causes unnecessary displacement and poses a surgical risk.

[0006] The puncture speed is so fast that the naked eye needs to slow down the video 10 times to observe it. Using a camera installed at a distance cannot achieve real-time tracking, positioning and observation. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for identifying changes in the position information of a body surface positioning member for surgery. The identification method and system of the present invention can realize real-time tracking, positioning and observation of the positioning member during the puncture process, so that the puncture needle is always in an observable / controllable / traceable state during the needle insertion process.

[0008] The method for identifying the change of the position information of the body surface positioning member for surgery in the present invention comprises the following steps: S1: A miniature camera is embedded at the end of the needle insertion device at the end of the puncture surgery robot mechanical arm, and the puncture surgery robot system is initialized and the environment is set; S2: the micro camera continuously photographs the positioning member located on the patient's body surface during the time period of the puncture needle insertion process, the positioning member is used for positioning the puncture robot during the puncture operation, all the images to be processed are acquired, all the acquired images to be processed are initialized, and are sequentially output as the first images to be processed; S3: sending all first images to be processed to a pre-trained first sample training model in sequence, and inferring the X and Y position information of the center points of the positioning parts in all first images to be processed in the image plane according to the first sample training model, to form a second image to be processed including the X and Y position information; S4: sending all the second images to be processed to the second sample training model in sequence, first inferring the longitudinal depth values ​​of all the second images to be processed from the micro camera according to the depth estimation method in the second sample training model, and further inferring the longitudinal Z position information of the center point of the positioning element; S5: Outputting the X, Y, and Z position information of the center point of the positioning member in all the second images to be processed in sequence to form a real-time output data stream, and further making a displacement curve that can be presented on the display screen.

[0009] Preferably, a step of determining whether the video transmission stream is normal is added after step S1. If it is abnormal, the program is closed and the puncture surgical robot exits the operation; if it is normal, the following steps are performed.

[0010] Preferably, in step S2, the first frame of the image to be processed is first converted into a first array with three channels, that is, the image is digitally processed. The first array with three channels contains the content information of each pixel point, color, and size of the first frame of the image to be processed, and the coordinate values ​​of the three channels in the first array are defined. Then, based on the coordinate values ​​of the three channels in the first array, the subsequent second frame of the image to be processed is converted into a second array with three channels. Based on the second array, the third frame of the image to be processed is converted into a third array with three channels. In this way, all the acquired images to be processed are initialized in sequence.

[0011] Preferably, all first to-be-processed graphs are repeatedly executed in the first sample training model according to the following steps in sequence: S3.1) converting the first image to be processed into a grayscale image, and applying Gaussian filtering for smoothing and noise reduction; S3.2) setting the dynamic threshold to 125-255, and performing dynamic threshold processing on the first image to be processed after the processing in step S3.1); S3.3) scanning the first image to be processed after being processed in step S3.2) using a canny edge detection algorithm, enhancing the shape, features and contour edges of the positioning member in the first image to be processed, and decomposing the image into smaller segments or regions according to the image features of the positioning member to form a plurality of first target regions; S3.4) connecting the contours of the plurality of first target regions to form a plurality of separate regions; S3.5) selecting a second target area from the plurality of separate areas according to the area size of the positioning element, drawing the contour of the second target area, and fitting the contour of the second target area according to a preset ellipse formula to form a plurality of ellipse frames; S3.6) Select a target ellipse frame according to the ratio of the major axis to the minor axis of the ellipse, and obtain the coordinate value of the center point of the target ellipse frame, which is the X, Y position information of the center point of the positioning member in the plane.

[0012] Preferably, in step S3.6), an ellipse having a major-minor axis ratio of 0.8-1.2 is selected.

[0013] Preferably, the training process of the second sample training model is as follows: To obtain training data, a large number of unlabeled monocular images are collected from public large-scale datasets, called pseudo-labeled sets, denoted as Du = {ui} N i=1; at the same time, a large number of labeled images are collected from public large-scale datasets, called labeled sets, denoted as Dl = {(xi, di)} M i=1; Construct a monocular deep recognition network, which includes a first set of encoders and a second set of decoders; A powerful monocular depth estimation model S1 is trained using the constructed monocular depth recognition network as follows: 3.1) Training of the initial depth estimation model Pre-train an initial depth estimation model T using labeled images from the labeled set D1; 3.2) Generate pseudo labels Use the initial depth estimation model T to predict the unlabeled images in the pseudo-label set Du and generate a pseudo depth label d * i ; 3.3) Joint Training Combine the labeled images in the labeled set D1 and the unlabeled images in the pseudo-label set Du to form a joint dataset D = Dl ∪ Du; 3.4) Training of depth estimation model Use the joint dataset D to train the initial depth estimation model T to obtain a depth estimation model S; 3.5) Feature Alignment Loss Use a 3D acquisition device to collect the grayscale image and depth image of the positioning part, use the pre-trained DINOv2 model to extract the features of the positioning part, and align the features extracted by the depth estimation model S with the features extracted by the DINOv2 encoder according to the following formula; in: cos(f i , f' i ) measures the cosine similarity between two feature vectors; f i Features extracted for the depth estimation model S; f' i are the features extracted from the DINOv2 model; 3.6) Through multiple rounds of training and optimization in steps 3.1-3.5), a powerful monocular depth estimation model S1 is obtained, which is the second sample training model.

[0014] The system for identifying the position information change of the body surface positioning member for surgery in the present invention comprises a preprocessing module, a startup module, a recognition module, a deep reasoning module and an output module. The preprocessing module consists of a package deployment environment preheating module and a neural network training module, and is used for the initialization and environment configuration of the puncture surgery robot system; The startup module is composed of a strong logic judgment module and a data stream monitoring module, which is used to judge whether the video stream shot by the micro camera can be transmitted normally. If the transmission stream cannot be found during the judgment process, the program is closed and exited; The recognition module is a first sample training module, which is used to recognize the X and Y position information of the center point of the positioning element in each frame of the image transmitted by the start module within the plane where the image is located; The depth reasoning module is a second sample training module, which uses zero-sample Panhua depth estimation model reasoning to identify the Z position information between the center point of the positioning part and the longitudinal direction of the micro camera in each frame of the image transmitted by the recognition module; The output module is used to synchronously output the X, Y position information obtained by the first sample training module and the Z position information obtained by the second sample training module to the display screen of the puncture surgery robot system to form a real-time output data stream.

[0015] In the field of thoracic and abdominal puncture surgery navigation, soft parts such as the chest and abdomen are large in size and fluctuate with the patient's posture and breathing. The method of the present invention can flexibly / unobstructedly track the needle insertion point of the positioning member used for positioning in real time. By using a high-precision, high-real-time positioning and tracking system, the movement error of the diseased organs in the body caused by observing the breathing fluctuations in close contact with the body surface can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the method for identifying changes in position information of a body surface positioning component for surgery in the present invention.

[0017] Figure 2 It is a schematic diagram of the method for identifying changes in position information of a body surface positioning component for surgery in the present invention.

[0018] Figure 3 It is a structural block diagram of the system for identifying changes in position information of a body surface positioning component for surgery in the present invention.

[0019] Figure 4 It is a working principle diagram of the method for identifying changes in position information of a body surface positioning member for surgery in the present invention.

[0020] Figure 5 It is a three-dimensional structural schematic diagram of the positioning member in the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of protection of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention. At the same time, the relational terms such as "first" and "second" do not have a substantial order relationship and are only used to distinguish different components.

[0022] like Figure 4 and Figure 5 As shown, the positioning member 1 in the present invention is made of aluminum oxide ceramic material, is white in color, has an outer diameter of 30 mm and a thickness of 5 mm, is provided in the middle with a center circle 2, and has a diameter of 10 mm. An inverted conical concave arc surface is formed between the inner wall surface of the center circle 2 and the outer circumference of the positioning member 1.

[0023] The method for identifying the position information change of the body surface positioning part for surgery in the present invention first embeds a micro camera at the end of the needle insertion device installed at the end of the mechanical arm of the puncture surgery robot. The shooting direction of the micro camera is parallel to the installation direction of the puncture needle, and the installation position of the micro camera is 10-30mm away from the vertical position of the puncture needle after installation, so that the movement process of the puncture needle is completely within the shooting range of the micro camera. Preferably, a micro camera with a focal length of 5-50mm, 30fps (Frames Per Second) and a diameter of only 1.8mm is embedded and installed at the end of the needle insertion device. The micro camera head has 4 LED light beads for illuminating the local environment during the operation. The micro camera in this embodiment is installed at the end of the mechanical arm. When the micro camera moves with the mechanical arm to a position 10-20mm away from the patient's body surface, it starts working, has the advantages of clear observation, can change with the surgical posture without being blocked, and can clearly capture the whole process of whether the positioning part changes position during the insertion of the puncture needle. The micro camera in this embodiment can be used to continuously capture multiple frames of 2D color images with a resolution of 400*400 and a resolution of 480p. The method and system of the present invention can be used to obtain the spatial position information of the center point of the positioning member in the image, that is, the X, Y, and Z position coordinate information. By continuously outputting the spatial position information of the positioning member in each frame of the image, close observation can be achieved, and the captured image can be transmitted in real time, which has good real-time performance and can effectively solve the problem of being unable to observe the entire process of puncture needle insertion in real time.

[0024] Based on the installation of the micro camera, the puncture surgical robot system needs to be initialized and the environment set up first, specifically importing the necessary libraries and configuration environment, such as: V4L2, RTSP, go2rtc, opencv or capvideo, and loading the pre-trained first sample training model and second sample training model (also called monocular depth estimation algorithm model or monocular depth estimation neural network training model).

[0025] When the puncture surgical robot system is started, the environment of the system for identifying the position information change of the surgical body surface positioning component in the present invention is preheated and necessary package libraries are deployed to ensure the stability of the subsequent processing flow.

[0026] like Figure 3 As shown, the system for identifying changes in position information of a body surface positioning component for surgery in the present invention includes a preprocessing module, a startup module, a recognition module, a deep reasoning module and an output module.

[0027] The preprocessing module consists of a package deployment environment preheating module and a neural network training module, which is used for the initialization and environment configuration of the puncture surgical robot system, and performs environment preheating and package deployment of the necessary package library for the system for identifying changes in position information of surgical surface locators in the present invention, thereby ensuring the stability of subsequent processing procedures.

[0028] The startup module consists of a strong logic judgment module and a data flow monitoring module, which are used to determine whether the video stream captured by the micro camera can be transmitted normally. If the video transmission stream cannot be found during the judgment process, the program is closed and exited, so as to further determine whether the recognition system in the present invention can operate normally.

[0029] The recognition module is the first sample training module, which is used to identify the X, Y position information of the center point of the positioning part in each frame image transmitted by the starting module in the plane where the image is located. The recognition module can be a pre-trained deep learning module or an image processing module that processes the image to be processed transmitted by the starting module.

[0030] The depth reasoning module is the second sample training module, which applies zero-sample Panhua depth estimation model reasoning to identify the Z position information between the center point of the positioning part and the longitudinal direction of the micro camera in each frame image transmitted by the recognition module.

[0031] The output module is used to synchronously output the X, Y position information obtained by the first sample training module and the Z position information obtained by the second sample training module to the display screen of the puncture surgery robot system. The position change of the needle insertion point on the patient's body surface can be further output through the coordinate values ​​of the X, Y, and Z of the center point of the positioning part in the image. With the help of the first frame image, the second frame image, the third frame..., multiple continuous frames form a real-time output data stream. After the puncture surgery robot system collects the data, a displacement curve is made based on the data. If the coordinate value of the center point of the positioning part does not change, a straight line will be presented on the display screen. If it changes, a displacement curve of change will be presented based on the specific values ​​of X, Y, and Z, which is convenient for medical staff to observe in real time, so that they can observe whether the patient's position changes during the puncture operation. Embodiment 1

[0032] like Figure 1 and Figure 2 As shown, the method for identifying changes in position information of a body surface positioning component for surgery in the present invention is performed according to the following steps.

[0033] S1: A miniature camera is embedded at the end of the needle insertion device at the end of the mechanical arm of the puncture surgery robot, and the puncture surgery robot system is initialized and the environment is set up at the same time; the end of the needle insertion device refers to the bottom of the needle insertion device facing the patient, and the installation position of the miniature camera is 20mm away from the vertical position of the puncture needle after installation, and the shooting direction of the miniature camera is parallel to the puncture needle.

[0034] S2: The micro camera takes real-time (continuous) photos of the positioning member 1 located on the patient's body surface within 2-4 seconds of the puncture needle insertion process. The positioning member is used to assist the puncture robot in positioning the patient's chest and abdomen during the puncture operation, and obtains all the captured 2D images to be processed. Under normal conditions, there are 20-60 frames of images to be processed, and 30 frames in this embodiment, and all the images to be processed are transmitted to the recognition module in real time and in sequence. The specific process of this step is as follows.

[0035] S2.1: The startup module first determines whether the video transmission stream is normal. If not, the program is closed and the puncture robot exits the surgery. If normal, proceed to the next step.

[0036] S2.2: Get the video stream from the RTSP source through the V4L2 interface or open the video file through the opencv library, get each frame of the video stream, that is, all the images to be processed, initialize all the images to be processed according to the transmission order, use the coordinate value of the previous frame of the image during the initialization process, and then read each frame of the image in sequence for repeated processing.

[0037] The specific initialization processing process is: first, the first frame image taken (the first to be transmitted) is converted into a first array with three channels, that is, the digital processing of the image. The first array with three channels contains each pixel point, color, size and other content information of the first frame image. At the same time, the coordinate values ​​of the three channels in the first array are defined by the system. Then, according to the coordinate values ​​of the three channels in the first array, the subsequent second frame image is converted into a second array with three channels. Based on the second array, the third frame image is converted into a third array with three channels. In this way, all the acquired images to be processed are initialized in sequence.

[0038] S2.3: Filter out all first images to be processed including the positioning piece from all images to be processed that have completed the initialization processing. Since the positioning piece 1 has a specific shape, color and size, all first images to be processed including the positioning piece can be filtered out according to the characteristics of the positioning piece in the image (for example, white, round, and the image area occupies 50,000-500,000 pixels). Then all the first images to be processed are transmitted to the recognition module in sequence, which can accelerate the processing speed of the recognition module, reduce the processing time, and achieve the purpose of real-time observation and monitoring.

[0039] S3: All first images to be processed are sent to the recognition module in sequence, and the pre-trained first sample training model extracts the contour data of the positioning part from each frame of the first image to be processed, and infers the X, Y position information of the center point of the positioning part in all first images to be processed in the image plane.

[0040] In this step S3, since the first sample training model is pre-trained through a neural network training module and integrated with a deep learning algorithm, it is only necessary to input all the first images to be processed in step S2 to directly output the X and Y coordinate values ​​of the positioning parts in each frame of the image, that is, to output the second image to be processed including the X and Y position information.

[0041] The training process of the first sample training model is as follows: a) The video recorded and shot in advance is captured frame by frame using a video tool, and each frame of the image is annotated to form an annotated data set. In this embodiment, the positioning member 1 placed on the patient's body surface needs to be photographed at a position of 10-20 mm from the patient's body surface, that is, it is required that images shot at a distance are not recognized, and only images shot at a close distance are recognized. Therefore, images shot at a close distance and after recognition are selected as annotated samples and are marked as positive samples.

[0042] b) Send all positive samples to the neural network training, and set the required parameters, such as learning rate, optimizer, loss function, activation function, training batch size, training image size, etc. In this embodiment, there are 25 parameters that can be set by default. The specific training process is easy to implement for technicians in the computer field and will not be described in detail here.

[0043] c) After training, many misjudgments are found, and the video tool is returned to capture the misjudged image frames and marked as negative samples.

[0044] d) Combine the negative samples and the positive samples into a balanced data set. In this embodiment, the data sets of positive samples and negative samples are constructed in a 1:1 ratio, and then the labeled data sets are used for model training to obtain the first sample model training. This is easy to implement for technicians in the computer field and will not be described in detail here.

[0045] S4: All the second images to be processed are sent to the depth reasoning module in sequence. The pre-trained second sample training model first infers the depth distance of each frame of the second image to be processed from the micro camera according to the depth estimation method, and then further deduces the longitudinal Z position information between the center point of the positioning part and the micro camera in each frame of the second image to be processed.

[0046] The training process of the second sample training model is as follows.

[0047] To obtain training data, 62 million unlabeled monocular images are collected from 6-8 public large-scale datasets, called the pseudo-labeled set, denoted as Du = {ui} N i=1; at the same time, 1.5 million labeled images are collected from 6-8 public large-scale datasets, called the labeled set, denoted as Dl = {(xi, di)} M i=1.

[0048] A monocular depth recognition network is constructed, which includes a first and a second set of encoders and decoders; each set of encoders includes a local encoder and a global encoder. The local encoder in the first set of encoders is used to extract the local information of each frame image in the tag set D1 and generate a local feature map, and the global encoder in the first set of encoders is used to extract the global information of each frame image in the tag set D1 and generate a global feature map; the local encoder in the second set of encoders is used to extract the local information of each frame image in the pseudo tag set Du and generate a local feature map, and the global encoder in the second set of encoders is used to extract the global information of each frame image in the pseudo tag set Du and generate a global feature map; the decoder is used to sample the feature maps provided by the first and second sets of encoders to obtain a recognized depth map.

[0049] A powerful monocular depth estimation model S1 is trained using the constructed monocular depth recognition network as follows: 3.1) Training of the initial depth estimation model An initial depth estimation model (also called teacher model T) is pre-trained using 1.5 million labeled images in the labeled set D1.

[0050] 3.2) Generate pseudo labels Use the initial depth estimation model T to predict the unlabeled images in the pseudo-label set Du and generate a pseudo depth label (predicted depth value) d * i .

[0051] 3.3) Joint Training Combine the labeled images in the labeled set D1 and the unlabeled images in the pseudo-label set Du to form a joint dataset D = Dl ∪ Du. During this training process, the depth value is converted to the disparity space through a specific formula, specifically d = f / z, where d represents the disparity value, f represents the focal length of the camera, and z represents the depth value (that is, the actual distance of the object from the camera). The depth information is converted into disparity information on the image plane.

[0052] 3.4) Training of depth estimation model Use the joint dataset D to train the initial depth estimation model T to obtain a depth estimation model (called the student model S).

[0053] During the training process of the depth estimation model S, the affine invariant mean absolute error loss ρ is used to calculate the predicted depth value d in the initial depth estimation model T according to the following formula (1): * i and the true depth value d i The loss between them is calculated, and then the depth value is obtained by weighted average method formula (2).

[0054] (1) (2) Among them: d * i and d i are the predicted depth value and the real depth value obtained in the initial depth estimation model T respectively; ρ is the affine invariant mean absolute error loss; t(d) and s(d) are the predicted depth value and the true depth value obtained in the depth estimation model S, respectively.

[0055]

[0056]

[0057] During the training process of the depth estimation model S, strong perturbations (such as strong color distortion and strong spatial distortion) are injected into the unlabeled images, forcing the depth estimation model to seek additional visual knowledge and learn robust representations.

[0058] Combined with pseudo depth label d * i and strong perturbations, the unlabeled loss Lu of the unlabeled image is calculated according to the following formula.

[0059]

[0060] The two losses are added together using a weighted average:

[0061] 3.5) Feature Alignment Loss Use a 3D acquisition device to collect the grayscale image and depth map of the positioning part, and use the pre-trained DINOv2 model to extract the features of the positioning part, such as the specific shape, color and size of the positioning part. Use the following formula to align the features extracted by the depth estimation model S with the features extracted from the DINOv2 encoder.

[0062] in: cos(f i , f' i ) measures the cosine similarity between two feature vectors.

[0063] f i Features extracted for the depth estimation model S.

[0064] f' i are the features extracted from the DINOv2 model.

[0065] 3.6) Through multiple rounds of training and optimization in steps 3.1-3.5), a powerful monocular depth estimation model S1 is obtained, namely the second sample training model, which effectively utilizes the comprehensive information of labeled and unlabeled images to improve the accuracy and generalization ability of depth estimation. The robustness and semantic understanding ability of the model are further enhanced by the introduction of strong perturbation injection and feature alignment loss, so that the depth of the positioning parts can be quickly and accurately estimated in various scenarios.

[0066] S5: Synchronously output the X, Y position information of the center point of the positioning member in each frame image in step S3 and step S4 within the plane where the image is located and the longitudinal Z position information between the center point of the positioning member and the micro camera, and display them on the display screen of the puncture robot control system. Embodiment 2

[0067] The second embodiment is basically the same as the first embodiment, except that the processing method of step S3 is different. However, both can quickly and accurately output the X, Y position information of the center point of the positioning member in all the first images to be processed in the plane where the image is located, and the processing process for each frame of the first image to be processed is the same, and all the first images to be processed are processed sequentially. The processing process of the first frame of the first image to be processed is listed below, such as Figure 2 As shown, the details are as follows: 1) Convert the first image to be processed into a grayscale image, and apply Gaussian filtering to perform smoothing and noise reduction according to the following formula.

[0068]

[0069] f(x, y) is the function mapping the value 2) The first image to be processed is further subjected to dynamic threshold processing, and the threshold value set by the dynamic threshold processing is 125-255.

[0070] 3) Use the Canny edge detection algorithm to process the contours and edges of the image 3.1) Detecting edges: The first image to be processed is scanned using the Canny algorithm, and the shape, features and contour edges of the positioning parts in the image are enhanced by applying a Gaussian filter (to reduce noise), calculating gradients (to identify potential edges) and then thinning the feature edges to improve clarity.

[0071] 3.2) Segment the image: Use the Canny algorithm to decompose the image into smaller segments or regions based on the color and shape features of the positioning piece. Based on the features of the positioning piece, the region of interest related to the positioning piece can be found, that is, multiple first target regions are formed.

[0072] 3.3) Dynamic parameter adjustment: Use the Canny algorithm to adjust the parameters of each first target area found, such as edge sensitivity.

[0073] 3.4) Adapting texture: Each first target region is processed separately using the Canny algorithm. For example, a relatively smooth first target region can be processed quickly with a lower sensitivity, while a first target region with a complex pattern requires a higher sensitivity. Ensure that the necessary details of each part of the image captured by the Canny algorithm can be utilized so that different parts of the image can be identified and adapted, making it effective and efficient for tasks such as feature extraction and segmentation.

[0074] 4) Connecting the contours of the multiple first target areas to form multiple separate areas; 5) Filtering contours according to area size, selecting contours with area ratios in the range of 50,000-500,000 pixels, that is, filtering the contours of the second target area that matches the positioning element from multiple separate areas according to the area size of the positioning element, drawing the contours of the second target area, and fitting the contours of the second target area according to a preset ellipse formula to form multiple ellipse frames; 6) Filter ellipses based on the ratio of the major axis to the minor axis, and select ellipses with a ratio of the major axis to the minor axis between 0.8 and 1.2.

[0075] 7) The X and Y coordinates of the center point of the ellipse can be obtained using the following ellipse formula.

[0076]

[0077] Where (x 0 , y 0 ) are the coordinates of the center point of the ellipse, a and b are the semi-axis lengths of the ellipse along the x-axis and y-axis respectively.

[0078] In summary, the present invention realizes effective processing and analysis of video frames through steps such as data collection, model training, edge detection, frame drawing and Gaussian filtering. It has the advantages of flexible and simple tracking, sharp and accurate capture of position changes, so that the picture and position information can be fed back in real time, and the position changes of the positioning parts can be monitored and observed in real time.

Claims

1. A method for identifying changes in position information of a body surface positioning member for surgery, characterized in that: include: S1: A miniature camera is embedded at the end of the needle insertion device at the end of the puncture surgery robot mechanical arm, and the puncture surgery robot system is initialized and the environment is set; S2: the micro camera continuously photographs the positioning member located on the patient's body surface during the time period of the puncture needle insertion process, the positioning member is used for positioning the puncture robot during the puncture operation, all the images to be processed are acquired, all the acquired images to be processed are initialized, and are sequentially output as the first images to be processed; S3: sending all first images to be processed to a pre-trained first sample training model in sequence, and inferring the X and Y position information of the center points of the positioning parts in all first images to be processed in the image plane according to the first sample training model, to form a second image to be processed including the X and Y position information; S4: sending all the second images to be processed to the second sample training model in sequence, first inferring the longitudinal depth values ​​of all the second images to be processed from the micro camera according to the depth estimation method in the second sample training model, and further inferring the longitudinal Z position information of the center point of the positioning element; S5: Outputting the X, Y, and Z position information of the center point of the positioning member in all the second images to be processed in sequence to form a real-time output data stream, and further making a displacement curve that can be presented on the display screen.

2. The method for identifying position information changes of a surgical body surface positioning member according to claim 1, characterized in that: After step S1, a step is added to determine whether the video transmission stream is normal. If it is abnormal, the program is closed and the puncture surgical robot exits the operation; if it is normal, the following steps are performed.

3. The method for identifying position information changes of a surgical body surface positioning member according to claim 2, characterized in that: In the step S2, the first frame of the image to be processed is first converted into a first array with three channels, that is, the image is digitally processed. The first array with three channels contains the content information of each pixel point, color, and size of the first frame of the image to be processed, and the coordinate values ​​of the three channels in the first array are defined. Then, according to the coordinate values ​​of the three channels in the first array, the subsequent second frame of the image to be processed is converted into a second array with three channels. On the basis of the second array, the third frame of the image to be processed is converted into a third array with three channels. In this way, all the acquired images to be processed are initialized in sequence.

4. The method for identifying position information changes of a surgical body surface positioning component according to any one of claims 1 to 3, characterized in that: All first to-be-processed graphs are repeatedly executed in the first sample training model according to the following steps in sequence: S3.1) converting the first image to be processed into a grayscale image, and applying Gaussian filtering for smoothing and noise reduction; S3.2) setting the dynamic threshold to 125-255, and performing dynamic threshold processing on the first image to be processed after the processing in step S3.1); S3.3) scanning the first image to be processed after being processed in step S3.2) using a canny edge detection algorithm, enhancing the shape, features and contour edges of the positioning member in the first image to be processed, and decomposing the image into smaller segments or regions according to the image features of the positioning member to form a plurality of first target regions; S3.4) connecting the contours of the plurality of first target regions to form a plurality of separate regions; S3.5) selecting a second target area from the plurality of separate areas according to the area size of the positioning element, drawing the contour of the second target area, and fitting the contour of the second target area according to a preset ellipse formula to form a plurality of ellipse frames; S3.6) Selecting a target ellipse frame according to the ratio of the major axis to the minor axis of the ellipse, and obtaining the coordinate value of the center point of the target ellipse frame, which is the X, Y position information of the center point of the positioning member in the plane.

5. The method for identifying position information changes of a surgical body surface positioning member according to claim 4, characterized in that: In step S3.6), an ellipse having a major-minor axis ratio between 0.8 and 1.2 is selected.

6. The method for identifying position information changes of a surgical body surface positioning member according to claim 4, characterized in that: The training process of the second sample training model is as follows: To obtain training data, a large number of unlabeled monocular images are collected from public large-scale datasets, called pseudo-labeled sets, denoted as Du = {ui} N i=1; at the same time, a large number of labeled images are collected from public large-scale datasets, called labeled sets, denoted as Dl = {(xi, di)} M i=1; Construct a monocular deep recognition network, which includes a first set of encoders and a second set of decoders; A powerful monocular depth estimation model S1 is trained using the constructed monocular depth recognition network as follows: 3.1) Training of the initial depth estimation model Pre-train an initial depth estimation model T using labeled images in the labeled set D1; 3.2) Generate pseudo labels Use the initial depth estimation model T to predict the unlabeled images in the pseudo-label set Du and generate a pseudo depth label d * i ; 3.3) Joint Training Combine the labeled images in the labeled set D1 and the unlabeled images in the pseudo-label set Du to form a joint dataset D = Dl ∪ Du; 3.4) Training of depth estimation model Use the joint dataset D to train the initial depth estimation model T to obtain a depth estimation model S; 3.5) Feature Alignment Loss Use a 3D acquisition device to collect the grayscale image and depth image of the positioning part, use the pre-trained DINOv2 model to extract the features of the positioning part, and align the features extracted by the depth estimation model S with the features extracted by the DINOv2 encoder according to the following formula; in: cos(f i , f' i ) measures the cosine similarity between two feature vectors; f i Features extracted for the depth estimation model S; f' i are the features extracted from the DINOv2 model; 3.6) Through multiple rounds of training and optimization in steps 3.1-3.5), a powerful monocular depth estimation model S1 is obtained, which is the second sample training model.

7. A system for identifying changes in position information of a body surface positioning member for surgery comprises a preprocessing module, a startup module, a recognition module, a deep reasoning module and an output module, characterized in that: The preprocessing module consists of a package deployment environment preheating module and a neural network training module, and is used for the initialization and environment configuration of the puncture surgery robot system; The startup module is composed of a strong logic judgment module and a data stream monitoring module, which is used to judge whether the video stream shot by the micro camera can be transmitted normally. If the transmission stream cannot be found during the judgment process, the program is closed and exited; The recognition module is a first sample training module, which is used to recognize the X and Y position information of the center point of the positioning element in each frame of the image transmitted by the start module within the plane where the image is located; The depth reasoning module is a second sample training module, which uses zero-sample Panhua depth estimation model reasoning to identify the Z position information between the center point of the positioning part and the longitudinal direction of the micro camera in each frame of the image transmitted by the recognition module; The output module is used to synchronously output the X, Y position information obtained by the first sample training module and the Z position information obtained by the second sample training module to the display screen of the puncture surgery robot system to form a real-time output data stream.

Images