Three-dimensional optical positioning pseudo point removal method based on improved lightweight Alexnet

Abstract

The present invention discloses a method for removing pseudo-points in trinocular optical positioning based on an improved lightweight AlexNet, comprising the following steps: 1) calibrating a near-infrared binocular camera and a single color camera; 2) selecting a small area color image containing candidate markers from the color image, labeling the scaled small area color image as true or false, forming the network's original data set, and performing data set enhancement; 3) dividing the enhanced data set into a training set and a test set, feeding the improved lightweight AlexNet network for training and testing, and obtaining the network's weight and bias files; 4) deploying the network model on an FPGA to obtain an accelerated model of the network; 5) during application, inputting the small area color image into the accelerated model, screening out the markers predicted to be true, and outputting their location information. The present invention can improve positioning accuracy and has universal applicability, and the improved lightweight AlexNet network is deployed and executed on an FPGA, ensuring the system's portability, low cost, and real-time performance.

Description

Technical Field

[0001] The present invention relates to the technical field of optical positioning, and in particular to a method for removing pseudo points in three-dimensional optical positioning based on an improved lightweight AlexNet. Background Art

[0002] In optical surgical navigation systems, surgical instruments equipped with luminous markers are located using an optical positioning device. Currently, near-infrared binocular optical positioning devices are commonly used in optical surgical navigation to locate surgical instruments. This optical positioning device uses two near-infrared cameras to form a binocular vision system. The left and right cameras simultaneously capture images of surgical instruments with multiple markers. The markers are then identified and located using grayscale centroid, stereo matching, and three-dimensional reconstruction methods. Finally, the coordinates and orientation of the surgical instrument are calibrated based on the three-dimensional coordinates of the markers. Therefore, in optical surgical navigation systems, it is crucial to ensure that the optical positioning device accurately locates the three-dimensional coordinates of the markers.

[0003] During the positioning process of this near-infrared binocular optical positioning device, there are often problems with false identification and false matching of points. A false identification point occurs when a reflective surface in the environment that is similar in size to a marker point is mistakenly identified as a marker point during the image marker recognition process. A false matching point occurs when, during stereo matching using binocular vision, when two or more marker points and the optical centers of two cameras are coplanar, the correct correspondence between the projections of the marker points in the binocular image cannot be determined, resulting in a false match and ultimately generating an unrealistic marker point in three-dimensional space.

[0004] To address the issue of false identification of pseudo points, some researchers have proposed a method to remove pseudo points based on the circular projection features of marker points, but this method is not applicable to surgical scenarios with nuts or circular medical instruments. To address the issue of false matching of pseudo points, some researchers have proposed using geometric distance constraints between marker points to remove pseudo points, but this method is not applicable to surgical scenarios with multiple surgical instruments. Other researchers have proposed using the reprojection error of marker points or removing pseudo points based on the relationship between projection size and lens-object distance, but these methods are too complex and time-consuming, and the algorithm stability is not high. NDI's OptotrakCertus trinocular optical positioning system uses three identical, precisely calibrated linear array CCD cameras to locate marker points using the principle of bipolar line constraints, which can effectively reduce the occurrence of false matching of pseudo points, but the system equipment is relatively complex and costly.

[0005] In summary, it is necessary to study a universally applicable, fast and stable method to remove pseudo-points, so as to solve the pseudo-point problem in the existing near-infrared binocular optical positioning system while ensuring the portability and low cost of the system. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art. A method for removing pseudo points in trinocular optical positioning based on an improved lightweight AlexNet is proposed. A small color camera is fixed on a near-infrared binocular optical positioning device. The position coordinates of the marker points identified in the binocular image in the color image can be calculated based on the positional relationship between the near-infrared binocular camera and the color camera. A neural network is then used to determine whether the color image at that position contains the marker point, thereby achieving the purpose of removing pseudo points identified in the binocular image and improving the positioning accuracy of the system. The method has universal applicability, and the neural network model is deployed on an FPGA to run, ensuring the real-time performance of the system.

[0007] To achieve the above-mentioned purpose, the technical solution provided by the present invention is: a method for removing pseudo points in three-dimensional optical positioning based on an improved lightweight Alexnet, wherein the method is based on an improved lightweight Alexnet network to realize accurate recognition of marker points in optical positioning, and the improved lightweight Alexnet network is an improvement on the traditional Alexnet network. The specific improvement method is: reducing the number of convolutional layers and fully connected layers of the Alexnet network, reducing the original five convolutional layers to two convolutional layers, each convolutional layer is followed by a pooling layer to reduce the dimension of the feature map and prevent overfitting, and reducing the original three fully connected layers to two fully connected layers to reduce the complexity of network calculation and reduce the occupancy of computing resources;

[0008] The specific implementation of this method includes the following steps:

[0009] S1: Calibrate a near-infrared binocular camera and a single color camera on the optical positioning device to obtain the internal and external parameters and distortion coefficients of the cameras, so as to calculate the mutual position conversion relationship between the three cameras, wherein the near-infrared binocular camera includes two cameras;

[0010] S2: Control the three calibrated cameras to take pictures simultaneously. Using the parameters and position relationships obtained from the calibration, the position coordinates of the candidate markers in the binocular image can be calculated in the color image. Then, a small area of ​​the color image with the candidate markers is selected in the color image and scaled to the same size. The small area of ​​the color image is labeled as true or false to form the original data set of the network. The original data set is then enhanced to obtain the enhanced data set.

[0011] S3: Divide the augmented dataset into training and test sets, feed the training and test data into the improved lightweight AlexNet network for training and testing, obtain the optimal model of the network, and export the network weight and bias files;

[0012] S4: Based on the network's weights and bias parameters, the optimal model is deployed on the FPGA to obtain the network's acceleration model. When applying prediction, the three cameras on the optical positioning device are used to obtain the input image and process it to obtain an image of the small area to be detected. The image of the small area to be detected is input into the acceleration model to obtain prediction information. Finally, the marker point information predicted to be true is screened out and output, thereby achieving the purpose of removing pseudo points from the candidate marker points in the binocular image.

[0013] Furthermore, in step S1, the three cameras on the optical positioning device are calibrated using Zhang Zhengyou's checkerboard calibration method. The three cameras are controlled to shoot the checkerboard at the same time. Checkerboard images at different positions are obtained by moving the checkerboard. The images are input into the calibration project to obtain the intrinsic parameters, distortion coefficients and extrinsic parameters of the three cameras relative to the world coordinate system. Then, the mutual position conversion relationship between the three cameras is obtained by calculation, thereby completing the calibration of the optical positioning device.

[0014] Furthermore, the specific operation steps of step S2 are as follows:

[0015] S21: Control the three calibrated cameras to take pictures simultaneously, and use the binocular image to obtain the position information of the candidate marker points. Use the parameters and position relationship of the three cameras obtained by calibration to calculate the position coordinates of the candidate marker points in the binocular image in the color image. Then, select a small area of ​​the color image centered on the position coordinates of the candidate marker points in the color image, and uniformly scale it to 32×32 size. For the selected area of ​​the color image, adjust the size of the candidate frame according to the longitudinal distance between the candidate marker points and the optical positioning device.

[0016] S22: Label the true and false images of small area color images to form the original data set of the network, and rotate and scale the original data set to obtain the enhanced data set.

[0017] Further, in step S3, the enhanced data set is divided into a training set and a test set in proportion, the data of the training set is fed into the improved lightweight Alexnet network for training, the feature information of the candidate marker points is obtained by the feature extraction module of the improved lightweight Alexnet network, the extracted feature information is input into the feature fusion module of the improved lightweight Alexnet network for integration, and then input into the prediction module of the improved lightweight Alexnet network to obtain the prediction result of the candidate marker point; wherein, the cross entropy loss CrossEntropyLoss function is used in back propagation to calculate the loss value between the output label and the actual label, and the stochastic gradient descent method is used for back propagation optimization, and it is continuously iterated until the loss value is minimized to obtain the optimal model of the network, and the weight and bias files of the network are exported;

[0018] The feature extraction module includes two convolutional layers and two pooling layers. The convolutional layers extract features from the input image, while the pooling layers reduce the dimensionality of the feature maps and prevent overfitting. First, the image feature maps are extracted through the convolutional layers, and then the pooling layers perform maximum pooling to reduce the dimensionality. The convolutional layers further extract features, and finally the pooling layers perform maximum pooling to reduce the dimensionality to obtain the output feature maps. A RELU function is added after each pooling layer to enhance the nonlinearity of the network.

[0019] The feature fusion module includes a fully connected layer for fusing the feature maps extracted by the feature extraction module, and then adding a RELU function to enhance the nonlinear ability of the network;

[0020] The prediction module includes a fully connected layer for predicting the feature map fused by the feature fusion module, and then adding a softmax function to convert the output of the network into a predicted probability distribution.

[0021] Furthermore, the specific operation steps of step S4 are as follows:

[0022] S41: Based on the network's weights and bias parameters, the network model is rewritten in C++. The C++ language is then converted into RTL using the XILINX HLS tool, enabling the network to run on the FPGA. During the network model rewrite, loop unrolling and array segmentation can be used to increase the network's running speed on the FPGA. FPGA resource usage can be reduced by converting floating-point numbers to fixed-point numbers and controlling loop unrolling parallelism. Expanding the memory bit width can also reduce memory usage.

[0023] S42: When applying prediction, the three cameras on the optical positioning device are used to obtain the input image, and then step S21 is processed to obtain the color image of the small area to be detected. The color image of the small area to be detected is input into the acceleration model to obtain prediction information, and the category of the candidate marker point is determined to be the category of the class with the largest prediction information value. Then, the candidate marker points predicted to be true are screened from the candidate marker points as real marker points. Finally, only the information of the real marker points is output, thereby achieving the purpose of removing pseudo points from the candidate marker points in the binocular image.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] 1. The present invention designs a method for removing false recognition points and false matching points in near-infrared binocular optical positioning using a color camera, which can effectively improve the accuracy of positioning and has universal applicability and strong practical application value.

[0026] 2. The present invention uses a neural network to determine whether a small area color image contains a marker point. The network is improved based on Alexnet. While ensuring the network detection accuracy, the network model is reduced to form a lightweight network structure, which improves the network operation speed and enables the network to be executed on a small FPGA.

[0027] 3. The present invention deploys the neural network on the FPGA integrated in the optical positioning device to run. The device has high computing power and can ensure the portability, low cost and real-time performance of the positioning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 To improve the structure of the lightweight Alexnet; in the figure, Image Input is the input 32×32×3 RGB small area color image; Conv1 is the first convolution layer, the convolution kernel size is 5×5, and the number of output channels is 10; Max Pool1 is the first pooling layer, the pooling method is maximum pooling, and the size is 2×2; Conv2 is the second convolution layer, the convolution kernel size is 3×3, and the number of output channels is 10; Max Pool2 is the second pooling layer, the pooling method is maximum pooling, and the size is 2×2; FC1 is the first fully connected layer, and the number of output channels is 128; FC2 is the second fully connected layer, and the number of output channels is 2; Softmax is a normalized exponential function used to convert the output of the network into a predicted probability distribution; and finally the probabilities of the two categories are output.

[0029] Figure 2 Schematic diagram of the prediction process for improving the lightweight Alexnet network in the application stage.

[0030] Figure 3 This is an application test scene diagram taken by a color camera. In the figure, 1, 2, 3, 4, and 5 are pseudo points, and 6, 7, 8, 9, 10, and 11 are marked points.

[0031] Figure 4 This is an appearance diagram of the optical positioning device used in the present invention; in the figure, ① represents the near-infrared left camera, ② represents the near-infrared right camera, and ③ represents the color camera. DETAILED DESCRIPTION

[0032] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0033] like Figures 1 to 4 As shown, this embodiment provides a method for removing pseudo points in three-dimensional optical positioning based on an improved lightweight Alexnet. The method is based on the improved lightweight Alexnet network to achieve accurate recognition of marker points in optical positioning; Figure 1As shown, the improved lightweight Alexnet network is an improvement on the traditional Alexnet network. The specific improvement method is as follows: reduce the number of convolutional layers and fully connected layers in the Alexnet network. The original five convolutional layers are reduced to two convolutional layers. Each convolutional layer is followed by a pooling layer to reduce the dimension of the feature map and prevent overfitting. The original three fully connected layers are reduced to two fully connected layers to reduce the computational complexity of the network and the occupation of computing resources.

[0034] The specific implementation of this three-vision optical positioning pseudo-point removal method includes the following steps:

[0035] 1) As Figure 4 shown, for the optical positioning device used in this invention, calibrate the near-infrared binocular camera and the single color camera on the optical positioning device to obtain the internal and external parameters of the camera and the distortion coefficient, so as to calculate the mutual position conversion relationship between the three cameras. Among them, use the Zhang Zhengyou checkerboard calibration method to calibrate the three cameras. Control the three-vision camera to simultaneously shoot the checkerboard, obtain checkerboard images at different positions by moving the checkerboard, input the images into the calibration project, and obtain the internal parameters, distortion coefficients of the three cameras and the external parameters relative to the world coordinate system.

[0036] 2) Control the three calibrated cameras to simultaneously shoot pictures. The position information of the candidate marker points can be obtained using the binocular images. Using the parameters and position relationships of the three cameras obtained by calibration, the position coordinates of the candidate marker points in the binocular images in the color image can be calculated. Among them, the size of the color image is 1280×720×3. As Figure 3 shown, it is the conversion of the color image to a grayscale image.

[0037] Select a small area color image centered on the position coordinates of the candidate marker points in the color image, and uniformly scale it to a size of 32×32. Manually label the authenticity of the small area color image to form the original dataset of the network. Perform rotation and scaling operations on the original dataset to obtain the enhanced dataset. Among them, for the selection of the size of the candidate box, the longitudinal distance of the candidate marker point from the optical positioning device is Z. According to the principle of pinhole imaging of the camera, considering the true size and field of view of the marker point, to reduce the complexity, when Z < 50 cm, select the candidate box size of 64×64; when 50 cm < Z < 150 cm, select the candidate box size of 32×32; when 150 cm < Z < 250 cm, select the candidate box size of 16×16; when Z > 250 cm, select the candidate box size of 8×8. [[ID=,17]]

[0038] 3) The enhanced dataset is divided into a training set and a test set at a ratio of 7:3, and the data of the training set and the test set are sent to the improved lightweight Alexnet network for training and testing. The feature information of the candidate marker points is obtained through the feature extraction module of the improved lightweight Alexnet network, and the extracted feature information is input into the feature fusion module of the improved lightweight Alexnet network for integration and then input into the prediction module of the improved lightweight Alexnet network to obtain the prediction result of the candidate marker point; wherein, the cross entropy loss CrossEntropyLoss function is used in the back propagation to calculate the loss value between the output label and the actual label, and the stochastic gradient descent method is used for back propagation optimization, and it is continuously iterated until the loss value is minimized to obtain the optimal model of the network, and the weight and bias files of the network are exported.

[0039] The feature extraction module consists of two convolutional layers and two pooling layers. The convolutional layers extract features from the input image, while the pooling layers reduce the dimensionality of the feature maps and prevent overfitting. First, the convolutional layers extract the image feature maps, which are then reduced in dimension using max pooling in the pooling layers. Further features are extracted using convolutional layers, and finally, the pooling layers perform max pooling to reduce the dimensionality and produce the output feature maps. A RELU function is added after each pooling layer to enhance the model's nonlinear capabilities.

[0040] The feature fusion module includes a fully connected layer for fusing the feature maps extracted by the feature extraction module, and then adding a RELU function to enhance the nonlinear ability of the network;

[0041] The prediction module includes a fully connected layer for predicting the feature map fused by the feature fusion module, and then adding a softmax function to convert the output of the network into a predicted probability distribution.

[0042] 4) Based on the network's weights and bias parameters, the optimal model is deployed on the FPGA to obtain the network's acceleration model. When applying prediction, the three cameras on the optical positioning device are used to acquire the input image and process it to obtain a color image of the small area to be detected. The color image of the small area to be detected is input into the acceleration model to obtain prediction information. Finally, the information of the marker points predicted to be true is screened and output, thereby achieving the purpose of removing false points from the candidate marker points in the binocular image. The specific situation is as follows:

[0043] Based on the network weights, bias parameters, and network structure, the network model was first rewritten in C++. Then, the HLS tool from XILINX was used to convert the C++ language into RTL language, enabling the network to run on the FPGA. When rewriting the network model, the unroll loop expansion instruction, pipeline pipeline execution instruction, combined with the array_partition array disassembly instruction and array_reshape array reshaping instruction, enabled the loop statements to be executed in parallel to speed up the loop execution rate. At the same time, by controlling the parallelism of the loop expansion, the usage of FPGA resources can be controlled. In addition, by converting the variables involved in the calculation from floating-point numbers to fixed-point numbers, the usage of DSP resources can be reduced. By expanding the memory bit width, the usage of memory can be reduced, thereby achieving a balance between the network operation rate and the usage of FPGA resources.

[0044] like Figure 2 As shown, when applying prediction, the three calibrated cameras on the optical positioning device are used to obtain the input image, and the processing shown in step 2) is performed to obtain a color image of the small area to be detected, and the color image of the small area to be detected is input into the acceleration model to obtain prediction information, and the category of the candidate marking point is determined to be the category of the class with the largest prediction information value, and then the candidate marking points predicted to be true are screened out from the candidate marking points as real marking points, and finally only the information of the real marking points is output, thereby achieving the purpose of removing pseudo points from the candidate marking points in the binocular image.

[0045] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A three-dimensional optical positioning pseudo-point removal method based on an improved lightweight Alexnet, characterized in that: This method is based on an improved lightweight Alexnet network to achieve accurate recognition of markers in optical positioning. The improved lightweight Alexnet network improves the traditional Alexnet network by reducing the number of convolutional layers and fully connected layers of the Alexnet network. The original five convolutional layers are reduced to two convolutional layers. Each convolutional layer is followed by a pooling layer to reduce the dimension of the feature map and prevent overfitting. The original three fully connected layers are reduced to two fully connected layers to reduce the complexity of network calculation and reduce the use of computing resources. The specific implementation of this method includes the following steps: S1: Calibrate a near-infrared binocular camera and a single color camera on the optical positioning device to obtain the internal and external parameters and distortion coefficients of the cameras, so as to calculate the mutual position conversion relationship between the three cameras, wherein the near-infrared binocular camera includes two cameras; S2: Control the three calibrated cameras to take pictures simultaneously. Using the parameters and position relationships obtained from the calibration, the position coordinates of the candidate markers in the binocular image can be calculated in the color image. Then, a small area of ​​the color image with the candidate markers is selected in the color image and scaled to the same size. The small area of ​​the color image is labeled as true or false to form the original data set of the network. The original data set is then enhanced to obtain the enhanced data set. S3: Divide the augmented dataset into training and test sets, feed the training and test data into the improved lightweight AlexNet network for training and testing, obtain the optimal model of the network, and export the network weight and bias files; S4: Based on the network's weights and bias parameters, the optimal model is deployed on the FPGA to obtain the network's acceleration model. When applying prediction, the three cameras on the optical positioning device are used to obtain the input image and process it to obtain a color image of the small area to be detected. The color image of the small area to be detected is input into the acceleration model to obtain prediction information. Finally, the marker point information predicted to be true is screened out and output, thereby achieving the purpose of removing pseudo points from the candidate marker points in the binocular image.

2. The method for removing pseudo points in three-dimensional optical positioning based on the improved lightweight Alexnet according to claim 1, characterized in that: In step S1, the three cameras on the optical positioning device are calibrated using Zhang Zhengyou's checkerboard calibration method. The three cameras are controlled to shoot the checkerboard at the same time. Checkerboard images at different positions are obtained by moving the checkerboard. The images are input into the calibration project to obtain the intrinsic parameters, distortion coefficients and extrinsic parameters of the three cameras relative to the world coordinate system. Then, the mutual position conversion relationship between the three cameras is obtained by calculation, thereby completing the calibration of the optical positioning device.

3. The method for removing pseudo points in three-dimensional optical positioning based on the improved lightweight Alexnet according to claim 2, characterized in that: The specific operation steps of step S2 are as follows: S21: Control the three calibrated cameras to take pictures simultaneously, and use the binocular image to obtain the position information of the candidate marker points. Use the parameters and position relationship of the three cameras obtained by calibration to calculate the position coordinates of the candidate marker points in the binocular image in the color image. Then, select a small area of ​​the color image centered on the position coordinates of the candidate marker points in the color image, and uniformly scale it to 32×32 size. For the selected area of ​​the color image, adjust the size of the candidate frame according to the longitudinal distance between the candidate marker points and the optical positioning device. S22: Label the small area images as true or false to form the original data set of the network, and rotate and scale the original data set to obtain the enhanced data set.

4. The method for removing pseudo points in three-dimensional optical positioning based on the improved lightweight Alexnet according to claim 3, characterized in that: In step S3, the enhanced data set is divided into a training set and a test set in proportion, the data of the training set is sent to the improved lightweight Alexnet network for training, the feature information of the candidate marker points is obtained by the feature extraction module of the improved lightweight Alexnet network, the extracted feature information is input into the feature fusion module of the improved lightweight Alexnet network for integration, and then input into the prediction module of the improved lightweight Alexnet network to obtain the prediction result of the candidate marker point; wherein, the cross entropy loss CrossEntropyLoss function is used in back propagation to calculate the loss value between the output label and the actual label, and the stochastic gradient descent method is used for back propagation optimization, and it is continuously iterated until the loss value is minimized to obtain the optimal model of the network, and the weight and bias files of the network are exported; The feature extraction module includes two convolutional layers and two pooling layers. The convolutional layers extract features from the input image, while the pooling layers reduce the dimensionality of the feature maps and prevent overfitting. First, the image feature maps are extracted through the convolutional layers, and then the pooling layers perform maximum pooling to reduce the dimensionality. The convolutional layers further extract features, and finally the pooling layers perform maximum pooling to reduce the dimensionality to obtain the output feature maps. A RELU function is added after each pooling layer to enhance the nonlinearity of the network. The feature fusion module includes a fully connected layer for fusing the feature maps extracted by the feature extraction module, and then adding a RELU function to enhance the nonlinear ability of the network; The prediction module includes a fully connected layer for predicting the feature map fused by the feature fusion module, and then adding a softmax function to convert the output of the network into a predicted probability distribution.

5. The method for removing pseudo points in three-dimensional optical positioning based on the improved lightweight Alexnet according to claim 4, characterized in that: The specific operation steps of step S4 are as follows: S41: Based on the network's weights and bias parameters, the network model is rewritten in C++. The C++ language is then converted into RTL using the XILINX HLS tool, enabling the network to run on the FPGA. During the network model rewrite, loop unrolling and array segmentation can be used to increase the network's running speed on the FPGA. FPGA resource usage can be reduced by converting floating-point numbers to fixed-point numbers and controlling loop unrolling parallelism. Expanding the memory bit width can also reduce memory usage. S42: When applying prediction, the three cameras on the optical positioning device are used to obtain the input image, and then step S21 is performed to obtain a color image of the small area to be detected. The color image of the small area to be detected is input into the acceleration model to obtain prediction information, and the category of the candidate marking point is determined to be the category of the class with the largest prediction information value. Then, the candidate marking points predicted to be true are screened out from the candidate marking points as real marking points. Finally, only the information of the real marking points is output, thereby achieving the purpose of removing pseudo points from the candidate marking points in the binocular image.

Images