Skull hole posture estimation method and device, electronic equipment and storage medium

Abstract

The present application provides a kind of skull hole posture estimation method, device, electronic equipment and storage medium, wherein skull hole posture estimation method includes: obtaining multiple groups of target binocular two-dimensional center coordinates of multiple holes in binocular skull hole image;Determine the multiple target three-dimensional center coordinates of multiple target holes not collinear in binocular skull hole image based on multiple groups of target binocular two-dimensional center coordinates;Determine the normal posture angle estimation value of multiple holes in binocular skull hole image based on multiple target three-dimensional center coordinates.The present application can realize the process of simplifying the estimation of skull hole posture, reduce manual intervention, improve the automation performance of estimating skull hole posture, and also have stronger robustness to scene transformation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a skull hole posture estimation method and device, electronic equipment and a storage medium. BACKGROUND

[0002] It is known that a brain-computer interface refers to a direct connection created between a human or animal and an external device to realize information exchange between the brain and the external device, and when the brain-computer interface is applied to the brain, the skull hole posture can be estimated to provide a theoretical basis for the diagnosis of skull base fracture and the comprehensive judgment and treatment of craniocerebral trauma based on the estimated skull hole posture.

[0003] In related technologies, for the skull hole posture, the skull hole image is usually first captured by a binocular camera, and the skull hole edge is extracted by adjusting the Gaussian blur kernel and the Canny algorithm threshold value, then the ellipse equation is fitted for the extracted skull hole edge, and the normal vector of the circular plane of the ellipse equation is determined as the normal posture angle of the skull hole.

[0004] However, since the Gaussian blur kernel and the Canny algorithm threshold value need to be manually adjusted by a person, and the ellipse equation can be successfully fitted only when the manually adjusted parameters are accurate enough for the extracted skull hole edge, the adjustment process is time-consuming and laborious, and the adjusted parameters are only applicable to the current scene, thereby resulting in low efficiency and limited scope of application of the skull hole posture estimation. SUMMARY

[0005] The present application provides a skull hole posture estimation method, device, electronic equipment and storage medium to solve the defects of low efficiency and limited scope of application of skull hole posture estimation caused by the need for manual adjustment of parameters to make the skull hole edge accurate enough for skull hole posture estimation in the prior art, to simplify the process of estimating the skull hole posture, reduce manual intervention, improve the automation performance of estimating the skull hole posture, and have stronger robustness to scene changes.

[0006] The present application provides a skull hole posture estimation method, comprising:

[0007] Obtaining a plurality of groups of target binocular two-dimensional center coordinates of a plurality of holes in a binocular skull hole image;

[0008] Based on the plurality of groups of target binocular two-dimensional center coordinates, determining a plurality of target three-dimensional center coordinates of a plurality of target holes that are not collinear in the binocular skull hole image;

[0009] Based on the plurality of target three-dimensional center coordinates, determining normal posture angle estimation values of the plurality of holes in the binocular skull hole image.

[0010] According to the skull hole posture estimation method provided by the application, the multiple groups of target binocular two-dimensional circle center coordinates of the multiple holes in the binocular skull hole image are obtained, which comprises:

[0011] The binocular skull hole image is input into a preset HoleNet network model to obtain multiple groups of predicted binocular two-dimensional circle center coordinates of the multiple holes output by the preset HoleNet network model;

[0012] The multiple groups of predicted binocular two-dimensional circle center coordinates are geometrically filtered to determine multiple groups of initially matched binocular two-dimensional circle center coordinates;

[0013] The multiple groups of initially matched binocular two-dimensional circle center coordinates are filtered according to image features to determine multiple groups of target binocular two-dimensional circle center coordinates of the multiple holes in the binocular skull hole image;

[0014] The preset HoleNet network model is used to extract the bottom image features of the binocular skull hole image, and the bottom image features are learned and predicted. Based on the learned local features of the circle center of each hole and the predicted circle center coordinates contained in each hole, multiple groups of predicted binocular two-dimensional circle center coordinates of the multiple holes are determined.

[0015] According to the skull hole posture estimation method provided by the application, the multiple groups of predicted binocular two-dimensional circle center coordinates comprise multiple predicted left binocular two-dimensional circle center coordinates distributed in multiple holes of a left binocular skull hole image and multiple predicted right binocular two-dimensional circle center coordinates distributed in multiple holes of a right binocular skull hole image. The multiple groups of predicted binocular two-dimensional circle center coordinates are geometrically filtered to determine multiple groups of initially matched binocular two-dimensional circle center coordinates, which comprises:

[0016] It is determined that a predicted left binocular two-dimensional circle center coordinate in the multiple predicted left binocular two-dimensional circle center coordinates corresponds to a first epipolar line in the right binocular skull hole image;

[0017] A predicted right binocular two-dimensional circle center coordinate in the multiple predicted right binocular two-dimensional circle center coordinates that is more than a first preset distance threshold away from the first epipolar line is deleted to determine multiple target predicted right binocular two-dimensional circle center coordinates;

[0018] It is determined that a target predicted right binocular two-dimensional circle center coordinate in the multiple target predicted right binocular two-dimensional circle center coordinates corresponds to a second epipolar line in the left binocular skull hole image;

[0019] A predicted left binocular two-dimensional circle center coordinate in the multiple predicted left binocular two-dimensional circle center coordinates that is more than a second preset distance threshold away from the second epipolar line is deleted to determine multiple target predicted left binocular two-dimensional circle center coordinates;

[0020] Determine a plurality of groups of binocular two-dimensional circle center coordinates of initial matching based on the plurality of groups of predicted right eye two-dimensional circle center coordinates and the plurality of groups of predicted left eye two-dimensional circle center coordinates.

[0021] According to the skull hole posture estimation method provided by the application, the plurality of groups of binocular two-dimensional circle center coordinates of initial matching are filtered by image features, and a plurality of groups of target binocular two-dimensional circle center coordinates of a plurality of holes in the binocular skull hole image are determined, which comprises the following steps:

[0022] Feature collection is performed on the bottom layer image features based on the plurality of groups of binocular two-dimensional circle center coordinates of initial matching, and a plurality of groups of image features are determined.

[0023] Cosine similarity of each group of image features is determined.

[0024] Based on the comparison result of the cosine similarity and the preset cosine similarity threshold, the plurality of groups of binocular two-dimensional circle center coordinates of initial matching are finely filtered, and a plurality of groups of target binocular two-dimensional circle center coordinates of a plurality of holes in the binocular skull hole image are determined.

[0025] According to the skull hole posture estimation method provided by the application, the training process of the preset HoleNet network model comprises the following steps:

[0026] A sample skull hole image and an initial HoleNet network model are determined, and the initial HoleNet network model has a bottom layer feature extraction function, a skull hole feature learning function and a skull hole circle center coordinate prediction function.

[0027] The initial HoleNet network model is trained using the sample skull hole image, and a preset HoleNet network model is determined.

[0028] According to the skull hole posture estimation method provided by the application, based on the plurality of groups of target binocular two-dimensional circle center coordinates, a plurality of target three-dimensional circle center coordinates of a plurality of non-collinear target holes in the binocular skull hole image are determined, which comprises the following steps:

[0029] Camera internal and external parameters of a binocular camera are obtained.

[0030] Based on the camera internal and external parameters and the plurality of groups of target binocular two-dimensional circle center coordinates, a plurality of target three-dimensional circle center coordinates of a plurality of non-collinear target holes in the binocular skull hole image are determined.

[0031] According to the skull hole posture estimation method provided by the application, based on the plurality of target three-dimensional circle center coordinates, normal posture angle estimation values of the plurality of holes in the binocular skull hole image are determined, which comprises the following steps:

[0032] determine a target plane equation of a plane where a plurality of target holes not in a same line are located in the binocular skull hole image based on the plurality of target three-dimensional circle center coordinates;

[0033] determine a normal vector of the target plane equation;

[0034] determine a normal pose angle estimation value of the plurality of holes in the binocular skull hole image based on the normal vector.

[0035] The application also provides a skull hole pose estimation device, comprising:

[0036] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the skull hole pose estimation method according to any one of the above when executing the program.

[0037] The application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the skull hole pose estimation method according to any one of the above.

[0038] The skull hole pose estimation method, device, electronic device and storage medium provided by the application, wherein the skull hole pose estimation method, the terminal device first acquires a plurality of groups of target binocular two-dimensional circle center coordinates of a plurality of holes in a binocular skull hole image, then determines a plurality of target three-dimensional circle center coordinates of a plurality of target holes not in a same line in the binocular skull hole image, and finally determines a normal pose angle estimation value of the plurality of holes in the binocular skull hole image, which not only simplifies the process of estimating the skull hole pose, reduces manual intervention, but also improves the automation performance of estimating the skull hole pose, thereby greatly improving the efficiency of skull hole pose estimation, and also has stronger robustness to scene transformation. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 is a flowchart of the skull hole pose estimation method provided by the application;

[0041] Figure 2 is a structural schematic diagram of the skull hole pose estimation device provided by the application;

[0042] Figure 3 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described below in connection with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0044] The present application will be described below in connection with the drawings in the present application. Figures 1-3 The skull hole posture estimation method, device, electronic equipment and storage medium of the present application are described below, wherein the execution subject of the skull hole posture estimation method can be a terminal device or a server. The server can refer to a server or a server cluster composed of multiple servers, a cloud computing center and the like. The terminal device can be a personal computer (PC), a portable device, a notebook computer, a smart phone, a tablet computer and other electronic devices such as a portable wearable device. The server or the terminal device at least has data acquisition function, data storage function and data processing function. The specific form of the server or the terminal device is not limited in the present application. The following method embodiments are described by taking the terminal device as an example.

[0045] Referring to Figure 1 The flowchart of the skull hole posture estimation method provided by the embodiments of the present application is shown in FIG. 1. The skull hole posture estimation method includes the following steps: Figure 1

[0046] Step 110, obtaining multiple groups of target binocular two-dimensional center coordinates of multiple holes in a binocular skull hole image.

[0047] ​The binocular skull hole image can be an image generated after the holes of the skull are photographed by the left and right cameras of a binocular camera and can include a left-eye skull hole image and a right-eye skull hole image, the number of holes of the skull photographed by the binocular camera can be at least 3, the holes of the skull can be holes of the skeleton on the head of the to-be-tested object, and the to-be-tested object can be a human or an animal. The number of holes distributed in the left-eye skull hole image and the right-eye skull hole image is the same and one-to-one corresponding, and there is a certain parallax between the left-eye skull hole image and the right-eye skull hole image. Moreover, each set of target binocular two-dimensional center coordinates in the plurality of sets of target binocular two-dimensional center coordinates corresponds to one hole and includes at least one target left-eye two-dimensional center coordinate and at least one target right-eye two-dimensional center coordinate, that is, each hole in the left-eye skull hole image is distributed with at least one target left-eye two-dimensional center coordinate, and each hole in the right-eye skull hole image is distributed with at least one target right-eye two-dimensional center coordinate. Each target left-eye two-dimensional center coordinate can be a left-eye two-dimensional center coordinate without noise and is a key center point in the corresponding hole in the left-eye skull hole image, and each target right-eye two-dimensional center coordinate can also be a right-eye two-dimensional center coordinate without noise and is a key center point in the corresponding hole in the right-eye skull hole image.

[0048] Specifically, the terminal device acquires a plurality of sets of target binocular two-dimensional center coordinates of a plurality of holes in the binocular skull hole image. When the terminal device detects that it is connected to the to-be-tested object through the brain-computer interface, the terminal device can acquire the binocular skull hole image of the skull of the to-be-tested object and a plurality of sets of target binocular two-dimensional center coordinates of a plurality of holes in the binocular skull hole image. The terminal device can directly select the binocular skull hole image of the skull of the to-be-tested object from a pre-stored image set, or can acquire the binocular skull hole image of the skull of the to-be-tested object by first starting the binocular camera function and then photographing the skull of the to-be-tested object. Moreover, for the acquired binocular skull hole image, the terminal device can further acquire a plurality of sets of target binocular two-dimensional center coordinates by first acquiring the two-dimensional center coordinates in the holes and then removing noise for a plurality of holes in the left-eye skull hole image and a plurality of holes in the right-eye skull hole image. The terminal device can also send the binocular skull hole image of the skull of the to-be-tested object to the user terminal and instruct the user terminal to feed back a plurality of sets of target binocular two-dimensional center coordinates of a plurality of holes in the binocular skull hole image corresponding to the user, so as to acquire a plurality of sets of target binocular two-dimensional center coordinates. The manner of acquiring a plurality of sets of target binocular two-dimensional center coordinates of a plurality of holes in the binocular skull hole image is not specifically limited here.

[0049] In step 120, a plurality of target three-dimensional center coordinates of a plurality of non-collinear target holes in the binocular skull hole image are determined based on the plurality of sets of target binocular two-dimensional center coordinates.

[0050] Specifically, since the diameter of the skull hole in the brain-computer interface task is about 300 microns, and the distance between adjacent holes is about 1000 microns, the region where the multiple holes in the binocular skull hole image are located can be considered as a plane, and the skull hole posture can be estimated by estimating the 3D center coordinates of 3 or more non-collinear holes to fit the plane equation of the region where the multiple holes are located. Based on this, the terminal device can determine the target 3D center coordinates of the multiple holes by using the triangulation method based on the multiple sets of target binocular 2D center coordinates of the multiple holes in the binocular skull hole image, and then determine the target 3D center coordinates of the multiple non-collinear target holes from the multiple target 3D center coordinates of the multiple holes.

[0051] Step 130, based on the multiple target 3D center coordinates, determining the normal posture angle estimation value of the multiple holes in the binocular skull hole image.

[0052] Specifically, the terminal device can calculate the target plane equation of the region where the multiple target holes are located based on the pre-stored plane equation construction algorithm based on the multiple target 3D center coordinates of the multiple non-collinear target holes in the binocular skull hole image, so as to determine the normal vector of the target plane equation as the normal posture angle estimation value of the multiple holes in the binocular skull hole image.

[0053] The skull hole posture estimation method provided by the embodiment of the application can first obtain the multiple sets of target binocular 2D center coordinates of the multiple holes in the binocular skull hole image, then determine the multiple target 3D center coordinates of the multiple non-collinear target holes in the binocular skull hole image, and finally determine the normal posture angle estimation value of the multiple holes in the binocular skull hole image, which not only simplifies the process of estimating the skull hole posture, reduces manual intervention, but also improves the automation performance of estimating the skull hole posture, thereby greatly improving the efficiency of estimating the skull hole posture, and also has stronger robustness to scene transformation.

[0054] Optionally, the specific implementation process of step 110 can include:

[0055] First, input the binocular skull hole image into the preset HoleNet network model to obtain multiple sets of predicted binocular 2D center coordinates of the multiple holes output by the preset HoleNet network model; further, perform geometric filtering on the multiple sets of predicted binocular 2D center coordinates to determine the multiple sets of initially matched binocular 2D center coordinates; then, perform image feature filtering on the multiple sets of initially matched binocular 2D center coordinates to determine the multiple sets of target binocular 2D center coordinates of the multiple holes in the binocular skull hole image.

[0056] The preset HoleNet network model is used to extract bottom image features of the binocular skull hole image, and the bottom image features are learned and predicted. Based on the learned local features of the center of each hole and the predicted center coordinates contained in each hole, a plurality of groups of predicted binocular two-dimensional center coordinates of the plurality of holes are determined.

[0057] Specifically, since the binocular skull hole image includes a left eye skull hole image and a right eye skull hole image, and the number of holes distributed in the left eye skull hole image and the right eye skull hole image is the same and one-to-one, in order to improve the accuracy of skull hole pose estimation, the preset HoleNet network model obtained by pre-training can be used to accurately predict the two-dimensional coordinates of the center point contained in each hole in the binocular skull hole image, thereby outputting a plurality of groups of predicted binocular two-dimensional center coordinates of the plurality of holes. Each group of predicted binocular two-dimensional center coordinates in the plurality of groups of predicted binocular two-dimensional center coordinates corresponds to one hole and includes at least one predicted left eye two-dimensional center coordinate and at least one predicted right eye two-dimensional center coordinate, that is, each hole in the left eye skull hole image is distributed with at least one predicted left eye two-dimensional center coordinate, and each hole in the right eye skull hole image is distributed with at least one predicted right eye two-dimensional center coordinate.

[0058] In addition, the preset HoleNet network model contains a trained ResNet network, a trained center convolution kernel, and a trained head1 network. When the binocular skull hole image is input into the preset HoleNet network model, the trained ResNet network is used to extract the bottom image features of the binocular skull hole image, the trained center convolution kernel is used to learn the bottom image features, and the learning result is the local feature of the center of each hole. The trained head1 network is used to predict the center coordinates of the skull hole frame center based on the local feature of the center, and the prediction result is the predicted center coordinates of each hole frame. Then, the center offset is determined based on the local feature of the center and the preset offset calculation method, and the predicted center coordinates and the center offset are added to obtain a group of predicted two-dimensional center coordinates corresponding to the hole. In this way, a plurality of groups of predicted binocular two-dimensional center coordinates of the plurality of holes are obtained.

[0059] Since the learning and prediction process of the preset HoleNet network model is inevitably disturbed by noise, for the multiple sets of predicted binocular two-dimensional center coordinates of the multiple holes output by the preset HoleNet network model, first, a geometric filter is used for geometric filtering, and noise center points contained in each hole in the left-eye skull hole image and the right-eye skull hole image are filtered out, so that the purpose of preliminarily matching the center points contained in the multiple holes in the binocular skull hole image is achieved, and multiple sets of initially matched binocular two-dimensional center coordinates are determined; then, an image feature filter is used to perform image feature filtering on the multiple sets of initially matched binocular two-dimensional center coordinates, that is, the feature similarity of each two-dimensional center coordinate in the multiple sets of initially matched binocular two-dimensional center coordinates is determined, and the binocular two-dimensional center coordinates with a feature similarity reaching a preset similarity requirement in the multiple sets of initially matched binocular two-dimensional center coordinates are determined as multiple sets of target binocular two-dimensional center coordinates of the multiple holes in the binocular skull hole image, so that the purpose of finely matching the multiple sets of initially matched binocular two-dimensional center coordinates is achieved.

[0060] It should be noted that when the number of target binocular two-dimensional center coordinates remaining after the geometric filtering and the image feature filtering is less than 2, the purpose of finding more holes or taking pictures of the holes from a better angle to increase the number of target binocular two-dimensional center coordinates to at least 3 can be achieved by slightly moving the pose of the binocular camera.

[0061] The skull hole pose estimation method provided by the embodiment of the application can improve the accuracy and reliability of obtaining multiple sets of target binocular two-dimensional center coordinates by using the preset HoleNet network model, the geometric filter, and the image feature filter in combination with network model learning and prediction and different filtering methods.

[0062] Optionally, in the case that the multiple sets of predicted binocular two-dimensional center coordinates include multiple predicted left-eye two-dimensional center coordinates distributed in the multiple holes of the left-eye skull hole image and multiple predicted right-eye two-dimensional center coordinates distributed in the multiple holes of the right-eye skull hole image, the multiple sets of predicted binocular two-dimensional center coordinates are geometrically filtered to determine multiple sets of initially matched binocular two-dimensional center coordinates, and the implementation process can include:

[0063] Firstly, a first epipolar line corresponding to a predicted left-eye two-dimensional circle center coordinate in the predicted left-eye two-dimensional circle center coordinates is determined in the right-eye skull hole image; then, a predicted right-eye two-dimensional circle center coordinate in the predicted right-eye two-dimensional circle center coordinates that exceeds a first preset distance threshold is deleted, and a plurality of target predicted right-eye two-dimensional circle center coordinates are determined; then, a second epipolar line corresponding to a target predicted right-eye two-dimensional circle center coordinate in the target predicted right-eye two-dimensional circle center coordinates is determined in the left-eye skull hole image; then, a predicted left-eye two-dimensional circle center coordinate in the predicted left-eye two-dimensional circle center coordinates that exceeds a second preset distance threshold is further deleted, and a plurality of target predicted left-eye two-dimensional circle center coordinates are determined; finally, based on the plurality of target predicted right-eye two-dimensional circle center coordinates and the plurality of target predicted left-eye two-dimensional circle center coordinates, a plurality of groups of binocular two-dimensional circle center coordinates that are initially matched are determined.

[0064] Specifically, based on the Zhang Zhengyou calibration method, the homogeneous coordinates of the predicted left-eye two-dimensional circle center coordinates in the predicted left-eye two-dimensional circle center coordinates can be determined to correspond to a first epipolar line in the right-eye skull hole image, and the first epipolar line can also be regarded as a line formed by the projection of the predicted left-eye two-dimensional circle center coordinates in the right-eye skull hole image; then, it is determined whether the distance between each predicted right-eye two-dimensional circle center coordinate and the first epipolar line exceeds a first preset distance threshold, and the predicted right-eye two-dimensional circle center coordinates that exceed the first preset distance threshold are filtered out from the plurality of predicted right-eye two-dimensional circle center coordinates; the remaining predicted right-eye two-dimensional circle center coordinates that do not exceed the first preset distance threshold are retained and determined as the plurality of target predicted right-eye two-dimensional circle center coordinates; similarly, the homogeneous coordinates of the target predicted right-eye two-dimensional circle center coordinates in the plurality of target predicted right-eye two-dimensional circle center coordinates can also be determined to correspond to a second epipolar line in the left-eye skull hole image; then, it is determined whether the distance between each predicted left-eye two-dimensional circle center coordinate and the second epipolar line exceeds a second preset distance threshold, and the predicted left-eye two-dimensional circle center coordinates that exceed the second preset distance threshold are filtered out from the plurality of predicted left-eye two-dimensional circle center coordinates; the remaining predicted left-eye two-dimensional circle center coordinates that do not exceed the second preset distance threshold are retained and determined as the plurality of target predicted left-eye two-dimensional circle center coordinates; then, the plurality of target predicted left-eye two-dimensional circle center coordinates and the plurality of target predicted right-eye two-dimensional circle center coordinates are determined as a plurality of groups of binocular two-dimensional circle center coordinates that are initially matched after geometric filtering of a plurality of groups of predicted binocular two-dimensional circle center coordinates, that is, the plurality of groups of predicted binocular two-dimensional circle center coordinates are initially matched. Optionally, the first preset distance threshold and the second preset distance threshold can both be 0.5.

[0065] The skull hole posture estimation method provided by the embodiment of the application can realize the purpose of geometric filtering of multiple groups of predicted binocular two-dimensional circle center coordinates by filtering the predicted binocular two-dimensional circle center coordinates deviating far from the first epipolar line from the multiple predicted right eye two-dimensional circle center coordinates and filtering the predicted binocular two-dimensional circle center coordinates deviating far from the second epipolar line from the multiple predicted left eye two-dimensional circle center coordinates, so as to effectively improve the accuracy and reliability of preliminary matching in combination with epipolar geometry and Zhang Zhengyou marking method.

[0066] Optionally, the multiple groups of binocular two-dimensional circle center coordinates of preliminary matching are subjected to image feature filtering to determine multiple groups of target binocular two-dimensional circle center coordinates of multiple holes in the binocular skull hole image, and the implementation process can include

[0067] Firstly, the image features of the bottom layer are collected based on the multiple groups of binocular two-dimensional circle center coordinates of preliminary matching to determine multiple groups of image features, and then the cosine similarity of each group of image features is further determined; then, the multiple groups of binocular two-dimensional circle center coordinates of preliminary matching are subjected to fine filtering based on the comparison result of the cosine similarity and the preset cosine similarity threshold to determine multiple groups of target binocular two-dimensional circle center coordinates of multiple holes in the binocular skull hole image.

[0068] Specifically, since the geometric filtering cannot filter the noise circle center points contained on the first epipolar line and the second epipolar line, the fine filtering can be realized by comparing the image feature similarity of the multiple groups of binocular two-dimensional circle center coordinates of preliminary matching, and based on this, the image features can be collected on the bottom layer image features extracted by the preset HoleNet network model based on the positions of the multiple groups of binocular two-dimensional circle center coordinates of preliminary matching, so as to determine multiple groups of image features, each group of image features corresponding to a group of binocular two-dimensional circle center coordinates of preliminary matching; then the cosine similarity of each group of image features is determined, and one of the groups of image features is set as L1 and L2, and the cosine similarity of this group of image features is cos(∠L1, L2), ∠L1, L2 representing the included angle of L1 and L2, when the value of ∠L1, L2 is 0°, the cosine similarity of L1 and L2 is 1; when the value of ∠L1, L2 is 90°, the cosine similarity of L1 and L2 is 0. At this time, the cosine similarity of each group of image features is compared with the preset cosine similarity threshold, and the binocular two-dimensional circle center coordinates corresponding to the cosine similarity exceeding the preset cosine similarity threshold are retained and determined as the target binocular two-dimensional circle center coordinates of the binocular skull hole image. In this way, multiple groups of target binocular two-dimensional circle center coordinates of multiple holes in the binocular skull hole image are determined. Thus, the purpose of fine matching for the multiple groups of binocular two-dimensional circle center coordinates of preliminary matching is achieved.

[0069] The skull hole posture estimation method provided by the embodiment of the present application, the terminal device determines a plurality of groups of image features through a method of corresponding acquisition of image features on the bottom layer image features based on the initial matching of a plurality of groups of binocular two-dimensional circle center coordinates, and performs fine filtering on the initial matching of the plurality of groups of binocular two-dimensional circle center coordinates based on the comparison result of the cosine similarity of each group of image features and the preset cosine similarity threshold, so as to effectively improve the reliability and accuracy of the fine filtering on the initial matching of the plurality of groups of binocular two-dimensional circle center coordinates, thereby laying a foundation for the accuracy of subsequent skull hole posture estimation.

[0070] Optionally, the training process of the preset HoleNet network model can include:

[0071] Firstly, a sample skull hole image and an initial HoleNet network model are determined, the initial HoleNet network model has a bottom layer feature extraction function, a skull hole feature learning function and a skull hole circle center coordinate prediction function; then, the initial HoleNet network model is trained using the sample skull hole image to determine the preset HoleNet network model.

[0072] Specifically, the sample skull hole image can be determined by shooting binocular images of a sample to-be-tested object, which can be a human or an animal, and the number of sample skull hole images is multiple; the initial HoleNet network model can be a network model containing a ResNet network, a circle center convolution kernel and a head1 network, the ResNet network is used to extract bottom features, the circle center convolution kernel is used to learn skull hole features, and the head1 network is used to predict the circle center of the skull hole frame. Based on this, the initial HoleNet network model is trained multiple times using sample skull hole images. For each training, the ResNet network is used to extract sample bottom image features of a single sample skull hole image, the circle center convolution kernel is used to learn sample bottom image features and the learning result is sample circle center local features of each hole, the head1 network is used to predict the sample circle center local features and the prediction result is the sample predicted circle center coordinates of each hole frame, then the sample circle center offset is determined based on the sample circle center local features and the preset offset calculation method, and the sample predicted circle center coordinates and the sample circle center offset are added to obtain the result, which is determined as the sample predicted two-dimensional circle center coordinates of the corresponding hole. Thus, one training for a single sample skull hole image is completed, and the initial HoleNet network model is trained for a preset number of times in this training manner to determine an intermediate HoleNet network model trained for a preset number of times. It is determined whether the loss value of the intermediate HoleNet network model reaches a preset loss threshold. If the loss value of the intermediate HoleNet network model reaches the preset loss threshold, the training is stopped, and the HoleNet network model corresponding to the training stop is determined as the preset HoleNet network model. Otherwise, if the loss value of the intermediate HoleNet network model does not reach the preset loss threshold, the training is continued until the training stop condition is met.

[0073] It should be noted that the training stop condition can be determined by the loss value of the intermediate HoleNet network model reaching the preset loss threshold, or by the cumulative training times reaching the preset training times when the current training is completed. Herein, no specific limitation is made.

[0074] The skull hole pose estimation method provided by the embodiment of the application determines the preset HoleNet network model obtained when the training converges and the accuracy is the highest by training the initial HoleNet network model containing the bottom feature extraction function, the skull hole feature learning function and the skull hole circle center coordinate prediction function using the sample skull hole image, thereby improving the accuracy and reliability of the model training, and further improving the accuracy of the skull hole pose estimation.

[0075] Optionally, the specific implementation process of step 120 can include:

[0076] obtain camera intrinsic and extrinsic parameters of the binocular camera; determine target three-dimensional circle center coordinates of each group of target binocular two-dimensional circle center coordinates based on the camera intrinsic and extrinsic parameters and the groups of target binocular two-dimensional circle center coordinates; and determine a plurality of target three-dimensional circle center coordinates of a plurality of non-collinear target holes in the binocular skull hole image based on the target three-dimensional circle center coordinates.

[0077] Specifically, a checkerboard calibration board is photographed by using a binocular camera to determine a binocular checkerboard image, and OpenCV library is used to recognize the binocular checkerboard image to determine 3D coordinates of 2n key points, where n is a positive integer greater than 1. Then, the z-axis coordinate value in each 3D coordinate is set to 0 to obtain a simplified projection conversion relationship from 3D space to 2D space as shown in the following formula.

[0078]

[0079] where [x y 1] represents the 2D space coordinates of each key point, [X Y 0 1] represents the 3D space coordinates of the corresponding key point, K represents the intrinsic matrix of the binocular camera, R represents the extrinsic rotation matrix of the binocular camera, and t represents the extrinsic translation vector of the binocular camera. Since the 2D space coordinates and the 3D space coordinates of each key point are known, when the photographed and recognized key points are sufficient, the camera intrinsic and extrinsic parameters of the binocular camera can be determined by constructing a linear equation and solving the optimal solution of the linear equation. The camera intrinsic and extrinsic parameters of the binocular camera include the intrinsic matrix K of the binocular camera, the extrinsic rotation matrix R of the binocular camera, and the extrinsic translation vector t of the binocular camera. Moreover, for the binocular camera, the intrinsic matrix of the left camera can be determined as K1, the intrinsic matrix of the right camera can be determined as K2, and the left camera coordinate system is taken as the reference coordinate system. Based on this, the homogeneous 2D image coordinates of the target left binocular two-dimensional circle center coordinates of the jth hole in the left binocular skull hole image are set as x1, the homogeneous coordinates of the target right binocular two-dimensional circle center coordinates of the jth hole in the right binocular skull hole image are set as x2, and when the target three-dimensional circle center coordinates of the jth hole in the left binocular skull hole image and the right binocular skull hole image are X, the following equation set is satisfied when X is projected onto the left binocular skull hole image and the right binocular skull hole image:

[0080] D1x1=K1X

[0081] D2x2=K2[R|t]X

[0082] where D1 represents the depth value of the jth hole in the left camera coordinate system, and D2 represents the depth value of the jth hole in the right camera coordinate system. The two depth values D1 and D2 are determined by solving the above equation set by using the least square method. Further, the target three-dimensional circle center coordinates of the jth hole in the reference camera coordinate system are calculated as follows: In this way, the target three-dimensional center coordinates of the M holes are calculated, j=1, 2, …, M, and M is the number of holes in the binocular skull hole image.

[0083] At this time, based on the target three-dimensional center coordinates of each hole, the distance value of each hole from the reference camera is calculated, M distance values are determined, then the average value u and the standard deviation s of each distance value are calculated, finally the holes in the range of [u-s, u+s] in the M distance values are determined as target holes, and the number of target holes is determined as N, N is less than M; for the N target holes, all combinations containing 3 or more target holes are traversed, and each combination can be determined as 3 or more non-collinear target holes, and then a plurality of target three-dimensional center coordinates of the 3 or more non-collinear target holes are determined.

[0084] The skull hole posture estimation method provided by the embodiment of the application determines the plurality of target three-dimensional center coordinates of the plurality of non-collinear target holes based on the camera internal and external parameters of the binocular camera and the plurality of target binocular two-dimensional center coordinates, and improves the reliable accuracy and effective stability of determining the target three-dimensional center coordinates.

[0085] Optionally, the specific implementation process of step 130 can include:

[0086] Firstly, based on the plurality of target three-dimensional center coordinates, a target plane equation of a plane where the plurality of non-collinear target holes in the skull hole image are located is determined; then a normal vector of the target plane equation is further determined; and then based on the normal vector, a normal posture angle estimation value of the plurality of holes in the binocular skull hole image is determined.

[0087] Specifically, based on the target three-dimensional center coordinates of the 3 or more non-collinear target holes, the target plane equation of the plane where the 3 or more non-collinear target holes are located can be fitted as Ax+By+Cz+D=0, and the normal vector of the target plane equation is determined as (A, B, C), which is also the target hole insertion direction, and the center of the target hole is the target three-dimensional center coordinate of the corresponding hole in the reference camera coordinate system. Therefore, the normal vector of the target plane equation can be determined as the normal posture angle estimation value of the plurality of holes in the binocular skull hole image.

[0088] It should be noted that for the N target holes, all combinations containing 3 or more target holes are traversed, each combination can fit a target plane equation, and the normal vector of each target plane equation can also be obtained, that is, the required hole insertion direction.

[0089] For example, for 4 target holes ABCD, 4 combinations of ABC, ABD, BCD and ACD can be combined, when A contains noise, it will cause the calculation of ABC, ABD and ACD planes to be wrong, therefore, the normal vector of ABC, ABD and ACD is not in the calculation range, so that the 4 target holes ABCD are filtered out; for example, when the number of target holes increases to 5, that is, ABCDE, when A contains noise, ABC, ACD, ACE and ADE are not in the reserved range, since the intersection of ABC, ACD, ACE and ADE is A, A is removed, and the remaining target hole BCDE is fitted to obtain the normal vector.

[0090] The skull hole posture estimation method provided by the embodiment of the application can effectively improve the efficiency of skull hole posture estimation and the automation performance of estimating skull hole posture, and has stronger robustness to scene transformation.

[0091] The skull hole posture estimation device provided by the embodiment of the application is described below, and the skull hole posture estimation device described below can be correspondingly referred to the skull hole posture estimation method described above.

[0092] Figure 2 The structure diagram of the skull hole posture estimation device provided by the embodiment of the application is shown in FIG. 2, the skull hole posture estimation device 200 comprises: Figure 2 The acquisition module 210 is configured to acquire a plurality of groups of target binocular two-dimensional center coordinates of a binocular skull hole image.

[0093] The determination module 220 is configured to determine a plurality of target three-dimensional center coordinates of a plurality of target holes not in a same line in the binocular skull hole image based on the plurality of groups of target binocular two-dimensional center coordinates.

[0094] The posture estimation module 230 is configured to determine a normal posture angle estimation value of the plurality of holes in the binocular skull hole image based on the plurality of target three-dimensional center coordinates.

[0095]

[0096] ​Optionally, the acquisition module 210 can be specifically configured to input the binocular skull hole image into a preset HoleNet network model to obtain a plurality of groups of predicted binocular two-dimensional circle center coordinates of a plurality of holes output by the preset HoleNet network model; perform geometric filtering on the plurality of groups of predicted binocular two-dimensional circle center coordinates to determine a plurality of groups of initially matched binocular two-dimensional circle center coordinates; and perform image feature filtering on the plurality of groups of initially matched binocular two-dimensional circle center coordinates to determine a plurality of groups of target binocular two-dimensional circle center coordinates of the plurality of holes in the binocular skull hole image; wherein the preset HoleNet network model is configured to extract bottom-layer image features of the binocular skull hole image, and learn and predict the bottom-layer image features, to determine the plurality of groups of predicted binocular two-dimensional circle center coordinates of the plurality of holes based on local features of a circle center of each hole learned and circle center coordinates contained in each hole predicted.

[0097] Optionally, the determination module 220 can be specifically configured to determine a first epipolar line in the right binocular skull hole image corresponding to a predicted left binocular two-dimensional circle center coordinate in the plurality of predicted left binocular two-dimensional circle center coordinates; delete a predicted right binocular two-dimensional circle center coordinate in the plurality of predicted right binocular two-dimensional circle center coordinates that is more than a first preset distance threshold away from the first epipolar line to determine a plurality of target predicted right binocular two-dimensional circle center coordinates; determine a second epipolar line in the left binocular skull hole image corresponding to a target predicted right binocular two-dimensional circle center coordinate in the plurality of target predicted right binocular two-dimensional circle center coordinates; delete a predicted left binocular two-dimensional circle center coordinate in the plurality of predicted left binocular two-dimensional circle center coordinates that is more than a second preset distance threshold away from the second epipolar line to determine a plurality of target predicted left binocular two-dimensional circle center coordinates; and determine the plurality of groups of initially matched binocular two-dimensional circle center coordinates based on the plurality of target predicted right binocular two-dimensional circle center coordinates and the plurality of target predicted left binocular two-dimensional circle center coordinates.

[0098] Optionally, the determination module 220 can be specifically configured to perform feature collection on the bottom-layer image features based on the plurality of groups of initially matched binocular two-dimensional circle center coordinates to determine a plurality of groups of image features; determine a cosine similarity of each group of image features; and perform fine filtering on the plurality of groups of initially matched binocular two-dimensional circle center coordinates based on a comparison result of the cosine similarity and a preset cosine similarity threshold to determine the plurality of groups of target binocular two-dimensional circle center coordinates of the plurality of holes in the binocular skull hole image.

[0099] Optionally, the device can further include a training module configured to determine a sample skull hole image and an initial HoleNet network model, the initial HoleNet network model having a bottom-layer feature extraction function, a skull hole feature learning function, and a skull hole circle center coordinate prediction function; and train the initial HoleNet network model using the sample skull hole image to determine the preset HoleNet network model.

[0100] Optionally, the determining module 220 can be further configured to acquire camera internal and external parameters of the binocular camera; and determine, based on the camera internal and external parameters and the multiple sets of target binocular two-dimensional circle center coordinates, multiple target three-dimensional circle center coordinates of the multiple target holes that are not collinear in the binocular skull hole image.

[0101] Optionally, the determining module 220 can be further configured to determine, based on the multiple target three-dimensional circle center coordinates, a target plane equation of a plane on which the multiple target holes that are not collinear in the binocular skull hole image are located; determine a normal vector of the target plane equation; and determine, based on the normal vector, the normal pose angle estimation value of the multiple holes in the binocular skull hole image.

[0102] Figure 3 An example of a schematic diagram of a physical structure of an electronic device is shown in FIG. 3. Figure 3 As shown in FIG. 3, the electronic device 300 can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340. The processor 310, the communications interface 320, and the memory 330 can communicate with each other via the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute a skull hole pose estimation method, which includes the following steps.

[0103] acquiring multiple sets of target binocular two-dimensional circle center coordinates of multiple holes in a binocular skull hole image;

[0104] determining, based on the multiple sets of target binocular two-dimensional circle center coordinates, multiple target three-dimensional circle center coordinates of multiple target holes that are not collinear in the binocular skull hole image;

[0105] determining, based on the multiple target three-dimensional circle center coordinates, a normal pose angle estimation value of the multiple holes in the binocular skull hole image.

[0106] In addition, the logical instruction in the memory 330 described above can be implemented in the form of a software functional unit and sold or used as an independent product. When stored in a computer-readable storage medium, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product stores a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0107] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program is executable by a processor to enable the computer to perform the skull hole pose estimation method provided by the above-mentioned methods, which comprises:

[0108] obtaining a plurality of sets of target binocular two-dimensional circle center coordinates of a plurality of holes in a binocular skull hole image;

[0109] determining a plurality of target three-dimensional circle center coordinates of a plurality of target holes not in a same line in the binocular skull hole image based on the plurality of sets of target binocular two-dimensional circle center coordinates;

[0110] determining normal pose angle estimation values of the plurality of holes in the binocular skull hole image based on the plurality of target three-dimensional circle center coordinates.

[0111] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program is executable by a processor to implement the skull hole pose estimation method provided by the above-mentioned methods, which comprises:

[0112] obtaining a plurality of sets of target binocular two-dimensional circle center coordinates of a plurality of holes in a binocular skull hole image;

[0113] determining a plurality of target three-dimensional circle center coordinates of a plurality of target holes not in a same line in the binocular skull hole image based on the plurality of sets of target binocular two-dimensional circle center coordinates;

[0114] determining normal pose angle estimation values of the plurality of holes in the binocular skull hole image based on the plurality of target three-dimensional circle center coordinates.

[0115] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0116] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0117] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A skull hole pose estimation method, characterized in that, include: Obtain multiple sets of target binocular two-dimensional circle center coordinates for multiple holes in binocular skull images; Based on the multiple sets of target binocular two-dimensional circle center coordinates, determine the multiple target three-dimensional circle center coordinates of multiple non-collinear target holes in the binocular skull hole image; Based on the three-dimensional center coordinates of the multiple targets, the estimated normal attitude angle of the multiple holes in the binocular skull hole image is determined; The acquisition of multiple sets of target binocular two-dimensional circle center coordinates for multiple holes in binocular skull images includes: The binocular skull hole images are input into a preset HoleNet network model to obtain multiple sets of predicted binocular two-dimensional circle center coordinates for multiple holes output by the preset HoleNet network model. Geometric filtering is performed on the multiple sets of predicted binocular 2D circle center coordinates to determine the initial matching multiple sets of binocular 2D circle center coordinates; Image feature filtering is performed on the initially matched multiple sets of binocular two-dimensional circle center coordinates to determine multiple sets of target binocular two-dimensional circle center coordinates for multiple holes in the binocular skull hole image; The preset HoleNet network model is used to extract the low-level image features of the binocular skull hole images, and to learn and predict the low-level image features. Based on the local features of the center of each hole obtained by learning and the predicted center coordinates contained in each hole, multiple sets of predicted binocular two-dimensional center coordinates of the multiple holes are determined. The multiple sets of predicted binocular 2D circle center coordinates include multiple predicted left-eye 2D circle center coordinates distributed in multiple holes in the left-eye skull hole image and multiple predicted right-eye 2D circle center coordinates distributed in multiple holes in the right-eye skull hole image. The step of geometrically filtering the multiple sets of predicted binocular 2D circle center coordinates to determine the initially matched multiple sets of binocular 2D circle center coordinates includes: Determine the first polar line in the right eye skull hole image corresponding to the predicted left eye two-dimensional circle center coordinates among the plurality of predicted left eye two-dimensional circle center coordinates; Delete the predicted right eye two-dimensional circle center coordinates from the plurality of predicted right eye two-dimensional circle center coordinates whose distance from the first epipolar line exceeds the first preset distance threshold, and determine the predicted right eye two-dimensional circle center coordinates of the plurality of targets. Determine the second polar line in the left eye skull hole image corresponding to the right eye two-dimensional circle center coordinates of the multiple target predictions; Delete the predicted left eye two-dimensional circle center coordinates from the plurality of predicted left eye two-dimensional circle center coordinates whose distance from the second polar line exceeds the second preset distance threshold, and determine the predicted left eye two-dimensional circle center coordinates of the plurality of targets. Based on the predicted right-eye 2D circle center coordinates and the predicted left-eye 2D circle center coordinates of the multiple targets, a set of initial matching binocular 2D circle center coordinates are determined.

2. The skull hole pose estimation method of claim 1, wherein, The step of performing image feature filtering on the initially matched multiple sets of binocular two-dimensional circle center coordinates to determine multiple sets of target binocular two-dimensional circle center coordinates for multiple holes in the binocular skull hole image includes: Based on the initial matching of multiple sets of binocular two-dimensional circle center coordinates, feature acquisition is performed on the underlying image features to determine multiple sets of image features; Determine the cosine similarity of features for each group of images; Based on the comparison results of the cosine similarity and the preset cosine similarity threshold, the multiple sets of binocular two-dimensional circle center coordinates of the initial matching are finely filtered to determine the multiple sets of target binocular two-dimensional circle center coordinates of multiple holes in the binocular skull hole image.

3. The skull hole pose estimation method of claim 1, wherein, The training process of the preset HoleNet network model includes: The sample skull hole image and the initial HoleNet network model are determined. The initial HoleNet network model has the functions of low-level feature extraction, skull hole feature learning and skull hole center coordinate prediction. The initial HoleNet network model is trained using the sample skull hole images to determine the preset HoleNet network model.

4. The skull hole attitude estimation method according to any one of claims 1 to 3, characterized in that, The step of determining the coordinates of multiple non-collinear target three-dimensional circles in the binocular skull hole image based on the multiple sets of target binocular two-dimensional circle center coordinates includes: Obtain the intrinsic and extrinsic parameters of the stereo camera; Based on the camera's intrinsic and extrinsic parameters and the multiple sets of target binocular two-dimensional circle center coordinates, the coordinates of multiple target three-dimensional circles of multiple non-collinear target holes in the binocular skull hole image are determined.

5. The skull hole attitude estimation method according to any one of claims 1 to 3, characterized in that, The step of determining the estimated normal attitude angle of the multiple holes in the binocular skull hole image based on the three-dimensional center coordinates of the multiple targets includes: Based on the three-dimensional center coordinates of the multiple targets, the target plane equation of the plane containing the multiple non-collinear target holes in the binocular skull hole image is determined. Determine the normal vector of the target plane equation; Based on the normal vector, the estimated normal attitude angle of the plurality of holes in the binocular skull hole image is determined.

6. A skull opening pose estimation device applying the skull opening pose estimation method as described in any one of claims 1 to 5, characterized in that, include: The acquisition module is used to acquire multiple sets of target binocular two-dimensional circle center coordinates for multiple holes in binocular skull hole images; The determination module is used to determine the coordinates of multiple non-collinear target three-dimensional circles in the binocular skull hole image based on the multiple sets of target binocular two-dimensional circle center coordinates; The attitude estimation module is used to determine the estimated normal attitude angle of the multiple holes in the binocular skull hole image based on the three-dimensional center coordinates of the multiple targets.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cranial hole pose estimation method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cranial hole pose estimation method as described in any one of claims 1 to 5.

Images