Camera positioning method, device and system and nonvolatile storage medium

Abstract

The embodiment of the invention discloses a camera positioning method, device and system and a nonvolatile storage medium. The camera positioning method comprises the following steps: determining a plurality of first tracking points in a first image, and determining a second tracking point corresponding to each first tracking point in the plurality of first tracking points in a second image; a depth map corresponding to the first image is determined, a three-dimensional point corresponding to each first tracking point is determined according to the depth map and the first tracking point coordinate of each first tracking point in a preset rectangular plane coordinate system, and the depth map is used for indicating the depth value of each pixel point in the first image; and according to the three-dimensional point corresponding to each first tracking point and the second tracking point, determining a target pose conversion relation of the pose of the camera during shooting of the second image relative to the pose of the camera during shooting of the first image, and according to the target pose conversion relation, determining pose estimation data of the camera during shooting of the second image.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image data processing, and in particular, to a camera positioning method, device, system and non-volatile storage medium. Background Art

[0002] In the related art, when the bronchoscope moves in the physiological channel, the camera tracking of the bronchoscope is prone to fail. That is to say, there is a significant deviation between the virtual bronchoscope image displayed in the window and the real bronchial image during real-time tracking. When this happens, relying solely on posture registration to adjust the posture of the virtual bronchoscope to achieve repositioning of the bronchoscope requires a lot of computing resources and is difficult to succeed. Summary of the invention

[0003] The embodiments of the present application provide a camera positioning method, device, system and non-volatile storage medium to at least solve the technical problem in the related art that only relying on posture alignment to adjust the posture of the virtual bronchoscope to achieve repositioning of the bronchoscope consumes a large amount of computing power resources and has a low positioning success rate.

[0004] An embodiment of the present application provides a camera positioning method, including: determining multiple first tracking points in a first image, and determining a second tracking point corresponding to each of the multiple first tracking points in a second image, wherein the first image and the second image are physiological channel images continuously captured by the camera in the physiological channel; determining a depth map corresponding to the first image, and determining a three-dimensional point corresponding to each first tracking point based on the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate the depth value of each pixel point in the first image; determining a target posture conversion relationship of the camera's posture when shooting the second image relative to the posture when shooting the first image based on the three-dimensional points corresponding to the first tracking points and the second tracking points, and determining target posture estimation data of the camera when shooting the second image based on the posture conversion relationship.

[0005] Optionally, determining a target posture conversion relationship of the posture of the camera when shooting the second image relative to the posture when shooting the first image based on the three-dimensional point corresponding to the first tracking point and the second tracking point includes: determining multiple point sets, wherein each point set contains a preset number of three-dimensional points and second tracking points corresponding to the three-dimensional points, wherein the three-dimensional points and the second tracking points corresponding to the three-dimensional points correspond to the same first tracking point; for each point set, determining an alternative posture conversion relationship corresponding to the point set based on the point set, wherein the alternative posture conversion relationship includes an estimated change matrix between the posture of the camera when shooting the second image relative to the posture when shooting the first image determined based on the point set; for each alternative posture conversion relationship, determining a projection point in the second image of the three-dimensional point corresponding to each first tracking point based on the alternative posture conversion relationship, and determining a position error value between the projection point and the second tracking point corresponding to each three-dimensional point; for each alternative posture conversion relationship, determining the number of three-dimensional points whose position error values ​​corresponding to the alternative posture conversion relationship are less than a preset value, and determining the alternative posture conversion relationship with the largest number of three-dimensional points whose corresponding position error values ​​are less than the preset value as the target posture conversion relationship.

[0006] Optionally, determining the projection point of the three-dimensional point corresponding to each first tracking point in the second image based on the alternative pose transformation relationship includes: determining the first three-dimensional point coordinates of the three-dimensional point in the first camera coordinate system, and determining the second three-dimensional point coordinates of the three-dimensional point in the second camera coordinate system based on the alternative pose transformation relationship and the first three-dimensional point coordinates, wherein the first camera coordinate system is the camera coordinate system when the camera shoots the first image, and the second camera coordinate system is the camera coordinate system when the camera shoots the second image; based on the second three-dimensional point coordinates of each three-dimensional point, determining the projection point coordinates of the projection point of each three-dimensional point in the second image in a preset plane rectangular coordinate system.

[0007] Optionally, determining the first three-dimensional point coordinates of the three-dimensional point in the first camera coordinate system includes: determining the camera intrinsic parameters of the camera, wherein the camera intrinsic parameters include focal length and principal point position; determining the depth value of each first tracking point based on the depth map; determining the first three-dimensional point coordinates of the three-dimensional point corresponding to each first tracking point based on the depth value of each first tracking point, the camera intrinsic parameters and the first tracking point coordinates of each first tracking point.

[0008] Optionally, determining the second tracking point corresponding to each first tracking point among multiple first tracking points in the second image includes: determining the second tracking point coordinates of the second tracking point in a preset plane rectangular coordinate system; determining the distance between each first tracking point and its corresponding second tracking point based on the first tracking point coordinates of each first tracking point and the second tracking point coordinates of the second tracking point corresponding to each first tracking point; calculating the average distance between each first tracking point and its corresponding second tracking point, and calculating the standard deviation of the distance between each first tracking point and its corresponding second tracking point based on the distance average; calculating a distance threshold based on the distance average and the standard deviation, and eliminating the first tracking points and their corresponding second tracking points whose corresponding distances are greater than the distance threshold.

[0009] Optionally, the camera positioning method also includes: obtaining a virtual image of the physiological channel model captured by a virtual camera corresponding to the camera in the posture indicated by each preset posture data, and screening the optimized posture data of the camera and the virtual camera from each preset posture data based on the similarity between the virtual image and the second image, wherein each preset posture data is the posture data obtained by adding a random posture offset to the posture estimation data as the initial value; and updating the posture estimation data to the optimized posture data.

[0010] Optionally, selecting optimized pose data of the camera and the virtual camera from various preset pose data based on the similarity between the virtual image and the second image includes: determining a root mean square error value between the virtual image and the second image, wherein the root mean square error value is used to characterize the similarity between the virtual image and the second image, and there is a negative correlation between the root mean square error value and the similarity; and determining that the preset pose data corresponding to the virtual image having the smallest root mean square error value is the optimized pose data.

[0011] Optionally, determining the root mean square error value between the virtual image and the second image includes: dividing the virtual image into a plurality of partially overlapping virtual image blocks, and dividing the second image into a plurality of partially overlapping real image blocks; determining virtual feature values ​​of the virtual image blocks, and real feature values ​​of the real image blocks, wherein both the virtual feature values ​​and the real feature values ​​include a standard deviation of the intensity value of the pixel points in the image block, and a whiteness value used to reflect the proportion of the specified pixel points in the image block; deleting the virtual image blocks and the real image blocks whose whiteness values ​​are greater than the preset whiteness value; for the retained virtual image blocks, determining the standard deviation of the intensity value of the pixel points in the virtual image blocks according to the standard deviation of the intensity value of the pixel points in the virtual image blocks; The method comprises the following steps: arranging the real image blocks according to the standard deviation of the pixel intensity values ​​of the real image blocks from large to small to obtain a first virtual image block sequence, and arranging the retained real image blocks according to the standard deviation of the pixel intensity values ​​of the real image blocks from large to small to obtain a first real image block sequence; selecting a preset number of virtual image blocks from the virtual image sequence in a front-to-back order to obtain a second virtual image block sequence, and selecting a preset number of real image blocks from the real image sequence to obtain a second real image block sequence; and determining that the root mean square error value between the second virtual image block sequence and the second real image block sequence is the root mean square error value between the virtual image and the second image.

[0012] Optionally, the designated pixel point in the image block is determined in the following manner: determining the saturation value and brightness value of the pixel point in the image block; determining the pixel point to be the designated pixel point when the saturation value of the pixel point is less than a preset saturation value and the brightness value of the pixel point is greater than a preset brightness value.

[0013] An embodiment of the present application also provides a camera positioning device, including: a matching module, used to determine multiple first tracking points in a first image, and determine the second tracking point corresponding to each first tracking point in the multiple first tracking points in the second image, wherein the first image and the second image are physiological channel images continuously captured by the camera in the physiological channel; a first pose estimation module, used to determine a depth map corresponding to the first image, and determine the three-dimensional point corresponding to each first tracking point based on the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate the depth value of each pixel point in the first image; a second pose estimation module, used to determine the target pose conversion relationship of the camera's pose when shooting the second image relative to the pose when shooting the first image based on the three-dimensional point corresponding to each first tracking point and the second tracking point, and determine the pose estimation data of the camera when shooting the second image according to the target pose conversion relationship.

[0014] An embodiment of the present application also provides a camera positioning system, which includes a camera and an electronic device, wherein the electronic device is used to determine multiple first tracking points in a first image, and determine a second tracking point corresponding to each of the multiple first tracking points in the second image, wherein the first image and the second image are physiological channel images continuously captured by the camera in the physiological channel; determine a depth map corresponding to the first image, and determine a three-dimensional point corresponding to each first tracking point based on the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate the depth value of each pixel point in the first image; determine a target posture conversion relationship between the camera's posture when shooting the second image and the posture when shooting the first image based on the three-dimensional point and the second tracking point corresponding to each first tracking point, and determine the posture estimation data of the camera when shooting the second image based on the target posture conversion relationship.

[0015] The embodiment of the present application further provides a non-volatile storage medium, in which a computer program is stored. When the computer program is executed by a processor, the camera positioning method is implemented.

[0016] An embodiment of the present application also provides a computer program product, including a computer program, which implements a camera positioning method when executed by a processor.

[0017] Based on the above scheme, it can be known that the present application determines the target posture conversion relationship when the camera shoots the second image relative to the first image through the first image and the second image continuously captured by the camera, and determines the posture estimation data when the camera shoots the second image based on the target posture conversion relationship, thereby providing an initial posture estimation value that is closer to the actual posture for the subsequent posture alignment process, reducing the computing power resources required for the alignment process, and improving the success rate of repositioning the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a structural schematic diagram of a camera positioning system in an exemplary embodiment of the present application;

[0020] Figure 2 A schematic diagram of a camera tracking and positioning process in an exemplary embodiment of the present application;

[0021] Figure 3 A schematic diagram of a flow chart of a camera positioning method in an exemplary embodiment of the present application;

[0022] Figure 4 A schematic diagram of a physiological channel image captured by a camera in an exemplary embodiment of the present application;

[0023] Figure 5 is a schematic diagram of image tracking points in an exemplary embodiment of the present application;

[0024] Figure 6 A schematic diagram of matching point pairs in an exemplary embodiment of the present application;

[0025] Figure 7 A schematic diagram of matching point pairs retained after screening in an exemplary embodiment of the present application;

[0026] Figure 8 A schematic diagram of a physiological channel image and its corresponding depth map in an exemplary embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of a camera posture in an exemplary embodiment of the present application;

[0028] Fig.10 It is a schematic diagram of the structure of a physiological channel model in an exemplary embodiment of the present application;

[0029] Fig.11 A schematic diagram of an image preprocessing process in an exemplary embodiment of the present application;

[0030] Fig.12 is a schematic diagram of an image sub-block in an exemplary embodiment of the present application;

[0031] Fig.13 A schematic diagram of a real physiological channel image and a virtual physiological channel image in an exemplary embodiment of the present application;

[0032] Fig.14 It is a structural schematic diagram of a camera positioning device in an exemplary embodiment of the present application;

[0033] Fig.15 It is a schematic diagram of the structure of an electronic device in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0037] Bronchoscopy navigation is an auxiliary technology used in bronchoscopy, which aims to improve the accuracy and safety of bronchoscopy by tracking and positioning the camera in the bronchoscope to guide the movement path of the bronchoscope in the physiological channel. Bronchoscopy is a medical procedure used to examine the trachea and bronchi, and is usually used to diagnose and treat respiratory diseases such as lung cancer, bronchitis and emphysema. When performing bronchoscopy navigation, the following requirements need to be met:

[0038] Improve inspection accuracy: By introducing navigation technology, the anatomical structure of physiological channels such as bronchi can be displayed in real-time images, helping operators to guide bronchoscopes more accurately, thereby improving inspection accuracy. This is crucial for early detection and diagnosis of lesions.

[0039] Reduced discomfort to patients: Traditional bronchoscopy may require repeated insertion of the bronchoscope, which may cause discomfort to the patient and the risk of complications. Bronchoscope navigation can help doctors find the target location more quickly and reduce the number of insertions, thereby reducing patient discomfort.

[0040] Overall, the background of bronchoscopic navigation is based on the pursuit of improving the effectiveness of bronchoscopy and patient experience, using modern technology to provide doctors with more information and tools to perform bronchoscopy more accurately and safely.

[0041] In the related technology, when tracking and positioning the camera in the camera, the commonly used process is offline calculation + posture registration + real-time tracking. The offline calculation is to reconstruct the three-dimensional bronchial tree according to the patient's CT data, calculate the center line and plan the route to the target; the posture registration is to adjust the posture of the virtual camera in the reconstructed three-dimensional bronchial tree, so that the virtual camera and the real camera are in the same position and posture in the bronchial tree and the real lungs respectively; the real-time tracking is to synchronize the virtual camera image and the image captured by the real camera in real time when the tracking is successful.

[0042] However, the camera real-time navigation algorithm in the related art is prone to tracking failure, that is, the virtual camera image displayed in the window and the real bronchial image have obvious deviations during the real-time tracking process. This is especially true when the camera moves too fast in physiological channels such as bronchi, hits the wall, and there are too few physiological channel features (such as airway openings, bifurcations, and folds) in the captured image. In these cases, it is difficult to successfully achieve re-matching and tracking positioning of the real camera and the virtual camera by adjusting the position of the virtual camera based solely on the matching between the virtual camera image and the real image captured by the real camera.

[0043] In order to solve the above problems, a camera positioning method is provided in an embodiment of the present application, so as to provide high-precision initial pose estimation data for the pose alignment process, thereby improving the success rate of re-alignment and tracking positioning of real cameras and virtual cameras, and reducing the required computing power resources.

[0044] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0045] Figure 1 This is a structural schematic diagram of a camera positioning system provided in an embodiment of the present application. The system can be used to execute the camera positioning method provided in an embodiment of the present application. Figure 1 The camera positioning system shown includes a camera 10 and an electronic device 12, wherein:

[0046] The electronic device 12 is used to determine multiple first tracking points in a first image, and to determine the second tracking point corresponding to each first tracking point in the second image, wherein the first image and the second image are physiological channel images continuously captured by the camera 10 in the physiological channel; determine a depth map corresponding to the first image, and determine the three-dimensional point corresponding to each first tracking point based on the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate the depth value of each pixel point in the first image; determine the target posture conversion relationship of the posture of the camera 10 when shooting the second image relative to the posture when shooting the first image based on the three-dimensional point corresponding to each first tracking point and the second tracking point, and determine the target posture estimation data of the camera 10 when shooting the second image based on the posture conversion relationship.

[0047] In some embodiments of the present application, the camera positioning system further includes a bronchoscope, wherein the camera 10 can be integrated into the bronchoscope.

[0048] As an optional method, the complete process of the camera positioning system when performing camera positioning is as follows: Figure 2 As shown in FIG, the process includes four steps: offline data calculation, pose estimation data determination, pose registration, and real-time tracking. In the offline data calculation step, the CT data is mainly processed to reconstruct the three-dimensional model of the physiological channel, such as the three-dimensional lung airway tree. In the pose estimation data determination step, the pose estimation data of the camera is determined based on two frames of images continuously collected by the camera, that is, the execution Figure 3 The camera positioning method shown in . In the pose registration step, the initial pose of the virtual camera is adjusted and optimized based on the pose estimation data, so that the similarity between the virtual image corresponding to the virtual camera and the image taken by the real camera 10 meets the preset requirements, wherein the preset requirements can be that the similarity is greater than a preset similarity threshold, or that the corresponding similarity is the highest among multiple optional initial poses. The real-time tracking step is to update the pose of the virtual camera in the three-dimensional model in real time, and synchronously display the image corresponding to the current virtual camera and the image taken by the real camera 10, so as to provide navigation information for the operator of the bronchoscope.

[0049] Figure 3 A flowchart of a camera positioning method provided in an embodiment of the present application specifically includes the following steps:

[0050] Step 302, determining a plurality of first tracking points in a first image, and determining a second tracking point corresponding to each of the plurality of first tracking points in a second image, wherein the first image and the second image are physiological channel images continuously captured by a camera in the physiological channel;

[0051] Optionally, the first image captured by the camera is as follows: Figure 4 As shown, the first tracking point determined in the first image is as follows Figure 5 As shown. Figure 5 It can be seen from FIG. 1 that when determining the first tracking point, the first image can be divided into a plurality of image blocks of the same size and shape, and the geometric center point of each image block is selected as the first tracking point.

[0052] Optionally, after determining the first tracking point, when determining the coordinates of the second tracking point, a sparse pyramid Lucas-Kanade optical flow algorithm and the first tracking point can be used to determine the position of the second tracking point corresponding to each first tracking point in the second image, and then determine the second tracking point coordinates of the second tracking point.

[0053] The sparse pyramid Lucas-Kanade optical flow algorithm specifically includes the following steps:

[0054] The first step is to initialize the parameters, including the number of layers of the image pyramid, the scale factor of the pyramid, the iteration termination condition, etc.

[0055] In the second step, the input image is pyramided according to the given number of pyramid layers and scale factors, thereby obtaining a series of images of different scales.

[0056] The third step is to use the sparse Lucas-Kanade optical flow algorithm to track the feature points on each layer of the image pyramid. This algorithm uses the grayscale changes in the local image area to calculate the displacement of the feature points and obtain the optical flow field.

[0057] The fourth step is to iteratively optimize the optical flow estimation results such as the optical flow field until the preset termination conditions are met, such as reaching the maximum number of iterations or reaching the specified accuracy requirements.

[0058] The fifth step is to output the final optical flow field, where the optical flow field is used to reflect the displacement information of the first tracking point on the image, that is, the offset information of the position of the second tracking point in the image relative to the position of the first tracking point in the image.

[0059] The second tracking point obtained by the above optical flow field is as follows Figure 6 As shown. Figure 6 The left side is the first tracking point in the first image. Figure 6 The second tracking point in the second image is shown on the right. Figure 6 It can be seen that the first tracking point and the second tracking point with corresponding relationship will be connected.

[0060] In the technical solution provided in step 302, after the second tracking point is determined, since there may be a false association in the process of determining the second tracking point, the first tracking point and the second tracking point that are considered to be associated may not be the projection of the same object. Therefore, the first tracking point and the second tracking point need to be screened. Figure 7 As shown, Figure 7 The left side is the first tracking point, the right side is the second tracking point, and the matched first tracking point and the second tracking point are connected. Optionally, the step of determining the second tracking point corresponding to each of the multiple first tracking points in the second image includes: determining the second tracking point coordinates of the second tracking point in a preset plane rectangular coordinate system; determining the distance between each first tracking point and its corresponding second tracking point based on the first tracking point coordinates of each first tracking point and the second tracking point coordinates of the second tracking point corresponding to each first tracking point; calculating the average distance between each first tracking point and its corresponding second tracking point, and calculating the standard deviation of the distance between each first tracking point and its corresponding second tracking point based on the distance average; calculating a distance threshold based on the distance average and the standard deviation, and eliminating the first tracking point and its corresponding second tracking point whose corresponding distance is greater than the distance threshold.

[0061] Optionally, assuming that set A contains all the first tracking points and set B contains all the second tracking points, in order to facilitate subsequent calculation and expression, the second tracking point matched by the i-th first tracking point in set A is limited to the i-th second tracking point in set B. Then the coordinates of the i-th first tracking point in set A are (x Ai ,y Ai ), the coordinates of the i-th second tracking point that matches the i-th first tracking point in set B are (x Bi ,y Bi ), the distance calculation formula is as follows:

[0062]

[0063] Then, the distance between the first tracking point and the second tracking point in each matching point pair can be determined according to the above formula, and then the average distance MeanDist of all matching point pairs can be determined according to the following formula:

[0064]

[0065] The standard deviation StdDevDist of all matching point pairs can then be determined according to the following formula:

[0066]

[0067] After that, for each matching point pair, if the distance between the matching point pairs meets the following conditions, they will be retained, otherwise they will be eliminated:

[0068] Dist<=MeanDist+Thresh*StdDevDist

[0069] In the above conditions, Thresh is a preset adjustable parameter, which can be set to 1, for example.

[0070] As an optional implementation, when determining the tracking point coordinates of each frame image in a preset plane rectangular coordinate system, the positions of each frame image in the preset plane rectangular coordinate system also remain consistent, for example, the lower left vertex of each image coincides with the origin of the preset plane rectangular coordinate system.

[0071] Step 304, determining a depth map corresponding to the first image, and determining a three-dimensional point corresponding to each first tracking point according to the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate a depth value of each pixel point in the first image;

[0072] In the technical solution provided in step 304, if Figure 8 As shown, the depth value corresponding to each pixel in the first image captured by the camera can be determined by the depth estimation model, so as to obtain Figure 8 The depth map shown on the right side shows that the grayscale value of each point in the depth map is the depth value of the corresponding pixel in the first image.

[0073] Step 306, determining a target posture conversion relationship of the camera's posture when shooting the second image relative to the posture when shooting the first image based on the three-dimensional point corresponding to the first tracking point and the second tracking point, and determining the posture estimation data of the camera when shooting the second image based on the target posture conversion relationship.

[0074] It should be noted that the camera posture data mentioned in this application are all the camera postures in the computed tomography coordinate system (CT coordinate system). It can also be the camera posture in the world coordinate system.

[0075] In the technical solution provided in step 306, determining the posture conversion relationship of the camera's posture when shooting the second image relative to the target posture when shooting the first image based on the three-dimensional point corresponding to each first tracking point and the second tracking point includes: determining multiple point sets, wherein each point set includes a preset number of three-dimensional points and second tracking points corresponding to the three-dimensional points, wherein the three-dimensional points and the second tracking points corresponding to the three-dimensional points correspond to the same first tracking point, and the preset number can be an integer greater than or equal to 4; for each point set, determining the alternative posture conversion relationship corresponding to the point set based on the point set, wherein the alternative posture conversion relationship includes the first tracking point determined based on the point set. an estimated change matrix between the posture of the determined camera when shooting the second image relative to the posture when shooting the first image; for each alternative posture transformation relationship, determine the projection point of each three-dimensional point corresponding to the first tracking point in the second image according to the alternative posture transformation relationship, and determine the position error value between the projection point corresponding to each three-dimensional point and the second tracking point; for each alternative posture transformation relationship, determine the number of three-dimensional points whose position error values ​​corresponding to the alternative posture transformation relationship are less than a preset value, and determine the alternative posture transformation relationship with the largest number of three-dimensional points whose corresponding position error values ​​are less than the preset value as the target posture transformation relationship.

[0076] As an optional implementation, the step of determining the projection point of the three-dimensional point corresponding to each first tracking point in the second image based on an alternative pose transformation relationship includes: determining the first three-dimensional point coordinates of the three-dimensional point in the first camera coordinate system, and determining the second three-dimensional point coordinates of the three-dimensional point in the second camera coordinate system based on the alternative pose transformation relationship and the first three-dimensional point coordinates, wherein the first camera coordinate system is the camera coordinate system when the camera shoots the first image, and the second camera coordinate system is the camera coordinate system when the camera shoots the second image; based on the second three-dimensional point coordinates of each three-dimensional point, determining the projection point coordinates of the projection point of each three-dimensional point in the second image in a preset plane rectangular coordinate system.

[0077] In some embodiments of the present application, the following posture estimation data determination process may also be adopted. The process includes the following stages:

[0078] Data preparation stage: in this stage, input data is prepared, wherein the input data includes first three-dimensional point coordinates of each three-dimensional point and second tracking point coordinates of a second tracking point corresponding to each three-dimensional point;

[0079] Random sampling stage: randomly select a point set from the input data, and the selected point set is used to estimate the alternative pose estimation data of the camera when shooting the second image. Among them, the point set includes a preset number of randomly selected point pairs, each point pair includes a three-dimensional point and a second tracking point corresponding to the three-dimensional point. The above preset number can be the minimum number of point pairs required to estimate the camera pose, usually three or four, and can also be set according to actual needs, and the embodiments of the present application are not limited to this.

[0080] Camera pose solution stage: Based on the point set selected in the random sampling stage, the pose transformation relationship is obtained by solving the PnP (Perspective-n-Point) problem;

[0081] Projection error calculation stage: determine the projection point coordinates of each three-dimensional point in the second image according to the alternative posture transformation relationship, and determine the error between the projection point coordinates and the corresponding second tracking point coordinates.

[0082] Specifically, assume that after filtering the first tracking points and the second tracking points, n point pairs are retained, each of which includes n first tracking points and n second tracking points. The n first tracking points in the first image correspond to n three-dimensional points in the first camera coordinate system, and the coordinates of one of the first tracking points are p i =(u i ,v i ), the corresponding three-dimensional point in the first camera coordinate system has the first three-dimensional point coordinate P i =(x i ,y i ,z i ), the second tracking point coordinates p′ of the corresponding second tracking point in the second image i =(u′ i ,v′ i ), and set the transformation matrix in the alternative pose transformation relationship to T, and find the second three-dimensional point coordinates P′ of each three-dimensional point in the second camera coordinate system i =(x′ i ,y′ i ,z′ i ), then the calculation formula is as follows: P′ i =TP i

[0083] Assume that the camera intrinsic parameters are expressed as a matrix: K , then the projection point coordinates p″ of the projection point of the three-dimensional point in the second image i =(u″ i ,v″ i ) is calculated as follows:

[0084]

[0085] The error calculation formula between the projection point coordinates and the corresponding second tracking point coordinates is as follows:

[0086] Projection Error

[0087] Inlier screening stage: point pairs whose corresponding projection errors are less than the preset error threshold are marked as inliers, and point pairs whose corresponding projection errors are not less than the preset error threshold are marked as outliers.

[0088] Iteration phase: Iterate the steps from the random sampling phase to the interior point screening phase, and after the preset iteration termination condition is met, select the candidate pose transformation relationship with the largest number of interior points as the final target pose transformation relationship. Fig. 9 As shown, the estimated pose data of the camera when shooting the second image is determined based on the finally determined target pose conversion relationship and the known pose of the camera when shooting the first image.

[0089] Specifically, Fig. 9 The subscript CT in the equation represents the world coordinate system, the subscript c represents the camera coordinate system based on the camera, and Q (i-1) is the position of the camera when taking the first image, Q (i) is the position and posture of the camera when taking the second image, and ΔQ(i) is the offset of the position and posture of the camera when taking the second image relative to the position and posture when taking the first image.

[0090] As an optional implementation, the step of determining the first three-dimensional point coordinates of the three-dimensional point in the first camera coordinate system includes: determining the camera intrinsic parameters of the camera, wherein the camera intrinsic parameters include the focal length and the principal point position, the focal length is the distance from the focus of the camera optical system to the imaging plane, and the principal point position is the intersection of the optical axis of the camera and the imaging plane, i.e., the center of the image; determining the depth value of each first tracking point based on the depth map; determining the first three-dimensional point coordinates of the three-dimensional point corresponding to each first tracking point based on the depth value of each first tracking point, the camera intrinsic parameters and the first tracking point coordinates of each first tracking point.

[0091] Optionally, the three-dimensional points (X, Y, Z) corresponding to the respective tracking points (u, v) retained after screening in the first image may be determined according to the depth map, wherein the value of Z may be obtained by accessing the corresponding depth map, and the calculation formulas of X and Y are as follows:

[0092]

[0093] In the above formula, c x , c y , f x , f y is the internal parameter of the camera, where c x, c y are the pixel coordinates of the principal point positions in the horizontal and vertical directions, respectively, and f x , f y are the focal lengths in the horizontal and vertical directions respectively.

[0094] As an optional implementation, during the process of continuous tracking and positioning of the camera, the camera will continuously capture multiple frames of images, and the camera's posture is usually different when capturing different images. In order to facilitate continuous positioning and tracking, the camera posture can be initialized when the camera captures the first frame of image. Specifically, when the bronchoscope where the camera is located reaches some parts with significant features in the physiological channel (such as the carina), the transformation matrix from the camera to the CT coordinate system can be calculated at this time T ; The subsequent update process of each frame is to calculate the position increment of the adjacent image frames through the above steps, that is, the transformation matrix of the camera from the previous moment to this moment I ,Finally, solve the current pose C=TI of the camera in the CT coordinate system (that is, the world coordinate system) when capturing the current frame.

[0095] In some embodiments of the present application, after determining the pose estimation data, the camera positioning method further includes: obtaining a virtual image of the physiological channel model captured by a virtual camera corresponding to the camera at the pose indicated by each preset pose data, and filtering out the optimized pose data of the camera and the virtual camera from each preset pose data according to the similarity between the virtual image and the second image, wherein each preset pose data is the pose data obtained by adding a random pose offset to the pose estimation data as the initial value; updating the pose estimation data to the optimized pose data. Wherein the physiological channel model is as follows: Fig.10 shown.

[0096] Specifically, in the process of tracking and positioning the camera, since the collected image data will contain certain noise, and the mismatched first tracking points and second tracking points may not be sufficiently eliminated in the process of screening the second tracking points, there will be a certain deviation between the determined pose estimation data and the actual pose data when the camera took the second image. Therefore, it is also necessary to align the virtual image and the second image captured by the virtual camera to accurately determine the pose data of the second camera when taking the second image, and synchronize the poses of the virtual camera and the real camera, that is, the relative position and orientation of the virtual camera in the physiological channel model are the same as the relative position and orientation of the real camera in the physiological channel.

[0097] When registering the virtual image and the second image, a random pose offset can be added to the pose estimation data as the initial value to obtain multiple preset pose data, and the virtual image captured by the virtual camera when the pose is the preset pose data can be determined. This step can be achieved by using rendering technology to obtain a 2D VB image. After that, the similarity between the rendered 2D image data and the second image captured by the real camera at this time can be compared, and the corresponding preset pose data with the largest similarity can be determined as the optimized pose data of the virtual camera in the physiological channel model, which is also the optimized pose data of the camera in the physiological channel.

[0098] As an optional implementation, the step of screening the optimized pose data of the camera and the virtual camera from various preset pose data based on the similarity between the virtual image and the second image includes: determining the root mean square error value between the virtual image and the second image, wherein the root mean square error value is used to characterize the similarity between the virtual image and the second image, and there is a negative correlation between the root mean square error value and the similarity; determining the preset pose data corresponding to the virtual image with the smallest root mean square error value as the optimized pose data.

[0099] Specifically, the optimization formula for optimizing the pose data according to the root mean square error value is as follows:

[0100]

[0101] In the above formula, ΔQ (i) is the offset of the camera's pose when taking the second image relative to the pose when taking the first image, C represents the pose estimation data, ΔQ represents the random pose offset, represents the second image, I V (CΔQ) represents the virtual image obtained after adding the random pose offset, represents the minimum value of the root mean square error between the second image and the virtual image. The specific meaning of the above optimization formula is to use C as the initial estimate, and then use the BOBYQA algorithm to calculate the update amount ΔQ so that the rendered image Images captured by real cameras Most similar.

[0102] ΔQ (i) It is the offset of the position of the camera when taking the second image relative to the position when taking the first image. arg is the abbreviation of minimizer, which means that the above formula is to find a ΔQ that minimizes the root mean square error value.

[0103] Among them, the BOBYQA (Bound Optimization BY Quadratic Approximation) algorithm is an algorithm for numerical optimization, especially suitable for solving nonlinear constrained optimization problems without gradient information. The BOBYQA algorithm will stop iterating when the stopping condition is met. The stopping condition can be that the change value of the objective function is less than the preset threshold or the iteration reaches the maximum number of iterations.

[0104] The BOBYQA algorithm uses an iterative optimization process to find possible local optimal solutions without relying on the objective function gradient. This makes the algorithm have a wider range of applications when solving optimization problems where gradient information cannot be directly obtained.

[0105] As an optional implementation, the registration process may also be implemented by manually adjusting the posture corresponding to the virtual image.

[0106] In some embodiments of the present application, Fig.11 As shown, the process of determining the root mean square error value between the virtual image and the second image can be divided into an image division stage, an eigenvalue calculation stage, a sub-block selection stage and a similarity calculation stage, and specifically includes the following steps: dividing the virtual image into a plurality of partially overlapping virtual image blocks, and dividing the second image into a plurality of partially overlapping real image blocks; determining virtual eigenvalues ​​of the virtual image blocks, and real eigenvalues ​​of the real image blocks, wherein both the virtual eigenvalues ​​and the real eigenvalues ​​include the standard deviation of the intensity value of the pixel points in the image block, and the whiteness value used to reflect the proportion of the specified pixel points in the image block; deleting the virtual image blocks and the real image blocks whose whiteness values ​​are greater than the preset whiteness value; and performing similarity calculation on the retained virtual image blocks. blocks, arranging the virtual image blocks in descending order according to the standard deviation of the pixel intensity values ​​of the virtual image blocks to obtain a first virtual image block sequence, and arranging the retained real image blocks in descending order according to the standard deviation of the pixel intensity values ​​of the real image blocks to obtain a first real image block sequence; selecting a preset number of virtual image blocks from the virtual image sequence in a front-to-back order to obtain a second virtual image block sequence, and selecting a preset number of real image blocks from the real image sequence to obtain a second real image block sequence; determining a root mean square error value between the second virtual image block sequence and the second real image block sequence as a root mean square error value between the virtual image and the second image.

[0107] Specifically, in Fig.11 In the image division stage, the k-th frame image B (k) (which can be an image captured by any camera, such as the first image or the second image) is divided into Fig.12 The M×N sub-blocks shown overlap each other. Overlapping the sub-blocks can prevent the small structure from being divided into two blocks.m,n is defined as:

[0108]

[0109] Where m and n satisfy the following constraints:

[0110] 2≤m≤M-1 and 2≤n≤N-1.

[0111] In the above constraints, W and H Respectively represent the width and height of the input image, M and N represent the number of image width and height divisions, respectively, and p and q represent the large sub-block D m,n The value range of the horizontal axis (width) and vertical axis (height), m and n represent the number of small blocks on the horizontal and vertical axes respectively.

[0112] When calculating the eigenvalue, for each sub-block D m,n , whose characteristic values ​​include the standard deviation of the intensity value of the sub-block and the whiteness of the sub-block. The intensity value can be a grayscale value or a pixel value. The whiteness of the sub-block is the number of specified pixels in the sub-block, where the specified pixels can be pixels that are considered white.

[0113] Sub-block D m,n Standard deviation of intensity values The calculation formula is as follows:

[0114]

[0115] Where |D m,n | is sub-block D m,n The number of pixels inside, is the average of all pixel intensities in this sub-block.

[0116] As an optional implementation, the designated pixel point in the image block is determined in the following manner: determining the saturation value and brightness value of the pixel point in the image block; determining the pixel point as the designated pixel point when the saturation value of the pixel point is less than a preset saturation value and the brightness value of the pixel point is greater than a preset brightness value.

[0117] Specifically, sub-block D m,n Whiteness The calculation formula is as follows:

[0118]

[0119] Where W is a function used to determine whether the pixel is white. The function is described as follows:

[0120]

[0121] In the formula, S and B correspond to the saturation value and brightness value of the pixel in the HSV space. The formula means that the saturation value S(v) of the pixel is less than or equal to a certain threshold T s And the brightness value B(v) is greater than a certain threshold T B When it is white, the value is 1.

[0122] In some embodiments of the present application, the following steps may be used to implement the selection of the sub-block in the (k)th image:

[0123] The first step is to add all sub-blocks of the (k)th image to the candidate list A (k) middle.

[0124] In the second step, remove the candidates whose whiteness value is greater than a certain threshold T from the candidate list. w The sub-blocks, that is

[0125] The third step is to calculate the standard deviation of the intensity value. Arrange the remaining sub-blocks in descending order.

[0126] Step 4. In List A (k) Keep the first α·M·N and remove the remaining sub-blocks. (α is a proportional coefficient, its value range is 0≤α≤1, M·N is the total number of small sub-blocks of the entire image).

[0127] Through the above steps, it is possible to select only sub-blocks with characteristic structures for similarity calculation when determining similarity, such as airway bifurcations or folded parts. Image sub-blocks with these characteristic structures have a higher intensity standard deviation. However, sub-blocks with alveoli also have a higher intensity standard deviation, but alveolar image sub-blocks will interfere with the similarity judgment results. Therefore, the influence of alveoli is excluded by comprehensively considering the saturation and brightness values ​​of the sub-blocks, that is, judging whether the whiteness value of the sub-block exceeds a certain threshold to exclude the image sub-block with alveoli.

[0128] The virtual image I obtained by rendering the virtual camera can then be processed in the above manner. V The same process is performed to obtain a set of real image sub-blocks arranged in descending order of intensity standard deviation and a set of virtual image sub-blocks arranged in descending order of intensity standard deviation. Then, the similarity between the real image and the virtual image can be calculated based on the two sets of image sub-blocks according to the following formula:

[0129]

[0130] Where |A (k) | is list A (k) The number of remaining sub-block pairs in |D| represents the number of pixels in the sub-block. and The average intensity of sub-block D corresponding to the real image and virtual image respectively. and Corresponding to the intensity value of each pixel in the real image and virtual image sub-block D. is the root mean square error between the virtual image and the real image, that is, the similarity between the real image and the virtual image.

[0131] The present application determines the posture conversion relationship when the camera shoots the second image relative to the posture when the camera shoots the first image through the first image and the second image continuously captured by the camera, and determines the posture estimation data when the camera shoots the second image based on the posture conversion relationship, thereby providing an initial posture estimation value that is closer to the actual posture for the subsequent posture alignment process, reducing the computing power resources required for the alignment process, and improving the success rate of repositioning the camera.

[0132] After adopting the camera positioning method provided in the embodiment of the present application, the images captured by the camera and its corresponding virtual camera are as follows: Fig.13 As shown, Fig.13 The left side shows the real physiological channel image, and the right side shows the virtual physiological channel image.

[0133] Fig.14 A structural block diagram of a camera positioning device provided in an embodiment of the present application, the device comprising:

[0134] A matching module 140 is used to determine a plurality of first tracking points in a first image, and to determine a second tracking point corresponding to each of the plurality of first tracking points in the second image, wherein the first image and the second image are physiological channel images continuously captured by a camera in a physiological channel; a first pose estimation module 142 is used to determine a depth map corresponding to the first image, and to determine a three-dimensional point corresponding to each first tracking point based on the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate the depth value of each pixel point in the first image; a second pose estimation module 144 is used to determine a target pose conversion relationship of the pose of the camera when shooting the second image relative to the pose when shooting the first image based on the three-dimensional point corresponding to each first tracking point and the second tracking point, and to determine target pose estimation data of the camera when shooting the second image based on the pose conversion relationship.

[0135] In some embodiments of the present application, the step of the matching module 140 determining the second tracking point corresponding to each first tracking point among multiple first tracking points in the second image includes: determining the second tracking point coordinates of the second tracking point in a preset plane rectangular coordinate system; determining the distance between each first tracking point and its corresponding second tracking point based on the first tracking point coordinates of each first tracking point and the second tracking point coordinates of the second tracking point corresponding to each first tracking point; calculating the average distance between each first tracking point and its corresponding second tracking point, and calculating the standard deviation of the distance between each first tracking point and its corresponding second tracking point based on the distance average; calculating a distance threshold based on the distance average and the standard deviation, and eliminating the first tracking points and their corresponding second tracking points whose corresponding distances are greater than the distance threshold.

[0136] In some embodiments of the present application, the second pose estimation module 144 determines the target pose conversion relationship of the pose of the camera when shooting the second image relative to the pose when shooting the first image based on the three-dimensional point corresponding to each first tracking point and the second tracking point, including: determining multiple point sets, wherein each point set contains a preset number of three-dimensional points and second tracking points corresponding to the three-dimensional points, wherein the three-dimensional points and the second tracking points corresponding to the three-dimensional points correspond to the same first tracking point; for each point set, determining the alternative pose conversion relationship corresponding to the point set based on the point set, wherein the alternative pose conversion relationship includes an estimated change matrix between the pose of the camera when shooting the second image relative to the pose when shooting the first image determined based on the point set; for each alternative pose conversion relationship, determining the projection point of the three-dimensional point corresponding to each first tracking point in the second image based on the alternative pose conversion relationship, and determining the position error value between the projection point and the second tracking point corresponding to each three-dimensional point; for each alternative pose conversion relationship, determining the number of three-dimensional points whose position error value corresponding to the alternative pose conversion relationship is less than the preset value, and determining the alternative pose conversion relationship with the largest number of three-dimensional points whose corresponding position error value is less than the preset value as the target pose conversion relationship.

[0137] In some embodiments of the present application, the step of the second pose estimation module 144 determining the projection point of the three-dimensional point corresponding to each first tracking point in the second image based on the alternative pose transformation relationship includes: determining the first three-dimensional point coordinates of the three-dimensional point in the first camera coordinate system, and determining the second three-dimensional point coordinates of the three-dimensional point in the second camera coordinate system based on the alternative pose transformation relationship and the first three-dimensional point coordinates, wherein the first camera coordinate system is the camera coordinate system when the camera shoots the first image, and the second camera coordinate system is the camera coordinate system when the camera shoots the second image; based on the second three-dimensional point coordinates of each three-dimensional point, determining the projection point coordinates of the projection point of each three-dimensional point in the second image in a preset plane rectangular coordinate system.

[0138] In some embodiments of the present application, the step in which the second pose estimation module 144 determines the first three-dimensional point coordinates of the three-dimensional point in the first camera coordinate system includes: determining the camera intrinsic parameters of the camera, wherein the camera intrinsic parameters include the focal length and the principal point position; determining the depth value of each first tracking point based on the depth map; determining the first three-dimensional point coordinates of the three-dimensional point corresponding to each first tracking point based on the depth value of each first tracking point, the camera intrinsic parameters and the first tracking point coordinates of each first tracking point.

[0139] In some embodiments of the present application, the camera positioning device is also used to: obtain a virtual image of the physiological channel model captured by a virtual camera corresponding to the camera in the posture indicated by each preset posture data, and filter out the optimized posture data of the camera and the virtual camera from each preset posture data based on the similarity between the virtual image and the second image, wherein each preset posture data is the posture data obtained by adding a random posture offset to the posture estimation data as the initial value; and update the posture estimation data to the optimized posture data.

[0140] In some embodiments of the present application, the step of the camera positioning device screening out the optimized pose data of the camera and the virtual camera from various preset pose data based on the similarity between the virtual image and the second image includes: determining the root mean square error value between the virtual image and the second image, wherein the root mean square error value is used to characterize the similarity between the virtual image and the second image, and there is a negative correlation between the root mean square error value and the similarity; determining the preset pose data corresponding to the virtual image with the smallest root mean square error value as the optimized pose data.

[0141] In some embodiments of the present application, the step of determining the root mean square error value between the virtual image and the second image by the camera positioning device includes: dividing the virtual image into a plurality of partially overlapping virtual image blocks, and dividing the second image into a plurality of partially overlapping real image blocks; determining virtual feature values ​​of the virtual image blocks, and real feature values ​​of the real image blocks, wherein both the virtual feature values ​​and the real feature values ​​include the standard deviation of the intensity value of the pixel points in the image block, and the whiteness value used to reflect the proportion of the specified pixel points in the image block; deleting the virtual image blocks and the real image blocks whose whiteness values ​​are greater than the preset whiteness value; for the retained virtual image blocks, according to each virtual image block The pixel intensity value standard deviations of the real image blocks are arranged in descending order to obtain a first virtual image block sequence, and the retained real image blocks are arranged in descending order according to the pixel intensity value standard deviations of each real image block to obtain a first real image block sequence; a preset number of virtual image blocks are selected from the virtual image sequence in a front-to-back order to obtain a second virtual image block sequence, and a preset number of real image blocks are selected from the real image sequence to obtain a second real image block sequence; a root mean square error value between the second virtual image block sequence and the second real image block sequence is determined as a root mean square error value between the virtual image and the second image.

[0142] In some embodiments of the present application, a designated pixel point in an image block is determined in the following manner: a saturation value and a brightness value of the pixel point in the image block are determined; and when the saturation value of the pixel point is less than a preset saturation value and the brightness value of the pixel point is greater than a preset brightness value, the pixel point is determined to be a designated pixel point.

[0143] See also Fig.15 , a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.15 As shown, the electronic device includes: a memory 1501 and a processor 1502 .

[0144] The memory 1501 stores an executable computer program 1503. The processor 1502 coupled to the memory 1501 calls the executable computer program 1503 stored in the memory to execute the camera positioning method provided in the above embodiment.

[0145] Exemplarily, the computer program 1503 may be divided into one or more modules / units, which are stored in the memory 1501 and executed by the processor 1502 to complete the present application. The one or more modules / units may include various modules in the camera positioning device in the above-mentioned embodiment, such as a matching module, a first pose estimation module, and a second pose estimation module.

[0146] Furthermore, the apparatus also includes: at least one input device and at least one output device.

[0147] The processor 1502 , memory 1501 , input device and output device may be connected via a bus.

[0148] The input device may specifically be a camera, a touch panel, a physical button or a mouse, etc. The output device may specifically be a display screen.

[0149] Furthermore, the device may include more components than those shown in the figure, or a combination of certain components, or different components, such as network access equipment, sensors, etc.

[0150] The processor 1502 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0151] The memory 1501 may be, for example, a hard disk drive memory, a non-volatile memory (such as a flash memory or other electronically programmable memory with limited deletion for forming a solid-state drive, etc.), a volatile memory (such as a static or dynamic random access memory, etc.), etc., and the embodiments of the present application are not limited thereto. Specifically, the memory 1501 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. The memory 1501 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 1501 may also include both an internal storage unit of the electronic device and an external storage device. The memory 1501 is used to store computer programs and other programs and data required by the terminal. The memory 1501 may also be used to temporarily store data that has been output or is to be output.

[0152] The present application also provides a computer-readable non-volatile storage medium, which can be disposed in the electronic device in the above embodiments. The computer-readable storage medium can be the above Fig.15The memory 1501 in the illustrated embodiment. The computer readable storage medium stores a computer program, and when the program is executed by the processor, the method for adjusting the target position based on the image described in the aforementioned embodiments is implemented. Further, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, etc., which can store program codes.

[0153] An embodiment of the present application also provides a computer program product, including a computer program, which implements the following camera positioning method when executed by a processor: determining multiple first tracking points in a first image, and determining a second tracking point corresponding to each of the multiple first tracking points in the second image, wherein the first image and the second image are physiological channel images continuously captured by the camera in the physiological channel; determining a depth map corresponding to the first image, and determining a three-dimensional point corresponding to each first tracking point based on the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate the depth value of each pixel point in the first image; determining a target posture conversion relationship of the camera's posture when shooting the second image relative to the posture when shooting the first image based on the three-dimensional point corresponding to each first tracking point and the second tracking point, and determining the posture estimation data of the camera when shooting the second image based on the target posture conversion relationship.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0155] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0157] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned readable storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0158] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples, without contradiction.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A camera positioning method, characterized in that: include: Determine a plurality of first tracking points in a first image, and determine a second tracking point corresponding to each of the plurality of first tracking points in a second image, wherein the first image and the second image are physiological channel images continuously captured by a camera in the physiological channel; Determine a depth map corresponding to the first image, and determine a three-dimensional point corresponding to each first tracking point according to the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate a depth value of each pixel point in the first image; A target posture conversion relationship between the posture of the camera when shooting the second image and the posture when shooting the first image is determined based on the three-dimensional point corresponding to the first tracking point and the second tracking point, and posture estimation data of the camera when shooting the second image is determined based on the target posture conversion relationship.

2. The camera positioning method according to claim 1, characterized in that: Determining a target posture conversion relationship of the posture of the camera when shooting the second image relative to the posture when shooting the first image based on the three-dimensional point corresponding to the first tracking point and the second tracking point includes: Determine a plurality of point sets, wherein each of the point sets includes a preset number of the three-dimensional points and the second tracking points corresponding to the three-dimensional points, wherein the three-dimensional points and the second tracking points corresponding to the three-dimensional points correspond to the same first tracking point; For each of the point sets, determining an alternative posture transformation relationship corresponding to the point set according to the point set, wherein the alternative posture transformation relationship includes an estimated change matrix between the posture of the camera when shooting the second image and the posture when shooting the first image, determined according to the point set; For each of the candidate posture conversion relationships, determine a projection point of the three-dimensional point corresponding to each of the first tracking points in the second image according to the candidate posture conversion relationship, and determine a position error value between the projection point corresponding to each of the three-dimensional points and the second tracking point; For each of the alternative posture transformation relationships, determine the number of three-dimensional points whose position error values ​​corresponding to the alternative posture transformation relationships are less than a preset value, and determine the alternative posture transformation relationship with the largest number of three-dimensional points whose corresponding position error values ​​are less than the preset value as the target posture transformation relationship.

3. The camera positioning method according to claim 2, characterized in that: Determining the projection point of the three-dimensional point corresponding to each first tracking point in the second image according to the candidate posture conversion relationship includes: Determine first three-dimensional point coordinates of the three-dimensional point in a first camera coordinate system, and determine second three-dimensional point coordinates of the three-dimensional point in a second camera coordinate system based on the alternative posture transformation relationship and the first three-dimensional point coordinates, wherein the first camera coordinate system is the camera coordinate system when the camera captures the first image, and the second camera coordinate system is the camera coordinate system when the camera captures the second image; According to the second three-dimensional point coordinates of each of the three-dimensional points, the projection point coordinates of the projection point of each of the three-dimensional points in the second image in the preset plane rectangular coordinate system are determined.

4. The camera positioning method according to claim 3, characterized in that: Determining the first three-dimensional point coordinates of the three-dimensional point in the first camera coordinate system includes: Determining the camera intrinsic parameters of the camera, wherein the camera intrinsic parameters include focal length and principal point position; Determining a depth value of each first tracking point according to the depth map; The first three-dimensional point coordinates of the three-dimensional point corresponding to each first tracking point are determined according to the depth value of each first tracking point, the camera intrinsic parameter and the first tracking point coordinates of each first tracking point.

5. The camera positioning method according to claim 1, characterized in that: Determining a second tracking point corresponding to each first tracking point in the second image includes: Determine the coordinates of the second tracking point of the second tracking point in the preset plane rectangular coordinate system; determining a distance between each first tracking point and the second tracking point corresponding to each first tracking point according to the first tracking point coordinates of each first tracking point and the second tracking point coordinates of the second tracking point corresponding to each first tracking point; Calculating an average value of the distance between each of the first tracking points and its corresponding second tracking point, and calculating a standard deviation of the distance between each of the first tracking points and its corresponding second tracking point based on the average value of the distance; A distance threshold is calculated according to the distance average and the standard deviation, and the first tracking point and the corresponding second tracking point whose corresponding distance is greater than the distance threshold are eliminated.

6. The camera positioning method according to claim 1, characterized in that: Also includes: Acquire a virtual image of the physiological channel model captured by a virtual camera corresponding to the camera at a posture indicated by each preset posture data, and select optimized posture data of the camera and the virtual camera from the preset posture data according to the similarity between the virtual image and the second image, wherein each preset posture data is posture data obtained by adding a random posture offset to the posture estimation data as an initial value; The estimated pose data is updated to the optimized pose data.

7. The camera positioning method according to claim 6, characterized in that: Filtering out the optimized pose data of the camera and the virtual camera from the respective preset pose data according to the similarity between the virtual image and the second image comprises: Determine a root mean square error value between the virtual image and the second image, wherein the root mean square error value is used to characterize the similarity between the virtual image and the second image, and the root mean square error value and the similarity are negatively correlated; Determine that the preset posture data corresponding to the virtual image with the smallest root mean square error value is the optimized posture data.

8. The camera positioning method according to claim 7, characterized in that: Determining a root mean square error value between the virtual image and the second image includes: Dividing the virtual image into a plurality of partially overlapping virtual image blocks, and dividing the second image into a plurality of partially overlapping real image blocks; Determine a virtual feature value of the virtual image block and a real feature value of the real image block, wherein both the virtual feature value and the real feature value include a standard deviation of intensity values ​​of pixels in the image block and a whiteness value used to reflect a proportion of specified pixels in the image block; Deleting the virtual image block and the real image block whose whiteness value is greater than a preset whiteness value; The retained virtual image blocks are arranged in descending order according to the standard deviation of the pixel point intensity values ​​of the virtual image blocks to obtain a first virtual image block sequence, and the retained real image blocks are arranged in descending order according to the standard deviation of the pixel point intensity values ​​of the real image blocks to obtain a first real image block sequence; Selecting a preset number of the virtual image blocks from the virtual image sequence in order from front to back to obtain a second virtual image block sequence, and selecting the preset number of the real image blocks from the real image sequence to obtain a second real image block sequence; A root mean square error value between the second virtual image block sequence and the second real image block sequence is determined as a root mean square error value between the virtual image and the second image.

9. The camera positioning method according to claim 8, characterized in that: The specified pixel point in the image block is determined by: Determine the saturation value and brightness value of the pixel points in the image block; When the saturation value of the pixel point is less than a preset saturation value and the brightness value of the pixel point is greater than a preset brightness value, the pixel point is determined to be the designated pixel point.

10. A camera positioning device, characterized in that: include: A matching module, configured to determine a plurality of first tracking points in a first image, and to determine a second tracking point corresponding to each of the plurality of first tracking points in a second image, wherein the first image and the second image are physiological channel images continuously captured by a camera in the physiological channel; a first pose estimation module, configured to determine a depth map corresponding to the first image, and determine a three-dimensional point corresponding to each first tracking point according to the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate a depth value of each pixel point in the first image; A second posture estimation module is used to determine the target posture transformation relationship of the posture of the camera when shooting the second image relative to the posture when shooting the first image based on the three-dimensional point corresponding to the first tracking point and the second tracking point, and determine the posture estimation data of the camera when shooting the second image based on the target posture transformation relationship.

11. A camera positioning system, characterized in that: The camera positioning system comprises a camera and an electronic device, wherein: The electronic device is used to determine multiple first tracking points in a first image, and determine the second tracking point corresponding to each first tracking point in the multiple first tracking points in the second image, wherein the first image and the second image are physiological channel images continuously captured by a camera in a physiological channel; determine a depth map corresponding to the first image, and determine the three-dimensional point corresponding to each first tracking point based on the depth map and the first tracking point coordinates of each first tracking point in a preset plane rectangular coordinate system, wherein the depth map is used to indicate the depth value of each pixel point in the first image; determine the target posture conversion relationship of the camera's posture when shooting the second image relative to the posture when shooting the first image based on the three-dimensional point corresponding to the first tracking point and the second tracking point, and determine the posture estimation data of the camera when shooting the second image based on the target posture conversion relationship.

12. A non-volatile storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images