Camera fusion positioning method, device and system and nonvolatile storage medium

Abstract

The embodiment of the invention discloses a camera fusion positioning method, device and system and a nonvolatile storage medium. The camera fusion positioning method comprises the steps that first pose data and at least one kind of second pose data when a camera in a physiological channel shoots a target image are acquired, and pose data determination modes corresponding to the first pose data and the second pose data are different; determining a first motion trend corresponding to the first pose data and a second motion trend corresponding to the second pose data; determining a first virtual image and a second virtual image collected by a virtual camera corresponding to the camera in a physiological channel model of the physiological channel, and determining a first similarity between the first virtual image and the target image and a second similarity between the second virtual image and the target image; and determining target pose data when the camera shoots the target image according to the first motion trend, the second motion trend, the first similarity and the second similarity.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of navigation and positioning, and in particular to a camera fusion positioning method, device, system and non-volatile storage medium. Background Art

[0002] In the related art, when positioning and tracking the camera at the end of the endoscope, a single positioning method is usually used, which easily leads to the problem of tracking failure. That is, during the real-time tracking process, the virtual physiological channel image displayed in the window and the real physiological channel image have obvious deviations, resulting in the inability to accurately determine the position of the camera in the physiological channel. Summary of the invention

[0003] The embodiments of the present application provide a camera fusion positioning method, device, system and non-volatile storage medium to at least solve the technical problem that the tracking and positioning of the camera is prone to failure due to the use of a single positioning method in the related art.

[0004] The embodiment of the present application provides a camera fusion positioning method, including: obtaining first pose data and at least one second pose data of a camera in a physiological channel when shooting a target image, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is the camera pose data determined based on the initial pose data and motion state data of a robotic arm, and the robotic arm is used to control the motion state of the camera; determining a first motion trend corresponding to the first pose data, and a second motion trend corresponding to the second pose data, wherein the trend types of the first motion trend and the second motion trend include forward and backward; determining a first virtual image and a second virtual image captured by a virtual camera corresponding to the camera in a physiological channel model of the physiological channel, and determining a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image obtained by the virtual camera at the pose indicated by the first pose data, and the second virtual image is an image obtained by the virtual camera at the pose indicated by the second pose data; determining the target pose data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity.

[0005] Optionally, each type of second posture data corresponds to a second motion trend and a second similarity; determining the target posture data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity includes: selecting alternative second posture data from at least one type of second posture data, wherein the alternative second posture data is second posture data whose corresponding second motion trend is consistent with the first motion trend; jointly determining the target posture data based on the alternative second posture data, the first posture data, the second similarity corresponding to the alternative second posture data, and the first similarity.

[0006] Optionally, determining the target pose data based on the alternative second pose data, the first pose data, the second similarity corresponding to the alternative second pose data, and the first similarity includes: taking the first pose data as the pose data default value, and taking the first similarity as the similarity default value; determining the alternative second pose data with the largest corresponding second similarity as the target alternative second pose data; when the second similarity corresponding to the target alternative second pose data is greater than the similarity default value, updating the similarity default value to the second similarity corresponding to the target alternative second pose data, and updating the pose data default value to the target alternative second pose data; determining the pose data default value to the target pose data, and determining the similarity default value to the target similarity when the camera shoots the target image.

[0007] Optionally, determining a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image includes: dividing the first virtual image into a plurality of partially overlapping first virtual image blocks, dividing the second virtual image into a plurality of partially overlapping second virtual image blocks, and dividing the target image into a plurality of partially overlapping real image blocks; determining a first virtual feature value of the first virtual image block, a second virtual feature value of the second virtual image block, and a real feature value of the real image block, wherein the first virtual feature value, the second virtual feature value, and the real feature value all include a standard deviation of pixel intensity values ​​within the image block, and a whiteness value used to reflect a proportion of specified pixels in the image block; deleting the first virtual image block, the second virtual image block, and the real image block whose whiteness values ​​are greater than a preset whiteness value; arranging the retained first virtual image blocks in descending order according to the standard deviation of the pixel intensity values ​​of each first virtual image block to obtain a first virtual image block sequence, and deleting the retained first virtual image blocks. The second virtual image blocks are arranged in descending order according to the standard deviation of the pixel intensity values ​​of each second virtual image block to obtain a second virtual image block sequence; the retained real image blocks are arranged in descending order according to the standard deviation of the pixel intensity values ​​of each real image block to obtain a first real image block sequence; a preset number of virtual image blocks are selected from the first virtual image sequence in a front-to-back order to obtain a third virtual image block sequence, and a preset number of virtual image blocks are selected from the second virtual image sequence in a front-to-back order to obtain a fourth virtual image block sequence; a preset number of real image blocks are selected from the real image sequence to obtain a second real image block sequence; a first root mean square error value between the third virtual image block sequence and the second real image block sequence is determined, and a second root mean square error value between the fourth virtual image block sequence and the second real image block sequence is determined, wherein the first root mean square error value is used to characterize the first similarity, and the second root mean square error value is used to characterize the second similarity.

[0008] Optionally, the designated pixel point in the image block is determined in the following manner: determining the saturation value and brightness value of the pixel point in the image block; determining the pixel point to be the designated pixel point when the saturation value of the pixel point is less than a preset saturation value and the brightness value of the pixel point is greater than a preset brightness value.

[0009] Optionally, determining the first motion trend corresponding to the first posture data and the second motion trend corresponding to the second posture data includes: determining a reference image and reference posture data of the camera when shooting the reference image, wherein the reference image and the target image are two frames of images continuously shot by the camera in the physiological channel, and the camera first captures the reference image and then captures the target image; determining a first translation increment in a preset direction relative to that when the reference image is shot based on the reference posture data and the first posture data when the camera shoots the target image, and determining a second translation increment in a preset direction relative to that when the reference image is shot based on the reference posture data and the second posture data when the camera shoots the target image; determining a trend type of the first motion trend based on the positive or negative value of the first translation increment, and determining a trend type of the second motion trend based on the positive or negative value of the second translation increment.

[0010] Optionally, the preset direction includes the extension direction of the lens of the camera when shooting the reference image. Optionally, after determining the target posture data of the camera when shooting the target image, the camera fusion positioning method also includes: obtaining a plurality of frames of real images shot continuously by the camera in time sequence, and the third similarity corresponding to each frame of the real image, wherein the target image is the last frame of the plurality of frames of real images, and the third similarity of the real image is the similarity determined based on the first similarity and the second similarity corresponding to the real image; determining the tracking state index of the target image based on the third similarity corresponding to each frame of the real image, wherein the tracking state index includes a state difference and a state excellence; and updating the target posture data when determining that the target image is a key frame image and the corresponding tracking state index is a state difference.

[0011] Optionally, determining the tracking state index of the target image based on the third similarity corresponding to each frame of real image includes: calculating the sum of the third similarities corresponding to each frame of real image; when the sum of the third similarities is not less than a preset threshold, determining the tracking state index to be in an excellent state; when the sum of the third similarities is less than a preset threshold, determining the tracking state index to be in a poor state.

[0012] Optionally, at least one second posture data includes second posture data determined based on a target image and a reference image, wherein the reference image and the target image are two frames of images continuously captured by a camera in a physiological channel, and the camera first captures the reference image and then captures the target image.

[0013] Optionally, the at least one second posture data includes second posture data determined based on a shape of a flexible instrument, wherein the flexible instrument is connected to a camera.

[0014] The embodiment of the present application also provides a camera fusion positioning device, including: a posture calculation module, used to determine a first posture data and at least one second posture data of a camera in a physiological channel when shooting a target image, wherein the posture data corresponding to the first posture data and the second posture data are determined in different ways, and the first posture data is the camera posture data determined based on the initial posture data and motion state data of a robotic arm, and the robotic arm is used to control the motion state of the camera; a motion trend calculation module, used to determine a first motion trend corresponding to the first posture data, and a second motion trend corresponding to the second posture data, wherein the trend types of the first motion trend and the second motion trend include The first virtual image is an image acquired by the virtual camera at the position indicated by the first pose data, and the second virtual image is an image acquired by the virtual camera at the position indicated by the second pose data; a processing module is used to determine the target pose data of the camera when shooting the target image according to the first motion trend, the second motion trend, the first similarity and the second similarity.

[0015] The embodiment of the present application also provides a camera fusion positioning system, including a camera, an electronic device, and a robotic arm, wherein the electronic device is used to determine a first pose data and at least one second pose data of a camera in a physiological channel when shooting a target image, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is the camera pose data determined based on the initial pose data and motion state data of the robotic arm, and the robotic arm is used to control the motion state of the camera; determine a first motion trend corresponding to the first pose data, and a second motion trend corresponding to the second pose data, wherein the trend types of the first motion trend and the second motion trend include forward and backward; determine a first virtual image and a second virtual image captured by a virtual camera corresponding to the camera in a physiological channel model of the physiological channel, and determine a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image obtained by the virtual camera at the pose indicated by the first pose data, and the second virtual image is an image obtained by the virtual camera at the pose indicated by the second pose data; determine the target pose data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity, and the second similarity.

[0016] An embodiment of the present application also provides a non-volatile storage medium, in which a computer program is stored. When the computer program is executed by a processor, a camera fusion positioning method is implemented.

[0017] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a camera fusion positioning method when executing the computer program.

[0018] An embodiment of the present application also provides a computer program product, including a computer program, which implements a camera fusion positioning method when executed by a processor.

[0019] Based on the above scheme, it can be seen that the present application estimates the target pose data of the camera when shooting the target image by fusing the camera pose data determined by multiple positioning methods, thereby improving the positioning accuracy of the camera and solving the technical problem that the tracking and positioning of the camera is prone to failure due to the use of a single positioning method in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 It is a schematic diagram of the structure of a camera fusion positioning system provided in an exemplary embodiment of the present application.

[0022] Figure 2 A schematic flow chart of a camera fusion positioning method provided in an exemplary embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of a physiological channel model provided in an exemplary embodiment of the present application;

[0024] Figure 4 A schematic diagram of a similarity calculation process provided in an exemplary embodiment of the present application;

[0025] Figure 5 A schematic diagram of an image sub-block provided in an exemplary embodiment of the present application;

[0026] Figure 6 A schematic diagram of a camera fusion positioning process provided in an exemplary embodiment of the present application;

[0027] Figure 7It is a flowchart of a camera real-time tracking process provided according to an embodiment of the present application;

[0028] Figure 8 It is a flowchart of a camera fusion positioning program provided according to an embodiment of the present application;

[0029] Fig. 9 It is a structural schematic diagram of a camera fusion positioning device provided in an exemplary embodiment of the present application;

[0030] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0034] Endoscopic navigation is an auxiliary technology commonly used in endoscopic examination of physiological channels such as bronchi, which aims to improve the accuracy and safety of endoscopic examination. Common endoscopes include bronchoscopes, anoscopes, sigmoidoscopes, nasal laryngoscopes and laryngoscopes. Taking bronchoscopes as an example, endoscopy is a medical procedure used to examine the trachea and bronchi, and is usually used to diagnose and treat respiratory diseases such as lung cancer, bronchitis and emphysema.

[0035] The background behind endoscopic navigation mainly involves the following aspects:

[0036] The need for accurate positioning: When performing an endoscopic examination, it is necessary to accurately locate and access specific bronchial branches in order to observe and collect tissue samples. Some bronchial branches may be small or located in a deep position, making it difficult to accurately locate them through conventional endoscopic examinations.

[0037] Improve inspection accuracy: By introducing navigation technology, the anatomical structure of the bronchi can be displayed in real-time images, and the endoscope can be guided more accurately, thereby improving the accuracy of the inspection. This is crucial for early detection and diagnosis of lesions.

[0038] Reduced discomfort to patients: Traditional endoscopy may require repeated insertion of the endoscope, which may cause patient discomfort and the risk of complications. Endoscopic navigation can find the target location more quickly and reduce the number of insertions, thereby reducing patient discomfort.

[0039] Driven by technological progress: With the continuous development of medical imaging technology and navigation technology, endoscopic navigation has become possible. Computer-assisted navigation systems can combine real-time images, computer-reconstructed three-dimensional anatomical structures, and electromagnetic or optical positioning technology to provide more comprehensive information to help doctors better navigate the bronchi.

[0040] Overall, endoscopic navigation technology can improve the effectiveness of endoscopic examinations and the pursuit of patient experience, and with the help of modern technology, provide doctors with more information and tools to perform endoscopic examinations more accurately and safely.

[0041] However, in the existing endoscopic navigation technology, a single positioning method is usually used to position the camera at the end of the endoscope, which can easily lead to tracking failure and cause obvious deviations between the virtual bronchial image rendered according to the posture data obtained by the endoscopic navigation technology and the real bronchial image. This situation usually occurs when the real endoscope moves too fast, hits a wall, or the image features captured by the endoscope (such as airway openings, bifurcations, and folds) are too few.

[0042] In order to solve the above problems, the technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0043] Figure 1 A schematic diagram of the structure of a camera fusion positioning system provided in an embodiment of the present application. Figure 1It can be seen that the system includes: a camera 10, an electronic device 12, and a robotic arm 14, wherein the electronic device 12 is used to obtain a first pose data and at least one second pose data of the camera 10 in the physiological channel when shooting a target image, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is the pose data of the camera 10 determined based on the initial pose data and motion state data of the robotic arm, and the robotic arm is used to control the motion state of the camera 10; determine a first motion trend corresponding to the first pose data, and a second motion trend corresponding to the second pose data, wherein the first motion trend and the second motion trend The trend types include forward and backward; determining a first virtual image and a second virtual image captured by the virtual camera 10 corresponding to the camera 10 in the physiological channel model of the physiological channel, and determining a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image acquired by the virtual camera 10 at a posture indicated by the first posture data, and the second virtual image is an image acquired by the virtual camera 10 at a posture indicated by the second posture data; determining the target posture data of the camera 10 when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity.

[0044] Figure 2 A flowchart of a camera fusion positioning method provided in an embodiment of the present application specifically includes the following steps:

[0045] Step 202, obtaining first pose data and at least one second pose data of a camera in a physiological channel when shooting a target image, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is the pose data of the camera determined according to the initial pose data and motion state data of the robot arm, and the robot arm is used to control the motion state of the camera;

[0046] In the technical solution provided in step S202, at least one second posture data includes second posture data determined based on a target image and a reference image, wherein the reference image and the target image are two frames of images continuously captured by a camera in a physiological channel, and the camera first captures the reference image and then captures the target image.

[0047] As an optional implementation, the following method can be used to determine the second pose data based on the target image and the reference image: a sparse pyramid optical flow algorithm is used to determine the matching point pairs between the reference image and the target image, and the displacement information of the matching point pairs in the reference image and the target image is calculated, and the matching point pairs are screened using the displacement information to determine the three-dimensional points corresponding to the screened matching point pairs. Then, the three-dimensional points are used to perform multiple iterative calculations to determine the conversion relationship between the pose data of the reference image and the pose data corresponding to the target image. Then, the pose data and conversion relationship corresponding to the known reference image are used to calculate the pose data of the camera in a preset three-dimensional space coordinate system (such as a CT coordinate system) when the target image is captured, that is, the second pose data.

[0048] In some embodiments of the present application, the target image and the set of virtual image data may be matched to determine the virtual image with the highest similarity to the target image, thereby determining the second pose data according to the pose of the virtual camera corresponding to the virtual image with the highest similarity. The set of virtual image data includes different virtual images rendered by the virtual camera in the physiological channel under various pose data.

[0049] As an optional embodiment, at least one second posture data includes second posture data determined based on the shape of a flexible instrument, wherein the flexible instrument is connected to a camera, that is, the endoscope includes a flexible instrument and a camera, and the camera is arranged at the end of the flexible instrument.

[0050] Optionally, when the flexible device is an optical fiber, the following process may be used to determine the second posture data according to the shape of the flexible device;

[0051] Determine the first coordinate system and the second coordinate system at the initial moment, wherein the first coordinate system is a spatial rectangular coordinate system with its origin in the endoscope, and the second coordinate system is a spatial rectangular coordinate system with its origin in the physiological channel, wherein the endoscope includes a camera and a flexible instrument, and the camera is arranged at the end of the flexible instrument; determine the rotation matrix and the translation matrix between the first coordinate system and the second coordinate system according to the first coordinate system and the second coordinate system; splice the first target part model of the flexible instrument into the first physiological channel model of the physiological channel according to the rotation matrix and the translation matrix, and obtain the spliced ​​second physiological channel model, wherein the first target part model is a model established according to the first target part of the flexible instrument, and the two endpoints of the first target part correspond to the first physiological feature point (epiglottis) and the second physiological feature point (carina) of the physiological channel respectively; perform ICP algorithm or SVD algorithm registration according to the spliced ​​second physiological channel model and the optical fiber signal, and determine the second posture data of the camera at the end of the current endoscope when shooting the target image. The optical fiber signal includes data such as the wavelength and curvature of each grating point.

[0052] In some embodiments of the present application, when determining the first pose data, the initial pose data of the robot arm connected to the camera can be acquired in advance, wherein the camera is arranged at the end of the endoscope. Robot (such as forward distance, rotation angle and bending angle), and obtain the first position data of the camera in a preset coordinate system (such as CT coordinate system) under the current robot motion state data.

[0053] Step 204, determining a first motion trend corresponding to the first posture data, and a second motion trend corresponding to the second posture data, wherein the trend types of the first motion trend and the second motion trend include forward and backward;

[0054] In the technical solution provided in step S204, the steps of determining the first motion trend corresponding to the first posture data and the second motion trend corresponding to the second posture data include: determining a reference image and reference posture data of the camera when shooting the reference image, wherein the reference image and the target image are two frames of images continuously shot by the camera in the physiological channel, and the camera first captures the reference image and then captures the target image; determining a first translation increment in a preset direction relative to that when the reference image is shot based on the reference posture data and the first posture data, and determining a second translation increment in a preset direction relative to that when the reference image is shot based on the reference posture data and the second posture data; determining a trend type of the first motion trend based on the positive or negative value of the first translation increment, and determining a trend type of the second motion trend based on the positive or negative value of the second translation increment.

[0055] In some embodiments of the present application, the calculation formula of the translation increment is as follows:

[0056]

[0057] In the above formula, t inc represents the translation increment, t cur and t prev They respectively represent the translation matrix when the camera shoots the target image and the translation matrix when the camera shoots the reference image in a preset coordinate system (such as the CT coordinate system). Represents the inverse of the rotation matrix in the preset coordinate system when the camera captures the reference image. The translation matrix of the camera when capturing the target image can be determined based on the pose data of the camera when capturing the target image. When the translation matrix is ​​determined based on the first pose data, the above translation increment is the first translation increment. And when the translation matrix is ​​determined based on the second pose data, the above translation increment is the second translation increment.

[0058] After determining the translation increment, the trend type of the specific movement trend can be further determined according to the positive or negative value of the translation increment. When the value of the first translation increment is positive, the first translation increment is assigned a value of true, and the trend type of the first movement trend is determined to be forward. When the value of the first translation increment is negative or zero, the negative value of the first translation increment is flase, and the trend type of the first movement trend is determined to be backward.

[0059] Similarly, when the value of the second translation increment is positive, the second translation increment is assigned a value of true, and the trend type of the second movement trend is determined to be forward. When the value of the second translation increment is negative or zero, the negative value of the second translation increment is flase, and the trend type of the second movement trend is determined to be backward.

[0060] As an optional implementation, the preset direction includes an extension direction of a lens of the camera when capturing the reference image.

[0061] In some embodiments of the present application, the preset direction may also be the positive direction of the Z axis of the camera coordinate system, where the Z axis of the camera coordinate system is the lens axis of the camera. The camera coordinate system refers to a three-dimensional rectangular coordinate system established with the camera as a reference. For example, the three-dimensional rectangular coordinate system may be established with the lens axis of the camera as the Z axis and the optical center of the camera (that is, the lens center) as the origin as the camera coordinate system.

[0062] Step 206, determining a first virtual image and a second virtual image captured by a virtual camera corresponding to the camera in the physiological channel model of the physiological channel, and determining a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image acquired by the virtual camera at a posture indicated by the first posture data, and the second virtual image is an image acquired by the virtual camera at a posture indicated by the second posture data;

[0063] In some embodiments of the present application, when the physiological channel is a bronchus, the physiological channel model is as follows: Figure 3 shown.

[0064] In the technical solution provided in step S206, the step of determining the first similarity between the first virtual image and the target image, and the step of determining the second similarity between the second virtual image and the target image includes: dividing the first virtual image into a plurality of partially overlapping first virtual image blocks, dividing the second virtual image into a plurality of partially overlapping second virtual image blocks, and dividing the target image into a plurality of partially overlapping real image blocks; determining a first virtual feature value of the first virtual image block, a second virtual feature value of the second virtual image block, and a real feature value of the real image block, wherein the first virtual feature value, the second virtual feature value, and the real feature value all include a standard deviation of the intensity value of the pixel point in the image block, and a whiteness value used to reflect the proportion of the specified pixel point in the image block; deleting the first virtual image block, the second virtual image block, and the real image block whose whiteness value is greater than the preset whiteness value; and arranging the retained first virtual image blocks in descending order according to the standard deviation of the intensity value of the pixel point of each first virtual image block to obtain a first virtual image block sequence. , and the retained second virtual image blocks are arranged in descending order according to the standard deviation of the pixel intensity values ​​of each second virtual image block to obtain a second virtual image block sequence; the retained real image blocks are arranged in descending order according to the standard deviation of the pixel intensity values ​​of each real image block to obtain a first real image block sequence; a preset number of virtual image blocks are selected from the first virtual image sequence in a front-to-back order to obtain a third virtual image block sequence, and a preset number of virtual image blocks are selected from the second virtual image sequence in a front-to-back order to obtain a fourth virtual image block sequence; a preset number of real image blocks are selected from the real image sequence to obtain a second real image block sequence; a first root mean square error value between the third virtual image block sequence and the second real image block sequence is determined, and a second root mean square error value between the fourth virtual image block sequence and the second real image block sequence is determined, wherein the first root mean square error value is used to represent the first similarity, and the second root mean square error value is used to represent the second similarity.

[0065] In some embodiments of the present application, a designated pixel point in an image block is determined in the following manner: a saturation value and a brightness value of the pixel point in the image block are determined; and when the saturation value of the pixel point is less than a preset saturation value and the brightness value of the pixel point is greater than a preset brightness value, the pixel point is determined to be a designated pixel point.

[0066] Optionally, the process of determining similarity is as follows Figure 4 As shown, it can be divided into the stages of image segmentation, eigenvalue calculation, sub-block selection and similarity calculation.

[0067] Taking the target image as an example, the target image can be divided into Figure 5The M×N overlapping image blocks shown in FIG. 1 avoid some small structures in the physiological channel from being divided into different image blocks. Image block D m,n is defined as:

[0068]

[0069] Where m and n satisfy the following constraints:

[0070] 2≤m≤M-1 and 2≤n≤N-1.

[0071] Where W and H represent the width and height of the target image, respectively; M and N represent the number of image width and height divisions, respectively; p and q represent the image block D, respectively. m,n The value range of the horizontal axis (width) and vertical axis (height), m and n represent the number of small blocks on the horizontal and vertical axes respectively.

[0072] In the eigenvalue calculation stage, for each image block D m,n , whose characteristic values ​​include the standard deviation of the intensity value and the whiteness value. Image block D m,n Standard deviation of intensity values The calculation formula is as follows:

[0073]

[0074] Where |D m,n | is sub-block D m,n The number of pixels inside, is the average of all pixel intensities in this sub-block.

[0075] Image block D m,n Whiteness The calculation formula is as follows:

[0076]

[0077] In the above formula, W is a function used to determine whether a pixel is a specified pixel. The description of the function is as follows:

[0078]

[0079] In the formula, S and B correspond to the saturation value and brightness value of the pixel in the HSV space. The formula means that the saturation value S(v) of the pixel v is less than or equal to a certain threshold T s And the brightness value B(v) is greater than a certain threshold T B When is the specified pixel point, the value of function W is 1.

[0080] After dividing the image blocks, it is necessary to select the image blocks according to the feature values ​​of the image blocks. Optionally, all the image blocks can be put into the candidate list A first. (k) Then remove the candidates whose whiteness value is greater than a certain threshold T from the candidate list. w The sub-blocks, that is Then the remaining image blocks are analyzed according to the standard deviation of the intensity values Sort in descending order. Finally, keep list A (k) The image blocks of α·M·N before sorting are selected, and the other image blocks are deleted. α is a proportional coefficient, and its value range is 0≤α≤1, and M·N is the total number of small sub-blocks of the entire image).

[0081] In this way, only image blocks with characteristic structures can be selected for similarity calculation, such as airway bifurcations or folded parts. The sub-blocks with these characteristic structures have higher intensity standard deviations, but the sub-blocks with alveoli also have higher intensity standard deviations. Therefore, the influence of alveoli is excluded by comprehensively considering the saturation and brightness values ​​of the sub-blocks, that is, judging whether the whiteness value of the sub-block exceeds a certain threshold to remove this special case.

[0082] It should be noted that the same processing flow as the target image is adopted for the first virtual image and the second virtual image, which will not be described in detail here.

[0083] When calculating the similarity, taking the first virtual image and the target image as an example, the root mean square error value calculation formula between the first virtual image and the target image is as follows:

[0084]

[0085] In the above formula, |A (k) | is the number of image blocks in the second real image block sequence and the third virtual image block sequence. |D| represents the number of pixels in any image block, and They correspond to the average intensities of the image blocks D in the second real image block sequence and the third virtual image block sequence respectively. and V i,j Corresponding to the intensity values ​​of each phase velocity in the image block D in the second real image block sequence and the third virtual image block sequence.

[0086] It should be noted that the calculation method of the root mean square error value between the second virtual image and the target image is the same as the calculation method of the root mean square error value between the first virtual image and the target image, which will not be repeated here.

[0087] Step 208 , determining target position data of the camera when shooting the target image according to the first motion trend, the second motion trend, the first similarity and the second similarity.

[0088] In the technical solution corresponding to step 208, each type of second posture data corresponds to a second motion trend and a second similarity; the step of determining the target posture data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity includes: selecting alternative second posture data from at least one type of second posture data, wherein the alternative second posture data is second posture data whose corresponding second motion trend is consistent with the first motion trend; jointly determining the target posture data based on the alternative second posture data, the first posture data, the second similarity corresponding to the alternative second posture data, and the first similarity.

[0089] As an optional implementation, the step of jointly determining the target pose data based on the alternative second pose data, the first pose data, the second similarity corresponding to the alternative second pose data, and the first similarity includes: taking the first pose data as the pose data default value, and taking the first similarity as the similarity default value; determining the alternative second pose data with the largest corresponding second similarity as the target alternative second pose data; when the second similarity corresponding to the target alternative second pose data is greater than the similarity default value, updating the similarity default value to the second similarity corresponding to the target alternative second pose data, and updating the pose data default value to the target alternative second pose data; determining the pose data default value to the target pose data, and determining the similarity default value to the target similarity when the camera shoots the target image.

[0090] In some embodiments of the present application, in summary, the process of performing fusion positioning on the camera is as follows: Figure 6 As shown, the following steps are included:

[0091] Step 602, synchronously determining the first pose data and a plurality of second pose data of the camera when shooting the target image;

[0092] In the scheme provided in step 602, the multiple second posture data include second posture data determined according to the shapes of the flexible instrument and the physiological channel, and second posture data determined according to the reference image and the target image, the reference image is an image taken by the camera before taking the target image, and the reference image and the target image are two frames of images taken continuously.

[0093] Step 604, determining a first similarity between the target image and the virtual image according to the first pose data, and determining a second similarity between the target image and the virtual image according to various second pose data, and determining a first motion trend indicator corresponding to the first pose data, and a second motion trend indicator corresponding to various second pose data;

[0094] It should be noted that the virtual image is an image rendered based on the first pose data or the second pose data in combination with the physiological channel model.

[0095] Step 606, based on the motion trend index and similarity corresponding to each pose data, various pose data are fused to obtain target pose data and target similarity of the target image.

[0096] Optionally, the first motion trend and the first similarity determined by the robot arm control signal are used as reference values, and the second motion trends and the second similarities corresponding to the respective second posture data are compared, so as to determine the final target posture data and the target similarity. Among them, since the robot arm moves under the instruction of the robot arm control signal, the initial posture data and the motion state data of the robot arm can be quickly and accurately determined by the robot arm control signal, so that the first motion trend can be determined according to the robot arm control signal.

[0097] As an optional implementation, the target similarity can be used to determine the tracking state corresponding to the target image, thereby determining whether the target pose data needs to be further optimized.

[0098] Optionally, in some embodiments of the present application, after determining the target pose data of the camera when shooting the target image, the camera fusion positioning method also includes: obtaining several frames of real images shot continuously by the camera in chronological order, and the third similarity corresponding to each frame of the real image, wherein the target image is the last frame image among the several frames of real images, and the third similarity of the real image is the similarity determined based on the first similarity and the second similarity corresponding to the real image; determining the tracking state index of the target image based on the third similarity corresponding to each frame of the real image, wherein the tracking state index includes a poor state and an excellent state; when it is determined that the target image is a key frame image and the corresponding tracking state indicator is a poor state, updating the target pose data.

[0099] As an optional implementation, determining the tracking state index of the target image based on the third similarity corresponding to each frame of real image includes: calculating the sum of the third similarities corresponding to each frame of real image; when the sum of the third similarities is not less than a preset threshold, determining the tracking state index to be in an excellent state; when the sum of the third similarities is less than a preset threshold, determining the tracking state index to be in a poor state.

[0100] In some embodiments of the present application, it is possible to determine whether the target image is a key frame image by inputting the target image into a pre-trained image classification neural network and obtaining a judgment result output by the image classification neural network.

[0101] As an optional implementation, when the tracking status index corresponding to the target image is poor and the target image is a key frame image, the target pose data can be optimized and the optimized data can be used as the new target pose data. In the case of good tracking status and key frame, good tracking status and non-key frame, poor tracking status and non-key frame, etc., the target pose data is not updated.

[0102] In some embodiments of the present application, when optimizing the target pose data, the target pose data can be used as the initial pose data, and different pose offsets are added to the initial pose data to obtain a number of different candidate pose data. Then, according to each candidate pose data, a virtual image obtained by the virtual camera in the pose indicated by the candidate pose data is rendered and generated, and then the similarity between each virtual image and the target image is determined. Finally, the candidate pose data corresponding to the maximum similarity is selected as the optimized target pose data.

[0103] It is not difficult to find from the above content that in the embodiments of the present application, the following Figure 7 The camera fusion positioning and real-time tracking process shown generates a physiological channel model of the physiological channel through offline calculation, and obtains a virtual image corresponding to the target image taken by the camera based on the physiological channel model. Afterwards, in the fusion positioning stage, the position data of the camera determined by multiple positioning methods are fused according to the similarity between the virtual image and the target image, and the motion trend corresponding to various positioning methods to estimate the target position data of the camera when shooting the target image, thereby improving the positioning accuracy of the camera. Then, in the real-time tracking stage, the position and posture of the virtual camera in the physiological channel model are updated in real time, and the target image and the virtual image are displayed synchronously to provide navigation information for the operator or doctor. This solves the technical problem that the tracking and positioning of the camera is prone to failure due to the use of a single positioning method in the related art.

[0104] In some embodiments of the present application, a camera fusion positioning program is also provided. The execution process of the program is as follows: Figure 8 As shown, including:

[0105] Step 802, determining first pose data and at least one second pose data when the camera captures the target image;

[0106] Specifically, the first position data is based on the initial position data and motion state data D of the robot arm. Robot Determined P Robot The second pose data is the image data D including the reference image and the target image according to the sparse pyramid optical flow algorithm image The second posture data P obtained after processing ImageThe second posture data is P determined according to the shape of the flexible device and the shape of the physiological channel. Fiber The shape of the flexible device can be determined by the flexible device signal D Fiber to obtain.

[0107] Step 804: determine the motion trend T corresponding to the first pose data Robot and similarity S Robot , the motion trend T corresponding to the second pose data determined by the sparse pyramid optical flow algorithm Image and similarity S Image , and the motion trend T corresponding to the second posture data determined according to the shape of the flexible device Fiber and similarity S Fiber ;

[0108] Step 806: P Robot As the reference pose data P, and S Robot As the benchmark similarity S, then update the benchmark posture data and the benchmark similarity according to the motion trends and similarities corresponding to the various second posture data and the first posture data, and use the final benchmark posture data as the target posture data, and use the final benchmark similarity as the target similarity;

[0109] Specifically, we can first compare T Image and T Robot If they are consistent, determine S Image Is it greater than the benchmark similarity S? If the judgment results of the above two conditions are both yes, the benchmark pose data P is updated to P Image , update the baseline similarity S to S Image .

[0110] In T Image and T Robot Inconsistent, or S Image If it is not greater than the benchmark similarity S, continue to judge T Fiber Whether it is related to T Robot Consistent and S Fiber Is it greater than the benchmark similarity S? If T Fiber With T Robot Consistent and S Fiber If it is greater than S, then update S to S Fiber , and update P to P Fiber .

[0111] If T Image and T Robot Consistent and S Image If it is greater than S, then update S to S first. Image , and update P to P ImageThen further judge T Fiber Whether it is related to T Robot Consistent and S Fiber Is it greater than the benchmark similarity S. That is, S Fiber Is it greater than S? Image . And in T Fiber With T Robot Consistent, and S Fiber Greater than S, that is, S Fiber Greater than S Image In the case of Fiber , and update P to P Fiber .

[0112] In some embodiments of the present application, P after the last update can be determined as the target pose data, and S after the last update can be determined as the target similarity.

[0113] Step 808, determining whether to update the target posture data P;

[0114] Specifically, n frames of real image 1, real image 2, real image 3, ... real image n, which are taken continuously in chronological order by the camera, can be obtained, where n is an integer greater than 0, and real image n is the target image. Then, it can be determined whether the target image is a key frame image K, and whether the tracking state determined by the sum of the target similarities corresponding to each of the n frames of images taken continuously is poor. When determining the tracking state, the sum of the above target similarities can be compared with a preset threshold, and the tracking state is determined to be good when the sum of the target similarities is not less than the preset threshold, otherwise it is poor.

[0115] When the target image is determined to be a key frame image and the tracking state is poor, the target pose data P can be directly used as the final pose data P of the target image. Cur Otherwise, the target pose data P is updated to obtain P Adjust As P Cur .

[0116] As an optional implementation, when determining P Adjust When P is used as the basis, several different pose data can be automatically generated by adding different random offsets, and then the generated pose data is used to render the corresponding virtual image, and the obtained virtual image is compared with the target image D Image The pose data corresponding to the virtual image with the greatest similarity is taken as the optimized pose P Adjust .

[0117] Figure 8A structural block diagram of a camera fusion positioning device provided in an embodiment of the present application, the device comprising: a posture calculation module 80, used to determine a first posture data and at least one second posture data of a camera in a physiological channel when shooting a target image, wherein the posture data corresponding to the first posture data and the second posture data are determined in different ways, and the first posture data is the camera posture data determined based on the initial posture data and motion state data of a robotic arm, and the robotic arm is used to control the motion state of the camera; a motion trend calculation module 82, used to determine a first motion trend corresponding to the first posture data, and a second motion trend corresponding to the second posture data, wherein the trend of the first motion trend and the second motion trend is The posture types include forward and backward; a similarity calculation module 84 is used to determine the first virtual image and the second virtual image captured by the virtual camera corresponding to the camera in the physiological channel model of the physiological channel, and determine the first similarity between the first virtual image and the target image, and the second similarity between the second virtual image and the target image, wherein the first virtual image is an image acquired by the virtual camera at the posture indicated by the first posture data, and the second virtual image is an image acquired by the virtual camera at the posture indicated by the second posture data; a processing module 86 is used to determine the target posture data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity.

[0118] In some embodiments of the present application, at least one second posture data includes second posture data determined based on a target image and a reference image, wherein the reference image and the target image are two frames of images continuously captured by a camera in a physiological channel, and the camera first captures the reference image and then captures the target image.

[0119] In some embodiments of the present application, the at least one second posture data includes second posture data determined based on a shape of a flexible instrument, wherein the flexible instrument is connected to a camera.

[0120] In some embodiments of the present application, the steps of the motion trend calculation module 82 determining the first motion trend corresponding to the first posture data and the second motion trend corresponding to the second posture data include: determining a reference image and reference posture data of the camera when shooting the reference image, wherein the reference image and the target image are two frames of images continuously shot by the camera in the physiological channel, and the camera first captures the reference image and then captures the target image; determining a first translation increment in a preset direction relative to the reference image when shooting the target image based on the reference posture data and the first posture data, and determining a second translation increment in a preset direction relative to the reference image when shooting the target image based on the reference posture data and the second posture data; determining a trend type of the first motion trend based on the positive or negative value of the first translation increment, and determining a trend type of the second motion trend based on the positive or negative value of the second translation increment.

[0121] In some embodiments of the present application, the preset direction includes an extension direction of a lens of a camera when capturing a reference image.

[0122] In some embodiments of the present application, the steps of determining the first similarity between the first virtual image and the target image, and the second similarity between the second virtual image and the target image by the similarity calculation module 84 include: dividing the first virtual image into a plurality of partially overlapping first virtual image blocks, dividing the second virtual image into a plurality of partially overlapping second virtual image blocks, and dividing the target image into a plurality of partially overlapping real image blocks; determining a first virtual feature value of the first virtual image block, a second virtual feature value of the second virtual image block, and a real feature value of the real image block, wherein the first virtual feature value, the second virtual feature value, and the real feature value all include a standard deviation of the intensity value of the pixel point in the image block, and a whiteness value used to reflect the proportion of the specified pixel point in the image block; deleting the first virtual image block, the second virtual image block, and the real image block whose whiteness value is greater than the preset whiteness value; and arranging the retained first virtual image blocks in descending order according to the standard deviation of the pixel point intensity value of each first virtual image block to obtain the first virtual image block. The method comprises the steps of: selecting a preset number of virtual image blocks from the first virtual image sequence and obtaining a second real image block sequence; arranging the retained second virtual image blocks in a descending order according to the standard deviation of the pixel point intensity values ​​of each second virtual image block to obtain a second virtual image block sequence; arranging the retained real image blocks in a descending order according to the standard deviation of the pixel point intensity values ​​of each real image block to obtain a first real image block sequence; selecting a preset number of virtual image blocks from the first virtual image sequence in a front-to-back order to obtain a third virtual image block sequence; and selecting a preset number of virtual image blocks from the second virtual image sequence in a front-to-back order to obtain a fourth virtual image block sequence; selecting a preset number of real image blocks from the real image sequence to obtain a second real image block sequence; determining a first root mean square error value between the third virtual image block sequence and the second real image block sequence, and determining a second root mean square error value between the fourth virtual image block sequence and the second real image block sequence, wherein the first root mean square error value is used to characterize the first similarity, and the second root mean square error value is used to characterize the second similarity.

[0123] In some embodiments of the present application, a designated pixel point in an image block is determined in the following manner: a saturation value and a brightness value of the pixel point in the image block are determined; and when the saturation value of the pixel point is less than a preset saturation value and the brightness value of the pixel point is greater than a preset brightness value, the pixel point is determined to be a designated pixel point.

[0124] In some embodiments of the present application, each type of second posture data corresponds to a second motion trend and a second similarity; the step in which the processing module 86 determines the target posture data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity includes: selecting alternative second posture data from at least one type of second posture data, wherein the alternative second posture data is second posture data whose corresponding second motion trend is consistent with the first motion trend; determining the target posture data based on the alternative second posture data, the first posture data, the second similarity corresponding to the alternative second posture data, and the first similarity.

[0125] In some embodiments of the present application, the step in which the processing module 86 determines the target pose data based on the alternative second pose data, the first pose data, the second similarity corresponding to the alternative second pose data, and the first similarity includes: taking the first pose data as the pose data default value, and taking the first similarity as the similarity default value; determining the alternative second pose data with the largest corresponding second similarity as the target alternative second pose data; when the second similarity corresponding to the target alternative second pose data is greater than the similarity default value, updating the similarity default value to the second similarity corresponding to the target alternative second pose data, and updating the pose data default value to the target alternative second pose data; determining the pose data default value to be the target pose data, and determining the similarity default value to be the target similarity when the camera takes the target image.

[0126] In some embodiments of the present application, after determining the target pose data of the camera when shooting the target image, the camera fusion positioning device is also used to: obtain several frames of real images shot continuously by the camera in chronological order, and the third similarity corresponding to each frame of the real image, wherein the target image is the last frame image of the several frames of real images, and the third similarity of the real image is the similarity determined based on the first similarity and the second similarity corresponding to the real image; determine the tracking state index of the target image based on the third similarity corresponding to each frame of the real image, wherein the tracking state index includes a poor state and an excellent state; when it is determined that the target image is a key frame image and the corresponding tracking state index is a poor state, update the target pose data.

[0127] In some embodiments of the present application, the step of the camera fusion positioning device determining the tracking state index of the target image based on the third similarity corresponding to each frame of real image includes: calculating the sum of the third similarities corresponding to each frame of real image; when the sum of the third similarities is not less than a preset threshold, determining the tracking state index as an excellent state; when the sum of the third similarities is less than a preset threshold, determining the tracking state index as a poor state.

[0128] According to an embodiment of the present application, a non-volatile storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the following camera fusion positioning method is implemented: obtaining first pose data and at least one second pose data of a camera in a physiological channel when shooting a target image, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is the camera pose data determined based on the initial pose data and motion state data of a robotic arm, and the robotic arm is used to control the motion state of the camera; determining a first motion trend corresponding to the first pose data, and a second motion trend corresponding to the second pose data, wherein , the trend types of the first motion trend and the second motion trend include forward and backward; determine the first virtual image and the second virtual image captured by the virtual camera corresponding to the camera in the physiological channel model of the physiological channel, and determine the first similarity between the first virtual image and the target image, and the second similarity between the second virtual image and the target image, wherein the first virtual image is an image acquired by the virtual camera at the posture indicated by the first posture data, and the second virtual image is an image acquired by the virtual camera at the posture indicated by the second posture data; determine the target posture data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity.

[0129] According to an embodiment of the present application, a computer program product is provided, including a computer program, which implements the following camera fusion positioning method when executed by a processor: obtaining first pose data and at least one second pose data of a camera in a physiological channel when shooting a target image, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is the camera pose data determined based on the initial pose data and motion state data of a robotic arm, and the robotic arm is used to control the motion state of the camera; determining a first motion trend corresponding to the first pose data and a second motion trend corresponding to the second pose data, wherein the trend types of the first motion trend and the second motion trend include forward and backward; determining a first virtual image and a second virtual image captured by a virtual camera corresponding to the camera in a physiological channel model of the physiological channel, and determining a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image obtained by the virtual camera at the pose indicated by the first pose data, and the second virtual image is an image obtained by the virtual camera at the pose indicated by the second pose data; determining the target pose data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity.

[0130] According to an embodiment of the present application, a method for performing a surgery is provided, comprising the following steps:

[0131] The first step is to determine the lesion location in the physiological channel;

[0132] The second step is to obtain first pose data and at least one second pose data of the camera in the physiological channel when shooting the target image during the process of the camera in the physiological channel moving toward the lesion position, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is the camera pose data determined based on the initial pose data and motion state data of the robotic arm, and the robotic arm is used to control the motion state of the camera;

[0133] The third step is to determine a first motion trend corresponding to the first posture data and a second motion trend corresponding to the second posture data, wherein the trend types of the first motion trend and the second motion trend include forward and backward;

[0134] The fourth step is to determine a first virtual image and a second virtual image captured by a virtual camera corresponding to the camera in the physiological channel model of the physiological channel, and determine a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image acquired by the virtual camera at a posture indicated by the first posture data, and the second virtual image is an image acquired by the virtual camera at a posture indicated by the second posture data;

[0135] Step 5, determining target posture data of the camera when shooting the target image according to the first motion trend, the second motion trend, the first similarity and the second similarity;

[0136] Step 6: After determining that the camera has reached the lesion location based on the target posture data, run the preset surgical procedure.

[0137] See also Fig.10 , a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.10 As shown, the electronic device includes: a memory 1001 and a processor 1002 .

[0138] The memory 1001 stores an executable computer program 1003. The processor 1002 coupled to the memory 1001 calls the executable computer program 1003 stored in the memory to execute the camera fusion positioning method provided in the above embodiment.

[0139] Exemplarily, the computer program 1003 may be divided into one or more modules / units, which are stored in the memory 1001 and executed by the processor 1002 to complete the present application. The one or more modules / units may include various modules in the camera fusion positioning device in the above embodiment, such as: a posture calculation module, a motion trend calculation module, a similarity calculation module, and a processing module.

[0140] Furthermore, the apparatus also includes: at least one input device and at least one output device.

[0141] The processor 1002 , memory 1001 , input device and output device may be connected via a bus.

[0142] The input device may specifically be a camera, a touch panel, a physical button or a mouse, etc. The output device may specifically be a display screen.

[0143] Furthermore, the device may include more components than those shown in the figure, or a combination of certain components, or different components, such as network access equipment, sensors, etc.

[0144] The processor 1002 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0145] The memory 1001 may be, for example, a hard disk drive memory, a non-volatile memory (such as a flash memory or other electronically programmable memory with limited deletion for forming a solid-state drive, etc.), a volatile memory (such as a static or dynamic random access memory, etc.), etc., and the embodiments of the present application are not limited thereto. Specifically, the memory 1001 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. The memory 1001 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 1001 may also include both an internal storage unit of the electronic device and an external storage device. The memory 1001 is used to store computer programs and other programs and data required by the terminal. The memory 1001 may also be used to temporarily store data that has been output or is to be output.

[0146] Furthermore, the present application also provides a computer-readable storage medium, which may be disposed in the electronic device in the above embodiments. Fig.10 The memory 1001 in the illustrated embodiment. The computer readable storage medium stores a computer program, and when the program is executed by the processor, the method for adjusting the target position based on the image described in the aforementioned embodiments is implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0148] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0150] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned readable storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0151] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0152] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0153] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A camera fusion positioning method, characterized in that: include: Acquire first pose data and at least one second pose data of a camera in a physiological channel when shooting a target image, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is camera pose data determined based on initial pose data and motion state data of a robotic arm, and the robotic arm is used to control the motion state of the camera; Determine a first motion trend corresponding to the first posture data, and a second motion trend corresponding to the second posture data, wherein trend types of the first motion trend and the second motion trend include forward and backward; Determine a first virtual image and a second virtual image captured by a virtual camera corresponding to the camera in the physiological channel model of the physiological channel, and determine a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image acquired by the virtual camera at the posture indicated by the first posture data, and the second virtual image is an image acquired by the virtual camera at the posture indicated by the second posture data; The target position data of the camera when shooting the target image is determined according to the first motion trend, the second motion trend, the first similarity and the second similarity.

2. The camera fusion positioning method according to claim 1, characterized in that: Each type of the second posture data corresponds to a second motion trend and a second similarity; determining the target posture data of the camera when shooting the target image according to the first motion trend, the second motion trend, the first similarity and the second similarity includes: Selecting candidate second posture data from the at least one second posture data, wherein the candidate second posture data is second posture data corresponding to the second motion trend being consistent with the first motion trend; The target pose data is determined based on the candidate second pose data, the first pose data, the second similarity corresponding to the candidate second pose data, and the first similarity.

3. The camera fusion positioning method according to claim 2, characterized in that: Determining the target pose data based on the candidate second pose data, the first pose data, the second similarity corresponding to the candidate second pose data, and the first similarity comprises: Using the first posture data as a default value of posture data, and using the first similarity as a default value of similarity; Determine the corresponding candidate second pose data with the largest second similarity as the target candidate second pose data; When the second similarity corresponding to the target candidate second pose data is greater than the similarity default value, updating the similarity default value to the second similarity corresponding to the target candidate second pose data, and updating the pose data default value to the target candidate second pose data; The default value of the pose data is determined to be the target pose data, and the default value of the similarity is determined to be the target similarity when the camera captures the target image.

4. The camera fusion positioning method according to claim 1, characterized in that: Determining a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image includes: Dividing the first virtual image into a plurality of partially overlapping first virtual image blocks, dividing the second virtual image into a plurality of partially overlapping second virtual image blocks, and dividing the target image into a plurality of partially overlapping real image blocks; Determine a first virtual feature value of the first virtual image block, a second virtual feature value of the second virtual image block, and a real feature value of the real image block, wherein the first virtual feature value, the second virtual feature value, and the real feature value all include a standard deviation of intensity values ​​of pixels in the image block, and a whiteness value used to reflect a proportion of specified pixels in the image block; Deleting the first virtual image block, the second virtual image block, and the real image block whose whiteness value is greater than a preset whiteness value; The retained first virtual image blocks are arranged in descending order according to the standard deviation of the pixel point intensity values ​​of the first virtual image blocks to obtain a first virtual image block sequence, and the retained second virtual image blocks are arranged in descending order according to the standard deviation of the pixel point intensity values ​​of the second virtual image blocks to obtain a second virtual image block sequence; The retained real image blocks are arranged in descending order according to the standard deviation of the pixel intensity values ​​of the real image blocks to obtain a first real image block sequence; Selecting a preset number of the virtual image blocks from the first virtual image sequence in a front-to-back order to obtain a third virtual image block sequence, and selecting a preset number of the virtual image blocks from the second virtual image sequence in a front-to-back order to obtain a fourth virtual image block sequence; Selecting the preset number of real image blocks from the real image sequence to obtain a second real image block sequence; Determine a first root mean square error value between the third virtual image block sequence and the second real image block sequence, and determine a second root mean square error value between the fourth virtual image block sequence and the second real image block sequence, wherein the first root mean square error value is used to characterize the first similarity, and the second root mean square error value is used to characterize the second similarity.

5. The camera fusion positioning method according to claim 4, characterized in that: The specified pixel point in the image block is determined by: Determine the saturation value and brightness value of the pixel points in the image block; When the saturation value of the pixel point is less than a preset saturation value and the brightness value of the pixel point is greater than a preset brightness value, the pixel point is determined to be the designated pixel point.

6. The camera fusion positioning method according to claim 1, characterized in that: Determining a first motion trend corresponding to the first posture data and a second motion trend corresponding to the second posture data includes: Determine a reference image and reference posture data of the camera when shooting the reference image, wherein the reference image and the target image are two frames of images continuously shot by the camera in the physiological channel, and the camera first captures the reference image and then captures the target image; Determine, based on the reference pose data and the first pose data, a first translation increment in a preset direction when the camera shoots the target image relative to that when the reference image is shot, and determine, based on the reference pose data and the second pose data, a second translation increment in a preset direction when the camera shoots the target image relative to that when the reference image is shot; The trend type of the first motion trend is determined according to the positive or negative value of the first translation increment, and the trend type of the second motion trend is determined according to the positive or negative value of the second translation increment.

7. The camera fusion positioning method according to claim 6, characterized in that: The preset direction includes an extending direction of a lens of the camera when shooting the reference image.

8. The camera fusion positioning method according to claim 1, characterized in that: After determining the target pose data of the camera when shooting the target image, the camera fusion positioning method further includes: Acquire a plurality of frames of real images continuously shot by the camera in time sequence, and a third similarity corresponding to each frame of the real image, wherein the target image is the last frame of the plurality of frames of real images, and the third similarity of the real image is a similarity determined based on the first similarity and the second similarity corresponding to the real image; Determining a tracking state index of the target image according to the third similarity corresponding to the real image of each frame, wherein the tracking state index includes a poor state and an excellent state; When it is determined that the target image is a key frame image and the corresponding tracking state indicator is a state difference, the target posture data is updated.

9. The camera fusion positioning method according to claim 8, characterized in that: Determining the tracking state index of the target image according to the third similarity corresponding to the real image of each frame includes: Calculating the sum of the third similarities corresponding to the real images of each frame; When the sum of the third similarities is not less than a preset threshold, determining that the tracking state indicator is in an excellent state; When the sum of the third similarities is less than the preset threshold, the tracking state indicator is determined to be a state difference.

10. The camera fusion positioning method according to claim 1, characterized in that: The at least one second posture data includes the second posture data determined based on the target image and the reference image, wherein the reference image and the target image are two frames of images continuously captured by the camera in the physiological channel, and the camera first captures the reference image and then captures the target image.

11. The camera fusion positioning method according to claim 1, characterized in that: The at least one second position pose data includes the second position pose data determined based on a shape of a flexible instrument, wherein the flexible instrument is connected to the camera.

12. A camera fusion positioning device, characterized in that: include: A posture calculation module, used to determine first posture data and at least one second posture data of a camera in a physiological channel when shooting a target image, wherein the posture data corresponding to the first posture data and the second posture data are determined in different ways, and the first posture data is camera posture data determined based on initial posture data and motion state data of a robotic arm, and the robotic arm is used to control the motion state of the camera; a motion trend calculation module, used to determine a first motion trend corresponding to the first posture data, and a second motion trend corresponding to the second posture data, wherein the trend types of the first motion trend and the second motion trend include forward and backward; a similarity calculation module, used to determine a first virtual image and a second virtual image captured by a virtual camera corresponding to the camera in the physiological channel model of the physiological channel, and to determine a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image acquired by the virtual camera at a posture indicated by the first posture data, and the second virtual image is an image acquired by the virtual camera at a posture indicated by the second posture data; A processing module is used to determine the target posture data of the camera when shooting the target image according to the first motion trend, the second motion trend, the first similarity and the second similarity.

13. A camera fusion positioning system, characterized in that: It includes cameras, electronic equipment, and robotic arms, among which, The electronic device is used to determine first pose data and at least one second pose data of a camera in a physiological channel when shooting a target image, wherein the pose data corresponding to the first pose data and the second pose data are determined in different ways, and the first pose data is camera pose data determined based on initial pose data and motion state data of a robotic arm, and the robotic arm is used to control the motion state of the camera; determine a first motion trend corresponding to the first pose data, and a second motion trend corresponding to the second pose data, wherein the trend types of the first motion trend and the second motion trend include forward and backward; determine a first virtual image and a second virtual image captured by a virtual camera corresponding to the camera in a physiological channel model of the physiological channel, and determine a first similarity between the first virtual image and the target image, and a second similarity between the second virtual image and the target image, wherein the first virtual image is an image obtained by the virtual camera at the pose indicated by the first pose data, and the second virtual image is an image obtained by the virtual camera at the pose indicated by the second pose data; determine the target pose data of the camera when shooting the target image based on the first motion trend, the second motion trend, the first similarity and the second similarity.

14. A non-volatile storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images