Positioning method and apparatus, nonvolatile storage medium, and electronic device

Abstract

The application discloses a positioning method and device, a nonvolatile storage medium and an electronic device. The method comprises the following steps: determining a first curvature peak point of a flexible instrument at a first time and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; determining a second curvature peak point of the flexible instrument at a second time and second position information of the second curvature peak point; determining displacement information of a flexible instrument end of the flexible instrument according to the first position information and the second position information; and determining pose information of the flexible instrument end at the second time according to the displacement information and pose information of the flexible instrument end at the first time. The application solves the technical problem that the position of the flexible instrument end cannot be accurately positioned at a position where the feature of the physiological channel is not obvious due to too many image features needing to be collected in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of navigation, in particular to a positioning method and device, a non-volatile storage medium and an electronic device. BACKGROUND

[0002] In the related art, when implementing navigation of a flexible instrument, a common method is to continuously collect images of a physiological channel during movement of the flexible instrument in the physiological channel, and to implement navigation through image registration. However, this method can accurately determine the position of the end of the flexible instrument at a branch of the physiological channel or other positions with obvious features, but cannot accurately determine the position of the end of the flexible instrument at positions in the middle of the physiological channel or other positions with non-obvious features.

[0003] To address the above problems, no effective solutions have been proposed. SUMMARY

[0004] Embodiments of the present application provide a positioning method and device, a non-volatile storage medium and an electronic device to at least solve the technical problem that the position of the end of the flexible instrument cannot be accurately determined at positions with non-obvious features in the physiological channel due to too many image features needing to be collected when positioning in the related art.

[0005] According to an aspect of an embodiment of the present application, a positioning method is provided, including: determining a first curvature peak point of a flexible instrument at a first time, and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; determining a second curvature peak point of the flexible instrument at a second time, and second position information of the second curvature peak point; determining displacement information of an end of the flexible instrument of the flexible instrument according to the first position information and the second position information; and determining pose information of the end of the flexible instrument at the second time according to the displacement information and pose information of the end of the flexible instrument at the first time.

[0006] Optionally, the step of determining the second curvature peak point of the flexible instrument at the second time includes: determining a search range corresponding to the second curvature peak point in the flexible instrument according to the first position information, wherein the search range contains points on the flexible instrument having a distance from the first curvature peak point within a preset distance range; and determining a point with a curvature difference corresponding to the first curvature peak point less than a preset curvature difference in the search range as the second curvature peak point.

[0007] Optionally, the step of determining the displacement information of the flexible instrument end of the flexible instrument according to the first position information and the second position information comprises: in the case that the number of the first curvature peak points is one, determining the displacement distance of the flexible instrument end of the flexible instrument as the distance difference between the second position information and the first position information; in the case that the number of the first curvature peak points is multiple, dividing any first curvature peak point and the second curvature peak point matched with the any first curvature peak point into the same curvature peak point group, determining the weight coefficient corresponding to each curvature peak point group, and performing weighted summation calculation on the distance difference of each curvature peak point group according to the weight coefficient to obtain the displacement information, wherein the distance difference of any curvature peak point group is calculated based on the second position information of the second curvature peak point and the first position information of the first curvature peak point contained in the curvature peak point group.

[0008] Optionally, the difference between the diameter of the target physiological channel corresponding to the first curvature peak point and the second curvature peak point and the diameter of the flexible instrument is less than a preset difference threshold.

[0009] Optionally, the first time is the time when the device connected with the flexible instrument collects the first channel image of the target physiological channel, and the second time is the time when the device connected with the flexible instrument collects the second channel image of the target physiological channel. The step of determining the pose information of the flexible instrument end at the second time according to the displacement information and the pose information of the flexible instrument end at the first time comprises: determining a specified virtual image corresponding to the second time and the second channel image, wherein the specified virtual image is an image collected in the channel model of the target physiological channel by a virtual camera corresponding to the flexible instrument; determining the angle change information of the flexible instrument end at the second time relative to the first time according to the second channel image and the specified virtual image; and determining the pose information of the flexible instrument end at the second time according to the angle change information, the displacement information and the pose information of the flexible instrument end at the first time.

[0010] Optionally, the step of determining the specified virtual image corresponding to the second time comprises: determining first camera pose information of the virtual camera in the channel model at the first time, wherein the first camera pose information is the same as the pose information of the flexible instrument end at the first time; determining second camera pose information of the virtual camera in the channel model at the second time in combination with the displacement information and the first camera pose information; and obtaining the specified virtual image collected by the virtual camera in the second camera pose information.

[0011] Optionally, obtaining the specified virtual image collected by the virtual camera in the second camera pose information comprises: obtaining electronic computed tomography information of the target physiological channel, and generating the channel model according to the electronic computed tomography information; and obtaining the specified virtual image corresponding to the second time according to the second camera pose information and the channel model.

[0012] According to a further aspect of the embodiments of the present application, a positioning apparatus is provided, comprising: a first positioning module configured to determine a first curvature peak point of a flexible instrument at a first time and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; a second positioning module configured to determine a second curvature peak point of the flexible instrument at a second time and second position information of the second curvature peak point; a calculation module configured to determine displacement information of a flexible instrument end of the flexible instrument according to the first position information and the second position information; and a processing module configured to determine pose information of the flexible instrument end at the second time according to the displacement information and pose information of the flexible instrument end at the first time.

[0013] According to a further aspect of the embodiments of the present application, a nonvolatile storage medium is provided, wherein the nonvolatile storage medium stores a program, and the program, when executed, controls a device in which the nonvolatile storage medium is located to perform the positioning method.

[0014] According to a further aspect of the embodiments of the present application, an electronic device is provided, comprising a memory and a processor, wherein the processor is configured to execute a program stored in the memory, and the program, when executed, performs the positioning method.

[0015] In the embodiments of the present application, the first curvature peak point of the flexible instrument at the first time and the first position information of the first curvature peak point are determined, wherein the flexible instrument is an instrument located in a target physiological channel; the second curvature peak point of the flexible instrument at the second time and the second position information of the second curvature peak point are determined; the displacement information of the flexible instrument end of the flexible instrument is determined according to the first position information and the second position information; and the pose information of the flexible instrument end at the second time is determined according to the displacement information and the pose information of the flexible instrument end at the first time. In this way, the position information of the curvature peak points of the flexible instrument at different times is determined, so as to determine the pose information of the flexible instrument according to the shape characteristics of the flexible instrument, thereby achieving the technical effect of positioning the flexible instrument end according to the shape characteristics of the flexible instrument, and further solving the technical problem that the position of the flexible instrument end cannot be accurately positioned at a position where the feature of the physiological channel is not obvious due to too many image features needing to be collected in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and the explanation of the present application, and do not constitute improper limitations to the present application. In the drawings:

[0017] Figure 1is a structural schematic diagram of a computer terminal according to an embodiment of the application;

[0018] Figure 2 is a flow schematic diagram of a positioning method according to an embodiment of the application;

[0019] Figure 3 is a model schematic diagram of a physiological channel model according to an embodiment of the application;

[0020] Figure 4 is a redundant space schematic diagram according to an embodiment of the application;

[0021] Figure 5 is a schematic diagram of a flexible instrument shape and corresponding curvature according to an embodiment of the application;

[0022] Figure 6 is a schematic diagram of a curvature peak point position information change according to an embodiment of the application;

[0023] Figure 7 is a multi-curvature peak point schematic diagram according to an embodiment of the application;

[0024] Figure 8 is a schematic diagram of a real image and a virtual image according to an embodiment of the application;

[0025] Figure 9 is a pose update flow schematic diagram according to an embodiment of the application;

[0026] Figure 10 is a structural schematic diagram of a positioning device according to an embodiment of the application. DETAILED DESCRIPTION

[0027] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should belong to the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second" and the like in the description and in the claims of the present application and the above-described accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] In the related art, a commonly used navigation method is to collect images in a target physiological channel and perform registration based on image features to determine the position of the flexible instrument tip in the physiological channel. However, image-based navigation needs to rely on the number of features of the real image currently seen, and when the flexible instrument tip reaches the bifurcation of the physiological channel, the image features in the collected physiological channel images are more obvious (the number of bifurcations, shapes, etc.), but before reaching the bifurcation, that is, when the flexible instrument tip is in the middle of a certain channel or just enters a certain channel, only the airway walls on both sides can be observed at this time, and the features of the image are less, which is not enough to perform matching calculation of identifiable structures, which may cause mismatch or inaccurate position information. In addition, before calculating the similarity of the identifiable structures, physiological channel pictures that do not meet the requirements are also removed, for example, photos with a large number of blurs and light spots cannot be used for calculation. Thus, when the flexible instrument tip advances too fast, a series of consecutive frame pictures cannot enter the calculation of structure similarity recognition due to insufficient camera frame numbers, thereby causing tracking breakpoints or incorrect connections, resulting in inconsistency between the subsequently tracked positions.

[0030] Therefore, in the actual use process in the related art, the surgeon needs to frequently manually correct the position of the flexible instrument tip, resulting in a cumbersome surgical process. In order to solve this problem, the related solutions are provided in the embodiments of the present application, which are described in detail below.

[0031] According to the embodiments of the present application, a method embodiment of a positioning method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0032] The method embodiment provided by the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1A hardware structure block diagram of a computer terminal (or mobile device) for implementing the positioning method is shown. As shown in Figure 1 The computer terminal 10 (or mobile device 10) can include one or more processors 102 (the processor 102 can include but not limited to a microprocessor MCU or a programmable logic device FPGA processing device, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports in the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can include more or less components than those shown in Figure 1 or have a different configuration than that shown in Figure 1 .

[0033] It should be noted that the one or more processors 102 and / or other data processing circuits described above can be referred to herein as "data processing circuits" in general. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any combination thereof. In addition, the data processing circuit can be a single independent processing module, or any one of the other elements combined into the computer terminal 10 (or mobile device) in whole or in part. As referred to in the embodiments of the present application, the data processing circuit is a processor control (for example, selection of a variable resistance terminal path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the positioning method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the positioning method of the application program described above. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 can further include a memory remotely disposed with respect to the processor 102, which can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0035] The transmission device 106 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) that can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module configured to communicate with the Internet in a wireless manner.

[0036] The display can be a liquid crystal display (LCD) in the form of a touch screen, which can enable a user to interact with the user interface of the computer terminal 10 (or the mobile device).

[0037] In the above operating environment, the embodiments of the present application provide a positioning method, as shown in the accompanying drawings, which includes the following steps: Figure 2

[0038] In step S202, a first curvature peak point of the flexible instrument at a first time is determined, and first position information of the first curvature peak point is determined, wherein the flexible instrument is an instrument located in a target physiological channel.

[0039] Before the technical solution provided in step S202 is executed, the computerized tomography information of the target physiological channel is acquired, and a channel model is generated according to the computerized tomography information, and a moving path of the flexible instrument in the target physiological channel is determined.

[0040] Specifically, taking the target physiological channel as a bronchial tract for example, before the first curvature peak point of the flexible instrument at the first time is determined, the three-dimensional bronchial tree model information of the bronchial tract is acquired, and the moving path of the flexible instrument in the bronchial tract is determined. When the three-dimensional bronchial tree model information is acquired, the CT data of the bronchial tract, that is, the computerized tomography information, can be acquired, and the three-dimensional bronchial tree model as shown in the accompanying drawings can be constructed according to the CT data. In this way, the three-dimensional bronchial tree model can be constructed according to the computerized tomography information of the bronchial tract. Figure 3 Figure 3 ​​The left model is a complete model, and the right model is a bronchial tree simplified model in which the center line of the bronchus is used to replace the bronchus. The bronchial tree simplified model can be used to determine a moving path, and the complete model can be used to obtain a virtual image. After obtaining the bronchial tree model, a lesion position can be marked in the bronchial tree model by an automatic lesion detection algorithm or manually by a target user. Then, a path reaching the lesion position can be generated according to the bronchial tree simplified model and the lesion position, and the path can be used as a planned path in a subsequent positioning and navigation process. When the virtual image is obtained by using the complete model, the inside of the three-dimensional model can be rendered by using a rendering technology, and a virtual image of the inside of the three-dimensional model after rendering can be collected by using a virtual camera.

[0041] In the technical solution provided in step S202, the flexible instrument can be an optical fiber instrument.

[0042] In step S204, a second curvature peak point of the flexible instrument at a second time is determined, and second position information of the second curvature peak point is determined.

[0043] In the technical solutions provided in steps S202 and S204, the first time is a time at which a device connected to the flexible instrument collects a first channel image of the target physiological channel, and the second time is a time at which the device connected to the flexible instrument collects a second channel image of the target physiological channel. The channel images collected at different times can be used to determine the pose information of the end of the flexible instrument at the time.

[0044] In some embodiments of the present application, the first channel image and the second channel image can be two images continuously collected by the device, that is, the first time and the second time are times at which the two images are continuously collected. In this case, the determined pose information of the end of the flexible instrument can be more accurate.

[0045] As an optional implementation, the first channel image and the second channel image can also be two images discontinuously collected, for example, can be two images collected at a time interval of a preset time length. The preset time length can be set according to actual needs, so as to save computing resources for calculating the pose information of the end of the flexible instrument.

[0046] As an optional implementation, the step of determining the second curvature peak point of the flexible instrument at the second time includes: determining a search range corresponding to the second curvature peak point in the flexible instrument according to the first position information, wherein the search range contains points on the flexible instrument having a distance from the first curvature peak point within a preset distance range; and determining a point in the search range as the second curvature peak point, where a curvature difference corresponding to the first curvature peak point is less than a preset curvature difference.

[0047] Specifically, during the movement of the flexible instrument, since various structures in the physiological channel do not change, and the length of the flexible instrument is fixed, as the flexible instrument moves, the relative position of the curvature peak point on the flexible instrument also changes, and the specific change form is as shown in Figure 6 The dashed line in the figure indicates the shape of the flexible instrument at the previous moment and the corresponding curvature information, and the solid line indicates the shape of the flexible instrument at the current moment and the curvature information. Figure 6

[0048] In order to facilitate the determination of the position of the curvature peak point, a one-dimensional coordinate system can be established on the flexible instrument with the end point of one end of the flexible instrument as a reference. The coordinates of each point on the flexible instrument in the coordinate system can be used to reflect the distance of the point from the reference point and the end of the flexible instrument. For example, in the case where the measurement of the coordinate system is 1 mm, the point with a coordinate of 78 indicates that the distance of the point from the reference point is 78 mm, and assuming that the total length of the optical fiber is 100 mm, the distance of the point from the end of the flexible instrument is 22 mm.

[0049] After the coordinate system is established, when the position of the second curvature peak point is determined, a time sequence prediction method can be used. Since the acquisition frequency of the flexible instrument is higher than that of the endoscope (the acquisition frequency of the endoscope is 30 HZ), and the speed at which the target user operates the flexible instrument is also limited to a certain extent. Therefore, it can be considered that the second curvature peak point will be within M coordinate difference intervals on the left and right of the first curvature peak. M is a positive integer and can be determined based on historical data.

[0050] It should be noted that the end of the flexible instrument in the embodiment refers to the end of the flexible instrument deep into the physiological channel.

[0051] In some embodiments of the present application, the difference between the diameter of the target physiological channel corresponding to the first curvature peak point and the second curvature peak point and the diameter of the flexible instrument is less than a preset difference threshold.

[0052] Specifically, as shown in Figure 4 When the optical fiber flexible instrument is bent, due to the difference between the diameter of the optical fiber flexible instrument and the diameter of the cavity, there is a certain excess space, which is called redundant space. When there is redundant space, especially at the bending part, different shapes are easily formed, which will interfere with the positioning and detection of the curvature peak point. As shown in the figure, when the optical fiber deepens into the bronchus, multiple peaks appear and tracking, and the wave peak closer to the left side on the curvature image has a larger redundant space. When the flexible instrument is slightly twisted or pushed forward by force F, due to the force transmission at the bending part, a component force is generated, which causes the flexible instrument to move to the left side close to the inner wall of the cavity while advancing.

[0053] ​To eliminate the influence of redundant space on positioning, the diameter of the physiological channel corresponding to the curvature peak point and the diameter of the optical fiber can be compared. When the difference between the two is less than a threshold D-Threshold, it can be considered that the flexible instrument does not have redundant space at this location. The wave peak offset after this location represents the distance of the advancement of the optical fiber, and the wave peak offset in front is discarded.

[0054] In determining the diameter of the physiological channel, the diameter of the physiological channel corresponding to each curvature peak point can be estimated by comparing the distance between each curvature peak point and some physiological feature points in the physiological channel, such as the epiglottis. Specifically, when obtaining the CT data of the target physiological channel, the change of the distance at each location in the target physiological channel relative to the epiglottis and the like can be obtained. Thus, the diameter of the target physiological channel corresponding to each curvature peak point can be determined according to the distance between the curvature peak point and the epiglottis.

[0055] As an optional implementation, when the diameter of the physiological channel corresponding to the curvature peak point is estimated based on the epiglottis as the reference point, the distance between the curvature peak point and the initial curvature peak point formed at the beginning of the movement of the flexible instrument can be considered as the distance between the curvature peak point and the epiglottis.

[0056] Step S206, determining the displacement information of the flexible instrument end of the flexible instrument according to the first position information and the second position information;

[0057] In the technical solution provided in step S206, the step of determining the displacement information of the flexible instrument end of the flexible instrument according to the first position information and the second position information includes: in the case where the number of the first curvature peak points is one, determining that the displacement distance of the flexible instrument end of the flexible instrument is the distance difference between the second position information and the first position information; in the case where the number of the first curvature peak points is multiple, dividing any first curvature peak point and the second curvature peak point matched with the any first curvature peak point into the same curvature peak point group, determining the weight coefficient corresponding to each curvature peak point group, and performing weighted summation calculation on the distance difference of each curvature peak point group according to the weight coefficient to obtain the displacement information, wherein the distance difference of any curvature peak point group is calculated based on the second position information of the second curvature peak point and the first position information of the first curvature peak point contained in the curvature peak point group.

[0058] In the case where multiple second curvature peak points exist, the second curvature peak point matched with any first curvature peak point can be determined in the following manner:

[0059] 1) Determine the search range corresponding to any first curvature peak point; the point in the search range, at which the curvature difference between the curvature and the first curvature peak point is less than a preset curvature difference, is determined as the second curvature peak point matched with any first curvature peak point.

[0060] 2) determine the second curvature peak value matched to any first curvature peak value according to the arrangement order of each peak value on the flexible instrument.

[0061] Specifically, as shown in Figure 7 , as the flexible instrument continues to penetrate, the shape of the flexible instrument will produce multiple bends, which corresponds to multiple peaks on the curvature curve, wherein, Figure 7 the upper half of the line represents the shape of the flexible instrument, and the lower half of the line represents the curvature of each point on the flexible instrument. The dots on the curvature graph represent the curvature peak values. As the depth of the physiological channel increases, the diameter of the physiological channel becomes smaller, that is, the redundant space becomes smaller, and the wave peak offset of the flexible instrument is closer to the actual advancement of the flexible instrument. Therefore, the advancement can be directly represented by the offset value of the last wave peak, or the offsets of multiple peaks can be calculated by weighted sum.

[0062] In the weighted sum, the same weight can be set for each curvature peak value, or the weight can be set according to the distance of each curvature peak value from the end of the flexible instrument. Specifically, the closer to the end of the flexible instrument, the greater the weight.

[0063] Step S208, according to the displacement information and the pose information of the end of the flexible instrument at the first time, determine the pose information of the end of the flexible instrument at the second time.

[0064] In determining the pose information, the virtual camera can be moved by the same displacement information determined above without changing the angle, and a specified virtual image can be collected, and then the rotation angle of the second channel image (real image) and the specified virtual image can be calculated to obtain the angle change of the end of the flexible instrument.

[0065] Specifically, in the technical solution provided in step S208, the step of determining the pose information of the end of the flexible instrument at the second time according to the displacement information and the pose information of the end of the flexible instrument at the first time includes: determining the specified virtual image corresponding to the second time and the second channel image, wherein the specified virtual image is an image collected by a virtual camera corresponding to the flexible instrument in the channel model of the target physiological channel; determining the angle change information of the end of the flexible instrument at the second time relative to the first time according to the real image and the specified virtual image; and determining the pose information of the end of the flexible instrument at the second time according to the angle change information, the displacement information and the pose information of the end of the flexible instrument at the first time.

[0066] In some embodiments of the present application, the real image and the specified virtual image are as shown in Figure 8 .

[0067] As an optional implementation, the step of determining the specified virtual image corresponding to the second time point comprises: determining first camera pose information of the virtual camera in the channel model at the first time point, wherein the first camera pose information is the same as the pose information of the distal end of the flexible instrument at the first time point; determining second camera pose information of the virtual camera in the channel model at the second time point in combination with the displacement information and the first camera pose information, wherein the angle information of the first camera pose information and the second camera pose information is the same; and obtaining the specified virtual image collected by the virtual camera in the second camera pose information.

[0068] In some embodiments of the present application, obtaining the specified virtual image collected by the virtual camera in the second camera pose information comprises the following steps: obtaining computed tomography information of the target physiological channel, and generating a channel model according to the computed tomography information; and obtaining the specified virtual image corresponding to the second time point according to the second camera pose information and the channel model.

[0069] Taking the bronchial tree as an example, the CT scan image of the patient's lung can be collected first, and a three-dimensional reconstruction is performed according to the CT scan image to obtain a virtual three-dimensional bronchial tree corresponding to the patient. The virtual three-dimensional bronchial tree obtained above and the pose data can be input into a renderer to obtain the virtual image corresponding to each pose data output by the renderer. For example, assuming that the channel model is the virtual three-dimensional bronchial tree, the second camera pose information and the virtual three-dimensional bronchial tree can be input into the renderer to obtain the specified virtual image output by the renderer. The renderer refers to a rendering engine for rendering a three-dimensional model file, including but not limited to: amold renderer, vary renderer, etc.

[0070] It should be noted that, since the angle change of the distal end of the flexible instrument needs to be determined using the real image and the specified virtual image, it is necessary to ensure that the real image and the specified virtual image are images corresponding to the same position in the physiological channel. Using the positioning method provided in the present application, it can be ensured that the problem of different corresponding positions of the real image and the specified virtual image does not occur during the movement of the flexible instrument, provided that the real image and the specified virtual image have corresponded to the same position. Therefore, in order to ensure the accuracy of the positioning result, the virtual camera and the real image acquisition device need to be manually aligned by the target object before the positioning process starts, so as to ensure that the virtual image collected by the virtual camera and the real image collected by the image acquisition device correspond to the same position in the physiological channel.

[0071] Specifically, when the optical fiber reaches the main airway through the mouth / nose, there is a larger bending angle at the epiglottis position. By processing the shape of the instrument fed back by the optical fiber, it can be found that the bending angle is embodied by a larger curvature, that is, the greater the bending degree, the greater the curvature, the closer to the straight line, and the smaller the curvature. As shown in FIG. 8, when the flexible instrument reaches the main channel of the physiological channel, Figure 5 As shown in FIG. 9, Figure 5 The upper half of the line is the bending shape of the instrument fed back by the optical fiber, and the lower half of the line is the curvature calculated according to the shape. The circle represents the epiglottis position. It can be seen that the position corresponding to the peak value of the curvature and the maximum bending exist corresponding relationship. Initialization is to artificially align the image of the virtual camera and the image of the real endoscope. In the case of good alignment, the image registration algorithm will fine-tune the virtual camera to achieve complete consistency between the real image and the virtual image. The virtual camera can obtain the position in the coordinate system of the CT reconstructed three-dimensional model by the rendering system, so as to obtain the position of the flexible instrument tip (the flexible instrument tip is the real endoscope position) in the three-dimensional model, and complete the initialization of navigation.

[0072] In the embodiments of the present application, a flowchart of a flexible instrument tip pose information updating flow is also provided as shown in FIG. 10. As shown in FIG. 11, the flowchart comprises the following steps: Figure 9 Figure 9 As shown in FIG. 12, the flowchart comprises the following steps:

[0073] Step S302, aligning the virtual camera and the endoscope;

[0074] After alignment, it can be ensured that the images collected by the virtual camera and the endoscope are images of the same position in the physiological channel.

[0075] Step S304, collecting a real image by the endoscope in the process of moving the flexible instrument;

[0076] Step S306, determining the shape feature of the flexible instrument, and tracking the curvature peak point of the flexible instrument according to the shape feature, so as to determine the displacement of the flexible instrument tip;

[0077] Step S308, determining the specified virtual image at the current time according to the displacement of the flexible instrument tip;

[0078] Step S310, determining the current time pose information of the flexible instrument tip according to the real image at the current time and the specified virtual image at the current time, and updating the specified virtual image at the current time according to the current time pose information;

[0079] Step S312, displaying the updated specified virtual image to the target object and displaying the pose information of the flexible instrument tip in the three-dimensional model.

[0080] ​The first curvature peak point of the flexible instrument at the first time is determined, and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; a second curvature peak point of the flexible instrument at a second time is determined, and second position information of the second curvature peak point is determined; displacement information of a flexible instrument end of the flexible instrument is determined according to the first position information and the second position information; and pose information of the flexible instrument end at the second time is determined according to the displacement information and pose information of the flexible instrument end at the first time. By determining the position information of the curvature peak points of the flexible instrument at different times, the purpose of determining the displacement information of the flexible instrument through the shape characteristics of the flexible instrument is achieved, thereby realizing the technical effect of positioning the flexible instrument end through the shape characteristics of the flexible instrument, and further solving the technical problem that the position of the flexible instrument end cannot be accurately positioned at a feature unclear part of the physiological channel due to too many image features needing to be collected in positioning in the related art.

[0081] The embodiment of the present application provides a positioning device. Figure 10 is a structural schematic diagram of the positioning device, as Figure 10 shown, the device comprises: a first positioning module 40, configured to determine a first curvature peak point of a flexible instrument at a first time, and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; a second positioning module 42, configured to determine a second curvature peak point of the flexible instrument at a second time, and second position information of the second curvature peak point; a calculation module 44, configured to determine displacement information of a flexible instrument end of the flexible instrument according to the first position information and the second position information; and a processing module 46, configured to determine pose information of the flexible instrument end at the second time according to the displacement information and pose information of the flexible instrument end at the first time.

[0082] In some embodiments of the present application, the first time is a time when a device connected with the flexible instrument collects a first channel image of the target physiological channel, and the second time is a time when the device connected with the flexible instrument collects a second channel image of the target physiological channel.

[0083] In some embodiments of the present application, a difference between a diameter of the target physiological channel corresponding to the first curvature peak point and the second curvature peak point and a diameter of the flexible instrument is less than a preset difference threshold.

[0084] In some embodiments of the present application, the second positioning module 42 determines the second curvature peak point of the flexible instrument at the second time, comprising: determining a search range corresponding to the second curvature peak point in the flexible instrument according to the first position information, wherein the search range contains points on the flexible instrument having a distance from the first curvature peak point within a preset distance range; determining the point in the search range as the second curvature peak point, where the curvature difference corresponding to the first curvature peak point is less than a preset curvature difference.

[0085] In some embodiments of the present application, the calculation module 44 determines the displacement information of the flexible instrument end of the flexible instrument according to the first position information and the second position information, comprising: in the case where the number of first curvature peak points is one, determining the displacement distance of the flexible instrument end of the flexible instrument as the distance difference between the second position information and the first position information; in the case where the number of first curvature peak points is more than one, dividing any first curvature peak point and the second curvature peak point matched with any first curvature peak point into the same curvature peak point group, determining the weight coefficient corresponding to each curvature peak point group, and performing weighted summation calculation on the distance difference of each curvature peak point group according to the weight coefficient to obtain the displacement information, wherein the distance difference of any curvature peak point group is calculated based on the second position information of the second curvature peak point and the first position information of the first curvature peak point contained therein.

[0086] In some embodiments of the present application, the processing module 46 determines the pose information of the flexible instrument end at the second time according to the displacement information and the pose information of the flexible instrument end at the first time, comprising: determining the specified virtual image corresponding to the second time and the second channel image, wherein the specified virtual image is an image collected in the channel model of the target physiological channel by the virtual camera corresponding to the flexible instrument; determining the angle change information of the flexible instrument end at the second time relative to the first time according to the real image and the specified virtual image; determining the pose information of the flexible instrument end at the second time according to the angle change information, the displacement information and the pose information of the flexible instrument end at the first time.

[0087] In some embodiments of the present application, the processing module 46 determines the specified virtual image corresponding to the second time, comprising: determining the first camera pose information of the virtual camera in the channel model at the first time, wherein the first camera pose information is the same as the pose information of the flexible instrument end at the first time; determining the second camera pose information of the virtual camera in the channel model at the second time in combination with the displacement information and the first camera pose information; and obtaining the specified virtual image collected by the virtual camera when it is in the second camera pose information.

[0088] It should be noted that each module in the positioning device described above can be a program module (for example, a program instruction set implementing a certain specific function) or a hardware module. For the latter, it can be in the form of, but not limited to, a processor, or the functions of the above-mentioned modules are implemented by a processor.

[0089] An embodiment of the present application provides a nonvolatile storage medium. The nonvolatile storage medium stores a program, and when the program is executed, the device in which the nonvolatile storage medium is located performs the following positioning method: determining a first curvature peak point of a flexible instrument at a first time and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; determining a second curvature peak point of the flexible instrument at a second time and second position information of the second curvature peak point; determining displacement information of a flexible instrument end of the flexible instrument according to the first position information and the second position information; and determining pose information of the flexible instrument end at the second time according to the displacement information and pose information of the flexible instrument end at the first time.

[0090] An embodiment of the present application provides an electronic device. The electronic device includes a memory and a processor, and the processor is configured to execute a program stored in the memory, and when the program is executed, the following positioning method is performed: determining a first curvature peak point of a flexible instrument at a first time and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; determining a second curvature peak point of the flexible instrument at a second time and second position information of the second curvature peak point; determining displacement information of a flexible instrument end of the flexible instrument according to the first position information and the second position information; and determining pose information of the flexible instrument end at the second time according to the displacement information and pose information of the flexible instrument end at the first time.

[0091] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0092] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0093] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0094] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0095] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0096] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the related art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0097] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A positioning device, characterized in that The method comprises the following steps: a first positioning module is configured to determine a first curvature peak point of a flexible instrument at a first time and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; a second positioning module is configured to determine a second curvature peak point of the flexible instrument at a second time and second position information of the second curvature peak point; a calculation module is configured to determine displacement information of a flexible instrument end of the flexible instrument according to the first position information and the second position information; a processing module is configured to determine pose information of the flexible instrument end at the second time according to the displacement information and pose information of the flexible instrument end at the first time.

2. The positioning device of claim 1, wherein, The second positioning module is further configured to: determine a search range corresponding to the second curvature peak point in the flexible instrument according to the first position information, wherein the search range contains points on the flexible instrument that are within a preset distance range from the first curvature peak point; determine that a point in the search range, at which a curvature difference corresponding to the first curvature peak point is less than a preset curvature difference, is the second curvature peak point.

3. The positioning device of claim 1, wherein, The calculation module is further configured to: in a case where the number of the first curvature peak points is one, determine that a displacement distance of the flexible instrument end of the flexible instrument is a distance difference between the second position information and the first position information; in a case where the number of the first curvature peak points is a plurality, divide any first curvature peak point and a second curvature peak point matched with the any first curvature peak point into a same curvature peak point group, determine a weight coefficient corresponding to each curvature peak point group, and perform weighted summation calculation on distance difference values of each curvature peak point group according to the weight coefficient to obtain the displacement information, wherein the distance difference value of any curvature peak point group is calculated based on the second position information of the second curvature peak point and the first position information of the first curvature peak point contained in the curvature peak point group.

4. The positioning device of claim 1, wherein, A difference between a diameter of the target physiological channel corresponding to the first curvature peak point and the second curvature peak point and a diameter of the flexible instrument is less than a preset difference threshold.

5. The positioning device of claim 1, wherein, The first time is a time at which a device connected with the flexible instrument collects a first channel image of the target physiological channel, and the second time is a time at which the device connected with the flexible instrument collects a second channel image of the target physiological channel; and the processing module is further configured to: determine a specified virtual image corresponding to the second time and the second channel image, wherein the specified virtual image is an image collected in a channel model of the target physiological channel by a virtual camera corresponding to the flexible instrument; determine angle change information of the flexible instrument end relative to the first time at the second time according to the second channel image and the specified virtual image; determine pose information of the flexible instrument end at the second time according to the angle change information, the displacement information, and the pose information of the flexible instrument end at the first time.

6. The positioning device of claim 5, wherein, The processing module is further configured to: determining first camera pose information of the virtual camera in the channel model at the first time point, wherein the first camera pose information is the same as the pose information of the flexible instrument tip at the first time point; determining second camera pose information of the virtual camera in the channel model at the second time point in combination with the displacement information and the first camera pose information; obtaining a specified virtual image collected by the virtual camera in the second camera pose information.

7. The positioning device of claim 6, wherein, The processing module is further configured to: obtain computed tomography information of the target physiological channel, and generate the channel model according to the computed tomography information; obtain a specified virtual image corresponding to the second time point according to the second camera pose information and the channel model.

8. A non-volatile storage medium, characterized by The non-volatile storage medium stores a program, and when the program is running, the device in which the non-volatile storage medium is located performs the following positioning method: determining a first curvature peak point of a flexible instrument at a first time point and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; determining a second curvature peak point of the flexible instrument at a second time point and second position information of the second curvature peak point; determining displacement information of a flexible instrument tip of the flexible instrument according to the first position information and the second position information; determining pose information of the flexible instrument tip at the second time point according to the displacement information and the pose information of the flexible instrument tip at the first time point.

9. An electronic device, comprising: comprise: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program performs the following positioning method when running: determining a first curvature peak point of a flexible instrument at a first time point and first position information of the first curvature peak point, wherein the flexible instrument is an instrument located in a target physiological channel; determining a second curvature peak point of the flexible instrument at a second time point and second position information of the second curvature peak point; determining displacement information of a flexible instrument tip of the flexible instrument according to the first position information and the second position information; determining pose information of the flexible instrument tip at the second time point according to the displacement information and the pose information of the flexible instrument tip at the first time point.

Images