Marker tracking method, system and device, navigation equipment and storage medium

Abstract

The invention relates to a marker tracking method, system and device, navigation equipment and a storage medium. The method comprises the following steps: determining a marker detection result obtained by detecting all markers in a working space in which the navigation equipment is located, if the marker detection result is that a first marker which is separated from the visual field of the navigation equipment exists, determining a target parameter of the navigation equipment, and adjusting the navigation equipment according to the target parameter of the navigation equipment, therefore, all the markers enter the visual field of the navigation equipment. By adopting the method, the marker can be tracked.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a marker tracking method, system, device, navigation equipment and storage medium. Background Art

[0002] Navigation technology is an emerging medical technology. Doctors can use navigation equipment to navigate the surgical process to improve the accuracy of the surgery.

[0003] Taking head-mounted navigation devices as an example, navigation devices usually use visual sensors in the navigation devices to track preset markers for navigation. If the doctor moves his head during the operation, the marker may be out of the field of view of the visual sensor, thereby reducing the navigation efficiency of the navigation device.

[0004] Therefore, how to track markers is a key research topic for those skilled in the art. Summary of the invention

[0005] Based on this, it is necessary to provide a marker tracking method, system, device, navigation equipment and storage medium that can track the marker in order to solve the above technical problems.

[0006] In a first aspect, the present application provides a marker tracking method, which is applied to a navigation device, and the method comprises:

[0007] Determine a marker detection result obtained after detecting all markers in the workspace where the navigation device is located;

[0008] If the marker detection result is that a first marker exists, determining a target parameter of the navigation device; the first marker is a marker that is out of the field of view of the navigation device;

[0009] The navigation device is adjusted according to the target parameters of the navigation device so that all the markers enter the field of view of the navigation device.

[0010] In one of the embodiments, the target parameter includes at least one of an angle adjustment parameter and a field of view adjustment parameter;

[0011] The angle adjustment parameter is used to adjust the attitude of the visual sensor in the navigation device, and the field of view adjustment parameter is used to adjust the field of view of the visual sensor in the navigation device.

[0012] In one embodiment, the target parameter includes a first parameter and a second parameter, the first parameter and the second parameter are respectively one of an angle adjustment parameter and a field of view adjustment parameter, and the first parameter and the second parameter are different; adjusting the navigation device according to the target parameter of the navigation device includes:

[0013] If the first parameter is within a first preset parameter adjustment range corresponding to the visual sensor in the navigation device, adjusting the navigation device according to the first parameter, and updating the marker detection result after adjusting the navigation device according to the first parameter;

[0014] If the updated marker detection result is that the first marker exists, the navigation device is adjusted according to the second parameter, or a prompt message is issued.

[0015] In one embodiment, the method further comprises:

[0016] If the first parameter is not within the first preset parameter adjustment range, the navigation device is adjusted according to the second parameter, or a prompt message is issued.

[0017] In one embodiment, adjusting the navigation device according to the second parameter includes:

[0018] If the second parameter is within a second preset parameter adjustment range corresponding to the visual sensor in the navigation device, adjusting the navigation device according to the second parameter, and updating the marker detection result after adjusting the navigation device according to the second parameter;

[0019] If the updated marker detection result is that the first marker exists, a prompt message is issued.

[0020] In one embodiment, the method further comprises:

[0021] If the second parameter is not within the second preset parameter adjustment range, a prompt message is issued.

[0022] In one embodiment, issuing a prompt message includes:

[0023] Determine, based on the image data acquired by the navigation device at the first moment, first relative position information of the first marker relative to the navigation device at the first moment; wherein the first moment is the moment before the first marker leaves the field of view of the navigation device;

[0024] Determine a first pose change amount of the navigation device according to the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment;

[0025] Determine second relative position information of the first marker relative to the navigation device at a second moment according to the first posture change amount and the first relative position information;

[0026] A prompt message is issued according to the second relative position information.

[0027] In one embodiment, determining a marker detection result obtained after detecting all markers in a workspace where the navigation device is located includes:

[0028] The marker detection result is determined according to preset parameters; the preset parameters include at least one of the image data acquired by the navigation device in the current field of view, the feature information corresponding to all the markers, and the global posture information of the navigation device.

[0029] In one embodiment, determining the marker detection result according to preset parameters includes:

[0030] If it is determined that the navigation device is moving based on the global position information of the navigation device, the marker detection result is determined based on preset parameters.

[0031] In a second aspect, the present application also provides a marker tracking system, the system comprising a navigation device and at least one marker;

[0032] The navigation device is used to determine the marker detection result obtained after detecting each marker in the working space where the navigation device is located. If the marker detection result is that the first marker exists, the target parameter of the navigation device is determined, and the navigation device is adjusted according to the target parameter of the navigation device so that each marker enters the field of view of the navigation device.

[0033] In a third aspect, the present application further provides a marker tracking device, which is applied to a navigation device, and the device comprises:

[0034] A first determination module is used to determine a marker detection result obtained after detecting all markers in the working space where the navigation device is located;

[0035] A second determination module is used to determine the target parameter of the navigation device if the marker detection result is that a first marker exists; the first marker is a marker that is out of the field of view of the navigation device;

[0036] The adjustment module is used to adjust the navigation device according to the target parameters of the navigation device so that all the markers enter the field of view of the navigation device.

[0037] In a fourth aspect, the present application further provides a navigation device, wherein the navigation device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above steps when executing the computer program.

[0038] In a fifth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0039] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0040] The above-mentioned marker tracking method, system, device, navigation device and storage medium, after determining the marker detection result obtained after detecting all markers in the workspace where the navigation device is located, can automatically determine the target parameters of the navigation device and adjust the navigation device according to the target parameters, because when the marker detection result is that there is a first marker that is out of the field of view of the navigation device, therefore, after adjusting the navigation device according to the target parameters, all markers in the workspace where the navigation device is located can be returned to the field of view of the navigation device, thereby achieving the tracking of the markers. Taking the navigation device as a head-mounted device as an example, the field of view of the navigation device can better adapt to the doctor's head movement. Even if the marker is out of the field of view of the navigation device due to head movement, the navigation device can track the marker in time, thereby improving the navigation efficiency of the navigation device. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is an application environment diagram of a marker tracking method in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a flow chart of a method for tracking a marker in an embodiment of the present application;

[0043] Figure 3 This is a schematic diagram of adjusting a navigation device according to an angle adjustment parameter in an embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of adjusting a navigation device according to a field of view adjustment parameter in an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of a process for adjusting a navigation device in an embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of another process of adjusting a navigation device in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of an adjustment process in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of a process for determining angle adjustment parameters in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of a workspace in an embodiment of the present application;

[0050] Fig.10 This is a schematic diagram of another process for determining angle adjustment parameters in an embodiment of the present application;

[0051] Fig.11This is a schematic diagram of another process for determining angle adjustment parameters in an embodiment of the present application;

[0052] Fig.12 This is a schematic diagram of a process of sending a prompt message in an embodiment of the present application;

[0053] Fig.13 A schematic diagram of a global space perception process according to an embodiment of the present application;

[0054] Fig.14 This is a schematic diagram of a prompt information in an embodiment of the present application;

[0055] Fig.15 A schematic diagram of a process of tracking a marker in an embodiment of the present application;

[0056] Fig.16 This is a schematic diagram of the structure of a tracking system for a marker in an embodiment of the present application;

[0057] Fig.17 This is a schematic diagram of the structure of another marker tracking system in an embodiment of the present application;

[0058] Fig.18 This is a structural block diagram of a tracking device for a marker in an embodiment of the present application;

[0059] Fig.19 This is a diagram of the internal structure of a navigation device in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] Figure 1 FIG. 1 is an application environment diagram of a method for tracking a marker in an embodiment of the present application, such as Figure 1 As shown, the doctor wears the navigation device 101 and uses the navigation device 101 to track at least one marker 102 in the workspace where the navigation device 101 is located, so as to complete the navigation of the surgical procedure.

[0062] The navigation device 101 includes a visual sensor. According to the number of cameras, the visual sensor can be at least one of a monocular vision sensor, a binocular vision sensor, or a multi-vision sensor; according to the function, the visual sensor can be at least one of a laser vision sensor, an infrared camera, a depth camera, an RGB red-green-blue camera (RGB camera), and a structured light camera.

[0063] In some embodiments, the navigation device 101 may further include a display component, which may be at least one of a display, an augmented reality (AR) device, a virtual reality (VR) device, and a holographic projection device.

[0064] It is understandable that the navigation device 101 may also include a processing unit, which may be a central processing unit (CPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices. The processing unit may also be a personal computer, a laptop computer, a smart phone, or a tablet computer disposed outside the navigation device, and the processing unit may also be implemented by an independent server or a server cluster consisting of multiple servers.

[0065] Please refer to Figure 1 (a) and Figure 1 (b) Taking the case where the navigation device is head-mounted, during the surgical navigation process, the relative position between the navigation device 101 and the doctor's head is fixed. If the doctor's head moves, the shooting angle and shooting position of the visual sensor in the navigation device will also change, which may cause the marker to be out of the field of view of the visual sensor in the navigation device, thereby reducing the navigation efficiency. Figure 1 The middle dotted line indicates that being out of the field of view of the navigation device may also be referred to as out of sight.

[0066] Based on this, it is necessary to provide a marker tracking method that can track the marker. The following will introduce the marker tracking method.

[0067] Figure 2 This is a flow chart of a method for tracking a marker in an embodiment of the present application. The method can be applied to Figure 1 In the navigation device shown, in one embodiment, as Figure 2 As shown, the following steps are included:

[0068] S201, determining a marker detection result obtained after detecting all markers in a workspace where a navigation device is located.

[0069] Before using the navigation device for surgical navigation, the doctor needs to wear the navigation device and install at least one marker in the workspace where the navigation device is located. Among them, the marker is used to provide a spatial reference for the navigation device, including but not limited to a plane target, a QR code marker, a reflective ball, and an LED (light-emitting diode). It should be noted that the "all markers" mentioned in step S201 and subsequent related descriptions only include markers used to provide a spatial reference for the navigation device, that is, markers used to be tracked by the navigation device; for example, marker 1, marker 2, marker 3 and marker 4 are installed in the workspace, but only marker 1, marker 2 and marker 3 are used to be tracked by the navigation device, then "all markers" only include marker 1, marker 2 and marker 3.

[0070] Then, the navigation device can construct a workspace using the installed markers and navigate using the relative positions between the markers and the navigation device.

[0071] In order to return the marker to the field of view of the navigation device after the marker is out of sight, the marker needs to be tracked. Therefore, in this embodiment, the navigation device detects all markers in the working space of the navigation device and obtains the marker detection result. The marker detection result is used to detect whether there is a marker out of the field of view of the navigation device.

[0072] Exemplarily, the marker detection result may be "the first marker exists" or "the first marker does not exist". The first marker is a marker that is out of the field of view of the navigation device. The field of view of the navigation device can be understood as the field of view of the visual sensor in the navigation device. In other words, if the marker is not in the picture of the visual sensor, the marker is out of the field of view of the navigation device. It should be noted that the "first marker" only means that a marker is out of view, and does not specifically refer to which marker it is. For example, assuming that "all markers" include marker 1, marker 2 and marker 3, at moment 1, it is determined that marker 1 is out of view, then the marker detection result at moment 1 is that the first marker exists; at moment 2, it is determined that marker 2 is out of view, then the marker detection result at moment 2 is also that the first marker exists.

[0073] Assume that there are markers 1, 2 and 3 in the working space of the navigation device. Initially, the three markers are in the field of view of the navigation device. However, due to the doctor's head movement, marker 1 is out of sight. Then the navigation device can determine that the marker detection result is the presence of the first marker.

[0074] Optionally, during the surgical navigation process, the navigation device may periodically detect all markers in the workspace where the navigation device is located, and obtain marker detection results. For example, the navigation device determines a marker detection result per second.

[0075] Optionally, the above S201, determining the marker detection result obtained after detecting all markers in the workspace where the navigation device is located, can be implemented in the following manner:

[0076] The marker detection result is determined according to preset parameters; the preset parameters include at least one of the image data acquired by the navigation device in the current field of view, the feature information corresponding to all the markers, and the global posture information of the navigation device.

[0077] The image data in the current field of view may be acquired by a visual sensor of the navigation device. For example, the processing unit of the navigation device is connected to the visual sensor in communication, and then the visual sensor may periodically acquire the image data in the current field of view and send the image data to the processing unit of the navigation device. Exemplarily, the visual sensor may acquire at least one frame of image data in the current field of view per second, and send the image data to the processing unit of the navigation device in real time, so that the processing unit of the navigation device may also acquire the image data in the current field of view per second.

[0078] Furthermore, the navigation device may detect the image data acquired in the current field of view using a preset algorithm or recognition model to determine the corresponding marker detection result. The preset algorithm may include but is not limited to at least one of an inter-frame difference method, a background modeling method, a point detection method, an image segmentation method, a cluster analysis method, and a motion vector field method. The recognition model may be a convolutional neural network (CNN), a recurrent neural network (RNN), or at least one of other deep learning networks and machine learning networks.

[0079] The characteristic information corresponding to all markers may include the characteristic information corresponding to each marker, which may be information stored in advance in the navigation device, or may be information determined by the navigation device based on image data corresponding to when all markers are not out of sight. The characteristic information includes but is not limited to at least one of a special pattern on the marker, a QR code mark, and complete outer contour information. The characteristic information of different markers may be the same or different.

[0080] Optionally, the navigation device searches for feature information corresponding to all markers from the image data acquired in the current field of view to perform feature matching. If the navigation device matches the feature information in the image data in the current field of view, it is determined that the marker detection result is that the first marker does not exist; if the navigation device does not match the feature information in the image data in the current field of view, it is determined that the marker detection result is that the first marker exists.

[0081] The global position information of the navigation device can be obtained by the odometer of the navigation device. The global position information of the navigation device is used to indicate the position of the navigation device in the global coordinate system. The global coordinate system can be a real world coordinate system.

[0082] Optionally, the navigation device may calculate the amount of change in the global pose information of the navigation device when the global pose information of the navigation device changes, and determine the marker detection result based on the amount of change in the pose. For example, when the amount of change in the pose is greater than a first preset amount of change, the marker detection result is determined to be the presence of a first marker. The first preset amount of change may be a parameter set in the navigation device in advance according to actual conditions.

[0083] In some embodiments, in order to improve the accuracy of the marker detection result determined by the navigation device, the navigation device may determine result 1 using the global position information of the navigation device, and determine result 2 using the image data acquired in the current field of view and the feature information corresponding to the marker, and determine the final marker detection result in combination with result 1 and result 2. For example, if both result 1 and result 2 indicate that the first marker exists, the marker detection result is determined to be the presence of the first marker; otherwise, the marker detection result is determined to be the absence of the first marker.

[0084] Since the preset parameters include at least one of the image data acquired by the navigation device in the current field of view, the feature information corresponding to all markers, and the global posture information of the navigation device, the navigation device can flexibly and accurately determine the marker detection result according to the preset parameters.

[0085] Optionally, in one embodiment, the above-mentioned “determining the marker detection result according to the preset parameters” can be implemented as follows:

[0086] If it is determined that the navigation device is moving based on the global position information of the navigation device, the marker detection result is determined based on preset parameters.

[0087] That is, when the navigation device can obtain its own global position information, the navigation device can determine whether it moves according to its own global position information. Optionally, the navigation device can determine that it moves when the change in its own global position information is greater than a preset change within a certain period of time.

[0088] If the navigation device determines that it is moving based on the global position information, the navigation device then determines the marker detection result based on the preset parameters. Figure 1 , the navigation device can determine the marker detection result when it detects its own movement, that is, the movement of the doctor's head. In this way, the marker detection result is triggered by the movement of the doctor's head. On the one hand, it can improve the accuracy of the marker detection result. On the other hand, the navigation device does not need to frequently determine the marker detection result, which avoids the resource consumption and occupation of the navigation device and improves the navigation efficiency.

[0089] S202: If the marker detection result shows that a first marker exists, determine a target parameter of the navigation device; the first marker is a marker that is out of the field of view of the navigation device.

[0090] In this embodiment, when the marker detection result is that the first marker exists, the navigation device determines the target parameter. The target parameter of the navigation device refers to a parameter that can adjust the field of view of the navigation device. In one embodiment, optionally, the target parameter includes at least one of an angle adjustment parameter and a field of view adjustment parameter.

[0091] The angle adjustment parameter is used to adjust the posture of the visual sensor in the navigation device. For example, the angle adjustment parameter is used to adjust at least one of the pitch angle, roll angle, and heading angle of the visual sensor.

[0092] The field of view adjustment parameter is used to adjust the field of view of the visual sensor in the navigation device. For example, the field of view adjustment parameter is used to adjust at least one of the resolution, focal length, and viewing angle of the visual sensor.

[0093] Optionally, the navigation device may determine the angle adjustment parameters according to its own movement when the marker detection result indicates that the first marker exists. For example, if the navigation device determines that it has moved 10 degrees to the left horizontally and 20 degrees upward horizontally, the angle adjustment parameters may be determined to be 15 degrees to the right horizontally and 20 degrees downward horizontally.

[0094] S203: Adjust the navigation device according to the target parameters of the navigation device so that all the markers enter the field of view of the navigation device.

[0095] In this embodiment, after the target parameters are determined, the navigation device can be adjusted according to the target parameters. For example, if the target parameters indicate 15 degrees to the right horizontally and 20 degrees to the downward horizontally, the navigation device will first adjust itself to the right by 15 degrees and then adjust it downward by 20 degrees. In some embodiments, the navigation device may also only adjust the visual sensor to the right by 15 degrees and adjust it downward by 20 degrees.

[0096] Optionally, a servo mechanism may be provided on the visual sensor of the navigation device, and after determining the target parameter, the processing unit of the navigation device sends an adjustment instruction to the servo mechanism, so that the servo mechanism adjusts the visual sensor according to the target parameter, thereby adjusting the navigation device according to the target parameter. The servo mechanism includes but is not limited to a pan / tilt, a slide rail, a gear, and a motor. Of course, in some embodiments, the processing unit of the navigation device may also directly send an adjustment instruction to the visual sensor to adjust the navigation device according to the target parameter.

[0097] Figure 3 FIG. 1 is a schematic diagram of adjusting a navigation device according to an angle adjustment parameter in an embodiment of the present application. Figure 3 As shown in (a), it is assumed that markers 2 and 3 are out of sight after the doctor's head moves. In this case, the marker detection result is that the first marker exists, and the navigation device determines the angle adjustment parameters as a heading angle of +15 degrees and a pitch angle of -20 degrees.

[0098] like Figure 3 As shown in (b), the navigation device controls the servo mechanism on the visual sensor to adjust horizontally 15 degrees to the right according to the heading angle of +15 degrees. After the adjustment, the marker 3 returns to the field of view of the navigation device.

[0099] Then, if Figure 3 As shown in (c), the navigation device controls the servo mechanism on the visual sensor to adjust horizontally downward by 20 degrees according to the pitch angle of -20 degrees. After the adjustment, the marker 2 returns to the field of view of the navigation device.

[0100] That is to say, when the doctor's head movement drives the navigation device to move, the navigation device moves horizontally to the left by 10 degrees, causing marker 3 to be out of sight, and moves horizontally upward by 20 degrees, causing marker 2 to be out of sight. The navigation device can adjust the visual sensor horizontally to the right by 15 degrees to return marker 3 to the field of view of the navigation device, and adjust the visual sensor horizontally downward by 20 degrees to return marker 2 to the field of view of the navigation device.

[0101] It can be understood that, in the above, the heading angle is adjusted first and then the pitch angle is adjusted as an example, and this embodiment does not limit the adjustment sequence.

[0102] Figure 4 This is a schematic diagram of adjusting a navigation device according to a field of view adjustment parameter in an embodiment of the present application. Taking the field of view adjustment parameter used to adjust the resolution of a visual sensor as an example, different resolutions correspond to different field of view sizes of the visual sensor.

[0103] like Figure 4As shown in the figure, when the resolution of the visual sensor is 120W pixels, the doctor's head movement causes markers 2 and 3 to be out of sight. If the navigation device adjusts the resolution of the visual sensor to 300W pixels, marker 3 can return to the field of view of the navigation device after the adjustment; if the navigation device adjusts the resolution of the visual sensor to 500W pixels, markers 3 and 2 can return to the field of view of the navigation device after the adjustment. Therefore, the field of view adjustment parameter can be used to adaptively adjust the field of view of the navigation device, thereby improving navigation efficiency and marker tracking performance.

[0104] Optionally, if the first marker exists, the navigation device may determine the field of view adjustment parameter according to the current field of view parameter of the visual sensor. It is understandable that the field of view adjustment parameter is to expand the field of view of the visual sensor in the navigation device. Therefore, the navigation device will adjust the current field of view parameter upward to determine the field of view adjustment parameter. The field of view parameter may include but is not limited to at least one of the resolution, focal length, and field of view angle of the visual sensor. For example, in combination with Figure 4 If marker 2 is out of sight and the current resolution of the visual sensor is 300W pixels, the navigation device can determine the field of view adjustment parameter to be 500W. If both marker 2 and marker 3 are out of sight and the current resolution of the visual sensor is 120W pixels, the navigation device determines the field of view adjustment parameter to be 300W or 500W.

[0105] In this way, after adjusting the navigation device according to the target parameters, all markers can be brought into the field of view of the navigation device.

[0106] The marker tracking method provided in this embodiment determines the marker detection result obtained after detecting all markers in the workspace where the navigation device is located. Since the marker detection result is that there is a first marker that is out of the field of view of the navigation device, the target parameters of the navigation device can be automatically determined, and the navigation device can be adjusted according to the target parameters. Therefore, after adjusting the navigation device according to the target parameters, all markers in the workspace where the navigation device is located can be returned to the field of view of the navigation device, thereby achieving the tracking of the markers. Taking the navigation device as a head-mounted device as an example, the field of view of the navigation device can better adapt to the doctor's head movement. Even if the marker is out of the field of view of the navigation device due to head movement, the navigation device can track the marker in time, thereby improving the navigation efficiency of the navigation device.

[0107] Figure 5 This is a flowchart of adjusting a navigation device in an embodiment of the present application, referring to Figure 5This embodiment relates to an optional implementation method of how to adjust the navigation device. Based on the above embodiment, the target parameter includes a first parameter and a second parameter, the first parameter and the second parameter are respectively one of the angle adjustment parameter and the field of view adjustment parameter, and the first parameter and the second parameter are different; the above S203, adjusting the navigation device according to the target parameter of the navigation device, includes the following steps:

[0108] S501: If the first parameter is within a first preset parameter adjustment range corresponding to a visual sensor in the navigation device, the navigation device is adjusted according to the first parameter, and after the navigation device is adjusted according to the first parameter, a marker detection result is updated.

[0109] In this embodiment, the first preset parameter adjustment range refers to the maximum adjustable range of the navigation device. If the first parameter is an angle adjustment parameter, the first preset parameter adjustment range is the adjustable angle range of the navigation device. If the first parameter is a field of view adjustment parameter, the first preset parameter adjustment range is the adjustable field of view range of the navigation device.

[0110] For example, taking the first parameter as the angle adjustment parameter, the upper and lower limits of the pitch angle of the visual sensor are [-50 degrees, 40 degrees], that is, the pitch angle of the visual sensor is up to 50 degrees downward from the horizontal, and the elevation angle is up to 40 degrees upward from the horizontal. Therefore, when the angle adjustment parameter is used to adjust the pitch angle of the visual sensor, it must be within the first preset parameter adjustment range [-50 degrees, 40 degrees].

[0111] Optionally, the navigation device can determine whether the angle adjustment parameter is within the first preset parameter adjustment range based on the current angle and angle adjustment parameter of the visual sensor. For example, the pitch angle of the current visual sensor is 20 degrees, and the pitch angle can be adjusted upward by a maximum of 20 degrees at this moment, but the determined angle adjustment parameter requires the pitch angle to be adjusted upward by 30 degrees, which exceeds the first preset parameter adjustment range.

[0112] Taking the first parameter as the field of view adjustment parameter as an example, the minimum resolution of the visual sensor is 120W and the maximum is 500W. When the field of view adjustment parameter is used to adjust the resolution of the visual sensor, it should be in the first preset parameter adjustment range {120W, 300W, 500W}. In other words, the resolution of the visual sensor needs to be one of 120W, 300W or 500W.

[0113] Similarly, the navigation device can determine whether the field of view adjustment parameter is within the first preset parameter adjustment range based on the current field of view parameter and the field of view adjustment parameter of the visual sensor. For example, if the current resolution of the visual sensor is 500W and the resolution still needs to be adjusted upward, the field of view adjustment parameter exceeds the first preset parameter adjustment range.

[0114] Further, taking the first parameter as the angle adjustment parameter and the second parameter as the field of view adjustment parameter as an example, the navigation device may first determine the angle adjustment parameter, and if the angle adjustment parameter is within the first preset parameter adjustment range, the navigation device adjusts the navigation device according to the first parameter, and after adjusting the navigation device according to the first parameter, updates the marker detection result. The process of updating the marker detection result can refer to S201, which will not be described in detail here.

[0115] Exemplarily, if the angle adjustment parameter indicates an upward adjustment of the pitch angle by 20 degrees, and the angle adjustment parameter is within the first preset parameter adjustment range, the processing component of the navigation device can control the servo mechanism on the visual sensor to adjust the pitch angle upward by 20 degrees, and update the marker detection result after the adjustment.

[0116] S502: If the updated marker detection result is that the first marker exists, adjust the navigation device according to the second parameter, or issue a prompt message.

[0117] In this embodiment, if the updated marker detection result shows that the first marker exists, that is, after adjusting the navigation device using the angle adjustment parameter, there is still an out-of-sight marker, the navigation device can change the adjustment method or directly issue a prompt message.

[0118] The prompt information is used to indicate that the marker of visual loss still exists. The prompt information can be any one or more of a voice broadcast, a buzzer alarm, a vibration prompt, a message prompt, and a display prompt, and this embodiment does not limit it, as long as the outside can know that the updated marker detection result is the presence of the first marker.

[0119] It should be noted that the above description is made by taking the first parameter as the angle adjustment parameter and the second parameter as the field of view adjustment parameter as an example. Alternatively, the first parameter may be the field of view adjustment parameter and the second parameter may be the angle adjustment parameter. For example, the navigation device may first determine the field of view adjustment parameter. If the field of view adjustment parameter is within the first preset parameter adjustment range corresponding to the visual sensor in the navigation device, the navigation device is adjusted according to the field of view adjustment parameter, and after the navigation device is adjusted according to the field of view adjustment parameter, the marker detection result is updated. If the updated marker detection result indicates that the first marker exists, the navigation device is adjusted according to the angle adjustment parameter, or a prompt message is issued.

[0120] In this embodiment, if the first parameter is within the first preset parameter adjustment range corresponding to the visual sensor in the navigation device, the navigation device is adjusted according to the first parameter, and after adjusting the navigation device according to the first parameter, the marker detection result is updated, and if the updated marker detection result is that the first marker exists, the navigation device is adjusted according to the second parameter, or a prompt message is issued. Since the target parameter includes the first parameter and the second parameter, the first parameter and the second parameter are respectively one of the angle adjustment parameter and the field of view adjustment parameter, and the first parameter and the second parameter are different. Therefore, this embodiment can flexibly adjust the navigation device in combination with the angle adjustment parameter and the field of view adjustment parameter to timely track the marker and improve the navigation efficiency of the navigation device.

[0121] In one embodiment, optionally, the above-mentioned marker tracking method further includes the following steps:

[0122] If the first parameter is not within the first preset parameter adjustment range, the navigation device is adjusted according to the second parameter, or a prompt message is issued.

[0123] In this embodiment, taking the first parameter as the angle adjustment parameter and the second parameter as the field of view adjustment parameter as an example, if the navigation device first determines the angle adjustment parameter, but the angle adjustment parameter is not within the first preset parameter adjustment range, the navigation device will continue to adjust the navigation device according to the field of view adjustment parameter, or the navigation device will issue a prompt message.

[0124] Taking the first parameter as the field of view adjustment parameter and the second parameter as the angle adjustment parameter as an example, if the navigation device first determines the field of view adjustment parameter, but the field of view adjustment parameter is not within the first preset parameter adjustment range, the navigation device will continue to adjust the navigation device according to the angle adjustment parameter, or the navigation device will issue a prompt message.

[0125] In this embodiment, when the first parameter is not within the first preset parameter adjustment range, the navigation device is adjusted according to the second parameter, or a prompt message is issued. That is to say, if the navigation device cannot be adjusted using one adjustment method, the navigation device will promptly switch to another adjustment method or notify the doctor in a timely manner, thereby improving the flexibility of the adjustment process.

[0126] Figure 6 This is another flowchart of adjusting the navigation device in the embodiment of the present application, referring to Figure 6 This embodiment relates to an optional implementation method of how to adjust the navigation device. Based on the above embodiment, the above "adjusting the navigation device according to the second parameter" includes the following steps:

[0127] S601: If the second parameter is within a second preset parameter adjustment range corresponding to the visual sensor in the navigation device, the navigation device is adjusted according to the second parameter, and after the navigation device is adjusted according to the second parameter, the marker detection result is updated.

[0128] S602: If the updated marker detection result is that the first marker exists, a prompt message is issued.

[0129] In this embodiment, the replacement adjustment method also needs to meet the maximum adjustable range. Similarly, the second preset parameter adjustment range also refers to the maximum adjustable range of the navigation device. If the second parameter is an angle adjustment parameter, the second preset parameter adjustment range is the range of angles that the navigation device can adjust. If the second parameter is a field of view adjustment parameter, the second preset parameter adjustment range is the range of fields of view that the navigation device can adjust.

[0130] Continuing with the example that the first parameter is the angle adjustment parameter and the second parameter is the field of view adjustment parameter, if the updated marker detection result shows that the first marker exists, that is, after the navigation device is adjusted using the angle adjustment parameter, there is still a marker that is out of sight, the navigation device will determine the field of view adjustment parameter. If the field of view adjustment parameter is within the second preset parameter adjustment range, the navigation device is adjusted according to the field of view adjustment parameter, and after the navigation device is adjusted according to the field of view adjustment parameter, the marker detection result is updated.

[0131] The first parameter is the field of view adjustment parameter, and the second parameter is the angle adjustment parameter. The principles are the same and will not be repeated here.

[0132] In this embodiment, when the second parameter is within the second preset parameter adjustment range corresponding to the visual sensor in the navigation device, the navigation device is continuously adjusted according to the second parameter, so the navigation device can be adjusted in combination with the two adjustment methods, thereby improving the success rate of the marker returning to the field of view of the navigation device. Furthermore, after the navigation device is adjusted according to the second parameter, the marker detection result is updated, and a prompt message is issued when the updated marker detection result indicates that the first marker exists. Therefore, even if the marker cannot be returned to the field of view of the navigation device, the doctor will be reminded in time, thereby improving navigation efficiency.

[0133] In one embodiment, optionally, the above-mentioned marker tracking method further includes the following steps:

[0134] If the second parameter is not within the second preset parameter adjustment range, a prompt message is issued.

[0135] In this embodiment, when the second parameter is not within the second preset parameter adjustment range, a prompt message is issued, that is, if the navigation device changes an adjustment method and the adjustment method cannot adjust the navigation device, the navigation device will promptly notify the doctor to improve the navigation efficiency of the navigation device.

[0136] In order to more clearly introduce the process of adjusting the navigation device according to the target parameters of the navigation device, Figure 7 illustrate. Figure 7 is a schematic diagram of an adjustment process in an embodiment of the present application, such as Figure 7 As shown, in one embodiment, the navigation device first determines the marker detection result. If the marker detection result is that the first marker does not exist, that is, there is no out-of-sight marker, the navigation device can end this adjustment and wait for the next marker detection result.

[0137] If the marker detection result is that the first marker exists, that is, there is an out-of-view marker, the navigation device first determines a first parameter, and determines whether the first parameter is within a first preset parameter adjustment range.

[0138] If the first parameter is not within the first preset parameter adjustment range, the navigation device may issue a prompt message or change the adjustment method. Changing the adjustment method means executing Figure 7 The process in the dashed box.

[0139] If the first parameter is within the first preset parameter adjustment range, the navigation device adjusts the navigation device using the first parameter and updates the marker detection result after the adjustment. If the updated marker detection result is that the first marker does not exist, the navigation device can end this adjustment and wait for the next marker detection result. If the updated marker detection result is that the first marker exists, the navigation device can issue a prompt message or change the adjustment method.

[0140] During the process of changing the adjustment mode, the navigation device will determine the second parameter, and determine whether the second parameter is within the second preset parameter adjustment range.

[0141] If the second parameter is not within the second preset parameter adjustment range, the navigation device may issue a prompt message.

[0142] If the second parameter is within the second preset parameter adjustment range, the navigation device adjusts the navigation device using the second parameter and updates the marker detection result after the adjustment. If the updated marker detection result is that the first marker does not exist, the navigation device can end this adjustment and wait for the next marker detection result. If the updated marker detection result is that the first marker exists, the navigation device can issue a prompt message.

[0143] In the above process, after the navigation device issues a prompt message, the current adjustment can be ended and the next marker detection result can be waited for. In this way, the navigation device can be adjusted flexibly and efficiently.

[0144] The following describes the process of determining the angle adjustment parameters. Figure 8 This is a schematic diagram of a process for determining angle adjustment parameters in an embodiment of the present application, referring to Figure 8 This embodiment relates to an optional implementation method of how to determine the angle adjustment parameter. Based on the above embodiment, if the target parameter includes the angle adjustment parameter, the "determining the target parameter of the navigation device" in S202 above includes the following steps:

[0145] S801, determining the global pose information corresponding to all markers.

[0146] In this embodiment, before starting navigation, the navigation device creates a workspace. After creating the workspace, the navigation device can determine the global pose information corresponding to all markers, that is, determine the global pose information of each marker relative to the navigation device in the global coordinate system when all markers are not out of sight.

[0147] Optionally, the navigation device may utilize the visual sensor and odometer therein to collect environmental information to create a workspace. Optionally, the navigation device may also create a workspace based on Simultaneous Localization And Mapping (SLAM) of a depth camera.

[0148] In some embodiments, the navigation device may also use the structured light camera therein to create a workspace. The structured light camera can not only collect the global pose information corresponding to all markers, but also collect the surface three-dimensional information of other objects in the workspace. Among them, other objects in the workspace are, for example, the patient's surgical site, the operating table, surgical instruments, etc., and the surface three-dimensional information can be in the form of point cloud or Mesh (three-dimensional model format).

[0149] Fig. 9 is a schematic diagram of a workspace in an embodiment of the present application, such as Fig. 9 As shown in (a), assuming that there are markers 1 and 2 in the workspace of the navigation device, the doctor wears the navigation device and adjusts the navigation device so that markers 1 and 2 enter the field of view of the navigation device. Then, the navigation device can obtain the global pose information 1 of marker 1 and the global pose information 2 of marker 2. In other words, at the beginning, the navigation device can know how many markers are in the workspace and where each marker is located.

[0150] S802: Determine angle adjustment parameters according to global pose information corresponding to all markers and image data acquired by the navigation device in the current field of view.

[0151] After starting navigation, the navigation device will obtain image data in the current field of view. Optionally, the navigation device can periodically obtain image data in the current field of view through a visual sensor therein. For example, if the visual sensor in the navigation device is an infrared camera, the infrared camera can obtain at least one frame of image data in the current field of view per second and send the image data to a processing component of the navigation device.

[0152] Furthermore, the navigation device can determine the out-of-view markers based on the image data acquired in the current field of view and the global pose information corresponding to all the markers. Fig. 9 As shown in (b), after the doctor's head moves, the navigation device can only obtain the global posture information 2 of the marker 1 at this time based on the image data obtained in the current field of view, which means that the marker 2 has been out of sight.

[0153] Furthermore, the navigation device can determine the angle adjustment parameters according to the global posture information corresponding to all markers and the image data obtained in the current field of view.

[0154] Continue with Fig. 9 For example, the navigation device can be based on Fig. 9 (a) the global position information 1 of the marker 1 relative to the navigation device, and Fig. 9 (b) The global posture information 1* of marker 1 relative to the navigation device is used to determine the posture change of marker 1, and the posture change can reflect the movement of the navigation device. Furthermore, the navigation device can determine the angle adjustment parameter based on the posture change of marker 1. For example, the navigation device determines that the navigation device has moved 10 degrees upward horizontally based on the posture change of marker 1, and the angle movement parameter can be set to move 10 degrees downward horizontally. In this way, after adjusting the navigation device according to the angle adjustment parameter, marker 2 can return to the field of view of the navigation device.

[0155] Since the present embodiment can determine the global pose information corresponding to all markers, the angle adjustment parameter can be determined according to the global pose information and the image data acquired by the navigation device in the current field of view. Since the angle adjustment parameter is determined based on the global pose information and the image data acquired by the navigation device in the current field of view, after adjusting the navigation device according to the angle adjustment parameter, all markers can enter the field of view of the navigation device.

[0156] Fig.10 This is another flow chart of determining the angle adjustment parameters in the embodiment of the present application, referring to Fig.10 This embodiment relates to an optional implementation method of how to determine the angle adjustment parameter. Based on the above embodiment, the above S802, according to the global posture information and the image data acquired by the navigation device in the current field of view, determines the angle adjustment parameter, including the following steps:

[0157] S1001, determining a second marker and third relative position information between the second marker and the navigation device based on image data acquired by the navigation device in the current field of view; the second marker is a marker within the field of view of the navigation device.

[0158] In this embodiment, the navigation device can determine the second marker based on the image data acquired in the current field of view. The second marker is a marker in the field of view of the navigation device. Optionally, the navigation device can determine the second marker in the image data in the current field of view by contour recognition, feature matching, deep learning algorithm, etc.

[0159] Furthermore, the navigation device can determine the third relative position information between the second marker and the navigation device based on the image data acquired in the current field of view. It should be noted that the third relative position information is the global position information of the second marker relative to the navigation device in the global coordinate system determined by the navigation device based on the image data in the current field of view.

[0160] Please continue to refer to Fig. 9 ,according to Fig. 9 (b) When the image data is obtained, the navigation device can determine that marker 1 is the second marker, and determine the global posture information of marker 1 in this case, that is, determine the third relative position information between marker 1 and the navigation device.

[0161] S1002: Determine fourth relative position information between the first marker and the second marker based on global pose information corresponding to all markers.

[0162] In this embodiment, the navigation device can determine the fourth relative position information between the first marker and the second marker according to the global pose information corresponding to all markers determined when the workspace is created. In other words, the navigation device can determine the relative position information between the marker that is not out of sight and the marker that has been out of sight when both are not out of sight.

[0163] Continue to refer Fig. 9 (a) When the workspace is first created, marker 1 and marker 2 are both within the field of view of the navigation device. Since the global pose information 1 of marker 1 and the global pose information 2 of marker 2 are known, the navigation device can determine the fourth relative position information between marker 2 and marker 1 in this case.

[0164] The above example uses two markers. In a scene with at least three markers, if there are multiple second markers, the navigation device can determine the fourth relative position information of the first marker and at least one second marker. For example, assuming that markers 1 to 3 are initially in the field of view of the navigation device, and then marker 1 is out of sight, the navigation device can determine the fourth relative position information of marker 1 and marker 2 and the fourth relative position information of marker 1 and marker 3, or it can only determine the fourth relative position information of marker 1 and marker 2 or the fourth relative position information of marker 1 and marker 3, and this embodiment does not impose any limitation.

[0165] S1003: Determine an angle adjustment parameter according to the third relative position information and the fourth relative position information.

[0166] Further, after determining the third relative position information and the fourth relative position information, the navigation device may determine the angle adjustment parameter according to the third relative position information and the fourth relative position information.

[0167] by Fig. 9 For example, Fig. 9 In the scenario shown in (b), the navigation device can determine the third relative position information of the marker 1 relative to the navigation device when the marker 2 is out of sight; Fig. 9 In the scenario shown in (a), the navigation device can determine the fourth relative position information between marker 2 and marker 1 when both marker 1 and marker 2 are not out of sight. Further, based on the third relative position information and the fourth relative position information, the angle that the navigation device needs to rotate when marker 2 returns to the field of view, that is, the angle adjustment parameter, can be determined.

[0168] by Figure 3 For example, marker 2 and marker 3 are out of sight due to the doctor's head movement. For marker 3, the navigation device Figure 3 (b) the third relative position information of the marker 1 and Figure 3 (a) The fourth relative position information between marker 3 and marker 1 can determine the heading angle that the navigation device needs to rotate to track marker 3 again. Figure 3 (c) the third relative position information of the marker 1, and Figure 3 The fourth relative position information between marker 2 and marker 1 at (a) can determine the pitch angle that the navigation device needs to rotate to re-track marker 2. In this way, the visual sensor uses the servo mechanism on it to adjust based on the determined heading angle and pitch angle, and the field of view of the navigation device can re-cover all markers.

[0169] Optionally, the navigation device may store in advance the correspondence between its own posture change and the angle adjustment parameters. Then, after the navigation device calculates the posture change of the navigation device based on the third relative position information and the fourth relative position information, the angle adjustment parameters may be determined using the posture change of the navigation device and the above correspondence.

[0170] In this embodiment, since the second marker is a marker within the field of view of the navigation device, the third relative position information of the second marker and the second marker and the navigation device is determined based on the image data acquired by the navigation device in the current field of view, and the fourth relative position information between the first marker and the second marker is determined based on the global posture information of all markers. Then, the angle adjustment parameters can be accurately and efficiently determined based on the third relative position information and the fourth relative position information.

[0171] Fig.11 This is another flow chart of determining the angle adjustment parameters in the embodiment of the present application, referring to Fig.11 This embodiment relates to an optional implementation method of how to determine the angle adjustment parameter. Based on the above embodiment, the above "determining the target parameter of the navigation device" includes the following steps:

[0172] S1101, determining a second posture change amount of the navigation device according to the global posture information of the navigation device.

[0173] In this embodiment, optionally, the navigation device can periodically obtain its own global posture information, and calculate the second posture change amount between the currently obtained global posture information and the global posture information obtained at the previous time point. Exemplarily, it is assumed that the navigation device can obtain its own global posture information every second, and calculate the second posture change amount between two adjacent seconds.

[0174] In some embodiments, the period for the navigation device to obtain global posture information and the period for calculating the second posture change amount may also be different. For example, the navigation device may obtain its own global posture information every second and calculate the second posture change amount every 2 seconds.

[0175] It can be understood that the second posture change can reflect the movement of the navigation device, that is, the movement of the doctor's head.

[0176] S1102: Determine an angle adjustment parameter according to the second posture change.

[0177] In this embodiment, after the navigation device determines the second posture change amount, it can determine the angle adjustment parameter according to the second posture change amount. Optionally, when the second posture change amount is greater than a preset value, the navigation device can determine the angle amount corresponding to the second posture change amount according to the conversion relationship between the posture change amount and the angle amount, and determine the angle adjustment parameter according to the angle amount.

[0178] Exemplarily, after the navigation device determines the second posture change, it determines that the angle corresponding to the second posture change is horizontally negative 15 degrees, that is, the navigation device changes from the initial position to the current position after being rotated 15 degrees horizontally to the left, and the posture of the navigation device also changes by the second posture change in the process. Further, the navigation device takes a negative value for the horizontal angle of negative 15 degrees to obtain an angle adjustment parameter, that is, the angle adjustment parameter is used to instruct the navigation device to rotate 15 degrees horizontally to the right.

[0179] This embodiment can determine the second posture change of the navigation device according to the global posture information of the navigation device, and determine the angle adjustment parameter according to the second posture change without using image data, thereby improving the efficiency of determining the angle adjustment parameter.

[0180] In one embodiment, optionally, if the target parameter includes an angle adjustment parameter, the above-mentioned “determining the target parameter of the navigation device” can also be implemented in the following manner:

[0181] The angle adjustment parameters are determined according to the image data acquired by the navigation device in the current field of view, the global pose information corresponding to all markers and the global pose information of the navigation device.

[0182] In this embodiment, another process of determining the angle adjustment parameter by combining the above two methods is also provided. That is, the navigation device determines the angle adjustment parameter according to the image data acquired by the navigation device in the current field of view, the global pose information corresponding to all markers, and the global pose information of the navigation device.

[0183] Optionally, the navigation device can determine the second marker and the third relative position information between the second marker and the navigation device based on image data acquired by the navigation device in the current field of view, and determine the fourth relative position information between the first marker and the second marker based on the global posture information, so as to determine the first candidate angle adjustment parameter based on the third relative position information and the fourth relative position information.

[0184] Furthermore, the navigation device determines a second posture change amount of the navigation device according to the global posture information of the navigation device, and determines a second angle adjustment parameter according to the second posture change amount.

[0185] After that, the navigation device can determine the angle adjustment parameter according to the first candidate angle adjustment parameter and the second candidate angle adjustment parameter. Optionally, the navigation device can use the second candidate angle parameter to correct the first candidate angle adjustment parameter, and use the corrected result as the final angle adjustment parameter. The navigation device can also perform weighted calculation or average calculation on the first candidate angle adjustment parameter and the second candidate angle parameter to obtain the final angle adjustment parameter.

[0186] This embodiment can determine the angle adjustment parameters based on the image data acquired by the navigation device in the current field of view, the global posture information corresponding to all markers and the global posture information of the navigation device, thereby improving the accuracy of the angle adjustment parameters.

[0187] In some application scenarios, the navigation device may not be able to adjust the parameters according to the angle and / or field of view so that all markers enter the field of view of the navigation device. In this case, the navigation device can perform global spatial perception, determine the location of the out-of-view markers, and promptly alert the doctor.

[0188] Fig.12 This is a schematic diagram of a process for sending a prompt message in an embodiment of the present application, referring to Fig.12 This embodiment relates to an optional implementation method of how to issue a prompt message. Based on the above embodiment, the above "issuing a prompt message" includes the following steps:

[0189] S1201, determining first relative position information of a first marker relative to the navigation device at the first moment based on image data acquired by the navigation device at the first moment; wherein the first moment is the moment before the first marker leaves the field of view of the navigation device.

[0190] In this embodiment, the first moment is the moment before the first marker leaves the field of view of the navigation device. That is, it is necessary to obtain the first relative position information between the marker that has left the field of view and the navigation device before leaving the field of view.

[0191] Optionally, the navigation device may determine the previous moment of the moment when the marker detection result corresponds to the presence of the first marker as the first moment, and determine the first relative position information of the first marker at the first moment corresponding to the first moment before it leaves the field of view of the navigation device based on the image data at the first moment.

[0192] Exemplarily, the navigation device acquires image data of the current field of view every second. Assuming that at the 5th second, the navigation device determines that there is a first marker, the first relative position information of the first marker can be determined based on the image data of the current field of view at the 4th second.

[0193] Optionally, the navigation device may determine the first relative position information based on the image data at the first moment, the global position information of the navigation device at the first moment, and the fixed position relationship between the navigation device and the visual sensor. In other words, the navigation device may determine the relative relationship between the first marker and the visual sensor based on the image data at the first moment, and determine the first relative position information based on the fixed position relationship between the navigation device and the visual sensor and the global position information of the navigation device at the first moment.

[0194] S1202: Determine a first pose change amount of the navigation device according to the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment.

[0195] In this embodiment, the navigation device also needs to determine its first position change amount between the first moment and the second moment, wherein the second moment refers to the current moment.

[0196] Optionally, the navigation device can periodically acquire its own global position information, so that it can determine its own global position information at a first moment and its global position information at a second moment.

[0197] Continuing with the above example, assuming that the current moment is the 5th second and the first moment is the 4th second, the navigation device can determine the first pose change based on the global pose information at the 5th second and the global pose information at the 4th second.

[0198] S1203: Determine second relative position information of the first marker relative to the navigation device at a second moment according to the first posture change amount and the first relative position information.

[0199] Fig.13 A schematic diagram of a global space perception process in an embodiment of the present application, Fig.13 The black thick solid line in the figure indicates the field of view of the navigation device. Fig.13 The coordinate system in represents the global coordinate system where the navigation device is located. Fig.13 (a) shows the situation where the marker 1 is within the field of view of the navigation device.

[0200] Assuming that at the 5th second, the doctor's head moves, and the position of the navigation device produces the first position change T1, Fig.13 (a) will follow Fig.13 (c) changes and eventually becomes Fig.13 (d) The situation shown in Figure 1 results in the loss of marker 1.

[0201] Fig.13 (b) shows the situation of the marker 1 at the first moment, that is, at the 4th second. Fig.13 (b) and Fig.13 As shown in (c), at the last moment before the marker 1 is out of sight, the first relative position information between the marker 1 and the navigation device is P1.

[0202] Please combine Fig.13 (a)~ Fig.13 (c) The second relative position information of the first marker relative to the navigation device can be determined based on the first relative position information and the second posture change.

[0203] Optionally, the navigation device can determine the second relative position information of the first marker relative to the navigation device according to the product of the first relative position information and the first pose change. Exemplarily, the first relative position information is recorded as P1, the first pose change is recorded as T1, and the second relative position information is recorded as P2, then P2 = P1*T1.

[0204] S1204: Issue a prompt message according to the second relative position information.

[0205] In this embodiment, since the second relative position information can reflect the relative position relationship between the navigation device and the first marker, the navigation device can issue a prompt information according to the second relative position information. The prompt information is any one or more of a voice broadcast, a buzzer alarm, a vibration prompt, a message prompt, and a display prompt.

[0206] Fig.14 is a schematic diagram of a prompt information in an embodiment of the present application, such as Fig.14 As shown, a head-mounted navigation device is used as an example to issue prompt information in the form of "near-eye display + speaker + stereo vibration".

[0207] When the marker 2 is out of sight and the marker 2 cannot be returned to the field of view of the navigation device using angle adjustment parameters and / or field of view adjustment parameters, the navigation device can acquire the second relative position information of the marker 2 according to S1201 to S1204.

[0208] Assuming that the marker 2 is determined to be located behind the doctor right according to the second relative position information of the marker 2, the navigation device can display the marker 2 to be located behind the doctor right in the form of an indicating arrow on its near-eye display screen, and the navigation device uses a speaker to prompt a voice message "marker 2 has left the field of vision and is located behind the doctor right". At the same time, the vibration point of the stereo vibration array located behind the doctor right in the navigation device will vibrate to prompt the doctor to turn his head to the right rear direction to adjust the field of vision.

[0209] Based on this, the navigation device can dynamically indicate the position of the out-of-view marker based on the holographic imaging technology of the near-eye display. After receiving the above prompt information, the doctor can adjust the head direction according to the prompt until the marker 2 returns to the field of view of the navigation device.

[0210] In some embodiments, the stereo vibration array may be replaced by a surround speaker array, and after the navigation device determines the second relative position information, a prompt message may be issued by the speaker in the corresponding direction to prompt the doctor to adjust the head direction. This embodiment is not limited to this.

[0211] This embodiment determines the first relative position information of the first marker relative to the navigation device at the first moment based on the image data acquired by the navigation device at the first moment, and determines the first posture change of the navigation device based on the global posture information of the navigation device at the current second moment and the global posture information of the navigation device at the first moment, and then determines the second relative position information of the first marker relative to the navigation device at the second moment based on the first posture change and the first relative position information, thereby issuing a prompt message based on the second relative position information. In other words, even if the marker is out of sight, the navigation device can perform multi-information fusion to complete global space perception, and then determine its position relative to itself, and promptly remind the doctor to improve navigation efficiency.

[0212] In order to more clearly introduce the tracking method of the marker in this application, Fig.15 illustrate. Fig.15 FIG. 1 is a schematic diagram of a method for tracking a marker in an embodiment of the present application. Fig.15 As shown, the doctor wears the navigation device and adjusts the field of view of the navigation device to adapt to different doctors' heights and different surgical working states, such as sitting and standing. After that, the navigation device can create a workspace and enter the navigation process to assist the doctor in completing the operation. Furthermore, during the navigation process, the navigation device can execute the marker tracking method according to the following process.

[0213] S1501, obtaining global position information of the navigation device.

[0214] S1502, determine whether the navigation device moves. If the navigation device moves, execute S1503; if the navigation device does not move, the navigation device or the doctor can determine whether the navigation is completed, and end the navigation if it is completed to complete the entire operation. If it is not completed, continue the navigation process.

[0215] S1503, determining the marker detection result according to the preset parameters.

[0216] S1504, determining whether the marker detection result is that the first marker exists. If the first marker exists, executing S1505; if the first marker does not exist, returning to executing S1501 to continue motion detection of the navigation device, that is, monitoring the doctor's head movement.

[0217] S1505, determining the global pose information corresponding to all markers.

[0218] S1506: Determine the second marker and third relative position information between the second marker and the navigation device according to the image data acquired by the navigation device in the current field of view.

[0219] S1507: Determine fourth relative position information between the first marker and the second marker based on the global pose information corresponding to all the markers.

[0220] S1508: Determine an angle adjustment parameter according to the third relative position information and the fourth relative position information.

[0221] S1509, determining whether the angle adjustment parameter is within the first preset parameter adjustment range. If the angle adjustment parameter is within the first preset parameter adjustment range, executing S1510; if the angle adjustment parameter is not within the first preset parameter adjustment range, executing S1513.

[0222] S1510, adjusting the navigation device according to the angle adjustment parameter.

[0223] S1511, update the marker detection result.

[0224] S1512, determining whether the updated marker detection result shows that the first marker exists. If the updated marker detection result still shows that the first marker exists, executing S1513; if the updated marker detection result does not show that the first marker exists, returning to executing S1501.

[0225] S1513, determine the field of view adjustment parameters.

[0226] S1514, determining whether the field of view adjustment parameter is within the second preset parameter adjustment range. If the field of view adjustment parameter is within the second preset parameter adjustment range, executing S1515; if the field of view adjustment parameter is not within the second preset parameter adjustment range, executing S1518.

[0227] S1515, adjusting the navigation device according to the field of view adjustment parameter.

[0228] S1516, update the marker detection result.

[0229] S1517, determining whether the updated marker detection result shows that the first marker exists. If the updated marker detection result still shows that the first marker exists, executing S1518; if the updated marker detection result does not show that the first marker exists, returning to executing S1501.

[0230] S1518: Determine first relative position information of the first marker relative to the navigation device at the first moment based on the image data acquired by the navigation device at the first moment.

[0231] S1519: Determine a first pose change amount of the navigation device according to the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment.

[0232] S1520: Determine second relative position information of the first marker relative to the navigation device at a second moment according to the first posture change amount and the first relative position information.

[0233] S1521, issuing a prompt message according to the second relative position information. The doctor can adjust the head direction based on the prompt message, and after the doctor makes the adjustment, the navigation device returns to step S1516 until all the markers are back in the field of view of the navigation device.

[0234] The processes of S1501 to S1521 may refer to the above-mentioned embodiment and will not be described in detail here.

[0235] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0236] Based on the same inventive concept, an embodiment of the present application also provides a marker tracking system. Fig.16 FIG. 1 is a schematic diagram of a tracking system for a marker in an embodiment of the present application. Fig.16 As shown, the marker tracking system 1600 includes a navigation device 1601 and at least one marker 1602 .

[0237] Among them, the navigation device 1601 is used to determine the marker detection result obtained after detecting each marker in the workspace where the navigation device 1601 is located. If the marker detection result is that there is a first marker, the target parameters of the navigation device 1601 are determined, and the navigation device 1601 is adjusted according to the target parameters of the navigation device 1601 so that each marker enters the field of view of the navigation device 1601.

[0238] Fig.17 This is a structural diagram of another marker tracking system in an embodiment of the present application. In one embodiment, optionally, the navigation device 1601 also includes a visual sensor 1701 and a servo mechanism 1702.

[0239] The servo device 1604 refers to a collection of mechanisms or components that can adjust parameters of the navigation device. The parameters that can be adjusted by the servo device 1604 include but are not limited to the angle, position, and focal length of the visual sensor on the navigation device.

[0240] Furthermore, the servo mechanism 1702 can be used to adjust the navigation device 1601 according to the target parameters of the navigation device 1601 so that each marker 1602 enters the field of view of the navigation device 1601, that is, each marker 1602 enters the field of view of the visual sensor 1701 on the navigation device.

[0241] Please continue to refer to Fig.17 In one embodiment, optionally, the navigation device 1601 also includes an odometer 1703 .

[0242] The odometer 1703 is used to obtain the global position information of the navigation device 1601. The odometer 1703 includes but is not limited to a gyroscope, an accelerometer, and a visual odometer camera.

[0243] Please continue to refer to Fig.17 In one embodiment, optionally, the marker tracking system 1600 further includes a prompt device 1704. The prompt device 1704 includes but is not limited to a speaker, an indicator light, a laser indicator light, and a stereo vibration array.

[0244] The navigation device 1601 is further used to determine the first relative position information of the first marker relative to the navigation device 1601 at the first moment according to the image data acquired by the navigation device 1601 at the first moment; determine the first position change of the navigation device 1601 according to the global position information of the navigation device 1601 at the current second moment and the global position information of the navigation device 1601 at the first moment; determine the second relative position information of the first marker relative to the navigation device 1601 at the second moment according to the first position change and the first relative position information. The first moment is the moment before the first marker leaves the field of vision of the navigation device 1601.

[0245] The prompt device 1704 is used to issue prompt information according to the second relative position information.

[0246] In one embodiment, the prompt device 1704 includes a stereoscopic vibration array 1705 and / or a display device 1706 disposed on the navigation device 1601. The stereoscopic vibration array 1705 includes a plurality of vibration points surrounding an area of ​​interest of the target object.

[0247] The target object's area of ​​interest is used to wear the navigation device 1601. Fig.13 As shown, taking the head-mounted device as an example, the navigation device is provided with multiple vibration points surrounding the doctor's head, and different vibration points indicate different orientations. After the navigation device determines the second relative position information, it can instruct the corresponding vibration point to vibrate based on the second relative position information. For example, if the second relative position information indicates that the first marker is located behind the right side of the navigation device, the vibration point set behind the right side of the doctor's head will vibrate.

[0248] In summary, the navigation device 1601 in the marker tracking system 1600 can execute the steps of any of the above-mentioned marker tracking methods.

[0249] The embodiment of the present application also provides a marker tracking device for implementing the marker tracking method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of the one or more marker tracking devices provided below can refer to the limitations of the marker tracking method above, and will not be repeated here.

[0250] Fig.18 is a structural block diagram of a tracking device for a marker in an embodiment of the present application, such as Fig.18 As shown, in an embodiment of the present application, a marker tracking device 1800 is provided, which is applied to a navigation device. The marker tracking device 1800 includes: a first determination module 1801, a second determination module 1802 and an adjustment module 1803, wherein:

[0251] The first determination module 1801 is used to determine a marker detection result obtained after detecting all markers in the working space where the navigation device is located.

[0252] The second determination module 1802 is used to determine the target parameter of the navigation device if the marker detection result is that a first marker exists; the first marker is a marker that is out of the field of view of the navigation device.

[0253] The adjustment module 1803 is used to adjust the navigation device according to the target parameters of the navigation device so that all the markers enter the field of view of the navigation device.

[0254] The marker tracking device provided in this embodiment can automatically determine the target parameters of the navigation device and adjust the navigation device according to the target parameters after determining the marker detection results obtained after detecting all markers in the workspace where the navigation device is located. Therefore, after adjusting the navigation device according to the target parameters, all markers in the workspace where the navigation device is located can be returned to the field of view of the navigation device, thereby achieving the tracking of the markers. Taking the navigation device as a head-mounted device as an example, the field of view of the navigation device can better adapt to the doctor's head movement. Even if the marker is out of the field of view of the navigation device due to head movement, the navigation device can track the marker in time, thereby improving the navigation efficiency of the navigation device.

[0255] Optionally, the target parameter includes at least one of an angle adjustment parameter and a field of view adjustment parameter; the angle adjustment parameter is used to adjust the posture of the visual sensor in the navigation device, and the field of view adjustment parameter is used to adjust the field of view of the visual sensor in the navigation device.

[0256] Optionally, the target parameter includes a first parameter and a second parameter, the first parameter and the second parameter are respectively one of an angle adjustment parameter and a field of view adjustment parameter, and the first parameter and the second parameter are different; the second determination module 1802 includes:

[0257] The first adjustment unit is used to adjust the navigation device according to the first parameter if the first parameter is within a first preset parameter adjustment range corresponding to the visual sensor in the navigation device, and update the marker detection result after adjusting the navigation device according to the first parameter.

[0258] The second adjustment unit is used to adjust the navigation device according to the second parameter, or issue a prompt message if the updated marker detection result is that the first marker exists.

[0259] Optionally, the second determining module 1802 further includes:

[0260] The third adjustment unit is used to adjust the navigation device according to the second parameter, or issue a prompt message if the first parameter is not within the first preset parameter adjustment range.

[0261] Optionally, the third adjustment unit includes:

[0262] The adjustment subunit is used to adjust the navigation device according to the second parameter if the second parameter is within a second preset parameter adjustment range corresponding to the visual sensor in the navigation device, and update the marker detection result after adjusting the navigation device according to the second parameter.

[0263] The first prompt subunit is used to issue a prompt message if the updated marker detection result is that the first marker exists.

[0264] Optionally, the third adjustment unit further includes:

[0265] The second prompt subunit is used to issue a prompt message if the second parameter is not within the second preset parameter adjustment range.

[0266] Optionally, if the target parameter includes an angle adjustment parameter, the second determining module 1802 includes:

[0267] The first determination unit is used to determine the global pose information corresponding to all markers.

[0268] The second determination unit is used to determine the angle adjustment parameter according to the global posture information and the image data acquired by the navigation device in the current field of view.

[0269] Optionally, the second determining unit includes:

[0270] The first determination subunit is used to determine the second marker and the third relative position information between the second marker and the navigation device according to the image data acquired by the navigation device in the current field of view; the second marker is a marker within the field of view of the navigation device.

[0271] The second determination subunit is used to determine the fourth relative position information between the first marker and the second marker according to the global posture information corresponding to all the markers.

[0272] The third determining subunit is used to determine the angle adjustment parameter according to the third relative position information and the fourth relative position information.

[0273] Optionally, if the target parameter includes an angle adjustment parameter, the second determining module 1802 includes:

[0274] The third determining unit is used to determine a second posture change amount of the navigation device according to the global posture information of the navigation device.

[0275] The fourth determination unit is used to determine the angle adjustment parameter according to the second posture change amount.

[0276] Optionally, if the target parameter includes an angle adjustment parameter, the second determining module 1802 includes:

[0277] The fifth determination unit is used to determine the angle adjustment parameter according to the image data acquired by the navigation device in the current field of view, the global posture information corresponding to all the markers and the global posture information of the navigation device.

[0278] Optionally, the first determining module 1801 includes:

[0279] The sixth determination unit is used to determine the marker detection result according to preset parameters; the preset parameters include at least one of the image data acquired by the navigation device in the current field of view, the feature information corresponding to all the markers, and the global posture information of the navigation device.

[0280] Optionally, the above-mentioned second adjustment unit, third adjustment unit, first prompt subunit and second prompt subunit are also used to determine the first relative position information of the first marker relative to the navigation device at the first moment based on the image data acquired by the navigation device at the first moment; determine the first posture change of the navigation device based on the global posture information of the navigation device at the current second moment and the global posture information of the navigation device at the first moment; determine the second relative position information of the first marker relative to the navigation device at the second moment based on the first posture change and the first relative position information; and issue a prompt information based on the second relative position information; wherein the first moment is the moment before the first marker leaves the field of view of the navigation device.

[0281] Optionally, the sixth determination unit is further configured to determine the marker detection result according to preset parameters if it is determined that the navigation device is moving according to the global posture information of the navigation device.

[0282] Each module in the above-mentioned marker tracking device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the navigation device in the form of hardware, or can be stored in the memory in the navigation device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0283] Fig.19 FIG. 1 is an internal structure diagram of a navigation device in an embodiment of the present application. In an embodiment of the present application, a navigation device is provided. The navigation device may be a server. The internal structure diagram thereof may be as shown in FIG. Fig.19 As shown. The navigation device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the navigation device is used to provide computing and control capabilities. The memory of the navigation device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the navigation device is used to store relevant data. The network interface of the navigation device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for tracking a marker is implemented.

[0284] Those skilled in the art will understand that Fig.19The structure shown in the figure is merely a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the navigation device to which the present application scheme is applied. The specific navigation device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0285] In one embodiment, a navigation device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0286] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0287] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0288] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0289] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0290] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for tracking a marker, characterized in that: Applied to a navigation device, the method comprises: Determining a marker detection result obtained after detecting all markers in the workspace where the navigation device is located; If the marker detection result is that a first marker exists, determining a target parameter of the navigation device; the first marker is a marker that is out of the field of view of the navigation device; The navigation device is adjusted according to the target parameters of the navigation device so that all the markers enter the field of view of the navigation device.

2. The method according to claim 1, characterized in that The target parameter includes at least one of an angle adjustment parameter and a field of view adjustment parameter; The angle adjustment parameter is used to adjust the posture of the visual sensor in the navigation device, and the field of view adjustment parameter is used to adjust the field of view of the visual sensor in the navigation device.

3. The method according to claim 1, characterized in that The target parameter includes a first parameter and a second parameter, the first parameter and the second parameter are respectively one of an angle adjustment parameter and a field of view adjustment parameter, and the first parameter and the second parameter are different; The adjusting the navigation device according to the target parameter of the navigation device comprises: If the first parameter is within a first preset parameter adjustment range corresponding to the visual sensor in the navigation device, adjusting the navigation device according to the first parameter, and updating the marker detection result after adjusting the navigation device according to the first parameter; If the updated marker detection result is that the first marker exists, the navigation device is adjusted according to the second parameter, or a prompt message is issued.

4. The method according to claim 3, characterized in that The method further comprises: If the first parameter is not within the first preset parameter adjustment range, the navigation device is adjusted according to the second parameter, or a prompt message is issued.

5. The method according to claim 3, characterized in that: The adjusting the navigation device according to the second parameter includes: If the second parameter is within a second preset parameter adjustment range corresponding to the visual sensor in the navigation device, adjusting the navigation device according to the second parameter, and updating the marker detection result after adjusting the navigation device according to the second parameter; If the updated marker detection result is that the first marker exists, a prompt message is issued.

6. The method according to claim 5, characterized in that The method further comprises: If the second parameter is not within the second preset parameter adjustment range, a prompt message is issued.

7. The method according to any one of claims 3 to 6, characterized in that: The sending of prompt information includes: Determine, according to the image data acquired by the navigation device at the first moment, first relative position information of the first marker relative to the navigation device at the first moment; wherein the first moment is the moment before the first marker leaves the field of view of the navigation device; Determine a first pose change amount of the navigation device according to the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment; Determine, according to the first posture change amount and the first relative position information, second relative position information of the first marker relative to the navigation device at the second moment; Prompt information is issued according to the second relative position information.

8. The method according to any one of claims 1 to 6, characterized in that: The determining of the marker detection result obtained after detecting all the markers in the working space where the navigation device is located includes: The marker detection result is determined according to preset parameters; the preset parameters include at least one of the image data acquired by the navigation device in the current field of view, the feature information corresponding to all the markers, and the global posture information of the navigation device.

9. The method according to claim 8, characterized in that Determining the marker detection result according to the preset parameters includes: If it is determined that the navigation device is moving according to the global position information of the navigation device, the marker detection result is determined according to the preset parameters.

10. A marker tracking system, characterized in that: The system includes a navigation device and at least one marker; The navigation device is used to determine the marker detection result obtained after detecting each marker in the working space where the navigation device is located. If the marker detection result is that a first marker exists, the target parameter of the navigation device is determined, and the navigation device is adjusted according to the target parameter of the navigation device so that each marker enters the field of view of the navigation device.

11. A tracking device for a marker, characterized in that: Applied to navigation equipment, the device comprises: A first determination module, used to determine a marker detection result obtained after detecting all markers in the working space where the navigation device is located; a second determination module, configured to determine a target parameter of the navigation device if the marker detection result indicates that a first marker exists; the first marker is a marker that is out of the field of view of the navigation device; The adjustment module is used to adjust the navigation device according to the target parameters of the navigation device so that all the markers enter the field of view of the navigation device.

12. A navigation device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images