Marker tracking method, system, device, navigation apparatus, and storage medium

Abstract

The application relates to a marker tracking method, system, device, navigation equipment and storage medium. The method comprises the following steps: determining a marker detection result obtained after detecting all markers in a working space where the navigation equipment is located; if the marker detection result is that a first marker is out of the field of view of the navigation equipment, determining a target parameter of the navigation equipment, and adjusting the navigation equipment according to the target parameter of the navigation equipment, so that all the markers enter the field of view of the navigation equipment. The method can track the markers.

Description

Technical Field

[0001] This application relates to the field of medical technology, and in particular to a method, system, apparatus, navigation device, and storage medium for tracking markers. Background Technology

[0002] Navigation technology is an emerging medical technology that allows doctors to use navigation devices to guide the surgical process and improve the accuracy of the surgery.

[0003] Taking head-mounted navigation devices as an example, navigation devices typically use visual sensors to track preset markers for navigation. If the doctor moves their head during surgery, the markers may move out of the field of view of the visual sensors, thereby reducing the navigation efficiency of the navigation device.

[0004] Therefore, how to track markers is a key research topic for those skilled in the art. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, system, apparatus, navigation device, and storage medium for tracking markers, which can track markers, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a marker tracking method applied to a navigation device, the method comprising:

[0007] The marker detection results are determined after detecting all markers within the workspace of the navigation equipment.

[0008] If the marker detection result indicates the presence of a first marker, then the target parameters of the navigation device are determined; the first marker is a marker that is out of the navigation device's field of view.

[0009] Adjust the navigation device according to the target parameters to bring all markers into the navigation device's field of view.

[0010] In one embodiment, the target parameters include at least one of angle adjustment parameters and field of view adjustment parameters;

[0011] Angle adjustment parameters are used to adjust the attitude of the visual sensor in the navigation device, while field of view adjustment parameters are used to adjust the field of view of the visual sensor in the navigation device.

[0012] In one embodiment, the target parameters include a first parameter and a second parameter, wherein the first parameter and the second parameter are respectively one of an angle adjustment parameter and a field of view adjustment parameter, and the first parameter and the second parameter are different; adjusting the navigation device according to the target parameters of the navigation device includes:

[0013] If the first parameter is within the adjustment range of the first preset parameter corresponding to the visual sensor in the navigation device, the navigation device is adjusted according to the first parameter, and the marker detection result is updated after the navigation device is adjusted according to the first parameter.

[0014] If the updated marker detection result indicates the presence of the first marker, then adjust the navigation device according to the second parameter, or issue a prompt message.

[0015] In one embodiment, the method further includes:

[0016] If the first parameter is not within the first preset parameter adjustment range, the navigation device is adjusted according to the second parameter, or a prompt message is issued.

[0017] In one embodiment, adjusting the navigation device according to the second parameter includes:

[0018] If the second parameter is within the adjustment range of the second preset parameter corresponding to the visual sensor in the navigation device, the navigation device is adjusted according to the second parameter, and the marker detection result is updated after the navigation device is adjusted according to the second parameter.

[0019] If the updated marker detection result indicates the presence of the first marker, a prompt message will be issued.

[0020] In one embodiment, the method further includes:

[0021] If the second parameter is not within the adjustment range of the second preset parameter, a prompt message will be issued.

[0022] In one embodiment, a prompt message is issued, including:

[0023] Based on the image data acquired by the navigation device at the first moment, the first relative position information of the first marker relative to the navigation device at the first moment is determined; wherein, the first moment is the moment before the first marker leaves the field of view of the navigation device;

[0024] Based on the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment, determine the first pose change of the navigation device;

[0025] Based on the first pose change and the first relative position information, determine the second relative position information of the first marker relative to the navigation device at the second time.

[0026] A prompt message is issued based on the second relative position information.

[0027] In one embodiment, determining the marker detection results obtained after detecting all markers within the workspace of the navigation device includes:

[0028] The marker detection results are determined based on preset parameters; the preset parameters include at least one of the following: image data acquired by the navigation device in the current field of view, feature information corresponding to all markers, and global pose information of the navigation device.

[0029] In one embodiment, determining the marker detection result based on preset parameters includes:

[0030] If the navigation device is determined to be moving based on its global pose information, then the marker detection result is determined based on preset parameters.

[0031] Secondly, this application also provides a marker tracking system, which includes a navigation device and at least one marker;

[0032] The navigation device is used to determine the marker detection results obtained after detecting each marker in the workspace where the navigation device is located. If the marker detection result indicates the presence of a first marker, the target parameters of the navigation device are determined, and the navigation device is adjusted according to the target parameters to bring each marker into the field of view of the navigation device.

[0033] Thirdly, this application also provides a marker tracking device for use in a navigation device, the device comprising:

[0034] The first determining module is used to determine the marker detection results obtained after detecting all markers in the workspace where the navigation device is located;

[0035] The second determining module is used to determine the target parameters of the navigation device if the marker detection result indicates the presence of a first marker; the first marker is a marker that is out of the field of view of the navigation device.

[0036] The adjustment module is used to adjust the navigation device according to the target parameters of the navigation device so that all markers are within the field of view of the navigation device.

[0037] Fourthly, this application also provides a navigation device. The navigation device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the steps described above.

[0038] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0039] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0040] The aforementioned marker tracking method, system, device, navigation equipment, and storage medium, after determining the marker detection results obtained by detecting all markers within the workspace of the navigation equipment, can automatically determine the target parameters of the navigation equipment when the detection result indicates the presence of a first marker that has moved out of the navigation equipment's field of view. The navigation equipment is then adjusted according to these target parameters. Therefore, after adjusting the navigation equipment according to the target parameters, all markers within the workspace of the navigation equipment can return to the navigation equipment's field of view, achieving marker tracking. Taking a head-mounted navigation equipment as an example, the field of view of the navigation equipment can better adapt to the doctor's head movements. Even if a marker moves out of the navigation equipment's field of view due to head movements, the navigation equipment can still track the marker in a timely manner, thereby improving the navigation efficiency of the navigation equipment. Attached Figure Description

[0041] Figure 1 This is an application environment diagram of a marker tracking method according to an embodiment of this application;

[0042] Figure 2 This is a flowchart illustrating a marker tracking method according to an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of a navigation device that adjusts according to angle adjustment parameters in an embodiment of this application;

[0044] Figure 4 This is a schematic diagram of a navigation device adjusting a field of view parameter according to an embodiment of this application;

[0045] Figure 5 This is a schematic diagram of a process for adjusting a navigation device according to an embodiment of this application;

[0046] Figure 6 This is a schematic diagram of another process for adjusting a navigation device in an embodiment of this application;

[0047] Figure 7 This is a schematic diagram of an adjustment process in an embodiment of this application;

[0048] Figure 8 This is a schematic diagram of a process for determining angle adjustment parameters in an embodiment of this application;

[0049] Figure 9 This is a schematic diagram of a workspace according to an embodiment of this application;

[0050] Figure 10 This is a schematic diagram of another process for determining angle adjustment parameters in an embodiment of this application;

[0051] Figure 11This is a schematic diagram of another process for determining angle adjustment parameters in an embodiment of this application;

[0052] Figure 12 This is a schematic diagram of a process for issuing a prompt message in an embodiment of this application;

[0053] Figure 13 This application provides a schematic diagram of a global spatial perception process according to an embodiment;

[0054] Figure 14 This is a schematic diagram of a prompt message in an embodiment of this application;

[0055] Figure 15 This is a schematic diagram illustrating the process of a marker tracking method in an embodiment of this application;

[0056] Figure 16 This is a schematic diagram of the structure of a marker tracking system according to an embodiment of this application;

[0057] Figure 17 This is a schematic diagram of the structure of another marker tracking system in the embodiments of this application;

[0058] Figure 18 This is a structural block diagram of a marker tracking device according to an embodiment of this application;

[0059] Figure 19 This is an internal structural diagram of a navigation device according to an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] Figure 1 This is an application environment diagram of a marker tracking method according to an embodiment of this application, such as... Figure 1 As shown, the doctor wears a navigation device 101 and uses the navigation device 101 to track at least one marker 102 in the workspace where the navigation device 101 is located in order to navigate the surgical procedure.

[0062] The navigation device 101 includes a vision sensor. Based on the number of cameras, the vision sensor can be at least one of a monocular field-of-view sensor, a binocular vision sensor, or a multi-view vision sensor; based on function, the vision sensor can be at least one of a laser vision sensor, an infrared camera, a depth camera, an RGB red-green-blue camera (RGB camera), or a structured light camera.

[0063] In some embodiments, the navigation device 101 may further include a display component, which may be at least one of a display, an augmented reality (AR) device, a virtual reality (VR) device, or a holographic projection device.

[0064] Understandably, the navigation device 101 may also include a processing unit, which may be a central processing unit (CPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices. The processing unit may also be a personal computer, laptop, smartphone, or tablet located externally to the navigation device. The processing unit may also be implemented by a standalone server or a server cluster consisting of multiple servers.

[0065] Please refer to Figure 1 (a) and Figure 1 (b) Taking a head-mounted navigation device as an example, during surgical navigation, the relative position between the navigation device 101 and the surgeon's head is fixed. If the surgeon's head moves, the shooting angle and position of the visual sensor in the navigation device will also change, which may cause the marker to fall out of the field of view of the visual sensor in the navigation device, thereby reducing navigation efficiency. The field of view of the visual sensor is as follows: Figure 1 The dashed line indicates that being out of the view of the navigation device can also be simply referred to as being out of sight.

[0066] Therefore, it is necessary to provide a marker tracking method capable of tracking markers. The marker tracking method will be introduced below.

[0067] Figure 2 This is a flowchart illustrating a marker tracking method according to an embodiment of this application. This method can be applied to... Figure 1 In one embodiment of the navigation device shown, such as Figure 2 As shown, it includes the following steps:

[0068] S201, Determine the marker detection results obtained after detecting all markers in the workspace where the navigation device is located.

[0069] Before using a navigation device for surgical navigation, the surgeon needs to wear the device and install at least one marker within the workspace where the device is located. This marker provides spatial reference for the navigation device and includes, but is not limited to, planar targets, QR code markers, reflective balls, and LEDs (light-emitting diodes). It should be noted that the "all markers" mentioned in step S201 and subsequent descriptions only include markers used to provide spatial reference for the navigation device, i.e., markers used for tracking by the navigation device; for example, if markers 1, 2, 3, and 4 are installed in the workspace, but only markers 1, 2, and 3 are used for tracking by the navigation device, then "all markers" only includes markers 1, 2, and 3.

[0070] Furthermore, the navigation device can then construct a workspace using the installed markers and navigate using the relative positions of the markers and the navigation device.

[0071] To ensure that a marker returns to the navigation device's field of view after it has fallen out of sight, marker tracking is necessary. Therefore, in this embodiment, the navigation device detects all markers within its workspace and obtains marker detection results. These results are used to detect whether any markers have fallen out of the navigation device's field of view.

[0072] For example, the marker detection result can be "the first marker exists" or "the first marker does not exist". Here, the first marker is a marker that has moved out of the navigation device's field of view. The navigation device's field of view can be understood as the field of view of the visual sensor within the navigation device. That is, if a marker is not in the view of the visual sensor, then the marker has moved out of the navigation device's field of view. It should be noted that "the first marker" only indicates that a marker has moved out of view, and does not specifically refer to which marker. For example, assuming that "all markers" includes marker 1, marker 2, and marker 3, if marker 1 is determined to be out of view at time 1, then the marker detection result at time 1 is "the first marker exists"; if marker 2 is determined to be out of view at time 2, then the marker detection result at time 2 is also "the first marker exists".

[0073] Assuming that the workspace where the navigation device is located originally contains marker 1, marker 2 and marker 3, and all three markers are initially within the field of view of the navigation device, but due to the doctor's head movement, marker 1 is out of sight, then the navigation device can determine that the marker detection result is that the first marker is present.

[0074] Optionally, the navigation device can periodically detect all markers within its workspace during surgical navigation and obtain marker detection results. For example, the navigation device determines one marker detection result per second.

[0075] Optionally, in step S201 above, determining the marker detection results obtained after detecting all markers within the workspace of the navigation device can be achieved in the following way:

[0076] The marker detection results are determined based on preset parameters; the preset parameters include at least one of the following: image data acquired by the navigation device in the current field of view, feature information corresponding to all markers, and global pose information of the navigation device.

[0077] The image data within the current field of view can be acquired by the navigation device's visual sensor. For example, the navigation device's processing unit is communicatively connected to the visual sensor, allowing the visual sensor to periodically acquire image data within its current field of view and send this image data to the navigation device's processing unit. Exemplarily, the visual sensor can acquire at least one frame of image data within the current field of view per second and send this image data to the navigation device's processing unit in real time, thus enabling the navigation device's processing unit to acquire image data within the current field of view per second.

[0078] Furthermore, the navigation device can utilize a preset algorithm or recognition model to detect image data acquired within the current field of view to determine the corresponding marker detection results. The preset algorithm may include, but is not limited to, at least one of the following: inter-frame differencing, background modeling, point detection, image segmentation, clustering analysis, and motion vector field methods. The recognition model may be a convolutional neural network (CNN), a recurrent neural network (RNN), or at least one of other deep learning networks and machine learning networks.

[0079] The feature information corresponding to all markers can include the feature information specific to each marker. This information can be pre-stored in the navigation device or determined by the navigation device based on image data when all markers are in sight. The feature information includes, but is not limited to, at least one of the following: special patterns on the markers, QR code markings, and complete outer contour information. The feature information of different markers can be the same or different.

[0080] Optionally, the navigation device searches for feature information corresponding to all markers in the image data acquired in the current field of view to perform feature matching. If the navigation device matches feature information in the image data in the current field of view, the marker detection result is determined to be that the first marker does not exist; if the navigation device does not match feature information in the image data in the current field of view, the marker detection result is determined to be that the first marker exists.

[0081] The global pose information of a navigation device can be obtained from its odometer. This information indicates the device's pose in a global coordinate system, which can be the real-world coordinate system.

[0082] Optionally, the navigation device can calculate the pose change of its global pose information when the global pose information of the navigation device changes, and determine the marker detection result based on the pose change. For example, if the pose change is greater than a first preset change, the marker detection result is determined to be the presence of a first marker. The first preset change can be a parameter pre-set in the navigation device according to actual conditions.

[0083] In some embodiments, to improve the accuracy of the marker detection results determined by the navigation device, the navigation device can use its global pose information to determine result 1, and use the image data acquired in the current field of view and the feature information corresponding to the marker to determine result 2. The final marker detection result is then determined by combining result 1 and result 2. For example, if both result 1 and result 2 indicate the presence of the first marker, the marker detection result is determined to indicate the presence of the first marker; otherwise, the marker detection result is determined to indicate the absence of the first marker.

[0084] Since the preset parameters include at least one of the image data acquired by the navigation device in the current field of view, the feature information corresponding to all markers, and the global pose information of the navigation device, the navigation device can flexibly and accurately determine the marker detection results based on the preset parameters.

[0085] Optionally, in one embodiment, the above-mentioned "determining the marker detection result according to preset parameters" can be achieved in the following way:

[0086] If the navigation device is determined to be moving based on its global pose information, then the marker detection result is determined based on preset parameters.

[0087] In other words, if a navigation device can obtain its own global pose information, it can determine whether it has moved based on this information. Optionally, the navigation device can determine that it has moved if the change in its global pose information exceeds a preset amount within a certain time period.

[0088] If the navigation device determines that it has moved based on global pose information, it then combines this information with preset parameters to determine the marker detection result. Figure 1 The navigation device can determine the marker detection result only after detecting its own movement, i.e., the doctor's head movement. Thus, triggering the acquisition of marker detection results by the doctor's head movement improves the accuracy of the marker detection results. Furthermore, the navigation device does not need to frequently determine the marker detection results, avoiding resource consumption and occupation, and improving navigation efficiency.

[0089] S202, if the marker detection result indicates the existence of a first marker, then the target parameters of the navigation device are determined; the first marker is a marker that is out of the field of view of the navigation device.

[0090] In this embodiment, if the marker detection result indicates the presence of a first marker, the navigation device determines target parameters. The target parameters of the navigation device refer to parameters that can adjust the field of view of the navigation device. In one embodiment, optionally, the target parameters include at least one of angle adjustment parameters and field of view adjustment parameters.

[0091] The angle adjustment parameters are used to adjust the attitude of the visual sensor in the navigation device. For example, the angle adjustment parameters are used to adjust at least one of the pitch angle, roll angle, and yaw angle of the visual sensor.

[0092] Field of view adjustment parameters are used to adjust the field of view of the visual sensor in a navigation device. For example, field of view adjustment parameters are used to adjust at least one of the visual sensor's resolution, focal length, and angle of view.

[0093] Optionally, the navigation device can determine the angle adjustment parameters based on its own movement if the marker detection result indicates the presence of a first marker. For example, if the navigation device determines that it has moved 10 degrees horizontally to the left and 20 degrees horizontally upward, it can determine the angle adjustment parameters as 15 degrees horizontally to the right and 20 degrees horizontally downward.

[0094] S203, Adjust the navigation device according to the target parameters of the navigation device so that all markers are within the field of view of the navigation device.

[0095] In this embodiment, after the target parameters are determined, the navigation device can be adjusted according to the target parameters. For example, if the target parameters indicate 15 degrees to the right and 20 degrees downward, the navigation device will first adjust itself 15 degrees to the right and then 20 degrees downward. In some embodiments, the navigation device may also adjust only the visual sensor 15 degrees to the right and 20 degrees downward.

[0096] Optionally, a servo mechanism can be installed on the vision sensor of the navigation device. After determining the target parameters, the processing unit of the navigation device sends an adjustment command to the servo mechanism, which then adjusts the vision sensor according to the target parameters, thereby adjusting the navigation device based on the target parameters. The servo mechanism includes, but is not limited to, a gimbal, a slide rail, gears, and a motor. Of course, in some embodiments, the processing unit of the navigation device can also directly send adjustment commands to the vision sensor to adjust the navigation device according to the target parameters.

[0097] Figure 3 This is a schematic diagram of a navigation device adjusting parameters based on angle, as described in an embodiment of this application. Figure 3 As shown in (a), assuming that markers 2 and 3 are out of sight after the doctor moves his head, the marker detection result is that the first marker exists. The navigation device determines the angle adjustment parameters as heading angle +15 degrees and pitch angle -20 degrees.

[0098] like Figure 3 As shown in (b), the navigation device controls the servo mechanism on the visual sensor to adjust horizontally to the right by 15 degrees based on the heading angle +15 degrees. After the adjustment, marker 3 returns to the field of view of the navigation device.

[0099] Next, as Figure 3 As shown in (c), the navigation device controls the servo mechanism on the visual sensor to adjust horizontally downward by 20 degrees based on the pitch angle of -20 degrees. After the adjustment, marker 2 returns to the field of view of the navigation device.

[0100] In other words, during the process of the doctor's head movement causing the navigation device to move, if the navigation device moves 10 degrees to the left horizontally, causing marker 3 to be out of sight, and moves 20 degrees to the top horizontally, causing marker 2 to be out of sight, then the navigation device can adjust the visual sensor 15 degrees to the right horizontally to bring marker 3 back into the navigation device's field of view, and adjust the visual sensor 20 degrees to the bottom horizontally to bring marker 2 back into the navigation device's field of view.

[0101] It is understandable that the above example, which takes adjusting the heading angle first and then the pitch angle, does not limit the order of adjustment in this embodiment.

[0102] Figure 4 This is a schematic diagram of a navigation device adjusted according to field of view adjustment parameters in an embodiment of this application. Taking the field of view adjustment parameters as an example of adjusting the resolution of the visual sensor, different resolutions correspond to different field of view sizes of the visual sensor.

[0103] like Figure 4As shown, when the resolution of the visual sensor is 1.2 million pixels, the doctor's head movement causes markers 2 and 3 to fall out of sight. If the navigation device adjusts the resolution of the visual sensor to 3 million pixels, marker 3 can return to the navigation device's field of view. If the navigation device adjusts the resolution of the visual sensor to 5 million pixels, both markers 2 and 3 can return to the navigation device's field of view. Therefore, by using the field of view adjustment parameter, the size of the navigation device's field of view can be adaptively adjusted, improving navigation efficiency and marker tracking performance.

[0104] Optionally, if a first marker exists, the navigation device can determine the field-of-view adjustment parameters based on the current field-of-view parameters of the visual sensor. It is understood that the field-of-view adjustment parameters are intended to expand the field of view of the visual sensor in the navigation device; therefore, the navigation device will adjust the current field-of-view parameters upwards to determine the field-of-view adjustment parameters. Field-of-view parameters may include, but are not limited to, at least one of the following: the resolution, focal length, and field of view angle of the visual sensor. For example, combined with... Figure 4 If marker 2 is out of sight, and the current visual sensor resolution is 300W pixels, the navigation device can determine the field of view adjustment parameter to be 500W. If both marker 2 and marker 3 are out of sight, and the current visual sensor resolution is 120W pixels, the navigation device can determine the field of view adjustment parameter to be either 300W or 500W.

[0105] In this way, after adjusting the navigation device according to the target parameters, all markers can be brought into the field of view of the navigation device.

[0106] The marker tracking method provided in this embodiment, after determining the marker detection results obtained by detecting all markers in the workspace of the navigation device, automatically determines the target parameters of the navigation device when the detection result indicates that a first marker has left the field of view of the navigation device. The navigation device is then adjusted according to these target parameters. Therefore, after adjusting the navigation device according to the target parameters, all markers in the workspace of the navigation device can return to the field of view of the navigation device, achieving marker tracking. Taking a head-mounted navigation device as an example, the field of view of the navigation device can better adapt to the doctor's head movements. Even if a marker leaves the field of view due to head movements, the navigation device can still track the marker in a timely manner, thereby improving the navigation efficiency of the navigation device.

[0107] Figure 5 This is a schematic diagram of a process for adjusting a navigation device according to an embodiment of this application. (Refer to...) Figure 5This embodiment relates to an optional implementation of how to adjust a navigation device. Based on the above embodiment, the target parameters include a first parameter and a second parameter, which are respectively one of an angle adjustment parameter and a field-of-view adjustment parameter, and the first parameter and the second parameter are different; S203 above, adjusting the navigation device according to the target parameters of the navigation device, includes the following steps:

[0108] S501, if the first parameter is within the adjustment range of the first preset parameter corresponding to the visual sensor in the navigation device, then the navigation device is adjusted according to the first parameter, and after adjusting the navigation device according to the first parameter, the marker detection result is updated.

[0109] In this embodiment, the first preset parameter adjustment range refers to the maximum range that the navigation device can adjust. If the first parameter is an angle adjustment parameter, then the first preset parameter adjustment range is the range of angles that the navigation device can adjust. If the first parameter is a field of view adjustment parameter, then the first preset parameter adjustment range is the range of the field of view that the navigation device can adjust.

[0110] For example, taking the first parameter as the angle adjustment parameter, the upper and lower limits of the pitch angle of the vision sensor are [-50 degrees, 40 degrees]. That is, the maximum pitch angle of the vision sensor is 50 degrees downward and the maximum elevation angle is 40 degrees upward. Therefore, when the angle adjustment parameter is used to adjust the pitch angle of the vision sensor, it must be within the adjustment range of the first preset parameter [-50 degrees, 40 degrees].

[0111] Optionally, the navigation device can determine whether the angle adjustment parameter is within the first preset parameter adjustment range based on the current angle of the visual sensor and the angle adjustment parameter. For example, if the current pitch angle of the visual sensor is 20 degrees, the pitch angle can be adjusted upwards by a maximum of 20 degrees. However, if the determined angle adjustment parameter requires the pitch angle to be adjusted upwards by 30 degrees, then it exceeds the first preset parameter adjustment range.

[0112] Taking the field of view adjustment parameter as an example, the minimum resolution of the visual sensor is 120W and the maximum is 500W. Therefore, when the field of view adjustment parameter is used to adjust the resolution of the visual sensor, it must be within the adjustment range of the first preset parameter {120W, 300W, 500W}. In other words, the resolution of the visual sensor needs to be one of 120W, 300W, or 500W.

[0113] Similarly, the navigation device can determine whether the field of view adjustment parameter is within the first preset parameter adjustment range based on the current field of view parameters and field of view adjustment parameters of the visual sensor. For example, if the current resolution of the visual sensor is 500W, and the resolution still needs to be adjusted upward, then the field of view adjustment parameter exceeds the first preset parameter adjustment range.

[0114] Furthermore, taking the first parameter as the angle adjustment parameter and the second parameter as the field of view adjustment parameter as an example, the navigation device can first determine the angle adjustment parameter. If the angle adjustment parameter is within the adjustment range of the first preset parameter, the navigation device will then adjust itself according to the first parameter, and after adjusting itself according to the first parameter, the marker detection result will be updated. The process of updating the marker detection result can be referred to in S201, and will not be elaborated here.

[0115] For example, if the angle adjustment parameter indicates an upward adjustment of the pitch angle by 20 degrees, and the angle adjustment parameter is within the first preset parameter adjustment range, the processing unit of the navigation device can control the servo mechanism on the visual sensor to adjust the pitch angle upward by 20 degrees, and update the marker detection result after the adjustment.

[0116] S502, if the updated marker detection result indicates the presence of the first marker, then adjust the navigation device according to the second parameter, or issue a prompt message.

[0117] In this embodiment, if the updated marker detection result indicates the presence of the first marker, meaning that even after adjusting the navigation device using angle adjustment parameters, there is still a marker out of sight, the navigation device can either change the adjustment method or directly issue a prompt message.

[0118] The prompt message is used to indicate that there is still an out-of-sight marker. The prompt message can be any one or more of the following: voice broadcast, buzzer alarm, vibration prompt, message prompt, and display prompt. This embodiment is not limited to any of these, as long as it enables external parties to know that the updated marker detection result shows the presence of the first marker.

[0119] It should be noted that the above explanation uses the first parameter as the angle adjustment parameter and the second parameter as the field of view adjustment parameter as an example. Alternatively, the first parameter can be the field of view adjustment parameter and the second parameter can be the angle adjustment parameter. For example, the navigation device can first determine the field of view adjustment parameter. If the field of view adjustment parameter is within the first preset parameter adjustment range corresponding to the visual sensor in the navigation device, the navigation device is adjusted according to the field of view adjustment parameter. After adjusting the navigation device according to the field of view adjustment parameter, the marker detection result is updated. If the updated marker detection result indicates the presence of the first marker, the navigation device is adjusted according to the angle adjustment parameter, or a prompt message is issued.

[0120] In this embodiment, if the first parameter is within the adjustment range of the first preset parameter corresponding to the visual sensor in the navigation device, the navigation device is adjusted according to the first parameter. After adjusting the navigation device according to the first parameter, the marker detection result is updated. If the updated marker detection result indicates the presence of the first marker, the navigation device is adjusted according to the second parameter, or a prompt message is issued. Since the target parameter includes the first parameter and the second parameter, which are one of the angle adjustment parameter and the field of view adjustment parameter, respectively, and the first parameter and the second parameter are different, this embodiment can flexibly combine the angle adjustment parameter and the field of view adjustment parameter to adjust the navigation device, track markers in a timely manner, and improve the navigation efficiency of the navigation device.

[0121] In one embodiment, optionally, the above-described marker tracking method further includes the following steps:

[0122] If the first parameter is not within the first preset parameter adjustment range, the navigation device is adjusted according to the second parameter, or a prompt message is issued.

[0123] In this embodiment, taking the first parameter as the angle adjustment parameter and the second parameter as the field of view adjustment parameter as an example, if the navigation device first determines the angle adjustment parameter, but the angle adjustment parameter is not within the adjustment range of the first preset parameter, the navigation device will continue to adjust the navigation device according to the field of view adjustment parameter, or the navigation device will issue a prompt message.

[0124] Taking the first parameter as the field of view adjustment parameter and the second parameter as the angle adjustment parameter as an example, if the navigation device first determines the field of view adjustment parameter, but the field of view adjustment parameter is not within the adjustment range of the first preset parameter, the navigation device will continue to adjust the navigation device according to the angle adjustment parameter, or the navigation device will issue a prompt message.

[0125] In this embodiment, when the first parameter is not within the adjustment range of the first preset parameter, the navigation device is adjusted according to the second parameter, or a prompt message is issued. In other words, if the navigation device cannot be adjusted using one adjustment method, the navigation device will promptly switch to another adjustment method or promptly notify the doctor, thus improving the flexibility of the adjustment process.

[0126] Figure 6 This is a schematic diagram of another process for adjusting a navigation device in an embodiment of this application, referred to... Figure 6 This embodiment relates to an optional implementation of how to adjust a navigation device. Based on the above embodiment, the aforementioned "adjusting the navigation device according to the second parameter" includes the following steps:

[0127] S601, if the second parameter is within the adjustment range of the second preset parameter corresponding to the visual sensor in the navigation device, then adjust the navigation device according to the second parameter, and update the marker detection result after adjusting the navigation device according to the second parameter.

[0128] S602, if the updated marker detection result indicates the presence of the first marker, a prompt message is issued.

[0129] In this embodiment, changing the adjustment method also needs to meet its maximum adjustable range. Similarly, the adjustment range of the second preset parameter also refers to the maximum adjustable range of the navigation device. If the second parameter is an angle adjustment parameter, then the adjustment range of the second preset parameter is the range of angles that the navigation device can adjust. If the second parameter is a field-of-view adjustment parameter, then the adjustment range of the second preset parameter is the range of the field of view that the navigation device can adjust.

[0130] Continuing with the example of using the first parameter as the angle adjustment parameter and the second parameter as the field of view adjustment parameter, if the updated marker detection result shows the presence of the first marker (meaning that even after adjusting the navigation device using the angle adjustment parameter, a marker still exists outside the field of view), the navigation device will determine the field of view adjustment parameter. If the field of view adjustment parameter is within the adjustment range of the second preset parameter, the navigation device will be adjusted according to the field of view adjustment parameter, and the marker detection result will be updated after adjusting the navigation device according to the field of view adjustment parameter.

[0131] The first parameter is for adjusting the field of view, and the second parameter is for adjusting the angle. The principle is the same, so it will not be repeated here.

[0132] In this embodiment, because the navigation device continues to adjust according to the second parameter when the second parameter is within the adjustment range of the second preset parameter corresponding to the visual sensor in the navigation device, it can combine two adjustment methods to adjust the navigation device, thus improving the success rate of the marker returning to the navigation device's field of view. Furthermore, after adjusting the navigation device according to the second parameter, the marker detection result is updated, and a prompt message is issued if the updated marker detection result shows the presence of the first marker. Therefore, even if the marker cannot return to the navigation device's field of view, it will promptly remind the doctor, improving navigation efficiency.

[0133] In one embodiment, optionally, the above-described marker tracking method further includes the following steps:

[0134] If the second parameter is not within the adjustment range of the second preset parameter, a prompt message will be issued.

[0135] In this embodiment, a prompt message is issued when the second parameter is not within the adjustment range of the second preset parameter. That is, if the navigation device changes to a different adjustment method and this method cannot adjust the navigation device, the navigation device will promptly notify the doctor to improve the navigation efficiency.

[0136] To more clearly illustrate the process of adjusting the navigation device according to its target parameters, this section combines... Figure 7 illustrate. Figure 7 This is a schematic diagram of an adjustment process in an embodiment of this application, such as... Figure 7 As shown, in one embodiment, the navigation device first determines the marker detection result. If the marker detection result is that there is no first marker, that is, there is no out-of-view marker, the navigation device can end the current adjustment and wait for the next marker detection result.

[0137] If the marker detection result indicates the presence of a first marker, i.e., an off-look marker, the navigation device first determines the first parameter and then determines whether the first parameter is within the first preset parameter adjustment range.

[0138] If the first parameter is not within the first preset parameter adjustment range, the navigation device can issue a prompt message or change the adjustment method. Changing the adjustment method means executing... Figure 7 The process is shown in the dashed box.

[0139] If the first parameter is within the first preset parameter adjustment range, the navigation device adjusts itself using the first parameter and updates the marker detection result after adjustment. If the updated marker detection result indicates that the first marker does not exist, the navigation device can end the adjustment and wait for the next marker detection result. If the updated marker detection result indicates that the first marker exists, the navigation device can issue a prompt or change the adjustment method.

[0140] During the process of changing the adjustment method, the navigation device will determine the second parameter and whether the second parameter is within the second preset parameter adjustment range.

[0141] If the second parameter is not within the adjustment range of the second preset parameter, the navigation device can issue a prompt message.

[0142] If the second parameter is within the adjustment range of the second preset parameter, the navigation device adjusts itself using the second parameter and updates the marker detection result after adjustment. If the updated marker detection result indicates that the first marker does not exist, the navigation device can end the adjustment and wait for the next marker detection result. If the updated marker detection result indicates that the first marker exists, the navigation device can issue a prompt message.

[0143] During the above process, once the navigation device issues a prompt, the adjustment can be terminated, and the system can wait for the next marker detection result. This allows for flexible and efficient adjustments to the navigation device.

[0144] The following describes the process of determining the angle adjustment parameters. Figure 8 This is a flowchart illustrating the process of determining angle adjustment parameters in an embodiment of this application. (Refer to...) Figure 8 This embodiment relates to an optional implementation of how to determine angle adjustment parameters. Based on the above embodiment, if the target parameters include angle adjustment parameters, then "determining the target parameters of the navigation device" in S202 above includes the following steps:

[0145] S801, determine the global pose information corresponding to all markers.

[0146] In this embodiment, the navigation device creates a workspace before navigation begins. After creating the workspace, the navigation device can determine the global pose information corresponding to all markers, that is, determine the global pose information of each marker relative to the navigation device in the global coordinate system when all markers are in sight.

[0147] Optionally, the navigation device can use its visual sensors and odometry to collect environmental information to create a workspace. Alternatively, the navigation device can also create a workspace based on Simultaneous Localization and Mapping (SLAM) using a depth camera.

[0148] In some embodiments, the navigation device can also utilize its structured light camera to create a workspace. The structured light camera can not only acquire global pose information corresponding to all markers, but also acquire surface 3D information of other objects in the workspace. These other objects in the workspace may include, for example, the patient's surgical site, the operating table, and surgical instruments; the surface 3D information can be in point cloud or mesh (3D model format) form.

[0149] Figure 9 This is a schematic diagram of a workspace according to an embodiment of this application, such as... Figure 9 As shown in (a), assuming there are markers 1 and 2 in the workspace of the navigation device, after the doctor wears the navigation device and adjusts it so that markers 1 and 2 enter the field of view, the navigation device can obtain the global pose information 1 of marker 1 and the global pose information 2 of marker 2. That is, at the beginning, the navigation device can know how many markers are in its workspace and the location of each marker.

[0150] S802 determines the angle adjustment parameters based on the global pose information corresponding to all markers and the image data acquired by the navigation device in the current field of view.

[0151] After starting navigation, the navigation device acquires image data within its current field of view. Optionally, the navigation device can periodically acquire image data within its current field of view using its visual sensor. Taking an infrared camera as an example, the infrared camera can acquire at least one frame of image data within the current field of view per second and send this image data to the processing unit of the navigation device.

[0152] Furthermore, navigation devices can determine out-of-sight markers based on image data acquired within their current field of view and the global pose information corresponding to all markers. For example... Figure 9 As shown in (b), after the doctor's head moves, the navigation device can only obtain the global pose information 2 of marker 1 at this time based on the image data it acquires in the current field of view, which means that marker 2 has been out of sight.

[0153] Furthermore, navigation devices can determine angle adjustment parameters based on the global pose information corresponding to all markers and the image data acquired in the current field of view.

[0154] Continue with Figure 9 For example, navigation devices can be based on Figure 9 (a) Global pose information 1 of time marker 1 relative to the navigation device, and Figure 9 (b) Based on the global pose information 1* of marker 1 relative to the navigation device, the pose change of marker 1 is determined, which reflects the movement of the navigation device. Then, based on the pose change of marker 1, the navigation device can determine the angle adjustment parameters. For example, if the navigation device determines that it has moved 10 degrees horizontally upwards based on the pose change of marker 1, the angle motion parameter can be set to move 10 degrees horizontally downwards. After adjusting the navigation device according to the angle adjustment parameters, marker 2 can return to the navigation device's field of view.

[0155] In this embodiment, since the global pose information corresponding to all markers can be determined, the angle adjustment parameters can be determined based on the global pose information and the image data acquired by the navigation device in the current field of view. Because the angle adjustment parameters are determined based on the global pose information and the image data acquired by the navigation device in the current field of view, adjusting the navigation device according to the angle adjustment parameters will allow all markers to enter the navigation device's field of view.

[0156] Figure 10 This is a flowchart illustrating another method for determining angle adjustment parameters in an embodiment of this application. (Refer to...) Figure 10 This embodiment relates to an optional implementation of how to determine angle adjustment parameters. Based on the above embodiment, step S802, which determines the angle adjustment parameters according to global pose information and image data acquired by the navigation device in the current field of view, includes the following steps:

[0157] S1001, based on the image data acquired by the navigation device in the current field of view, determine the second marker and the third relative position information between the second marker and the navigation device; the second marker is a marker within the field of view of the navigation device.

[0158] In this embodiment, the navigation device can determine the second marker based on image data acquired within its current field of view. The second marker is a marker located within the navigation device's field of view. Optionally, the navigation device can determine the second marker in the image data within its current field of view using methods such as contour recognition, feature matching, or deep learning algorithms.

[0159] Furthermore, the navigation device can determine the third relative position information between the second marker and the navigation device based on the image data acquired within its current field of view. It should be noted that the third relative position information is also the global pose information of the second marker relative to the navigation device in the global coordinate system, determined by the navigation device based on the image data within its current field of view.

[0160] Please continue to refer to this. Figure 9 ,according to Figure 9 (b) The image data obtained at this time allows the navigation device to determine that marker 1 is the second marker and to determine the global pose information of marker 1 in this case, that is, to determine the third relative position information between marker 1 and the navigation device.

[0161] S1002, based on the global pose information corresponding to all markers, determine the fourth relative position information between the first marker and the second marker.

[0162] In this embodiment, the navigation device determines the fourth relative position information between the first and second markers based on the global pose information corresponding to all markers determined when the workspace is created. In other words, the navigation device can determine the relative position information between markers that are not out of sight and markers that have been out of sight, when neither is out of sight.

[0163] Continue to refer to Figure 9 (a) When the workspace is first created, both marker 1 and marker 2 are within the field of view of the navigation device. Since the global pose information 1 of marker 1 and the global pose information 2 of marker 2 are known, the navigation device can determine the fourth relative position information between marker 2 and marker 1 in this case.

[0164] The above example uses two markers. In scenarios with at least three markers, if there are multiple second markers, the navigation device can determine the fourth relative position information of the first marker with each of the at least one second marker. For example, assuming that markers 1 to 3 are initially within the navigation device's field of view, and then marker 1 is out of sight, the navigation device can determine the fourth relative position information of marker 1 with marker 2 and the fourth relative position information of marker 1 with marker 3. Alternatively, it can determine only the fourth relative position information of marker 1 with marker 2 or the fourth relative position information of marker 1 with marker 3. This embodiment does not impose any limitations.

[0165] S1003, determine the angle adjustment parameters based on the third relative position information and the fourth relative position information.

[0166] Furthermore, after determining the third and fourth relative position information, the navigation device can determine the angle adjustment parameters based on the third and fourth relative position information.

[0167] by Figure 9 For example, Figure 9 In the scenario shown in (b), the navigation device can determine the third relative position information of marker 1 relative to the navigation device when marker 2 is out of sight; Figure 9 In the scenario shown in (a), the navigation device can determine the fourth relative position information between marker 2 and marker 1 when neither marker 1 nor marker 2 is out of sight. Then, based on the third and fourth relative position information, the angle that the navigation device needs to rotate when marker 2 returns to the field of view can be determined, i.e., the angle adjustment parameter.

[0168] by Figure 3 For example, due to the doctor's head movement, markers 2 and 3 are out of sight. For marker 3, the navigation device... Figure 3 (b) The third relative position information of time marker 1 and Figure 3 (a) Based on the fourth relative position information between marker 3 and marker 1, the navigation device can determine the heading angle required to re-track marker 3. Similarly, the navigation device... Figure 3 (c) The third relative position information of time marker 1, and Figure 3 (a) The fourth relative position information between marker 2 and marker 1 can determine the pitch angle that the navigation device needs to rotate to re-track marker 2. In this way, the vision sensor uses its servo mechanism to adjust based on the determined heading and pitch angles, and the navigation device's field of view can then cover all markers again.

[0169] Optionally, the navigation device can store the correspondence between its own pose changes and angle adjustment parameters in advance. Then, after calculating the pose changes of the navigation device based on the third relative position information and the fourth relative position information, the navigation device can use the pose changes of the navigation device and the above correspondence to determine the angle adjustment parameters.

[0170] Since the second marker is within the field of view of the navigation device, the second marker and the third relative position information between the second marker and the navigation device are determined based on the image data acquired by the navigation device in the current field of view. After determining the fourth relative position information between the first marker and the second marker based on the global pose information of all markers, the angle adjustment parameters can be accurately and efficiently determined based on the third and fourth relative position information.

[0171] Figure 11 This is a flowchart illustrating another method for determining angle adjustment parameters in an embodiment of this application. (Refer to...) Figure 11 This embodiment relates to an optional implementation of how to determine angle adjustment parameters. Based on the above embodiment, the above-mentioned "determining the target parameters of the navigation device" includes the following steps:

[0172] S1101, determine the second pose change of the navigation device based on the global pose information of the navigation device.

[0173] In this embodiment, optionally, the navigation device can periodically acquire its own global pose information and calculate the second pose change between the currently acquired global pose information and the global pose information acquired at the previous time point. For example, assume the navigation device can acquire its own global pose information every second and calculate the second pose change between two adjacent seconds.

[0174] In some embodiments, the period for the navigation device to acquire global pose information and the period for calculating the second pose change can also be different. For example, the navigation device can acquire its own global pose information every second and calculate the second pose change every 2 seconds.

[0175] Understandably, the second pose change can reflect the movement of the navigation device, that is, the movement of the doctor's head.

[0176] S1102, determine the angle adjustment parameters based on the second pose change.

[0177] In this embodiment, after determining the second pose change, the navigation device can determine the angle adjustment parameters based on the second pose change. Optionally, if the second pose change is greater than a preset value, the navigation device can determine the angle corresponding to the second pose change based on the conversion relationship between the pose change and the angle, and then determine the angle adjustment parameters based on that angle.

[0178] For example, after determining the second pose change, the navigation device determines that the corresponding angle is -15 degrees horizontally. That is, the navigation device rotates 15 degrees horizontally to the left from its initial position to its current position, and during this process, the navigation device's pose also changes according to the second pose change. Further, the navigation device takes a negative value for the -15 degree horizontal angle to obtain an angle adjustment parameter; that is, the angle adjustment parameter is used to instruct the navigation device to rotate 15 degrees horizontally to the right.

[0179] This embodiment can determine the second pose change of the navigation device based on the global pose information of the navigation device, and determine the angle adjustment parameters based on the second pose change, without using image data, thus improving the efficiency of determining the angle adjustment parameters.

[0180] In one embodiment, optionally, if the target parameters include angle adjustment parameters, then the above-mentioned "determining the target parameters of the navigation device" can also be achieved in the following way:

[0181] The angle adjustment parameters are determined based on the image data acquired by the navigation device in the current field of view, the global pose information corresponding to all markers, and the global pose information of the navigation device.

[0182] In this embodiment, another process combining the above two methods to determine the angle adjustment parameters is also provided. That is, the navigation device determines the angle adjustment parameters based on the image data acquired by the navigation device in the current field of view, the global pose information corresponding to all markers, and the global pose information of the navigation device.

[0183] Optionally, the navigation device can determine the second marker and the third relative position information between the second marker and the navigation device based on the image data acquired by the navigation device in the current field of view, and determine the fourth relative position information between the first marker and the second marker based on the global pose information, so as to determine the first candidate angle adjustment parameter based on the third relative position information and the fourth relative position information.

[0184] Furthermore, the navigation device determines the second pose change amount based on the global pose information of the navigation device, and determines the second angle adjustment parameters based on the second pose change amount.

[0185] Then, the navigation device can determine the angle adjustment parameters based on the first candidate angle adjustment parameters and the second candidate angle parameters. Optionally, the navigation device can use the second candidate angle parameters to correct the first candidate angle adjustment parameters and use the corrected result as the final angle adjustment parameters. Alternatively, the navigation device can perform weighted calculations or average calculations on the first and second candidate angle adjustment parameters to obtain the final angle adjustment parameters.

[0186] This embodiment improves the accuracy of angle adjustment parameters by determining them based on image data acquired by the navigation device in the current field of view, global pose information corresponding to all markers, and global pose information of the navigation device.

[0187] In some application scenarios, navigation devices may not be able to adjust parameters based on angle and / or field of view to bring all markers into the navigation device's field of view. In such cases, the navigation device can perform global spatial perception, determine the location of out-of-sight markers, and promptly alert the doctor.

[0188] Figure 12 This is a schematic diagram of a process for issuing a prompt message in an embodiment of this application. (Refer to...) Figure 12 This embodiment relates to an optional implementation of how to issue a prompt message. Based on the above embodiment, the "issuing a prompt message" includes the following steps:

[0189] S1201, Based on the image data acquired by the navigation device at the first moment, determine the first relative position information of the first marker relative to the navigation device at the first moment; wherein, the first moment is the moment before the first marker leaves the field of view of the navigation device.

[0190] In this embodiment, the first moment is the moment before the first marker leaves the field of view of the navigation device. That is, it is necessary to obtain the first relative position information between the marker that has left the field of view and the navigation device before it leaves the field of view.

[0191] Optionally, the navigation device can take the previous moment corresponding to the moment when the marker detection result is determined to be present as the first moment, and determine the first relative position information of the first marker at the first moment before it leaves the field of view of the navigation device based on the image data at the first moment.

[0192] For example, the navigation device acquires image data of the current field of view every second. Suppose that the navigation device determines the existence of the first marker at the 5th second, it can determine the first relative position information of the first marker based on the image data of the current field of view at the 4th second.

[0193] Optionally, the navigation device can determine the first relative position information based on the image data at the first moment, the global pose information of the navigation device at the first moment, and the fixed pose relationship between the navigation device and the visual sensor. In other words, the navigation device can determine the relative relationship between the first marker and the visual sensor based on the image data at the first moment, and determine the first relative position information based on the fixed pose relationship between the navigation device and the visual sensor and the global pose information of the navigation device at the first moment.

[0194] S1202, based on the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment, determine the first pose change of the navigation device.

[0195] In this embodiment, the navigation device also needs to determine its first attitude change between the first time point and the second time point, where the second time point refers to the current time point.

[0196] Optionally, the navigation device can periodically acquire its own global pose information, thereby determining its global pose information at the first moment and at the second moment.

[0197] Continuing with the example above, assuming the current time is the 5th second and the first time is the 4th second, the navigation device can determine the first pose change based on the global pose information at the 5th second and the global pose information at the 4th second.

[0198] S1203, based on the first pose change and the first relative position information, determine the second relative position information of the first marker relative to the navigation device at the second time.

[0199] Figure 13 This application provides a schematic diagram of a global spatial perception process according to an embodiment. Figure 13 The thick black solid line in the image represents the field of view of the navigation device. Figure 13 The coordinate system in the text represents the global coordinate system in which the navigation device is located. Figure 13 (a) shows the situation where marker 1 is within the field of view of the navigation device.

[0200] Assuming the doctor's head moves at the 5th second, the navigation device will generate its first pose change T1 after that. Figure 13 (a) will follow Figure 13 (c) It changes, and eventually changes to Figure 13 The situation shown in (d) leads to marker 1 being out of sight.

[0201] Figure 13 (b) shows the situation of marker 1 at the first moment, that is, at the 4th second. Figure 13 (b) and Figure 13 As shown in (c), at the last moment before marker 1 is out of sight, the first relative position information between marker 1 and the navigation device is P1.

[0202] Please combine Figure 13 (a)~ Figure 13 (c) can then determine the second relative position information of the first marker relative to the navigation device based on the first relative position information and the second pose change.

[0203] Optionally, the navigation device can determine the second relative position information of the first marker relative to the navigation device based on the product of the first relative position information and the first pose change. For example, if the first relative position information is denoted as P1, the first pose change is denoted as T1, and the second relative position information is denoted as P2, then P2 = P1 * T1.

[0204] S1204, issue a prompt message based on the second relative position information.

[0205] In this embodiment, since the second relative position information reflects the relative positional relationship between the navigation device and the first marker, the navigation device can issue a prompt based on the second relative position information. The prompt can be any one or more of the following: voice announcement, buzzer alarm, vibration alert, message notification, or display notification.

[0206] Figure 14 This is a schematic diagram of a prompt message in an embodiment of this application, such as... Figure 14 As shown, a head-mounted navigation device uses a combination of "near-eye display + speaker + stereo vibration" to issue prompts, as an example.

[0207] When marker 2 is out of sight and cannot be brought back into the navigation device's field of view using angle adjustment parameters and / or field of view adjustment parameters, the navigation device can obtain the second relative position information of marker 2 in accordance with S1201 to S1204.

[0208] Assuming that marker 2 is located to the doctor's right rear based on its second relative position information, the navigation device can display the location of marker 2 to the right rear on its near-eye display screen as an indicator arrow. Furthermore, the navigation device will use a speaker to announce, "Marker 2 has been removed from the field of vision and is located to the right rear." Simultaneously, the vibration point of the stereo vibration array located to the right rear in the navigation device will vibrate to prompt the doctor to turn their head to the right rear to adjust their field of vision.

[0209] Based on this, the navigation device can dynamically indicate the location of a marker that has been out of sight from all angles, using near-eye holographic imaging technology. After receiving the above prompts, the doctor can adjust their head orientation as instructed until marker 2 returns to the navigation device's field of view.

[0210] In some embodiments, the stereo vibration array can be replaced with a surround speaker array. After the navigation device determines the second relative position information, the speaker in the corresponding direction can emit a prompt message to prompt the doctor to adjust the head direction. This embodiment is not limited to this.

[0211] This embodiment determines the first relative position information of the first marker relative to the navigation device at the first moment based on the image data acquired by the navigation device at the first moment. Then, based on the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment, it determines the first pose change of the navigation device. Finally, based on the first pose change and the first relative position information, it determines the second relative position information of the first marker relative to the navigation device at the second moment, and issues a prompt based on the second relative position information. In other words, even if the marker is out of sight, the navigation device can perform multi-information fusion to achieve global spatial perception, thereby determining its position relative to itself and promptly alerting the doctor, improving navigation efficiency.

[0212] To more clearly illustrate the marker tracking method in this application, this paper combines... Figure 15 illustrate. Figure 15 This is a schematic diagram illustrating a marker tracking method in an embodiment of this application, as shown below. Figure 15 As shown, the doctor wears a navigation device and adjusts its field of view to suit different doctors' heights and surgical postures, such as sitting or standing. The navigation device then creates a workspace and initiates the navigation process to assist the doctor in completing the surgery. Furthermore, during navigation, the device can execute the marker tracking method according to the following procedure.

[0213] S1501, Obtain the global pose information of the navigation device.

[0214] S1502, determine whether the navigation device has moved. If the navigation device has moved, proceed to S1503; if the navigation device has not moved, the navigation device or the doctor can determine whether navigation is complete. If complete, navigation ends to complete the entire surgery. If incomplete, the navigation process continues.

[0215] S1503, determine the marker detection result according to the preset parameters.

[0216] S1504, determine whether the marker detection result indicates the presence of a first marker. If the first marker exists, proceed to S1505; if the first marker does not exist, return to S1501 to continue motion detection of the navigation device, i.e., monitoring the doctor's head movements.

[0217] S1505, determine the global pose information corresponding to all markers.

[0218] S1506, Based on the image data acquired by the navigation device in the current field of view, determine the second marker and the third relative position information between the second marker and the navigation device.

[0219] S1507, Based on the global pose information corresponding to all markers, determine the fourth relative position information between the first marker and the second marker.

[0220] S1508, determine the angle adjustment parameters based on the third and fourth relative position information.

[0221] S1509, determine whether the angle adjustment parameter is within the first preset parameter adjustment range. If the angle adjustment parameter is within the first preset parameter adjustment range, execute S1510; if the angle adjustment parameter is not within the first preset parameter adjustment range, execute S1513.

[0222] S1510, adjusts the navigation device according to the angle adjustment parameters.

[0223] S1511, Update marker detection results.

[0224] S1512, determine whether the updated marker detection result indicates the existence of the first marker. If the updated marker detection result still indicates the existence of the first marker, proceed to S1513; if the updated marker detection result indicates the absence of the first marker, return to S1501.

[0225] S1513, Determine the field of view adjustment parameters.

[0226] S1514, determine whether the field of view adjustment parameter is within the second preset parameter adjustment range. If the field of view adjustment parameter is within the second preset parameter adjustment range, execute S1515; if the field of view adjustment parameter is not within the second preset parameter adjustment range, execute S1518.

[0227] S1515, adjust the navigation device according to the field of view adjustment parameters.

[0228] S1516, Update marker detection results.

[0229] S1517, determine whether the updated marker detection result indicates the existence of the first marker. If the updated marker detection result still indicates the existence of the first marker, proceed to S1518; if the updated marker detection result indicates the absence of the first marker, return to S1501.

[0230] S1518, Based on the image data acquired by the navigation device at the first moment, determine the first relative position information of the first marker relative to the navigation device at the first moment.

[0231] S1519, Based on the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment, determine the first pose change of the navigation device.

[0232] S1520, based on the first pose change and the first relative position information, determine the second relative position information of the first marker relative to the navigation device at the second time.

[0233] S1521, a prompt is issued based on the second relative position information. The doctor can adjust the head orientation based on this prompt. After the doctor makes the adjustment, the navigation device will return to step S1516 until all markers are back in the navigation device's field of view.

[0234] The processes S1501 to S1521 can be referred to in the above embodiments, and will not be repeated here.

[0235] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0236] Based on the same inventive concept, embodiments of this application also provide a tracking system for markers. Figure 16 This is a schematic diagram of the structure of a marker tracking system according to an embodiment of this application, as shown below. Figure 16 As shown, the marker tracking system 1600 includes a navigation device 1601 and at least one marker 1602.

[0237] The navigation device 1601 is used to determine the marker detection results obtained after detecting each marker in the workspace where the navigation device 1601 is located. If the marker detection result is that a first marker exists, the target parameters of the navigation device 1601 are determined, and the navigation device 1601 is adjusted according to the target parameters of the navigation device 1601 so that each marker enters the field of view of the navigation device 1601.

[0238] Figure 17 This is a schematic diagram of the structure of another marker tracking system in the embodiments of this application. In one embodiment, optionally, the navigation device 1601 further includes a visual sensor 1701 and a servo mechanism 1702.

[0239] Servo device 1604 refers to a collection of mechanisms or components that can adjust the parameters of a navigation device. The parameters that servo device 1604 can adjust include, but are not limited to, the angle, position, and focal length of the visual sensor on the navigation device.

[0240] Furthermore, the servo mechanism 1702 can be used to adjust the navigation device 1601 according to the target parameters of the navigation device 1601 so that each marker 1602 enters the field of view of the navigation device 1601, that is, each marker 1602 enters the field of view of the vision sensor 1701 on the navigation device.

[0241] Please continue to refer to this. Figure 17 In one embodiment, the navigation device 1601 may optionally include an odometer 1703.

[0242] The odometer 1703 is used to acquire the global pose information of the navigation device 1601. The odometer 1703 includes, but is not limited to, a gyroscope, an accelerometer, and a visual odometry camera.

[0243] Please continue to refer to this. Figure 17 In one embodiment, optionally, the marker tracking system 1600 further includes a prompting device 1704. The prompting device 1704 includes, but is not limited to, a speaker, an indicator light, a laser indicator light, or a stereo vibration array.

[0244] The navigation device 1601 is further configured to: determine, based on image data acquired by the navigation device 1601 at a first moment, a first relative position information of the first marker relative to the navigation device 1601 at the first moment; determine, based on the global pose information of the navigation device 1601 at the current second moment and the global pose information of the navigation device 1601 at the first moment; and determine, based on the first pose change and the first relative position information, a second relative position information of the first marker relative to the navigation device 1601 at the second moment. The first moment is the moment before the first marker leaves the field of view of the navigation device 1601.

[0245] The prompting device 1704 is used to issue a prompting message based on the second relative position information.

[0246] In one embodiment, the prompting device 1704 includes a stereo vibration array 1705 disposed on the navigation device 1601 and / or a display device 1706. The stereo vibration array 1705 includes a plurality of vibration points surrounding a region of interest of the target object.

[0247] The region of interest of the target object is used for wearing navigation device 1601. For example... Figure 13 As shown, taking a head-mounted navigation device as an example, the device has multiple vibration points surrounding the doctor's head. Different vibration points indicate different directions. Once the navigation device determines the second relative position information, it can instruct the corresponding vibration point to vibrate based on this information. For example, if the second relative position information indicates that the first marker is located to the right rear of the navigation device, then the vibration point located to the right rear of the doctor's head will vibrate.

[0248] In summary, the navigation device 1601 in the marker tracking system 1600 is capable of performing the steps of any of the marker tracking methods described above.

[0249] This application also provides a marker tracking device for implementing the marker tracking method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more marker tracking device embodiments provided below can be found in the limitations of the marker tracking method described above, and will not be repeated here.

[0250] Figure 18 This is a structural block diagram of the marker tracking device in the embodiments of this application, such as... Figure 18 As shown, this application provides a marker tracking device 1800 for use in navigation devices. The marker tracking device 1800 includes: a first determining module 1801, a second determining module 1802, and an adjusting module 1803, wherein:

[0251] The first determining module 1801 is used to determine the marker detection results obtained after detecting all markers in the workspace where the navigation device is located.

[0252] The second determining module 1802 is used to determine the target parameters of the navigation device if the marker detection result is that a first marker exists; the first marker is a marker that is out of the field of view of the navigation device.

[0253] The adjustment module 1803 is used to adjust the navigation device according to the target parameters of the navigation device so that all markers are within the field of view of the navigation device.

[0254] The marker tracking device provided in this embodiment, after determining the marker detection results obtained by detecting all markers in the workspace of the navigation device, can automatically determine the target parameters of the navigation device when the marker detection result indicates that there is a first marker that has left the field of view of the navigation device. The navigation device is then adjusted according to the target parameters. Therefore, after adjusting the navigation device according to the target parameters, all markers in the workspace of the navigation device can return to the field of view of the navigation device, thus achieving marker tracking. Taking a head-mounted navigation device as an example, the field of view of the navigation device can better adapt to the doctor's head movements. Even if a marker leaves the field of view of the navigation device due to head movements, the navigation device can still track the marker in a timely manner, thereby improving the navigation efficiency of the navigation device.

[0255] Optionally, the target parameters include at least one of angle adjustment parameters and field of view adjustment parameters; the angle adjustment parameters are used to adjust the attitude of the visual sensor in the navigation device, and the field of view adjustment parameters are used to adjust the field of view of the visual sensor in the navigation device.

[0256] Optionally, the target parameters include a first parameter and a second parameter, wherein the first parameter and the second parameter are one of the angle adjustment parameter and the field of view adjustment parameter, respectively, and the first parameter and the second parameter are different; the second determining module 1802 includes:

[0257] The first adjustment unit is used to adjust the navigation device according to the first parameter if the first parameter is within the adjustment range of the first preset parameter corresponding to the visual sensor in the navigation device, and to update the marker detection result after adjusting the navigation device according to the first parameter.

[0258] The second adjustment unit is used to adjust the navigation device according to the second parameter if the updated marker detection result indicates the presence of the first marker, or to issue a prompt message.

[0259] Optionally, the second determining module 1802 further includes:

[0260] The third adjustment unit is used to adjust the navigation device according to the second parameter if the first parameter is not within the adjustment range of the first preset parameter, or to issue a prompt message.

[0261] Optionally, the aforementioned third adjustment unit includes:

[0262] The adjustment subunit is used to adjust the navigation device according to the second parameter if the second parameter is within the adjustment range of the second preset parameter corresponding to the visual sensor in the navigation device, and to update the marker detection result after adjusting the navigation device according to the second parameter.

[0263] The first prompt subunit is used to issue a prompt message if the updated marker detection result indicates the presence of the first marker.

[0264] Optionally, the third adjustment unit mentioned above also includes:

[0265] The second prompt subunit is used to issue a prompt message if the second parameter is not within the adjustment range of the second preset parameter.

[0266] Optionally, if the target parameters include angle adjustment parameters, then the second determining module 1802 described above includes:

[0267] The first determining unit is used to determine the global pose information corresponding to all markers.

[0268] The second determining unit is used to determine the angle adjustment parameters based on global pose information and image data acquired by the navigation device in the current field of view.

[0269] Optionally, the second determining unit includes:

[0270] The first determining subunit is used to determine the second marker and the third relative position information between the second marker and the navigation device based on the image data acquired by the navigation device in the current field of view; the second marker is a marker that is within the field of view of the navigation device.

[0271] The second determining subunit is used to determine the fourth relative position information between the first marker and the second marker based on the global pose information corresponding to all markers.

[0272] The third determining subunit is used to determine the angle adjustment parameters based on the third relative position information and the fourth relative position information.

[0273] Optionally, if the target parameters include angle adjustment parameters, then the second determining module 1802 described above includes:

[0274] The third determining unit is used to determine the second pose change of the navigation device based on the global pose information of the navigation device.

[0275] The fourth determining unit is used to determine the angle adjustment parameters based on the second pose change.

[0276] Optionally, if the target parameters include angle adjustment parameters, then the second determining module 1802 described above includes:

[0277] The fifth determining unit is used to determine the angle adjustment parameters based on the image data acquired by the navigation device in the current field of view, the global pose information corresponding to all markers, and the global pose information of the navigation device.

[0278] Optionally, the first determining module 1801 includes:

[0279] The sixth determining unit is used to determine the marker detection result according to preset parameters; the preset parameters include at least one of the image data acquired by the navigation device in the current field of view, feature information corresponding to all markers, and global pose information of the navigation device.

[0280] Optionally, the aforementioned second adjustment unit, third adjustment unit, first prompting subunit, and second prompting subunit are further configured to: determine the first relative position information of the first marker relative to the navigation device at the first moment based on the image data acquired by the navigation device at the first moment; determine the first pose change of the navigation device based on the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment; determine the second relative position information of the first marker relative to the navigation device at the second moment based on the first pose change and the first relative position information; and issue a prompting message based on the second relative position information; wherein, the first moment is the moment before the first marker leaves the field of view of the navigation device.

[0281] Optionally, the sixth determining unit described above is also used to determine the marker detection result according to preset parameters if it is determined that the navigation device has moved based on the global pose information of the navigation device.

[0282] Each module in the aforementioned marker tracking device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware within or independently of the processor in the navigation device, or stored in software within the memory of the navigation device, so that the processor can invoke and execute the corresponding operations of each module.

[0283] Figure 19 This is an internal structure diagram of a navigation device in an embodiment of this application. This application provides a navigation device, which can be a server, and its internal structure diagram can be as shown below. Figure 19 As shown, the navigation device includes a processor, memory, and network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores relevant data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a marker tracking method.

[0284] Those skilled in the art will understand that Figure 19The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the navigation device to which the present application is applied. A specific navigation device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0285] In one embodiment, a navigation device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0286] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0287] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0288] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0289] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0290] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for tracking markers, characterized in that, Applied to navigation devices, the method includes: The marker detection results are determined after detecting all markers within the workspace of the navigation device. If the marker detection result indicates the presence of a first marker, then the target parameters of the navigation device are determined; the first marker is a marker that is out of the field of view of the navigation device. The navigation device is adjusted according to its target parameters to bring all the markers into the field of view of the navigation device. Determining the target parameters of the navigation device includes: The target parameters of the navigation device are determined based on the relative position information between the second marker and the navigation device, and the relative position information between the first marker and the second marker; the second marker is a marker that is within the field of view of the navigation device.

2. The method according to claim 1, characterized in that, The target parameters include at least one of angle adjustment parameters and field of view adjustment parameters; The angle adjustment parameter is used to adjust the attitude of the visual sensor in the navigation device, and the field of view adjustment parameter is used to adjust the field of view of the visual sensor in the navigation device.

3. The method according to claim 1, characterized in that, The target parameters include a first parameter and a second parameter, wherein the first parameter and the second parameter are one of the angle adjustment parameter and the field of view adjustment parameter, respectively, and the first parameter and the second parameter are different; Adjusting the navigation device according to the target parameters of the navigation device includes: If the first parameter is within the first preset parameter adjustment range corresponding to the visual sensor in the navigation device, then the navigation device is adjusted according to the first parameter, and after the navigation device is adjusted according to the first parameter, the marker detection result is updated; If the updated marker detection result indicates the presence of the first marker, then the navigation device is adjusted according to the second parameter, or a prompt message is issued.

4. The method according to claim 3, characterized in that, The method further includes: If the first parameter is not within the adjustment range of the first preset parameter, then the navigation device is adjusted according to the second parameter, or a prompt message is issued.

5. The method according to claim 3, characterized in that, The step of adjusting the navigation device according to the second parameter includes: If the second parameter is within the second preset parameter adjustment range corresponding to the visual sensor in the navigation device, then the navigation device is adjusted according to the second parameter, and after the navigation device is adjusted according to the second parameter, the marker detection result is updated; If the updated marker detection result indicates the presence of the first marker, a prompt message will be issued.

6. The method according to claim 5, characterized in that, The method further includes: If the second parameter is not within the adjustment range of the second preset parameter, a prompt message will be issued.

7. The method according to any one of claims 3-6, characterized in that, The notification message includes: Based on the image data acquired by the navigation device at a first moment, the first relative position information of the first marker relative to the navigation device at the first moment is determined; wherein, the first moment is the moment before the first marker leaves the field of view of the navigation device; Based on the global pose information of the navigation device at the current second moment and the global pose information of the navigation device at the first moment, determine the first pose change of the navigation device; Based on the first pose change and the first relative position information, determine the second relative position information of the first marker relative to the navigation device at the second moment; A prompt message is issued based on the second relative position information.

8. The method according to any one of claims 1-6, characterized in that, The determination of the marker detection results obtained after detecting all markers within the workspace of the navigation device includes: The marker detection result is determined according to preset parameters; the preset parameters include at least one of the image data acquired by the navigation device in the current field of view, the feature information corresponding to all markers, and the global pose information of the navigation device.

9. The method according to claim 8, characterized in that, The step of determining the marker detection result according to preset parameters includes: If it is determined that the navigation device has moved based on the global pose information of the navigation device, then the marker detection result is determined based on the preset parameters.

10. A tracking system for markers, characterized in that, The system includes a navigation device and at least one marker; The navigation device is used to determine the marker detection results obtained after detecting each of the markers in the workspace where the navigation device is located. If the marker detection result is that a first marker exists, the target parameters of the navigation device are determined, and the navigation device is adjusted according to the target parameters of the navigation device so that each of the markers enters the field of view of the navigation device. Determining the target parameters of the navigation device includes: The target parameters of the navigation device are determined based on the relative position information between the second marker and the navigation device, and the relative position information between the first marker and the second marker; the second marker is a marker that is within the field of view of the navigation device.

11. A tracking device for markers, characterized in that, Applied to navigation devices, the device includes: The first determining module is used to determine the marker detection results obtained after detecting all markers in the workspace where the navigation device is located; The second determining module is used to determine the target parameters of the navigation device if the marker detection result indicates the presence of a first marker; the first marker is a marker that is out of the field of view of the navigation device. An adjustment module is used to adjust the navigation device according to the target parameters of the navigation device, so that all the markers enter the field of view of the navigation device; The second determining module is further configured to determine the target parameters of the navigation device based on the relative position information between the second marker and the navigation device and the relative position information between the first marker and the second marker; the second marker is a marker that is within the field of view of the navigation device.

12. A navigation device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images