Intracranial target spot positioning auxiliary system

Abstract

The embodiment of the invention discloses an intracranial target spot positioning auxiliary system. The intracranial target spot positioning auxiliary system comprises a target spot positioning device which comprises a main operation mechanical arm and an auxiliary mechanical arm, and a server generates mechanical arm modeling information; a target point positioning device for generating target execution track information; the main operation mechanical arm is controlled to obtain the positioning label; the main operation mechanical arm is controlled to move; collecting target environment feedback information at the current moment; generating movement compensation information; the main operation mechanical arm is controlled to move; the main operation mechanical arm is controlled to carry out target spot positioning treatment; generating moving track information of the auxiliary mechanical arm; the auxiliary mechanical arm is controlled to carry out positioning compensation treatment; and the augmented reality device displays the calibration point position information. According to the embodiment, the stability of the positioning system can be improved, the situation that the delay of the positioning system is long is reduced, the positioning difficulty is reduced, the positioning steps are simplified, and the consumed time is shortened.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to an intracranial target localization assistance system. Background Technology

[0002] Intracranial target localization assistance systems are computer systems that control robotic arms to assist in the localization of intracranial targets. Currently, existing intracranial target localization assistance systems typically include: systems that use a control panel for localization via a cloud processor; systems that transmit localization commands via a remote expert network; or systems that directly use a traditional metal stereoscopic positioning head for target localization.

[0003] However, in practice, the following technical problems often arise when using the existing systems for locating intracranial targets: When using a system that transmits positioning commands over a network via a cloud processor or remote experts for positioning, the inability to execute positioning commands due to network failure results in poor stability and long latency in the positioning system. When using a metal stereo positioning head system for positioning, the assembly and debugging of the metal stereo positioning head involves many steps and is quite difficult, consuming a lot of time. When using a control panel for positioning, it is necessary to repeatedly look up to check the positioning status and look down to use the control panel, making the positioning process cumbersome and resulting in long positioning times.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose an intracranial target localization assistance system to address one or more of the technical problems mentioned in the background section above.

[0007] Some embodiments of this disclosure provide an intracranial target localization assistance system, which includes a server, a target localization device, and an augmented reality device. The target localization device includes a main manipulator and an auxiliary manipulator. The server is configured to: generate manipulator modeling information based on manipulator point cloud information from the main manipulator and the auxiliary manipulator; generate target execution trajectory information based on the manipulator modeling information, scene environment information, and image target location information; and store the target execution trajectory information and the manipulator modeling information in a target storage space included in the target localization device. The target localization device is configured to: control the main manipulator to acquire a localization tag in response to detecting target localization initiation information; and control the main manipulator to acquire a localization tag in response to detecting network environment conditions. Pre-set network fault conditions, and based on the target execution trajectory information, control the main manipulator to move; collect target environmental feedback information at the current moment; in response to determining that the target environmental feedback information meets preset correction conditions, generate movement compensation information based on the target environmental feedback information; control the main manipulator to move based on the movement compensation information; in response to detecting target positioning operation information, control the main manipulator to perform target positioning processing; in response to detecting that the target environmental feedback information meets preset auxiliary conditions, generate the corresponding running trajectory information of the auxiliary manipulator based on auxiliary position information; control the auxiliary manipulator to perform positioning compensation processing based on the running trajectory information; the augmented reality device is used to display calibration point information.

[0008] The above-described embodiments of this disclosure have the following beneficial effects: the intracranial target localization assistance system of some embodiments of this disclosure improves the stability of the localization system, reduces the long latency of the localization system, reduces the difficulty of localization, simplifies the localization steps, and thus shortens the time consumed. Specifically, the reasons for poor stability and long latency of the localization system, and the numerous and difficult steps in assembling and debugging the metal stereoscopic localization headframe, resulting in long time consumption, are as follows: when using a system that transmits localization commands via a network using a cloud processor or remote experts, the localization operation commands cannot be executed when the network fails, resulting in poor stability and long latency of the localization system; when using a metal stereoscopic localization headframe localization system, the assembly and debugging steps of the metal stereoscopic localization headframe are numerous and difficult, resulting in long time consumption; when using a control panel for localization, it is necessary to repeatedly look up to check the localization status and look down to use the control panel for localization, making the localization steps cumbersome and resulting in long localization time consumption. Based on this, in the intracranial target localization assistance system of some embodiments of this disclosure, firstly, the server is used to generate robotic arm modeling information based on the robotic arm point cloud information of the main robotic arm and the auxiliary robotic arm. Thus, a model of the robotic arm can be obtained. Then, based on the aforementioned robotic arm modeling information, scene environment information, and image target location information, target execution trajectory information is generated. Thus, the execution trajectory of the robotic arm can be obtained. Next, the aforementioned target execution trajectory information and the aforementioned robotic arm modeling information are stored in the target storage space included in the aforementioned target localization device. Thus, information can be stored to enable the localization processing of the aforementioned target localization device when offline. Afterwards, the aforementioned target localization device, in response to the detection of target localization initiation information, controls the aforementioned main operating robotic arm to acquire a localization tag. Thus, the localization tag can be acquired for localization. Then, in response to the detection that the network environment meets preset network fault conditions, the aforementioned main operating robotic arm is controlled to move according to the aforementioned target execution trajectory information. Thus, the robotic arm can be moved to the localization area. Next, target environment feedback information at the current moment is collected. Thus, the current target localization environment can be obtained. Afterwards, in response to determining that the aforementioned target environment feedback information meets preset correction conditions, motion compensation information is generated based on the aforementioned target environment feedback information and a pre-trained localization operation generation model. Thus, the compensation information required for the robotic arm can be obtained. Then, in response to determining that the target environment feedback information meets the preset correction conditions, motion compensation information is generated based on the target environment feedback information. From this, instructions for controlling the auxiliary robotic arm can be obtained. Next, based on the motion compensation information, the main robotic arm is controlled to perform motion processing. Thus, the main robotic arm can be controlled to move to correct its position. Next, in response to detecting target positioning operation information, the main robotic arm is controlled to perform target positioning processing.Therefore, positioning tags can be affixed for location tracking. Then, in response to the detection that the target point's environmental feedback information meets preset auxiliary conditions, the corresponding trajectory information of the auxiliary robotic arm is generated based on the auxiliary position information. This allows control of the robotic arm's movement. Finally, based on the trajectory information, the auxiliary robotic arm is controlled to perform positioning compensation processing. This allows control of the auxiliary robotic arm to press the tag. Furthermore, the augmented reality device is used to display calibration point information. This can be used to calibrate the user's gaze position. Because positioning is not achieved through cloud servers or remote experts transmitting data over a network, but rather by storing the target execution trajectory information and the robotic arm modeling information in the target point positioning device's storage space, the target point positioning device can still perform positioning processing using the stored information even in the event of a network failure. Also, because positioning is not achieved through a metal stereoscopic positioning head, but through a robotic arm, the assembly and debugging of the metal stereoscopic positioning head is eliminated, thus reducing the difficulty of positioning, simplifying the positioning steps, and shortening the positioning time. Furthermore, because location tracking is performed using augmented reality (AR) devices, users can directly wear these devices for positioning without repeatedly looking up to check the location status or looking down to perform the operation, thus improving the convenience of location tracking. As a result, users can issue location commands promptly, shortening the time the positioning system takes to locate the target. This improves the stability of the positioning system, reduces latency issues, simplifies the positioning process, and further reduces the time required for location tracking. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0010] Figure 1 This is a schematic diagram of an application scenario of the intracranial target localization assistance system disclosed herein; Figure 2 This is a timing diagram of some embodiments of the intracranial target localization assistance system according to the present disclosure. Detailed Implementation

[0011] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0012] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0013] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0014] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0015] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0016] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] Figure 1 This is a schematic diagram of an application scenario of an intracranial target localization assistance system according to some embodiments of the present disclosure.

[0018] like Figure 1As shown, the intracranial target localization assistance system provided in this disclosure may include: a server, a target localization device, and an augmented reality device. The server may be configured to generate robotic arm modeling information based on the robotic arm point cloud information of the main robotic arm and the auxiliary robotic arm; generate target execution trajectory information based on the robotic arm modeling information, scene environment information, and image target position information; and store the target execution trajectory information and the robotic arm modeling information in the target storage space included in the target localization device. The aforementioned target localization device can be used to: control the main robotic arm to acquire a positioning tag in response to detecting target localization initiation information; control the main robotic arm to move according to the target execution trajectory information in response to detecting that the network environment meets preset network fault conditions; collect target environment feedback information at the current moment; generate movement compensation information according to the target environment feedback information in response to determining that the target environment feedback information meets preset correction conditions; control the main robotic arm to perform movement processing according to the movement compensation information; control the main robotic arm to perform target localization processing in response to detecting target localization operation information; generate corresponding running trajectory information of the auxiliary robotic arm according to the auxiliary position information in response to detecting that the target environment feedback information meets preset auxiliary conditions; and control the auxiliary robotic arm to perform positioning compensation processing according to the running trajectory information. The aforementioned augmented reality device can be AR glasses used to display calibration point information.

[0019] The following is for reference. Figure 2 The diagram shows a timing diagram of an intracranial target localization assistance system of the present disclosure.

[0020] like Figure 2 As shown, an intracranial target localization assistance system includes: a server, a target localization device, and an augmented reality device. The target localization device includes a main robotic arm, an auxiliary robotic arm, and a main controller. The interaction steps between the server, the target localization device, and the augmented reality device may include the following steps: Step 201: The server generates robotic arm modeling information based on the point cloud information of the main robotic arm and the auxiliary robotic arm.

[0021] In some embodiments, the server can generate robotic arm modeling information based on the point cloud information of the main robotic arm and the auxiliary robotic arm. The robotic arm modeling information can represent the three-dimensional models of the main robotic arm and the auxiliary robotic arm. The target localization device can be a device for assisting in intracranial target localization. The main robotic arm can be a robotic arm for attaching a positioning tag to the electrode needle inlet. The electrode needle inlet can be the entrance for inserting the electrode needle into the skull for target localization. The location where the positioning tag is attached can represent the electrode needle inlet. The auxiliary robotic arm can be a robotic arm for pressing the positioning tag attached to the main robotic arm to assist the main robotic arm in attaching the positioning tag, thereby making the positioning tag more firmly attached to the patient's skull. Here, the specific types of the main robotic arm and the auxiliary robotic arm are not limited and can be adjusted according to actual needs. For example, the main robotic arm can be a multi-joint serial robotic arm, or a six-axis robotic arm. The aforementioned auxiliary robotic arm can be a lightweight, multi-joint, serial robotic arm. It can also be a six-axis robotic arm. The aforementioned augmented reality device can be AR glasses. The point cloud information of the robotic arm can represent the individual 3D point clouds of both the main robotic arm and the auxiliary robotic arm.

[0022] In some optional implementations of certain embodiments, the server described above is configured as follows: The first step is to obtain the point cloud information of the main manipulator and the auxiliary manipulator. In practice, the server can obtain the point cloud information of the main manipulator and the auxiliary manipulator from a database that stores the 3D point clouds of the main manipulator and the auxiliary manipulator.

[0023] The second step is to preprocess the above robotic arm point cloud information to obtain the preprocessed robotic arm point cloud information as the robotic arm point cloud information.

[0024] The third step involves generating a semantic model of the robotic arm based on the semantic information corresponding to the main and auxiliary robotic arms. This semantic model can represent the BIM models of both the main and auxiliary robotic arms. The semantic information can represent the geometric and semantic information of both the main and auxiliary robotic arms. The geometric information of both the main and auxiliary robotic arms can represent the three-dimensional shape, size, spatial position, and topological relationships of the robotic arm. The topological relationships can represent the connection methods (structural relationships) and motion transmission logic (degrees of freedom association) of the various joints and links of the robotic arm. For example, the joints of the robotic arm can be shoulder, elbow, and wrist joints; the three-dimensional shape can be a slender, serial arm; the connection method of the links can be bolted connections; and the motion transmission logic (degrees of freedom association) can be the rotation degree of each joint included in the robotic arm. For example, the rotation angle of a joint can be ±90 degrees for the shoulder joint. The semantic information of both the primary and auxiliary robotic arms can characterize their tasks, states, control parameters, and instruction interpretation semantics. The primary robotic arm's task can be identified by the task identifier of attaching positioning tags. This task identifier can be 0 or 1. The states of both the primary and auxiliary robotic arms can be identified as standby or motion modes. The control parameters of the primary robotic arm can characterize its operating speed and the pressure applied to the positioning tags. For example, the pressure applied to the positioning tags can range from 0.3 to 0.5 N. The control parameters of the auxiliary robotic arm can characterize its operating speed and the pressure applied to the attached positioning tags. For example, the pressure applied to the attached positioning tags can range from 0.8 to 1.2 N. The instruction interpretation semantics can characterize the interpretation of instructions issued by the AR glasses. For example, the instruction interpretation semantics can be that the primary robotic arm stops operating and the auxiliary robotic arm starts operating. In practice, the server can use BIM software to generate a robotic arm semantic model based on the semantic information of the primary and auxiliary robotic arms.

[0025] The fourth step involves coordinate system transformation of the aforementioned robotic arm semantic model. In practice, the server can perform coordinate system transformation on the robotic arm semantic model using a coordinate transformation method based on rigid transformations (including scaling, rotation, and translation) to transform the coordinate system of the robotic arm semantic model to the target coordinate system within the current scene. This target coordinate system can represent the patient's head coordinate system. This head coordinate system can be a three-dimensional coordinate system established with the midpoint of the line connecting the patient's external auditory canals as the origin. Furthermore, the target coordinate system is rigidly associated with the coordinate system of the augmented reality device through reflective markers (relatively fixed to the patient's head) affixed to the edge of the operating table. The relative fixation of these reflective markers to the patient's head indicates that the relative position between the reflective markers (markers) affixed to the edge of the operating table and the patient's head remains constant (distance and angle do not change).

[0026] The fifth step involves fusing the aforementioned robotic arm point cloud information and the robotic arm semantic model after coordinate system transformation to obtain the robotic arm digital twin information. This digital twin information can represent the digital twins of the aforementioned main robotic arm and the aforementioned auxiliary robotic arm. In practice, the server can spatially align the robotic arm semantic model after coordinate system transformation and the aforementioned robotic arm point cloud information to obtain the robotic arm digital twin information.

[0027] Step 6: Based on the aforementioned digital twin information of the robotic arm, create robotic arm modeling information corresponding to the primary and auxiliary robotic arms. In practice, firstly, the server can use the Delaunay triangulation algorithm to create triangulation network structure information corresponding to the primary and auxiliary robotic arms based on the digital twin information of the robotic arms. This triangulation network structure information can represent the triangular network of the primary robotic arm and the triangular mesh of the auxiliary robotic arm. Finally, the created triangulation network structure information can be updated using the Laplace smoothing algorithm to obtain the updated triangulation network structure information as the robotic arm modeling information.

[0028] In addressing the technical problems mentioned above, and considering the application scenario: when using a positioning system to locate epileptogenic foci, which are often tiny targets deep within the brain, requiring sub-millimeter level (≤0.5mm) accuracy, the following technical problem often arises: directly using scanned 3D point clouds of a robotic arm for intracranial target localization results in raw, disordered, and noisy robotic arm point clouds, leading to low accuracy in the constructed robotic arm model. This results in low accuracy of the applied positioning tags, necessitating repeated detection and reapplying, increasing the number of repetitive robotic arm operations, consuming significant computational resources, and increasing the risk of collisions with the operating table, causing substantial hardware resource wear. Given the specific requirements of this application scenario—adapting to the high difficulty of target localization and the high accuracy requirements—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the server is configured to preprocess the robotic arm point cloud information through the following steps to obtain preprocessed robotic arm point cloud information as robotic arm point cloud information: The first step is to perform the following steps for each point cloud information in the above robotic arm point cloud information: The first sub-step involves determining the number of neighboring points in the aforementioned point cloud information based on preset neighborhood radius information. The preset neighborhood radius information can be 0.1 mm. This neighboring point number information represents the number of three-dimensional point clouds included in a region centered on the aforementioned point cloud information and with a radius equal to the preset neighborhood radius information. The aforementioned point cloud information represents the three-dimensional point cloud within the aforementioned robotic arm point cloud information. The target positioning device can determine the number of three-dimensional point clouds within a region centered on the aforementioned point cloud information and with a radius equal to the preset neighborhood radius information as the neighboring point number information.

[0029] The second sub-step involves deleting the aforementioned point cloud information from the robotic arm's point cloud information in response to the determination that the number of neighboring points is less than a preset number of neighboring points, thereby updating the robotic arm's point cloud information. Here, the preset number of neighboring points is not specifically limited and can be adjusted according to actual needs. For example, the preset number of neighboring points can be 10.

[0030] The second step involves dividing the updated robotic arm point cloud information into voxel information. Each voxel can represent a cube mesh of length A × A × A. This cube mesh can be a voxel. In practice, the server can use a voxelization algorithm to divide the updated robotic arm point cloud information into voxel information based on preset side lengths. The specific value of the preset side lengths is not limited and can be adjusted according to actual needs.

[0031] The third step is to determine the centroid information of each grid voxel in the aforementioned grid voxel information, thus obtaining the centroid information for each individual voxel. The centroid information in the aforementioned centroid information can represent the centroid. In practice, the server can determine the centroid information of each grid voxel in the aforementioned grid voxel information using the direct mean method, thus obtaining the centroid information for each individual voxel.

[0032] The fourth step involves modifying the voxel information of each mesh based on the centroid information, resulting in modified voxel information. In practice, for each voxel, the server can replace each point cloud information with its corresponding centroid to modify the voxel information, thus obtaining modified voxel information. The point cloud information included in each voxel can represent a 3D point cloud.

[0033] The fifth step involves performing initial alignment processing on the modified mesh voxel information and the preset point cloud information to obtain aligned mesh voxel information. The preset point cloud information represents the 3D point cloud of the midpoint of the external auditory canal connection in the target coordinate system. In practice, the server can use a centroid alignment algorithm to perform initial alignment processing on the modified mesh voxel information and the preset point cloud information.

[0034] Step 6: For each point cloud information in the mesh voxel information after the above alignment process, perform the following update steps: The first sub-step involves identifying the point cloud information with the smallest distance from the aforementioned preset point cloud information as the target point cloud information. Here, "smallest distance" refers to the smallest Euclidean distance.

[0035] The second sub-step involves identifying the aforementioned point cloud information and the aforementioned target point cloud information as a point cloud information group.

[0036] Step 7: Perform a rigid transformation on each obtained point cloud information group to obtain rotation matrix information, translation vector information, and objective function value. The rotation matrix information represents the rotation matrix. The translation vector information represents the translation vector. The objective function value represents the minimum objective function value. The objective function value represents the sum of squared Euclidean distances between each point cloud information group before and after the rigid transformation. In practice, the server can use the Singular Value Decomposition (SVD) algorithm based on least squares to perform the rigid transformation on each obtained point cloud information group to obtain the rotation matrix information, translation vector information, and objective function value.

[0037] Step 8: Based on the aforementioned rotation matrix information and translation vector information, update the information of each mesh voxel to obtain the updated information of each mesh voxel. In practice, for each point cloud information in the aforementioned mesh voxel information, the server can determine the first product as the product between the aforementioned rotation matrix information and the coordinate vector of the aforementioned point cloud information. Then, the sum of the obtained first product and the aforementioned translation vector information is determined as the updated point cloud information, which is used to update the aforementioned mesh voxel information to obtain the updated information of each mesh voxel.

[0038] The ninth step is to determine the updated voxel information of each mesh as the preprocessed point cloud information of the robotic arm.

[0039] The above-described technical solution, as an inventive point of this disclosure, solves technical problem two: "The accuracy of the constructed robotic arm model is low, resulting in low accuracy of the applied positioning tags, requiring repeated detection and reapplying, leading to numerous repetitive executions by the robotic arm, consuming significant computational resources, and causing the robotic arm to easily collide with the operating table, resulting in substantial hardware resource consumption." The reasons for these defects are as follows: Directly using the scanned 3D point cloud of the robotic arm for intracranial target localization results in a raw, disordered, and noisy robotic arm point cloud, leading to low accuracy of the constructed robotic arm model, resulting in low accuracy of the applied positioning tags, requiring repeated detection and reapplying, numerous repetitive executions by the robotic arm, consuming significant computational resources, and causing substantial hardware resource consumption due to the robotic arm's tendency to collide with the operating table. Solving these factors can improve the accuracy of the constructed robotic arm model, improve the accuracy of the applied positioning tags, reduce the number of repeated detections and reapplying, reduce the number of repetitive executions by the robotic arm, reduce computational resource consumption, and reduce hardware resource consumption. To achieve this effect, the intracranial target localization assistance system disclosed herein preprocesses the scanned 3D point cloud using mesh voxels, preset neighborhood radius information, and singular value decomposition (SVD) algorithm before modeling. This improves the accuracy of the 3D point cloud, thereby improving the accuracy of the robotic arm model, the accuracy of the constructed robotic arm model, the accuracy of the applied positioning tags, reducing the number of repeated detections and re-applications, reducing the number of repetitive executions by the robotic arm, reducing computational resources consumed, and reducing hardware resource consumption.

[0040] Step 202: The server generates target execution trajectory information based on the robotic arm modeling information, scene environment information, and image target location information.

[0041] In some embodiments, the server can generate target execution trajectory information based on the robotic arm modeling information, scene environment information, and image target location information. The scene environment information can represent a 3D model of the detected current scene. This current scene can be a 3D scene for intracranial target localization. For example, the 3D scene for intracranial target localization could be an intracranial target localization laboratory / operating room. The image target location information can represent the transformation of the intracranial target's coordinates from the image coordinate system to the target coordinate system. The target execution trajectory information can represent the trajectory of the main robotic arm when attaching the positioning tag. The image coordinate system can represent the 3D spatial coordinate system inherent in preoperative CT / MRI and other medical imaging equipment.

[0042] In addressing the technical problems mentioned above, and considering the application scenario: when using a positioning system to locate epileptogenic foci, the small size, depth, and ambiguous boundaries of the epileptogenic focus, along with sub-millimeter-level positioning requirements, leave no room for trajectory deviation. This places extremely high demands on the accuracy of the robotic arm's positioning, especially given the large number of target points and the short timeframe. This often leads to the following technical problem: directly generating an execution trajectory based on the target point's location without considering obstacles and resulting in poor trajectory smoothness, causes the robotic arm to stall and collide, making positioning difficult and time-consuming. To meet the following requirements of this application scenario—adapting to the high difficulty of target point positioning, high accuracy requirements, and short execution timeframes—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the server described above is configured as follows: The first step is to obtain the current position information of the main manipulator. This current position information represents the three-dimensional coordinates of the main manipulator at the current moment. In practice, the server can obtain the three-dimensional coordinates of the main manipulator at the current moment from a database storing these coordinates.

[0043] The second step involves segmenting the map information of the current scene to obtain segmented map region information. The current scene map information represents a 3D map of the current scene. The segmented map region information represents regions segmented from the 3D map. In practice, the server can use a geometric feature-based method to segment the current scene map information to obtain segmented map region information. For example, a geometric feature-based method could be a region growing algorithm.

[0044] The third step is to determine the segmented map region information corresponding to the current location information from the aforementioned segmented map region information as the first segmented map region information. In practice, the server can determine the segmented map region information that includes the three-dimensional coordinates represented by the current location information from the aforementioned segmented map region information as the first segmented map region information.

[0045] The fourth step is to determine the segmented map region information corresponding to the image target location information from the aforementioned segmented map region information as the second segmented map region information. In practice, the server can determine the segmented map region information that includes the three-dimensional coordinates represented by the image target location information from the aforementioned segmented map region information as the second segmented map region information.

[0046] The fifth step involves generating path guidance map information based on the first and second segmented map region information described above. This path guidance map information represents the guidance path map for the current scene. In practice, the server can generate the path guidance map information using a path planning algorithm based on the first and second segmented map region information. For example, the path planning algorithm could be Dijkstra's algorithm.

[0047] The sixth step involves determining the initial path information for the main robotic arm based on the path guidance map information described above. This initial path information characterizes the execution path of the generated robotic arm. In practice, the server can determine the initial path information for the main robotic arm based on the path guidance map information using a minimum time trajectory optimization algorithm.

[0048] The seventh step is to perform dilation processing on the obstacles in the map information of the current scene. In practice, the server can perform dilation processing on the obstacles in the map information of the current scene based on a geometric dilation algorithm.

[0049] Step 8: Construct a beacon point information set based on the initial target point information, target point information, and the aforementioned initial path information. The initial target point information represents the starting position (3D coordinates) of the main manipulator. The target point information represents the ending position (3D coordinates) of the main manipulator. The beacon point information in the beacon point information set represents the reference point between the initial target point information and the target point information. In practice, the server can construct the beacon point information set based on the initial target point information, target point information, and the aforementioned initial path information using a spline curve interpolation method. Here, the reference point is not specifically defined; it can be set according to actual needs.

[0050] Step nine: Based on the aforementioned beacon point information set, generate the running trajectory of the main manipulator as the target execution trajectory information. In practice, the server can use the B-spline interpolation algorithm to generate the running trajectory of the main manipulator as the target execution trajectory information based on the aforementioned beacon point information set.

[0051] The above-described technical solution, as an inventive point of this disclosure, solves technical problem three: "causing the robotic arm to jam or collide, resulting in significant positioning difficulty and time consumption." The reasons for these defects are as follows: When using a positioning system to locate epileptogenic foci, the small size, deep depth, and ambiguous boundaries of the epileptogenic foci, along with sub-millimeter-level positioning requirements, leave no room for trajectory deviation. This places extremely high demands on the precision of the robotic arm positioning, and there are numerous target points to be located with a short timeframe. Solving these factors can reduce the occurrence of jamming and collisions during robotic arm execution, lower the positioning difficulty, and shorten the positioning time. To achieve this effect, the intracranial target positioning assistance system of this disclosure generates an execution trajectory through initial path planning → global obstacle expansion → beacon point construction → B-spline trajectory generation. This generates a highly accurate, safe, smooth, and time-efficient robotic arm execution trajectory, thereby reducing the occurrence of jamming and collisions during robotic arm execution, lowering the positioning difficulty, and shortening the time consumption.

[0052] Step 203: The server stores the target execution trajectory information and the robotic arm modeling information into the target storage space included in the target positioning device.

[0053] In some embodiments, the server can store the target execution trajectory information and the robotic arm modeling information in the target storage space included in the target localization device. The target storage space can be a solid-state drive (SSD). The server and the target localization device can be connected via an industrial bus. In practice, the server can send the target execution trajectory information and the robotic arm modeling information to the target storage space included in the target localization device for storage processing. It should be noted that the generation of the robotic arm modeling information and the target execution trajectory information by the server occurs during the preparation phase, assuming no network failure. For example, the preparation phase can be a phase where data is stored in advance before target localization begins.

[0054] Optionally, the augmented reality device is configured to send target localization activation information to the target localization device in response to detecting a user's activation operation. The activation operation can be the user selecting a button to activate target localization. The method of selecting the button is not specifically limited; for example, it can be a click. The user can be a user performing target localization operations and wearing the augmented reality device. For example, the user could be a doctor.

[0055] Step 204: The target positioning device is used to control the main operating robotic arm to acquire the positioning tag in response to the detection of target positioning start information.

[0056] In some embodiments, in response to detecting target positioning activation information, the target positioning device can control the main robotic arm to acquire the positioning tag. The target positioning activation information can represent receiving a command from the AR glasses to activate the main robotic arm. The end effector of the main robotic arm integrates a soft silicone suction cup (0.8~1cm in diameter, Shore hardness 30°) and a force sensor (to control labeling pressure). The soft silicone suction cup has a micro-vent hole (0.1mm) at its center. The soft silicone suction cup is connected to a micro-vacuum pump. The end effector of the auxiliary robotic arm can integrate a disposable sterile silicone pressure head and a force sensor. Both the main robotic arm and the auxiliary robotic arm have three reflective markers. For example, the force sensor can be a micro-strain gauge force sensor. The positioning tag can be a sterile medical self-adhesive label. In practice, the main controller of the aforementioned target localization device can determine the trajectory of the main manipulator based on the current three-dimensional coordinates of the main manipulator and the three-dimensional coordinates of the plate where the positioning tag is placed, using the aforementioned B-spline interpolation algorithm. Then, it controls the main manipulator to move to the plate where the positioning tag is placed. Afterward, it activates the connected miniature vacuum pump, controlling a soft silicone suction cup to pick up a positioning tag. It should be noted that the positioning tag can be pre-placed in a sterile plate. The type of main controller is not specifically limited here and can be adjusted according to actual needs. For example, the main controller can be an industrial-grade embedded controller.

[0057] Step 205: The target positioning device is used to control the main operating robotic arm to move in response to the detection that the network environment meets the preset network fault conditions, based on the target execution trajectory information.

[0058] In some embodiments, in response to detecting that the network environment meets preset network fault conditions, the main robotic arm is controlled to move according to the target execution trajectory information. The preset network fault conditions can be a network failure at the current moment, for example, a network disconnection or a wireless signal RSSI below -80dBm. In practice, the main controller of the target positioning device can control the main robotic arm to move according to the target execution trajectory information. The connection between the AR glasses and the target positioning device can be a direct USB-C connection.

[0059] Step 206: The target location device is used to collect target environment feedback information at the current moment.

[0060] In some embodiments, the target localization device can collect target environment feedback information at the current moment. The target localization device may include an optical positioning sensor and a vision sensor. The main controller can control the optical positioning sensor to collect the three-dimensional coordinates of the main manipulator and the auxiliary manipulator in real time. Here, the type of optical positioning sensor is not specifically limited; for example, it can be a passive optical positioning sensor. In practice, the target localization device can determine the three-dimensional coordinates of the main manipulator and the auxiliary manipulator collected in real time by the optical positioning sensor as the target environment feedback information.

[0061] Step 207: The target location device is used to generate motion compensation information in response to determining that the target environment feedback information meets the preset correction conditions.

[0062] In some embodiments, the target positioning device is configured to generate motion compensation information based on the target environment feedback information in response to determining that the target environment feedback information meets preset correction conditions. The preset correction conditions can be that the difference between the position (three-dimensional coordinates) of the main manipulator in the target environment feedback information and a preset position is greater than a preset difference. Here, the preset position and the preset difference are not specifically limited and can be set according to actual needs. For example, the preset difference can be Δx = 0.2 mm, Δy = 0.1 mm, and Δz = 0.2 mm, and the preset position can be x: 2 mm, y: 5 mm, and z: 6 mm. The preset position can be the position (three-dimensional coordinates) in the target coordinate system where the positioning label needs to be affixed. The motion compensation information can characterize the trajectory of controlling the main manipulator to move to the preset position. In practice, the aforementioned target positioning device can use a cubic B-spline interpolation algorithm to generate the running trajectory of the aforementioned main operating robot as motion compensation information based on the real-time collected three-dimensional coordinates of the aforementioned main operating robot and the aforementioned preset position, which are included in the target environment feedback information.

[0063] Step 208: The target positioning device is used to control the main operating robotic arm to perform movement processing based on the movement compensation information.

[0064] In some embodiments, the target localization device is used to control the main manipulator to perform target localization processing based on the aforementioned motion compensation information. In practice, the target localization device can control the main manipulator to perform movement processing according to the aforementioned motion compensation information.

[0065] Optionally, the augmented reality device is configured to send target localization operation information to the target localization device in response to detecting a user's tag placement initiation operation. The tag placement initiation operation can be the user clicking a button to initiate tag placement.

[0066] Step 209: In response to the detection of target localization operation information, control the main operating robotic arm to perform target localization processing.

[0067] In some embodiments, in response to detecting target positioning operation information, the target positioning device controls the main robotic arm to perform target positioning processing. The target positioning operation information may represent a command sent by the AR glasses to begin affixing a positioning tag. In practice, the target positioning device can control the main robotic arm to affix the picked-up positioning tag according to a pre-set pressure value for affixing the positioning tag.

[0068] Step 210: The target positioning device is used to generate the running trajectory information of the corresponding auxiliary robotic arm in response to the detection that the target target environmental feedback information meets the preset auxiliary conditions, based on the auxiliary position information.

[0069] In some embodiments, in response to detecting that the target point environmental feedback information meets a preset auxiliary condition, trajectory information corresponding to the auxiliary robotic arm is generated based on the auxiliary position information. The preset auxiliary condition may be that the detection result of the positioning tag in the target image included in the target point environmental feedback information indicates that the positioning tag is not attached. The target image may represent the image of the positioning tag attached to the patient's head. The auxiliary position information may represent the three-dimensional coordinates of the positioning tag in the target image. In practice, firstly, the target point positioning device can control the vision sensor to acquire images of the current scene environment and the target image in real time as target point environmental feedback information. Then, the image of the positioning tag attached to the patient's head is detected using a sub-pixel edge detection algorithm and a gap grayscale analysis algorithm to obtain a detection result. The detection result may indicate that the positioning tag is not attached or that the positioning tag is attached. The absence of a positioning tag may indicate that the edge of the positioning tag is raised and not attached to the patient's head. In response to determining that the detection result indicates that the positioning tag is not attached, it is determined that the target point environmental feedback information meets the preset auxiliary condition. Then, the target localization device can use the PnP algorithm to determine the three-dimensional coordinates of the localization tag in the image of the localization tag attached to the patient's head. Afterwards, the target localization device can use a cubic B-spline interpolation algorithm to generate the corresponding trajectory information of the auxiliary robotic arm based on the real-time acquired three-dimensional coordinates of the auxiliary robotic arm and the three-dimensional coordinates of the localization tag, including the target environment feedback information.

[0070] Step 211: The target positioning device is used to control the auxiliary robotic arm to perform positioning compensation processing based on the running trajectory information.

[0071] In some embodiments, the target positioning device can be used to control the auxiliary robotic arm to perform positioning compensation processing based on the aforementioned trajectory information. In practice, the target positioning device can control the auxiliary robotic arm to move according to the aforementioned trajectory information. Then, the auxiliary robotic arm is controlled to press the affixed positioning label according to a preset pressure value to perform positioning compensation processing.

[0072] Optionally, the above-mentioned target localization device is configured to: The first step is to receive the marker coordinate information collected by the augmented reality device. This marker coordinate information represents the three-dimensional coordinates of each reflective marker affixed within the scanned current scene. The number of reflective markers is not specifically limited and can be adjusted according to actual needs. The affixed reflective markers may include each marker affixed to the edge of the operating table, three markers affixed to the main robotic arm, and three markers affixed to the auxiliary robotic arm. The number of reflective markers affixed to the edge of the operating table is not limited and can be set according to actual needs.

[0073] The second step is to generate marker coordinate mapping information based on the aforementioned marker point coordinate information. This marker coordinate mapping information can represent the transformation matrix between the coordinate system of the main manipulator and the coordinate system of the augmented reality device, and the transformation matrix between the coordinate system of the auxiliary manipulator and the coordinate system of the augmented reality device. The aforementioned main manipulator coordinate system represents the coordinate system of the main manipulator. The aforementioned auxiliary manipulator coordinate system represents the coordinate system of the auxiliary manipulator. In practice, the target localization device can generate marker coordinate mapping information based on the aforementioned marker point coordinate information using the SVD decomposition algorithm.

[0074] The third step is to store the above-mentioned marker coordinate mapping information into the target storage space.

[0075] In some optional implementations of some embodiments, the target localization device described above is configured to: The first step is to determine the target model information. This target model information can represent a three-dimensional model of the patient's intracranial target. In practice, the target localization device can determine the target model information using a deep learning segmentation algorithm based on pre-set intracranial target image data. This pre-set intracranial target image data can represent the image data of the intracranial target. For example, the image data can be a CT scan.

[0076] The second step involves truncating the aforementioned robotic arm modeling information and target model information to obtain truncated robotic arm modeling information and target model information. In practice, the aforementioned target positioning device can use a planar cutting method to truncate the aforementioned robotic arm modeling information and target model information to obtain truncated robotic arm modeling information and target model information.

[0077] The third step involves performing coordinate alignment processing on the aforementioned scene environment information, the aforementioned image target point location information, the processed robotic arm modeling information, and the processed target point model information to obtain coordinate alignment association information. This coordinate alignment association information can represent the 3×3 rotation matrix and 3×1 translation vector of the augmented reality device in the aforementioned target coordinate system. The aforementioned scene environment information may include the coordinates of reflective markers affixed to the edge of the operating table. In practice, the aforementioned target point localization device can use the PnP algorithm to perform coordinate alignment processing on the aforementioned scene environment information, the aforementioned image target point location information, the processed robotic arm modeling information, and the processed target point model information.

[0078] The fourth step is to send the aforementioned coordinate alignment information to the aforementioned augmented reality device.

[0079] In addressing the technical problems mentioned above, and considering the application scenario: when using a positioning system to locate epileptogenic foci, network malfunctions and the small, deep, and ambiguous boundaries of the epileptogenic foci often lead to the following technical problem: when directly extracting target features from medical images using a network, the large network latency and the small, ambiguous boundaries of the target make feature extraction difficult and unstable, resulting in low accuracy. This necessitates repeated changes to the positioning tags on the robotic arm, leading to longer operating times and greater wear and tear. Furthermore, the system's control over the robotic arm is unstable, resulting in frequent collisions due to erroneous operation. To meet the following requirements of this application scenario—adapting to the difficulty of target location, high accuracy requirements, and the small, ambiguous boundaries of the target—we have decided to adopt the following solution: In some optional implementations of some embodiments, the target localization device described above is configured to: The first step involves performing voxel segmentation on the extracted robotic arm modeling information and target model information to obtain voxel-segmented robotic arm modeling information and target model information. In practice, the target localization device can use the voxel segmentation algorithm of VoxelNet to perform voxel segmentation on the extracted robotic arm modeling information and target model information.

[0080] The second step involves grouping the voxel-based modeling information of the robotic arm and the target model after voxel segmentation to obtain semantic voxel clusters for the robotic arm and functional voxel clusters for the target. The semantic voxel clusters for the robotic arm can include voxel clusters for the end effector, the base, and the links. The functional voxel clusters for the target can include core voxel clusters, boundary voxel clusters, and surface projection voxel clusters. The core voxel clusters represent the essential region of the intracranial target (e.g., the core of an epileptogenic focus or tumor). The boundary voxel clusters represent the edge transition region of the intracranial target (e.g., the boundary of an epileptogenic focus or tumor capsule). The surface projection voxel clusters represent the vertical projection region of the intracranial target onto the scalp surface (the actual location of the label). In practice, the aforementioned target localization device can use the VoxelNet point cloud grouping algorithm based on semantic labels and geometric features to perform voxel grouping processing on the robotic arm modeling information and target model information after voxel block processing, to obtain the robotic arm semantic voxel cluster and the target functional voxel cluster.

[0081] The third step is to determine the numbering information for the corresponding semantic voxel clusters of the robotic arm and the functional voxel clusters of the target. This numbering information represents the number of each voxel cluster included in the semantic voxel cluster of the robotic arm and the number of each voxel cluster included in the functional voxel cluster of the target. In practice, the target positioning device can determine the numbers of each voxel cluster included in the semantic voxel cluster of the robotic arm and the number of each voxel cluster included in the functional voxel cluster of the target by retrieving the numbers from a preset set of numbering information. The preset numbering information in the preset set of numbering information represents the correspondence between voxel clusters and their numbers. For example, if the voxel cluster is the voxel cluster of the end effector, its corresponding number is 02.

[0082] The fourth step involves performing attribute association mapping on the aforementioned semantic volume cluster of the robotic arm and the aforementioned functional volume cluster of the target point to obtain an attribute association mapping table. This table characterizes the attribute correspondence between the semantic volume cluster of the robotic arm and the functional volume cluster of the target point. For example, the attribute correspondence could be the correspondence between the function of the end effector volume cluster of the master robotic arm in picking up or affixing positioning tags and the requirement for the target point surface projection volume cluster to affix positioning tags. In practice, the target positioning device can use a rule-based attribute mapping algorithm to perform attribute association mapping on the semantic volume cluster of the robotic arm and the functional volume cluster of the target point.

[0083] The fifth step involves performing spatial boundary verification on the aforementioned semantic volume cluster of the robotic arm and the aforementioned functional volume cluster of the target point to obtain the verification result information. In practice, the aforementioned target localization device can use the AABB axis-aligned bounding box algorithm to perform spatial boundary verification on the aforementioned semantic volume cluster of the robotic arm and the aforementioned functional volume cluster of the target point.

[0084] The sixth step involves encapsulating the aforementioned verification results in metadata to obtain the target geometry group. In practice, the target localization device can use an attribute-label-based voxel cluster encapsulation and fusion algorithm to merge the robotic arm semantic voxel cluster and the target functional voxel cluster.

[0085] Step 7: Perform scene segmentation processing on the above target geometry group to obtain the scene-segmented target geometry group. In practice, the above target localization device can perform scene segmentation processing on the above target geometry group based on semantic rule segmentation algorithms that consider surgical tasks and spatial constraints to obtain the scene-segmented target geometry group. For example, the semantic rule segmentation algorithm based on surgical tasks and spatial constraints can be the EndoARSS multi-task semantic segmentation algorithm.

[0086] The eighth step involves merging the target geometry groups after the scene segmentation process described above to obtain the target geometry. This target geometry can represent a single geometry with a unified structure and complete parameters after merging. In practice, the target localization device can use a voxel-level direct fusion algorithm to merge the target geometry groups after the scene segmentation process to obtain the target geometry.

[0087] The ninth step involves calibrating and associating the coordinate information of the target geometry with the target coordinate system. The coordinate information of the target geometry represents its three-dimensional coordinates within the AR glasses coordinate system. In practice, the target localization device can use an ICP algorithm based on bony landmarks (first performing coarse registration using bony landmarks, then iterating to the nearest point) to calibrate and associate the coordinate information of the target geometry with the target coordinate system. Specifically, the coarse registration using bony landmarks involves coarsely registering the origin, the coordinates of the bony prominence on the lateral side of the eyebrow, and the coordinates of the occipital prominence in the target coordinate system with the coordinates of the midpoint of the line connecting the external auditory canals, the coordinates of the bony prominence on the lateral side of the eyebrow, and the coordinates of the occipital prominence in the coordinate system of the augmented reality device.

[0088] The above-described technical solution, as an inventive point of this disclosure, solves technical problem four: "The robotic arm's operating time is long and its wear and tear is significant, and the system's control stability of the robotic arm is poor, resulting in frequent collisions due to malfunctions." The reasons for these defects are as follows: When using a network to directly extract target features from medical images of the target, the network latency is large, and the target's boundary features are vague and minute, making it difficult and unstable for the system to extract target features. Consequently, the accuracy of the extracted target features is low, requiring the robotic arm to repeatedly change the attached positioning tags, resulting in long robotic arm operating times and significant wear and tear. Furthermore, the system's control stability of the robotic arm is poor, leading to frequent collisions due to malfunctions. Solving these factors can reduce the need for repeated changes to the attached positioning tags, shorten the robotic arm's operating time, and reduce wear and tear. To achieve this effect, the intracranial target localization assistance system disclosed herein performs voxelization processing to improve the accuracy of target characterization and voxel-level fusion, thereby reducing data errors. Furthermore, the target data processing can be performed even in the event of a network failure, resulting in strong stability of the localization system. This improves the accuracy of extracted target features, thereby improving the accuracy of the attached localization tags. Ultimately, this reduces the need for the robotic arm to repeatedly change the attached localization tags, improves the stability of the robotic arm's operation, and reduces the likelihood of collisions with the robotic arm.

[0089] Step 212: The augmented reality device is used to display calibration point information.

[0090] In some embodiments, the augmented reality device described above can display calibration point information. This calibration point information can represent a preset number of white, semi-transparent dots. Each of these preset number of dots corresponds to coordinates. The transparency of these dots can be 70%. The specific number of these preset number is not limited; for example, it can be 5. In practice, for each of these preset number of dots, the augmented reality device can display the dot based on its corresponding coordinates. It should be noted that the aforementioned preset number of dots includes one dot displayed in the preset gaze area information, with coordinates (0, 0); one dot displayed on the left edge of the preset gaze area information, with coordinates (15°, 0); one dot displayed on the right edge of the preset gaze area information, with coordinates (-15°, 0); one dot displayed on the upper side of the auxiliary area information, with coordinates (0, 22.5°); and one dot displayed on the lower side of the auxiliary area information, with coordinates (0, -22.5°). The preset gaze area information represents the area that the user (doctor) wearing the AR glasses should gaze at, and is used to display the target core voxel cluster and the target boundary voxel cluster within the target functional voxel cluster. The auxiliary area information represents the preset auxiliary gaze area in the AR glasses, and is used to display the voxel cluster of the robotic arm end effector and the voxel cluster of the robotic arm linkage within the robotic arm semantic voxel cluster. The aforementioned edge region information can characterize the pre-defined edge gaze region of the AR glasses, and can be used to display the target surface projection voxel cluster and the voxel cluster of the robotic arm base. For example, the region with the center point of the AR glasses display area as the center point and a first preset distance as the radius is the preset gaze region information; the region between the region with the center point of the AR glasses display area as the center point and a second preset distance as the radius and the aforementioned preset gaze region information is the auxiliary region information; the region between the region with the center point of the AR glasses display area as the center point and the aforementioned auxiliary region information and the aforementioned auxiliary region information is the edge gaze region. The aforementioned third preset distance is greater than the aforementioned second preset distance. The aforementioned second preset distance is greater than the aforementioned first preset distance. Here, the division method of the aforementioned preset gaze region information, the aforementioned auxiliary region information, and the aforementioned edge region information, and the specific values ​​of the aforementioned first preset distance, second preset distance, and third preset distance are not specifically limited, and can be set according to actual conditions.

[0091] Optionally, in response to determining that the detected eye-tracking data meets preset calibration conditions, the augmented reality device can hide the calibration point information and render the preset gaze area information, auxiliary area information, and edge area information according to preset rendering method information. The preset rendering method information can characterize the rendering method of the preset gaze area information, the rendering method of the auxiliary area information, and the rendering method of the edge area information. In practice, the aforementioned augmented reality device calls the target core voxel cluster number and the target boundary voxel cluster number, and displays the target core voxel cluster and the target boundary voxel cluster in the preset gaze area information through the aforementioned preset rendering method information; calls the voxel cluster number of the robotic arm end effector and the voxel cluster number of the robotic arm link, and displays the voxel cluster number of the robotic arm end effector and the voxel cluster of the robotic arm link in the auxiliary area information through the aforementioned preset rendering method information; and calls the target surface projection voxel cluster number and the voxel cluster number of the robotic arm base, and displays the target surface projection voxel cluster and the voxel cluster of the robotic arm base in the edge area information through the aforementioned preset rendering method information. For example, the preset rendering method information could be: "The rendering method for the above preset gaze area information is: frame rate: 60fps (maximum, synchronized with the robotic arm control frequency); resolution: native maximum resolution (e.g., HoloLens 2's 2K resolution); rendering algorithm: ray tracing (high-precision scenes) / rasterization (real-time priority scenes). The rendering method for the above auxiliary area information could be: frame rate: 30fps (balancing precision and load); resolution: 75% of the native resolution (e.g., 1.5K); rendering algorithm: rasterization (reducing GPU load). The rendering method for the above edge area information could be: frame rate: 15~20fps (minimum, does not affect the overall experience); resolution: 50% of the native resolution (e.g., 1K); rendering algorithm: simplified rasterization (e.g., disabling anti-aliasing and shadows)."

[0092] In some alternative implementations of some embodiments, the augmented reality device described above is configured to: The first step involves identifying the pupil center gaze information and corneal reflector gaze information detected by the aforementioned infrared auxiliary component as eye-tracking data. Specifically, the pupil center gaze information characterizes the two-dimensional angular coordinates of the user's gaze point within the AR glasses coordinate system and the direction of their gaze. The corneal reflector gaze information characterizes the two-dimensional angular coordinates of the bright reflector point formed after the augmented reality device illuminates the cornea within the AR glasses coordinate system. The augmented reality device may include an infrared auxiliary component. This component may include an infrared emitting device and a miniature infrared camera. The infrared emitting device may be a near-infrared LED.

[0093] The second step involves determining the gaze point coordinates based on the aforementioned pupil center gaze information and corneal reflector gaze information. These gaze point coordinates characterize the angular coordinates of the user's gaze point within the AR glasses' coordinate system. In practice, the augmented reality device can use the pupil-corneal reflector vector method to determine the gaze point coordinates based on the pupil center gaze information and corneal reflector gaze information. It should be noted that the gaze point coordinates are generated by further updating the coordinates of the gaze point based on the aforementioned pupil center gaze information and corneal reflector gaze information.

[0094] The third step involves determining the calibration result information based on the coordinates of the calibration point information and the gaze point coordinates. This calibration result information indicates whether the gaze point coordinates and the coordinates of the calibration point information have been calibrated. The coordinates of the calibration point information represent the two-dimensional angular coordinates of the field of view in the AR glasses coordinate system corresponding to the calibration point information. The calibration result information can indicate whether the calibration is correct or incorrect. For example, the gaze point coordinates can be (0, 20°). In practice, the augmented reality device can use a least-squares rigid transformation fitting algorithm to determine the mean square error between the coordinates of the calibration point information and the gaze point coordinates. Then, in response to determining that the mean square error is less than or equal to a preset threshold, the calibration is determined to be correct as the calibration result information. In response to determining that the mean square error is greater than the preset threshold, the calibration is determined to be incorrect as the calibration result information. The specific value of the preset threshold is not limited and can be adjusted according to actual needs. For example, the preset threshold can be 0.3°.

[0095] Fourth, in response to determining that the above calibration result information indicates correct calibration, determine that the above eye-tracking data information meets preset calibration conditions. These preset calibration conditions can be the obtained calibration result information indicating correct calibration.

[0096] The above-described embodiments of this disclosure have the following beneficial effects: the intracranial target localization assistance system of some embodiments of this disclosure improves the stability of the localization system, reduces the long latency of the localization system, reduces the difficulty of localization, simplifies the localization steps, and thus shortens the time consumed. Specifically, the reasons for poor stability and long latency of the localization system, and the numerous and difficult steps in assembling and debugging the metal stereoscopic localization headframe, resulting in long time consumption, are as follows: when using a system that transmits localization commands via a network using a cloud processor or remote experts, the localization operation commands cannot be executed when the network fails, resulting in poor stability and long latency of the localization system; when using a metal stereoscopic localization headframe localization system, the assembly and debugging steps of the metal stereoscopic localization headframe are numerous and difficult, resulting in long time consumption; when using a control panel for localization, it is necessary to repeatedly look up to check the localization status and look down to use the control panel for localization, making the localization steps cumbersome and resulting in long localization time consumption. Based on this, in the intracranial target localization assistance system of some embodiments of this disclosure, firstly, the server is used to generate robotic arm modeling information based on the robotic arm point cloud information of the main robotic arm and the auxiliary robotic arm. Thus, a model of the robotic arm can be obtained. Then, based on the aforementioned robotic arm modeling information, scene environment information, and image target location information, target execution trajectory information is generated. Thus, the execution trajectory of the robotic arm can be obtained. Next, the aforementioned target execution trajectory information and the aforementioned robotic arm modeling information are stored in the target storage space included in the aforementioned target localization device. Thus, information can be stored to enable the localization processing of the aforementioned target localization device when offline. Afterwards, the aforementioned target localization device, in response to the detection of target localization initiation information, controls the aforementioned main operating robotic arm to acquire a localization tag. Thus, the localization tag can be acquired for localization. Then, in response to the detection that the network environment meets preset network fault conditions, the aforementioned main operating robotic arm is controlled to move according to the aforementioned target execution trajectory information. Thus, the robotic arm can be moved to the localization area. Next, target environment feedback information at the current moment is collected. Thus, the current target localization environment can be obtained. Afterwards, in response to determining that the aforementioned target environment feedback information meets preset correction conditions, motion compensation information is generated based on the aforementioned target environment feedback information and a pre-trained localization operation generation model. Thus, the compensation information required for the robotic arm can be obtained. Then, in response to determining that the target environment feedback information meets the preset correction conditions, motion compensation information is generated based on the target environment feedback information. From this, instructions for controlling the auxiliary robotic arm can be obtained. Next, based on the motion compensation information, the main robotic arm is controlled to perform motion processing. Thus, the main robotic arm can be controlled to move to correct its position. Next, in response to detecting target positioning operation information, the main robotic arm is controlled to perform target positioning processing.Therefore, positioning tags can be affixed for location tracking. Then, in response to the detection that the target point's environmental feedback information meets preset auxiliary conditions, the corresponding trajectory information of the auxiliary robotic arm is generated based on the auxiliary position information. This allows control of the robotic arm's movement. Finally, based on the trajectory information, the auxiliary robotic arm is controlled to perform positioning compensation processing. This allows control of the auxiliary robotic arm to press the tag. Furthermore, the augmented reality device is used to display calibration point information. This can be used to calibrate the user's gaze position. Because positioning is not achieved through cloud servers or remote experts transmitting data over a network, but rather by storing the target execution trajectory information and the robotic arm modeling information in the target point positioning device's storage space, the target point positioning device can still perform positioning processing using the stored information even in the event of a network failure. Also, because positioning is not achieved through a metal stereoscopic positioning head, but through a robotic arm, the assembly and debugging of the metal stereoscopic positioning head is eliminated, thus reducing the difficulty of positioning, simplifying the positioning steps, and shortening the positioning time. Furthermore, because positioning is achieved through augmented reality devices, users can directly wear these devices for positioning without repeatedly looking up to check the status or looking down to perform the operation, thus improving the convenience of positioning. This allows for timely rendering of the gaze area, improving rendering accuracy, and enabling users to issue positioning commands promptly, shortening the positioning system's processing time.

[0097] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An intracranial target localization assistance system, wherein, The intracranial target localization assistance system includes a server, a target localization device, and an augmented reality device. The target localization device includes a main robotic arm and an auxiliary robotic arm, wherein: The server is configured to generate robotic arm modeling information based on the robotic arm point cloud information of the main robotic arm and the auxiliary robotic arm; generate target execution trajectory information based on the robotic arm modeling information, scene environment information and image target position information; and store the target execution trajectory information and the robotic arm modeling information in the target storage space included in the target positioning device. The target positioning device is configured to: control the main robotic arm to acquire a positioning tag in response to detecting target positioning initiation information; control the main robotic arm to move according to the target execution trajectory information in response to detecting that the network environment meets preset network fault conditions; collect target environment feedback information at the current moment; generate movement compensation information according to the target environment feedback information in response to determining that the target environment feedback information meets preset correction conditions; control the main robotic arm to perform movement processing according to the movement compensation information; control the main robotic arm to perform target positioning processing in response to detecting target positioning operation information; generate running trajectory information corresponding to the auxiliary robotic arm according to the auxiliary position information in response to detecting that the target environment feedback information meets preset auxiliary conditions; and control the auxiliary robotic arm to perform positioning compensation processing according to the running trajectory information. The augmented reality device is used to display calibration point information.

2. The intracranial target localization assistance system according to claim 1, wherein, The augmented reality device is configured to: In response to determining that the detected eye-tracking data meets the preset calibration conditions, the calibration point information is hidden and the preset gaze area information, auxiliary area information, and edge area information are rendered according to the preset rendering method information.

3. The intracranial target localization assistance system according to claim 1, wherein, The target localization device is configured to: Receive the coordinate information of the marker points collected by the augmented reality device; Based on the coordinate information of the marked points, generate marked coordinate mapping information; The marker coordinate mapping information is stored in the target storage space.

4. The intracranial target localization assistance system according to claim 1, wherein, The target localization device is configured to: Determine target model information; The robotic arm modeling information and the target point model information are truncated to obtain truncated robotic arm modeling information and target point model information.

5. The intracranial target localization assistance system according to claim 4, wherein, The target localization device is configured to: The scene environment information, the image target point location information, the processed robotic arm modeling information, and the processed target point model information are subjected to coordinate alignment processing to obtain coordinate alignment association information; The coordinate alignment association information is sent to the augmented reality device.

6. The intracranial target localization assistance system according to claim 1, wherein, The augmented reality device includes an infrared assist component; and The augmented reality device is configured to: The pupil center fixation information and corneal reflector fixation information detected by the infrared auxiliary component are determined as eye-tracking data information; Based on the pupil center fixation information and the corneal reflector fixation information, determine the fixation point coordinate information; The calibration result information is determined based on the coordinate information of the calibration point and the coordinate information of the gaze point; In response to determining that the calibration result information indicates that the calibration is correct, it is determined that the eye-tracking data information meets the preset calibration conditions.

7. The intracranial target localization assistance system according to claim 1, wherein, The target localization device is configured to: Obtain the point cloud information of the robotic arms corresponding to the main operating robotic arm and the auxiliary robotic arm; The robotic arm point cloud information is preprocessed to obtain preprocessed robotic arm point cloud information as robotic arm point cloud information. Based on the semantic information of the robotic arms corresponding to the main operating robotic arm and the auxiliary robotic arm, a semantic model of the robotic arm is generated; The semantic model of the robotic arm is subjected to coordinate system transformation. The point cloud information of the robotic arm and the semantic model of the robotic arm after coordinate system transformation are fused to obtain the digital twin information of the robotic arm. Based on the digital twin information of the robotic arm, robotic arm modeling information corresponding to the main operating robotic arm and the auxiliary robotic arm is created.

Images