A pharmacy picking navigation method based on augmented reality

CN122841697APending Publication Date: 2026-09-29BAISE PEOPLES HOSPITAL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611135755.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-29
Publication Date
2026-09-29

AI Technical Summary

Technical Problem

[0003]现有增强现实技术通常侧重于将虚拟信息按照空间位置关系叠加至真实环境中,并随用户位置、视角及观察方向变化进行同步更新,在实际操作中更偏向完成虚实配准和显示同步,而对多个目标对象之间的空间关联、操作者注视意图和提示形态变化缺少细粒度处理;在药房货架场景中,药盒数量密集、外包装相似、摆放层级接近,仅依赖静态标签、箭头或固定框选提示时,多个虚拟提示容易在视野内相互遮挡,操作者需要在真实货架纹理和虚拟标记之间频繁切换注意力,造成目标判断时间延长;当操作者短暂扫视某一区域时,现有显示方式往往难以区分普通经过视线和稳定关注视线,容易出现提示过早触发或目标强调不足的问题,导致操作者在多药盒连续拣选时产生误判

Benefits of technology

[0014]与现有技术相比,本发明的优点和积极效果在于:

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122841697A_ABST
    Figure CN122841697A_ABST
Patent Text Reader

Abstract

The application relates to the technical field of augmented reality, in particular to a pharmacy picking navigation method based on augmented reality, which comprises the following steps: obtaining three-dimensional space coordinates of each target medicine box to be picked and mapping the three-dimensional space coordinates into space control nodes in a world coordinate system, calculating scalar values of each voxel point by taking each space control node as a potential source and superimposing to obtain a global scalar field, and constructing and generating a three-dimensional fluid fusion grid wrapping all the space control nodes based on the scalar sum of each voxel point in the global scalar field. According to the application, after the eye movement ray vector deviates from the independent three-dimensional boundary box and reaches a preset delay limit value, the independent three-dimensional boundary box is replaced by an independent fluid grid, and the local scalar field is driven to return to the global scalar field along a damping shrinkage track, so that the target prompt can be smoothly shrunk and recovered after losing attention, interface residues and misleading prompts are reduced, and the spatial continuity in the medicine picking process is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of augmented reality technology, and more particularly to an augmented reality-based method for pharmacy drug picking navigation. Background Technology

[0002] Augmented reality (AR) technology is a comprehensive technology that integrates multiple technologies such as computer vision, 3D graphics processing, spatial positioning, environmental perception, human-computer interaction, and real-time rendering. Its core is to overlay computationally generated virtual information onto the user's field of vision in real time according to spatial positional relationships based on the real physical environment. This ensures that the virtual objects maintain a consistent spatial registration relationship with the real scene and is updated synchronously as the user's position, viewpoint, and observation direction change, thereby achieving collaborative display and interaction between the real environment and virtual information.

[0003] Current augmented reality (AR) technologies typically focus on overlaying virtual information onto the real environment according to spatial relationships and updating synchronously with changes in user position, viewpoint, and observation direction. In practice, they tend to prioritize virtual-real registration and display synchronization, lacking fine-grained processing of spatial relationships between multiple target objects, operator gaze intent, and changes in the form of prompts. In pharmacy shelf scenarios, where medicine boxes are densely packed, have similar packaging, and are arranged at similar levels, relying solely on static labels, arrows, or fixed selection boxes can easily lead to multiple virtual prompts obscuring each other within the field of view. Operators need to frequently switch their attention between the real shelf texture and virtual markers, resulting in prolonged target judgment time. When operators briefly scan a certain area, existing display methods often struggle to distinguish between ordinary passing gaze and stable focused gaze, easily leading to premature prompt triggering or insufficient target emphasis, causing misjudgments when operators continuously pick up multiple medicine boxes. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an augmented reality-based pharmacy medication selection navigation method.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a pharmacy medication picking navigation method based on augmented reality, comprising the following steps: The three-dimensional spatial coordinates of each target medicine box to be picked are obtained and mapped to spatial control nodes in the world coordinate system. The scalar values ​​of each voxel point are calculated using each of the spatial control nodes as potential sources and superimposed to obtain a global scalar field. Based on the sum of the scalar values ​​of each voxel point in the global scalar field, a three-dimensional fluid fusion mesh that encloses all the spatial control nodes is constructed. The operator's eye-tracking ray vector is tracked in real time, and the first intersection point between the eye-tracking ray vector and the surface of the three-dimensional fluid fusion mesh is calculated. An influence sphere is constructed with the first intersection point as its center. The local isosurface limit within the influence sphere is lowered to re-extract the isosurface and generate a local bulge deformation mesh oriented toward the direction of the eye-tracking ray vector. The eye-tracking ray vector is continuously tracked. When the gaze duration reaches a preset trigger limit, the scalar field contribution of the target space control node is separated from the global scalar field and a local scalar field is established. The local scalar field and the global scalar field are driven to move away from each other in space and generate independent fluid meshes in real time. When it is determined that the intersection influence area of ​​the local scalar field and the global scalar field is zero, the independent fluid mesh is replaced with an independent three-dimensional bounding box indicating the position of the target medicine box.

[0006] Preferably, the method further includes: The eye-tracking ray vector is monitored. When it is determined that the vector deviates from the independent 3D bounding box and reaches a preset delay limit, the independent 3D bounding box is replaced with the independent fluid mesh, and the local scalar field is driven to move towards the global scalar field along a damped contraction trajectory. When the local scalar field and the global scalar field re-intersect spatially, the scalar contribution of the target spatial control node in the global scalar field is restored, and the original three-dimensional fluid fusion mesh is restored.

[0007] Preferably, the step of acquiring the space control node is as follows: Obtain the three-dimensional spatial coordinates of each target medicine box to be picked, read the horizontal, vertical and longitudinal coordinates of each target medicine box in the pharmacy shelf space, and uniformly transform the horizontal, vertical and longitudinal coordinates according to the coordinate origin, coordinate axis direction and scale reference of the world coordinate system. Establish a unique node number for each target medicine box after transformation, and bind the unique node number to the corresponding target medicine box to form a spatial control node in the world coordinate system.

[0008] Preferably, the step of obtaining the three-dimensional fluid fusion mesh is as follows: Each of the spatial control nodes is calibrated as a potential source. The horizontal, vertical, and longitudinal coordinates of each voxel point in the three-dimensional space are read one by one. The spatial distance of each voxel point relative to each of the spatial control nodes is calculated. According to the increasing relationship of spatial distance, a distance attenuation contribution is applied to each of the spatial control nodes. All distance attenuation contributions received by the same voxel point are superimposed item by item to obtain the global scalar field. The scalar sum of each voxel point in the global scalar field is extracted one by one. The scalar sum of each voxel point is compared with the preset global isosurface limit. The boundary positions of voxel points that reach the global isosurface limit are retained. The mesh vertex topology is constructed according to the connection relationship between the boundary positions of adjacent voxel points. The mesh vertex topology is closed into a continuous three-dimensional surface that wraps all the spatial control nodes to generate a three-dimensional fluid fusion mesh.

[0009] Preferably, the step of obtaining the influencing sphere is as follows: The operator's gaze origin and gaze direction are read in real time. The gaze origin is used as the starting position of the eye movement ray vector, and the gaze direction is used as the extension direction of the eye movement ray vector. The mesh patches on the surface of the three-dimensional fluid fusion mesh are detected segment by segment along the extension direction of the eye movement ray vector. The mesh patches that come into contact are selected in order of distance from the gaze origin from near to far. The spatial position of the first contact is located on the mesh patches that come into contact, and the first intersection point is obtained. The first intersection point is set as the center of the sphere. The preset influence radius and preset Gaussian weight decay attribute are read. A spatial coverage area is defined around the center of the sphere according to the preset influence radius. The distance from each position within the spatial coverage area to the center of the sphere is mapped to the corresponding weight decay degree, so that the weight decay degree is more obvious for positions farther away from the center of the sphere, thus forming an influence sphere.

[0010] Preferably, the step of obtaining the locally raised deformation mesh is as follows: By comparing the node coordinates of each target spatial control node with the spatial coverage of the influencing sphere, target spatial control nodes located within the influencing sphere are selected. The Euclidean distance from each target spatial control node to the center of the sphere is extracted. According to the Gaussian weight decay attribute corresponding to the Euclidean distance, the isosurface limit of the local region within the influencing sphere is dynamically lowered. The position of the local isosurface is then re-determined according to the lowered isosurface limit, causing the local isosurface to form an outward expansion deformation along the direction of the eye-tracking ray vector, generating a local bulging deformation mesh.

[0011] Preferably, the step of obtaining the local scalar field is as follows: The eye-tracking ray vector is continuously tracked, and the start time and current gaze time of the continuous gaze pointing to the region corresponding to the target spatial control node are recorded. The gaze dwell time between the start time and the current gaze time is calculated, and the gaze dwell time is compared with a preset trigger limit. If the gaze dwell time does not reach the preset trigger limit, the current gaze time is recorded. If the gaze dwell time reaches the preset trigger limit, the target spatial control node is locked, and the scalar field contribution corresponding to the target spatial control node is extracted from the global scalar field. The scalar field contribution corresponding to the target spatial control node is separated from the global scalar field to establish a local scalar field.

[0012] Preferably, the step of obtaining the independent 3D bounding box is as follows: Read the current node coordinates of the target space control node in the world coordinate system, extract the line of sight of the eye-tracking ray vector, reverse the line of sight as the translation traction direction, apply translation traction force to the target space control node according to the translation traction direction, so that the target space control node drives the local scalar field away from the global scalar field frame by frame, update the spatial position of the local scalar field frame by frame, and generate an independent fluid mesh with droplet peeling morphology based on the updated local scalar field; The intersection influence area of ​​the local scalar field and the global scalar field in three-dimensional space is detected frame by frame. The number of voxel points still affected by the common scalar contribution within the intersection influence area is counted. If the number of voxel points is not zero, the local scalar field continues to move along the translation traction direction. If the number of voxel points is zero, it is determined that the local scalar field has been stripped. The target pillbox position corresponding to the stripped independent fluid mesh is read. The bounding box center, length, width and height dimensions and the orientation of the bounding box are determined according to the target pillbox position. The stripped independent fluid mesh is replaced with an independent three-dimensional bounding box indicating the target pillbox position.

[0013] Preferably, the specific steps for seamlessly restoring the original three-dimensional fluid fusion mesh are as follows: The eye-tracking ray vector is monitored, and the gaze origin and gaze direction of the eye-tracking ray vector are read frame by frame. The gaze direction extension line is projected onto the outer surface area of ​​the independent 3D bounding box. It is detected whether the gaze direction extension line passes through the outer surface area of ​​the independent 3D bounding box. If the gaze direction extension line passes through the outer surface area of ​​the independent 3D bounding box, the deviation timing value is reset to zero. If the gaze direction extension line does not pass through the outer surface area of ​​the independent 3D bounding box, the deviation timing value is continuously accumulated. The deviation timing value is compared with a preset delay limit. When the deviation timing value reaches the preset delay limit, the independent 3D bounding box is replaced with the independent fluid mesh. The current node coordinates of the target space control node in the independent fluid grid are read, and the initial coordinates of the target space control node in the three-dimensional fluid fusion grid are read. The direction from the current node coordinates to the initial coordinates is determined as the return direction. The frame-by-frame displacement of the target space control node is calculated according to the return direction. The frame-by-frame displacement is corrected by decreasing the elastic rebound strength and the damping attenuation strength, so that the return speed of the target space control node decreases frame by frame as it approaches the initial coordinates, forming a damped contraction trajectory. The local scalar field is driven to move frame by frame toward the global scalar field according to the damped contraction trajectory. Frame by frame, it is checked whether there are overlapping voxel points in the voxel point ranges corresponding to the local scalar field and the global scalar field. If there are no overlapping voxel points, the local scalar field continues to move. If there are overlapping voxel points, the scalar contribution of the target space control node in the global scalar field is restored, the isosurface limit of the local region is restored to the initial global isosurface limit, the fused scalar field distribution after the target space control node returns is recalculated, and the continuous closed mesh surface is reconstructed based on the fused scalar field distribution, restoring it to the initial three-dimensional fluid fused mesh.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This invention maps the three-dimensional spatial coordinates of the target medicine box to spatial control nodes, and uses these spatial control nodes as potential sources to form a global scalar field. This allows multiple discrete medicine box positions to no longer be displayed as isolated individual marker points, but rather integrated into a continuous three-dimensional fluid fusion mesh. This creates a comprehensive, holistic visual navigation range in a real pharmacy space, reducing the burden on operators who repeatedly identify scattered prompts in multi-target medicine picking scenarios. By tracking eye-tracking ray vectors in real time, the invention locates the first intersection point between the eye-tracking ray vector and the surface of the three-dimensional fluid fusion mesh, and constructs an influence sphere around this first intersection point. The local isosurface limits are dynamically lowered, causing the operator's gaze area to generate a locally bulging deformation mesh oriented towards the line of sight. This transforms visual attention into spatial deformation feedback, improving the visibility of the target area against complex shelf backgrounds. After the gaze dwell time reaches a preset trigger limit, the scalar field contribution of the target space control node is separated, a local scalar field is established, and the local scalar field is driven away from the global scalar field to generate an independent fluid mesh. This allows the target of interest to be gradually separated from the overall navigation form. Furthermore, when the intersection influence area is zero, it is replaced with an independent three-dimensional bounding box, realizing a continuous transition from group navigation, attention confirmation, target prominence to position locking, avoiding sudden changes in navigation prompts that cause interruption of understanding. After the eye-tracking ray vector deviates from the independent three-dimensional bounding box and reaches a preset delay limit, the independent three-dimensional bounding box is replaced with an independent fluid mesh again, and the local scalar field is driven to return to the global scalar field along a damped contraction trajectory. This allows the target prompt to smoothly contract and recover after losing attention, reducing interface residue and misleading prompts, and improving the spatial continuity in the drug picking process. Attached Figure Description

[0015] Figure 1 A graph showing the relationship between the distance attenuation contribution of spatial control nodes and the global isosurface limit. Figure 2 This is a diagram showing the change in the number of voxels in the region affected by the intersection of the local and global scalar fields. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] Please see Figure 1-2 This invention provides a technical solution: a pharmacy medication picking navigation method based on augmented reality, comprising the following steps: The three-dimensional spatial coordinates of each target medicine box to be picked are obtained and mapped to spatial control nodes in the world coordinate system. The scalar values ​​of each voxel point are calculated using each spatial control node as a potential source and superimposed to obtain a global scalar field. Based on the sum of the scalar values ​​of each voxel point in the global scalar field, a three-dimensional fluid fusion mesh that wraps all spatial control nodes is constructed. The operator's eye-tracking ray vector is tracked in real time, and the first intersection point between the eye-tracking ray vector and the surface of the 3D fluid fusion mesh is calculated. An influence sphere is constructed with the first intersection point as the center. The local isosurface limit within the influence sphere is lowered to re-extract the isosurface and generate a local bulge deformation mesh oriented towards the direction of the eye-tracking ray vector. Continuously track eye-tracking ray vectors. When the gaze duration reaches a preset trigger limit, separate the scalar field contribution of the target space control node from the global scalar field and establish a local scalar field. Drive the local scalar field and the global scalar field to move away from each other in space and generate independent fluid meshes in real time. When it is determined that the intersection influence area of ​​the local scalar field and the global scalar field is zero, replace the independent fluid mesh with an independent three-dimensional bounding box indicating the position of the target pillbox. Monitor the eye-tracking ray vector. When it is determined that the deviation from the independent 3D bounding box has reached the preset delay limit, replace the independent 3D bounding box with an independent fluid mesh. Drive the local scalar field to move towards the global scalar field along the damped contraction trajectory. When the local scalar field and the global scalar field re-intersect, restore the scalar contribution of the target spatial control node in the global scalar field and restore it to the initial 3D fluid fusion mesh.

[0018] The steps for obtaining space control nodes are as follows: Obtain the three-dimensional spatial coordinates of each target medicine box to be picked, read the horizontal, vertical and longitudinal coordinates of each target medicine box in the pharmacy shelf space, and uniformly transform the horizontal, vertical and longitudinal coordinates according to the coordinate origin, coordinate axis direction and scale reference of the world coordinate system. Establish a unique node number for each target medicine box after transformation, and bind the unique node number to the corresponding target medicine box to form a spatial control node in the world coordinate system.

[0019] Specifically, to obtain the three-dimensional spatial coordinates of each target medicine box to be picked, the pharmacy management system interface is used to batch retrieve the picking task orders. The inventory location code of each medicine box within the task order is parsed. This code uniquely corresponds to its physical location on the shelf. For example, code A-03-05-12 represents shelf A, column 3, layer 5, slot 12. Based on pre-mapped and stored 3D model data of the pharmacy shelves, this location code is converted into coordinate values ​​(x_shelf, y_shelf, z_shelf) in the shelf coordinate system. The horizontal, vertical, and z-axis coordinates represent the left / right, high / low, and deep positions of the medicine box's center point within the shelf, respectively. Then, a coordinate system transformation is performed to unify all coordinate points from the shelf coordinate system to the world coordinate system. This process uses a 4x4 homogeneous transformation matrix. To achieve this, the matrix is ​​determined during the initialization of augmented reality applications by calibrating at least three physical markers that are not on the same straight line. It contains rotation and translation information from the shelf coordinate system to the world coordinate system. The origin of the world coordinate system is usually set at the center of the ground at the pharmacy entrance, with its X-axis pointing along the length of the pharmacy, its Y-axis pointing upwards, and its Z-axis pointing along the width of the pharmacy. The scale is uniformly set to meters. After transformation, the coordinates in the world coordinate system are obtained as (x_world, y_world, z_world). Subsequently, a unique node number is generated for each transformed target medicine box coordinate. For example, a Universal Unique Identifier (UUID) generation algorithm is used to generate a 128-bit unique string for each coordinate point. This unique node number is then bound to the corresponding medicine box information (such as drug name, specifications, batch number) and its world coordinates in a data structure. For example, a JSON object is constructed, which contains fields such as id, name, batch, and position:[x_world, y_world, z_world], forming a spatial control node in the world coordinate system.

[0020] The steps for obtaining a 3D fluid fusion mesh are as follows: Each spatial control node is calibrated as a potential source. The horizontal, vertical, and longitudinal coordinates of each voxel point in the three-dimensional space are read one by one. The spatial distance of each voxel point relative to each spatial control node is calculated. According to the increasing relationship of spatial distance, a distance attenuation contribution is applied to each spatial control node. All distance attenuation contributions received by the same voxel point are superimposed one by one to obtain the global scalar field. Extract the scalar sum of each voxel in the global scalar field one by one, compare the scalar sum of each voxel with the preset global isosurface limit item by item, retain the boundary positions of voxel points that reach the global isosurface limit, construct the mesh vertex topology according to the connection relationship between the boundary positions of adjacent voxel points, close the mesh vertex topology into a continuous three-dimensional surface that wraps all spatial control nodes, and generate a three-dimensional fluid fusion mesh.

[0021] Specifically, each spatial control node is designated as a potential source. First, the three-dimensional space within the augmented reality device's field of view is discretized to construct a uniform voxel grid. For example, a 10m x 5m x 3m space is divided into cubic voxels with sides of 1cm, resulting in a set containing 15 million voxel points. Then, the center three-dimensional coordinates of each voxel point in this three-dimensional space are read one by one, denoted as... For each voxel, calculate its relationship with all voxels. Space control nodes Euclidean distance This involves calculating the straight-line distance from each voxel to each potential source, and then applying a distance attenuation contribution to each spatial control node based on the increasing spatial distance relationship. A Gaussian function is used as the attenuation model, and the scalar contribution value generated by each spatial control node to the voxel is... ,in, This is the fundamental potential field strength, which is a preset constant, for example, set to 100, to ensure a sufficient range of scalar values. The radius of influence determines the rate at which the potential field decays. For example, setting it to 0.2 meters means that at a distance of 0.2 meters from the node, the field strength decays to approximately 60%. Then, the field strength received by the same voxel from all... The total scalar sum of the distance attenuation contributions of each spatial control node is calculated by linearly superimposing them one by one. After performing this calculation on all voxel points, a smooth and continuous scalar field covering the entire picking area is formed, resulting in the global scalar field.

[0022] The scalar sums of each voxel point in the global scalar field are extracted one by one. The moving cube algorithm is used to generate a 3D mesh model from the voxel data. First, the scalar sum of each voxel point is bounded to a preset global isosurface value. A point-by-point comparison is performed. This limit is a key parameter, and its setting is based on a specific percentage of the average scalar values ​​at all spatial control node locations. For example, if there are 5 nodes with scalar values ​​of 98, 99, 100, 97, and 99, and an average value of 98.6, then this limit can be set. It is 30% of that average, that is The purpose of setting this limit is to generate an initial merged shape that can smoothly wrap all nodes and has a full form. If the limit is too high, the mesh will be broken or too small; if the limit is too low, the mesh will be too bloated and lose details. During the comparison process, the algorithm moves in the entire scalar field in units of 2x2x2 voxel cubes, checking the 8 vertices of each cube, and determining whether the scalar value of each vertex is higher or lower than the limit. An 8-bit index value (range 0-255) is generated for the cube. Then, the mesh vertex topology is constructed according to the connection relationships between the boundary positions of adjacent voxel points. Specifically, a pre-computed lookup table containing 256 entries is used. This table defines one or more sets of triangle facet vertex connections for each index value. These vertices are located on the edges of the cube, and their positions are determined by the scalar values ​​of the voxel points at the two ends of the edge. Linear interpolation calculations are performed, and then all the triangular facets generated by the cubes are seamlessly stitched together to topologically close the mesh vertices into a continuous three-dimensional surface that encloses all spatial control nodes, thus generating a three-dimensional fluid fusion mesh.

[0023] The steps that affect obtaining the sphere are: The system reads the operator's gaze origin and gaze direction in real time. The gaze origin is used as the starting position of the eye movement ray vector, and the gaze direction is used as the extension direction of the eye movement ray vector. The mesh patches on the surface of the three-dimensional fluid fusion mesh are detected segment by segment along the extension direction of the eye movement ray vector. The mesh patches that come into contact are selected in order of distance from the gaze origin from near to far. The spatial position of the first contact is located on the mesh patches that come into contact, and the first intersection point is obtained. The first intersection point is set as the center of the sphere. The preset influence radius and preset Gaussian weight decay attribute are read. A spatial coverage range is defined around the center of the sphere according to the preset influence radius. The distance from each position within the spatial coverage range to the center of the sphere is mapped to the corresponding weight decay degree, so that the weight decay degree is more obvious for positions farther away from the center of the sphere, thus forming an influence sphere.

[0024] Specifically, the system reads the operator's gaze origin and gaze direction in real time. Raw eye-tracking data is acquired at a frequency of 90Hz through an eye-tracking module integrated into the augmented reality glasses. The data includes the three-dimensional coordinates of the gaze origin representing the eye position. And the unit vector representing the direction of the line of sight. By taking the gaze origin as the starting position of the eye-tracking ray vector and the gaze direction as the extension direction of the eye-tracking ray vector, a mathematical three-dimensional ray is formed. ,in Then, the mesh patches on the surface of the 3D fluid fusion mesh are detected segment by segment along the extension direction of the eye-tracking ray vector. This process uses the Möller–Trumbore algorithm. To improve efficiency, an octree is first used to remove a large number of mesh parts that cannot intersect. Intersection tests are only performed on candidate triangular patches near the ray path. Mesh patches that have contact are selected in order of distance from the gaze origin from near to far. Specifically, for each triangular patch that has been found to intersect, the ray parameters corresponding to its intersection point are calculated. Value, that The value is proportional to the distance from the intersection point to the origin of the gaze point, and the smallest positive value among all intersection results is recorded. The one that, and locate the spatial position where the first contact occurs on the mesh surface where the contact occurs, that position is the The first intersection point is obtained.

[0025] Set the first intersection point as the center of the sphere, and read the preset influence radius and preset Gaussian weight decay attributes. It is an empirical value set based on the visual range of the human eye's fovea and interaction comfort, for example, set to 15 centimeters, to ensure that the user's gaze area can cover one or more nearby pillbox targets. The preset Gaussian weight decay attribute defines the functional form that affects the weight distribution within the sphere. A spherical virtual space coverage area is delineated around the center of the sphere according to a preset influence radius. Any point within this range... Each location within the spatial coverage area will be assigned a weight value, and the distance from each location to the center of the sphere will be mapped to the corresponding weight decay level. Specifically, the calculation... To the center of the ball distance ,like If the weight is 0, then the weight is 0. Then its weight Calculated using the Gaussian function: ,in It is the standard deviation of the Gaussian function, which controls the severity of the decay, and is usually set as a part of the radius of influence, for example... The weight is set to centimeters, so that the weight attenuation is more obvious the farther away from the center of the sphere. The weight is 1 at the center of the sphere and close to 0 at the boundary of the sphere. This forms a soft influence area with the influence intensity smoothly decreasing from the inside to the outside, centered on the gaze point, thus forming the influence sphere.

[0026] The steps for obtaining the mesh for localized bulge deformation are as follows: By comparing the node coordinates of each target spatial control node with the spatial coverage of the influencing sphere, target spatial control nodes located within the influencing sphere are selected. The Euclidean distance from each target spatial control node to the center of the sphere is extracted. According to the Gaussian weight decay attribute corresponding to the Euclidean distance, the isosurface limit of the local area within the influencing sphere is dynamically lowered. The position of the local isosurface is then redefined according to the lowered isosurface limit, so that the local isosurface forms an outward expansion deformation along the direction of the eye-tracking ray vector, generating a local bulging deformation mesh.

[0027] Specifically, the node coordinates of each target space control node and the spatial coverage area affecting the sphere are compared one by one, traversing all... Coordinates of each spatial control node Calculate the distance from each node to the center of the sphere. distance and the radius of influence (For example, 15 cm) Comparison, if If so, the node is selected as the target space control node located within the influence sphere, and then for each voxel point within the influence sphere... Extract its Euclidean distance to the center of the sphere. And calculate its weight based on this distance. Based on this weight, the isosurface limits affecting local areas within the sphere are dynamically adjusted downwards, resulting in new local limits. The calculation method is as follows ,in It is the global isosurface limit (e.g., 29.58). This is the maximum reduction amount, which determines the maximum extent of the bulge. It can be set to 50% of the global limit, i.e. and in accordance with the reduced equivalent surface limit The location of the local isosurface is redefined due to the weight. The presence of this feature means that the closer a voxel is to the gaze point, the greater the reduction in its isosurface limit. This causes the isosurface to need to move further outward to meet the new limit conditions, resulting in a smooth outward deformation of the local isosurface along the direction of the eye-tracking ray vector. The moving cube algorithm is then re-run to generate the mesh of the affected area and stitched with the unaffected mesh to generate a locally raised deformed mesh.

[0028] The steps to obtain a local scalar field are as follows: The eye-tracking ray vector is continuously tracked, and the start time and current gaze time of the continuous gaze pointing to the region corresponding to the target space control node are recorded. The gaze dwell time between the start time and the current gaze time is calculated, and the gaze dwell time is compared with a preset trigger limit one by one. If the gaze dwell time does not reach the preset trigger limit, the current gaze time is recorded. If the gaze dwell time reaches the preset trigger limit, the target space control node is locked, and the scalar field contribution corresponding to the target space control node is extracted from the global scalar field. The scalar field contribution corresponding to the target space control node is separated from the global scalar field to establish a local scalar field.

[0029] Specifically, the eye-tracking ray vector is continuously tracked, and the gaze point position returned by the eye-tracking module for each frame (e.g., every 1 / 90th of a second) is used to determine whether the gaze point falls on a target space control node. Within the localized bulge deformation mesh region caused by the fixation point, the system timestamp at the moment the fixation point first enters the region is recorded as the start time of continuous fixation. And continue to record the current gaze moment in subsequent frames. The time difference between the two is calculated as the fixation duration. The calculated fixation duration Same as preset trigger limit Successive comparisons are performed. This trigger limit is designed to distinguish between unconscious squinting and intentional staring. A typical value is 800 milliseconds. This value effectively avoids false triggers caused by rapid eye movement while ensuring responsiveness of the interaction. (Regarding gaze duration...) If 800 milliseconds have not been reached, the current gaze duration will continue to be updated and recorded. If the user's gaze moves out of the area, the recording will be reset. If the gaze lingers for a period of time If the time reaches or exceeds 800 milliseconds, the user's intention is determined to have been achieved, and the target space control node corresponding to the currently gazed area is immediately locked. Extract the scalar field contribution corresponding to the target space control node from the global scalar field. The specific operation involves, at the data level, converting the global scalar field... Updated to Simultaneously, the scalar field contribution corresponding to the target space control node will be... Separate from the global scalar field, establish an independent field data structure that can be spatially transformed independently, and build a local scalar field.

[0030] The steps to obtain an independent 3D bounding box are as follows: Read the current node coordinates of the target space control node in the world coordinate system, extract the line of sight direction of the eye-tracking ray vector, reverse the line of sight direction as the translation traction direction, apply translation traction force to the target space control node according to the translation traction direction, so that the target space control node drives the local scalar field away from the global scalar field frame by frame, update the spatial position of the local scalar field frame by frame, and generate an independent fluid mesh with droplet peeling morphology based on the updated local scalar field. The system detects the intersection influence region of the local scalar field and the global scalar field in 3D space frame by frame, counts the number of voxels still affected by the common scalar contribution within the intersection influence region, and continues to move the local scalar field along the translation traction direction if the number of voxels is not zero. If the number of voxels is zero, it is determined that the local scalar field has been stripped. The target pillbox position corresponding to the stripped independent fluid mesh is read, and the bounding box center, length, width and height dimensions and orientation are determined according to the target pillbox position. The stripped independent fluid mesh is replaced with an independent 3D bounding box indicating the target pillbox position.

[0031] Specifically, read the current node coordinates of the target space control node in the world coordinate system. Simultaneously, the eye-tracking module obtains the gaze direction of the operator's eye-tracking ray vector in the current frame. By reversing the unit vector of this line of sight, the translational traction direction can be obtained. The direction is roughly from the target medicine box towards the operator's eyes, following the direction of translational traction. A virtual translational traction force is applied to the target space control node. This traction force does not follow Newtonian mechanics but instead drives a frame-by-frame position update process, causing the target space control node to move its bound local scalar field at a constant speed (e.g., 0.5 meters per second) along... Directional movement, that is, updating its position in each frame: ,in It's speed. The frame interval is used to update the spatial position of the local scalar field frame by frame. Based on the updated local scalar field, the moving cube algorithm is reapplied in each frame, and a low, independent isosurface limit (e.g., 10% of the global limit) is used to generate a mesh that tightly wraps around the moving node. As the scalar field moves away from the main field, the superposition area of ​​its contribution from other fields gradually decreases. The generated mesh will visually present a shape similar to liquid being stretched, necked, and eventually broken apart from the main droplet, generating an independent fluid mesh with a droplet peeling shape.

[0032] The system detects the intersection and influence region of the local and global scalar fields in 3D space frame by frame. This detection process is performed at the voxel level. For the local scalar field, its influence range is the set of all voxels whose index value is greater than a minimum value (e.g., 0.01), denoted as […]. Similarly, the influence range of the global scalar field is denoted as... Calculate the intersection of two voxel sets. Then, count the number of voxel points within the intersection influence region that are still affected by the common scalar contribution, i.e., the set. If the number of voxel points is not zero, it means that the two scalar fields still overlap in space, and the meshes they generate may be visually connected or very close. In this case, the local scalar field continues to move along the translation traction direction in the next frame, and the intersection detection is repeated. If the number of voxel points decreases to zero, it is determined that the local scalar field has been completely separated from the global scalar field in terms of energy distribution, and the stripping process is completed. At this time, the size information (length, width, height) of the target medicine box corresponding to the stripped independent fluid mesh in the pharmacy management system is immediately read, and the independent fluid mesh at this moment is used as the basis for the determination. The center of the bounding box is used as the center. At the same time, in order to make the orientation of the bounding box more intuitive, one of its principal axes (such as the Y-axis) can be aligned with the vertical axis of the world coordinate system, and the other principal axis (such as the X-axis) can be aligned with the projection of the direction from the center of the bounding box to the operator's eyes on the horizontal plane. This determines the final orientation of the bounding box. The stripped-out independent fluid mesh is smoothly faded out, and at the same time, a three-dimensional cube with clear edges and semi-transparent surfaces generated according to the above parameters is faded in at its original position. The stripped-out independent fluid mesh is replaced with an independent three-dimensional bounding box indicating the location of the target medicine box.

[0033] The specific steps for seamlessly restoring the initial 3D fluid fusion mesh are as follows: The eye-tracking ray vector is monitored, and the gaze origin and gaze direction are read frame by frame. The gaze direction extension line is projected onto the outer surface area of ​​the independent 3D bounding box. It is detected whether the gaze direction extension line passes through the outer surface area of ​​the independent 3D bounding box. If the gaze direction extension line passes through the outer surface area of ​​the independent 3D bounding box, the deviation timing value is reset to zero. If the gaze direction extension line does not pass through the outer surface area of ​​the independent 3D bounding box, the deviation timing value is continuously accumulated. The deviation timing value is compared with the preset delay limit. When the deviation timing value reaches the preset delay limit, the independent 3D bounding box is replaced with an independent fluid mesh. Read the current node coordinates of the target space control node in the independent fluid grid, read the initial coordinates of the target space control node in the three-dimensional fluid fusion grid, determine the direction from the current node coordinates to the initial coordinates as the regression direction, calculate the frame-by-frame displacement of the target space control node according to the regression direction, and make the frame-by-frame displacement a decreasing correction according to the elastic rebound strength and damping attenuation strength, so that the regression speed of the target space control node decreases frame by frame as it approaches the initial coordinates, forming a damped contraction trajectory; The local scalar field is driven to move frame by frame toward the global scalar field according to the damped contraction trajectory. Frame by frame, it is checked whether there are overlapping voxel points in the corresponding voxel point ranges of the local scalar field and the corresponding voxel point ranges of the global scalar field. If there are no overlapping voxel points, the local scalar field continues to move. If there are overlapping voxel points, the scalar contribution of the target space control node in the global scalar field is restored, the isosurface limit of the local region is restored to the initial global isosurface limit, the fused scalar field distribution after the target space control node returns is recalculated, and the continuous closed mesh surface is reconstructed based on the fused scalar field distribution, restoring it to the initial three-dimensional fluid fused mesh.

[0034] Specifically, eye-tracking ray vectors are monitored, and the gaze origin of the eye-tracking ray vectors is read frame by frame in each frame. and gaze direction Constructing gaze rays The ray is then intersected with the generated independent 3D bounding boxes. This intersecting detection is performed by calculating the intersection points of the ray with the six planes constituting the bounding box. The system also checks whether the gaze-direction extension line passes through the outer surface region of the independent 3D bounding box. If the calculation results show that the ray intersects the bounding box at any point, meaning the user's gaze is focused on the target pillbox, then a deviation timing value is reset to zero. If the calculation results show that the ray does not intersect the bounding box at any point, meaning the user's gaze has moved away, then the deviation timing value is started or continuously accumulated, with the accumulation amount being the interval time per frame (e.g., 1 / 90 second). This deviation timing value is then compared with a preset delay limit. The limit is designed to provide users with a brief window to "change their minds" or reconfirm, preventing the immediate triggering of the rollback animation when the user's gaze is briefly removed, which could cause interaction jitter. It can be set to 1.5 seconds based on the user experience. When the deviation from the timing value continues to accumulate and reaches 1.5 seconds, the recovery process is triggered. The currently displayed independent 3D bounding box is immediately faded out with a fade-out animation effect and replaced with the independent fluid mesh that was in the same position before the animation started, serving as the starting point for the subsequent rollback animation.

[0035] Read the current node coordinates of the target space control node in the independent fluid grid. And read the initial coordinates of the node in the unseparated 3D fluid fusion mesh from the initial configuration. The direction vector that points the current node's coordinates to the initial coordinates. Determine the regression direction and calculate the total regression distance. The frame-by-frame displacement of the target space control node is calculated according to the regression direction. To achieve a smooth damping effect, uniform motion is not used; instead, a physical model based on damped vibration is employed for displacement calculation. The frame-by-frame displacement is progressively corrected according to the elastic rebound strength and damping attenuation strength. For example, the regression process can be likened to a spring-damped system, where the displacement is calculated in the first frame... Frame position It can be obtained through iterative calculation: And speed The update follows ,in It is the frame interval. It is the elastic modulus, which determines the "tensile force" of the rebound. For example, let's set it to 50. It is the damping coefficient, which determines how fast the speed decays. For example, if it is set to 15, by adjusting the ratio of these two coefficients, the return speed of the target space control node can be made faster at the beginning. As it gets closer to the initial coordinates, the speed will decrease smoothly frame by frame due to the reduction of "tension" and the damping effect, and finally reach the initial position at a speed close to zero, forming a seamless damped contraction trajectory.

[0036] The local scalar field is driven to move frame-by-frame towards the global scalar field along a damped contraction trajectory. Specifically, in each frame, the position of the local scalar field center (i.e., the target spatial control node) is updated, and its scalar distribution in the voxel grid is recalculated. Simultaneously, frame-by-frame, it is checked whether there are overlapping voxel points in the influence ranges of the corresponding voxel points in the local and global scalar fields. This detection is achieved by calculating whether the intersection of the voxel sets influenced by the two scalar fields is an empty set. If no overlapping voxel points are found, it indicates that the two fields are still spatially separated, and the local scalar field continues to move along the calculated damped trajectory. If overlapping voxel points are detected, it indicates that the two fields have begun to contact. At this point, the scalar contribution of the target spatial control node in the global scalar field is immediately restored. Specifically, the data from the local scalar field is linearly superimposed back into the global scalar field. At the same time, the local isosurface limits that were lowered during fixation to produce a bulging effect will be restored to the initial global isosurface limits. (For example, restore to 29.58), recalculate the fused scalar field distribution after the regression of the target space control nodes, which includes the contributions of all nodes, and then, based on this fully restored fused scalar field distribution, execute the moving cube algorithm again to reconstruct a continuous closed mesh surface that is completely consistent with the initial state according to the fused scalar field distribution, complete the visually smooth and seamless merging process, and restore it to the initial three-dimensional fluid fused mesh.

[0037] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A pharmacy medication selection navigation method based on augmented reality, characterized in that, Includes the following steps: The three-dimensional spatial coordinates of each target medicine box to be picked are obtained and mapped to spatial control nodes in the world coordinate system. The scalar values ​​of each voxel point are calculated using each of the spatial control nodes as potential sources and superimposed to obtain a global scalar field. Based on the sum of the scalar values ​​of each voxel point in the global scalar field, a three-dimensional fluid fusion mesh that encloses all the spatial control nodes is constructed. The operator's eye-tracking ray vector is tracked in real time, and the first intersection point between the eye-tracking ray vector and the surface of the three-dimensional fluid fusion mesh is calculated. An influence sphere is constructed with the first intersection point as its center. The local isosurface limit within the influence sphere is lowered to re-extract the isosurface and generate a local bulge deformation mesh oriented toward the direction of the eye-tracking ray vector. The eye-tracking ray vector is continuously tracked. When the gaze duration reaches a preset trigger limit, the scalar field contribution of the target space control node is separated from the global scalar field and a local scalar field is established. The local scalar field and the global scalar field are driven to move away from each other in space and generate independent fluid meshes in real time. When it is determined that the intersection influence area of ​​the local scalar field and the global scalar field is zero, the independent fluid mesh is replaced with an independent three-dimensional bounding box indicating the position of the target medicine box.

2. The augmented reality-based pharmacy medication navigation method according to claim 1, characterized in that, The method further includes: The eye-tracking ray vector is monitored. When it is determined that the vector deviates from the independent 3D bounding box and reaches a preset delay limit, the independent 3D bounding box is replaced with the independent fluid mesh, and the local scalar field is driven to move towards the global scalar field along a damped contraction trajectory. When the local scalar field and the global scalar field re-intersect spatially, the scalar contribution of the target spatial control node in the global scalar field is restored, and the original three-dimensional fluid fusion mesh is restored.

3. The augmented reality-based pharmacy medication navigation method according to claim 1, characterized in that, The steps for obtaining the space control node are as follows: Obtain the three-dimensional spatial coordinates of each target medicine box to be picked, read the horizontal, vertical and longitudinal coordinates of each target medicine box in the pharmacy shelf space, and uniformly transform the horizontal, vertical and longitudinal coordinates according to the coordinate origin, coordinate axis direction and scale reference of the world coordinate system. Establish a unique node number for each target medicine box after transformation, and bind the unique node number to the corresponding target medicine box to form a spatial control node in the world coordinate system.

4. The augmented reality-based pharmacy medication navigation method according to claim 1, characterized in that, The steps for obtaining the three-dimensional fluid fusion mesh are as follows: Each of the spatial control nodes is calibrated as a potential source. The horizontal, vertical, and longitudinal coordinates of each voxel point in the three-dimensional space are read one by one. The spatial distance of each voxel point relative to each of the spatial control nodes is calculated. According to the increasing relationship of spatial distance, a distance attenuation contribution is applied to each of the spatial control nodes. All distance attenuation contributions received by the same voxel point are superimposed item by item to obtain the global scalar field. The scalar sum of each voxel point in the global scalar field is extracted one by one. The scalar sum of each voxel point is compared with the preset global isosurface limit. The boundary positions of voxel points that reach the global isosurface limit are retained. The mesh vertex topology is constructed according to the connection relationship between the boundary positions of adjacent voxel points. The mesh vertex topology is closed into a continuous three-dimensional surface that wraps all the spatial control nodes to generate a three-dimensional fluid fusion mesh.

5. The augmented reality-based pharmacy medication navigation method according to claim 1, characterized in that, The steps for obtaining the sphere are as follows: The operator's gaze origin and gaze direction are read in real time. The gaze origin is used as the starting position of the eye movement ray vector, and the gaze direction is used as the extension direction of the eye movement ray vector. The mesh patches on the surface of the three-dimensional fluid fusion mesh are detected segment by segment along the extension direction of the eye movement ray vector. The mesh patches that come into contact are selected in order of distance from the gaze origin from near to far. The spatial position of the first contact is located on the mesh patches that come into contact, and the first intersection point is obtained. The first intersection point is set as the center of the sphere. The preset influence radius and preset Gaussian weight decay attribute are read. A spatial coverage area is defined around the center of the sphere according to the preset influence radius. The distance from each position within the spatial coverage area to the center of the sphere is mapped to the corresponding weight decay degree, so that the weight decay degree is more obvious for positions farther away from the center of the sphere, thus forming an influence sphere.

6. The augmented reality-based pharmacy medication navigation method according to claim 1, characterized in that, The steps for obtaining the locally raised deformation mesh are as follows: By comparing the node coordinates of each target spatial control node with the spatial coverage of the influencing sphere, target spatial control nodes located within the influencing sphere are selected. The Euclidean distance from each target spatial control node to the center of the sphere is extracted. According to the Gaussian weight decay attribute corresponding to the Euclidean distance, the isosurface limit of the local region within the influencing sphere is dynamically lowered. The position of the local isosurface is then re-determined according to the lowered isosurface limit, causing the local isosurface to form an outward expansion deformation along the direction of the eye-tracking ray vector, generating a local bulging deformation mesh.

7. The augmented reality-based pharmacy medication navigation method according to claim 1, characterized in that, The steps for obtaining the local scalar field are as follows: The eye-tracking ray vector is continuously tracked, and the start time and current gaze time of the continuous gaze pointing to the region corresponding to the target spatial control node are recorded. The gaze dwell time between the start time and the current gaze time is calculated, and the gaze dwell time is compared with a preset trigger limit. If the gaze dwell time does not reach the preset trigger limit, the current gaze time is recorded. If the gaze dwell time reaches the preset trigger limit, the target spatial control node is locked, and the scalar field contribution corresponding to the target spatial control node is extracted from the global scalar field. The scalar field contribution corresponding to the target spatial control node is separated from the global scalar field to establish a local scalar field.

8. The augmented reality-based pharmacy medication navigation method according to claim 1, characterized in that, The steps for obtaining the independent 3D bounding box are as follows: Read the current node coordinates of the target space control node in the world coordinate system, extract the line of sight of the eye-tracking ray vector, reverse the line of sight as the translation traction direction, apply translation traction force to the target space control node according to the translation traction direction, so that the target space control node drives the local scalar field away from the global scalar field frame by frame, update the spatial position of the local scalar field frame by frame, and generate an independent fluid mesh with droplet peeling morphology based on the updated local scalar field; The intersection influence area of ​​the local scalar field and the global scalar field in three-dimensional space is detected frame by frame. The number of voxel points still affected by the common scalar contribution within the intersection influence area is counted. If the number of voxel points is not zero, the local scalar field continues to move along the translation traction direction. If the number of voxel points is zero, it is determined that the local scalar field has been stripped. The target pillbox position corresponding to the stripped independent fluid mesh is read. The bounding box center, length, width and height dimensions and the orientation of the bounding box are determined according to the target pillbox position. The stripped independent fluid mesh is replaced with an independent three-dimensional bounding box indicating the target pillbox position.

9. The augmented reality-based pharmacy medication navigation method according to claim 2, characterized in that, The specific steps for seamlessly restoring the original three-dimensional fluid fusion mesh are as follows: The eye-tracking ray vector is monitored, and the gaze origin and gaze direction of the eye-tracking ray vector are read frame by frame. The gaze direction extension line is projected onto the outer surface area of ​​the independent 3D bounding box. It is detected whether the gaze direction extension line passes through the outer surface area of ​​the independent 3D bounding box. If the gaze direction extension line passes through the outer surface area of ​​the independent 3D bounding box, the deviation timing value is reset to zero. If the gaze direction extension line does not pass through the outer surface area of ​​the independent 3D bounding box, the deviation timing value is continuously accumulated. The deviation timing value is compared with a preset delay limit. When the deviation timing value reaches the preset delay limit, the independent 3D bounding box is replaced with the independent fluid mesh. The current node coordinates of the target space control node in the independent fluid grid are read, and the initial coordinates of the target space control node in the three-dimensional fluid fusion grid are read. The direction from the current node coordinates to the initial coordinates is determined as the return direction. The frame-by-frame displacement of the target space control node is calculated according to the return direction. The frame-by-frame displacement is corrected by decreasing the elastic rebound strength and the damping attenuation strength, so that the return speed of the target space control node decreases frame by frame as it approaches the initial coordinates, forming a damped contraction trajectory. The local scalar field is driven to move frame by frame toward the global scalar field according to the damped contraction trajectory. Frame by frame, it is checked whether there are overlapping voxel points in the voxel point ranges corresponding to the local scalar field and the global scalar field. If there are no overlapping voxel points, the local scalar field continues to move. If there are overlapping voxel points, the scalar contribution of the target space control node in the global scalar field is restored, the isosurface limit of the local region is restored to the initial global isosurface limit, the fused scalar field distribution after the target space control node returns is recalculated, and the continuous closed mesh surface is reconstructed based on the fused scalar field distribution, restoring it to the initial three-dimensional fluid fused mesh.