Buccal needle positioning navigation method and system based on multi-mode perception driving

Abstract

The invention discloses a buccal needle positioning navigation method and system based on multi-mode perception driving, and the method comprises the steps: collecting the image information and depth information of the face of a patient; generating an individualized face three-dimensional model according to the image information and the depth information; determining a standard meridian map according to a meridian database, and registering the standard meridian map to the individualized face three-dimensional model through non-rigid deformation to obtain acupuncture point positioning; according to the acupuncture point positioning and a pre-stored organization hierarchy model, needle insertion path guiding information is calculated and generated, and in an interactive interface, the acupuncture point positioning and the needle insertion path guiding information are displayed in an overlapped mode so as to execute the needle insertion guiding process; and in the needle insertion guiding process, collision detection is carried out according to the needle insertion path guiding information and the organization level model, and safety early warning is carried out when a collision risk is detected. The precision and safety of the acupuncture treatment process can be effectively improved.

Description

Technical Field

[0001] This application relates to the technical field of acupuncture treatment, and in particular to a cheek needle positioning and navigation method and system based on multimodal perception-driven approach. Background Technology

[0002] Acupuncture, especially facial acupuncture, is widely used due to its significant therapeutic effects. However, current clinical practice mainly relies on traditional acupoint location methods, with accuracy depending on subjective experience and lacking individualized adjustments. These methods primarily rely on "surface landmarks" and "finger measurement," which are highly dependent on the physician's personal experience and visual judgment. Due to significant individual differences in facial bone structure, muscle and fat distribution, these methods struggle to achieve precise, repeatable millimeter-level acupoint location, especially in densely populated areas like the face, where location errors can lead to reduced therapeutic efficacy.

[0003] In acupuncture procedures using related technologies, the lack of real-time visualization and guidance in three-dimensional space means that physicians cannot "see" the location of the target acupoints under the skin or the ideal needle insertion path throughout the treatment. The needle insertion angle and depth rely entirely on feel and experience, lacking a real-time, visual reference system to assist in the operation. This undoubtedly increases the skill requirements and uncertainty of the operation, affecting the effectiveness of acupuncture treatment and posing a risk of accidental injury. Summary of the Invention

[0005] The purpose of this application is to provide a cheek needle positioning and navigation method and system based on multimodal perception, which can improve the accuracy and safety of the acupuncture process.

[0006] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a cheek acupuncture positioning and navigation method based on multimodal perception-driven approach. The method includes: acquiring image information and depth information of a patient's face; generating an individualized three-dimensional facial model based on the image information and the depth information; confirming a standard meridian atlas based on a meridian database, and registering the standard meridian atlas onto the individualized three-dimensional facial model through non-rigid deformation to obtain acupoint locations; calculating and generating needle insertion path guidance information based on the acupoint locations and a pre-stored tissue hierarchy model, and overlaying and displaying the acupoint locations and the needle insertion path guidance information in an interactive interface to execute the needle insertion guidance process; during the needle insertion guidance process, performing real-time collision detection between the needle insertion path guidance information and the tissue hierarchy model, and issuing a safety warning when a collision risk is detected.

[0007] For example, generating a personalized 3D facial model based on the image information and the depth information includes: preprocessing the image information and the depth information, the preprocessing including at least spatiotemporal alignment and denoising; generating a 3D point cloud based on the preprocessed depth information; detecting facial key points using a deep learning algorithm and mapping the facial key points onto the 3D point cloud for mapping processing; converting the mapped 3D point cloud into a triangular mesh model using a surface reconstruction algorithm; and mapping the image information as a texture map onto the triangular mesh model to form a textured personalized 3D facial model.

[0008] For example, the standard meridian map in the meridian database is a pre-constructed digital map, which includes at least: meridian pathways defined on a standard face model based on traditional Chinese medicine meridian theory; acupoints associated with the meridian pathways, each acupoint having three-dimensional coordinates on the standard face model; and a data structure defining the topological relationship between the meridian pathways and the acupoints.

[0009] For example, the standard meridian atlas is registered onto the individualized facial 3D model through non-rigid deformation to obtain acupoint locations. This includes: calculating a rigid transformation matrix based on the anatomical key points on the individualized facial 3D model and the corresponding standard key points on the standard meridian atlas; initially aligning the standard meridian atlas to the individualized facial 3D model for initial global alignment; performing deformation adjustment based on the initial global alignment, including coarse-grained deformation based on thin-plate spline transformation and fine-grained deformation based on Laplacian surface editing, so that the geometry of the standard meridian atlas fits the surface of the individualized facial 3D model; during the deformation adjustment process, a smoothness constraint term is used to limit the curvature change of the meridian pathway to maintain the physiological characteristics of the meridian pathway.

[0010] For example, the tissue hierarchy model is a three-dimensional model constructed based on anatomical data, which includes hierarchical spatial distribution information of facial skin, muscles, blood vessels and nerves.

[0011] For example, the calculation and generation of needle insertion path guidance information includes: using the acupoint location as the target point, calculating the optimal needle insertion point from the skin surface to the target point; determining the direction vector and safe depth threshold of the needle insertion based on medical standards and the tissue hierarchy model; and generating a virtual guide line connecting the needle insertion point and the target point, extending to the safe depth threshold, as the needle insertion path guidance information.

[0012] For example, the real-time collision detection based on the needle insertion path guidance information and the tissue hierarchy model includes: constructing a collision body in three-dimensional space based on the virtual guide line; calculating in real time the spatial distance relationship between the collision body and the marked dangerous tissue area in the tissue hierarchy model; and determining that there is a collision risk when the collision body is detected to have intruded into the safe distance range of the dangerous tissue area.

[0013] For example, the safety warning when a collision risk is detected includes at least one of the following: changing the visual representation of the needle insertion path guidance information in the interactive interface; popping up a visual warning sign in the interactive interface; or triggering an audio-visual alarm.

[0014] Secondly, this application provides a cheek acupuncture positioning and navigation system based on multimodal perception, comprising: an image acquisition module for acquiring image information and depth information of a patient's face; a control module for generating an individualized three-dimensional facial model based on the image information and the depth information; and for confirming a standard meridian atlas based on a meridian database, registering the standard meridian atlas onto the individualized three-dimensional facial model through non-rigid deformation to obtain acupoint positioning; the control module is further configured to calculate and generate needle insertion path guidance information based on the acupoint positioning and a pre-stored tissue hierarchy model, and overlay and display the acupoint positioning and the needle insertion path guidance information in an interactive interface to execute the needle insertion guidance process; and a collision warning module for performing real-time collision detection between the needle insertion path guidance information and the tissue hierarchy model during the needle insertion guidance process, and issuing a safety warning when a collision risk is detected.

[0015] For example, the system further includes a storage module and a remote communication module, the storage module being used to store data, including a meridian database, an organizational hierarchy model, and an individualized 3D facial model; the remote communication module is used to communicate with the control module and the cloud platform.

[0016] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a cheek acupuncture positioning and navigation method and system based on multimodal perception. It generates an individualized 3D facial model by collecting image and depth information of the patient's face, and then registers the model onto this individualized 3D facial model. This accurately adapts abstract meridian maps from the standard facial space to the patient's 3D facial model, solving the problem of inaccurate positioning caused by individual anatomical differences. By displaying acupoint positioning and needle insertion path guidance information in the interactive interface, it provides visual guidance and improves the accuracy of the acupuncture process. During needle insertion, real-time collision detection is performed, and a safety warning is issued when a collision risk is detected, effectively improving safety. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the cheek needle localization and navigation method based on multimodal perception in the embodiments of this application.

[0019] Figure 2 This is a block diagram of a cheek needle positioning and navigation system based on multimodal perception in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] like Figure 1 As shown in the figure, this application provides a cheek needle localization and navigation method based on multimodal perception driving, including the following steps: S110. Collect image and depth information of the patient's face.

[0023] S120. Generate an individualized 3D facial model based on image and depth information.

[0024] S130. Based on the meridian database, confirm the standard meridian map and register it onto the individualized facial 3D model through non-rigid deformation to obtain the acupoint location.

[0025] S140. Based on the acupoint location and the pre-stored tissue hierarchy model, calculate and generate needle insertion path guidance information, and overlay and display the acupoint location and needle insertion path guidance information in the interactive interface to execute the needle insertion guidance process.

[0026] S150. During the needle insertion guidance process, real-time collision detection is performed based on the needle insertion path guidance information and the tissue hierarchy model, and a safety warning is issued when a collision risk is detected.

[0027] This application presents a multimodal perception-driven cheek acupuncture positioning and navigation method. It generates an individualized 3D facial model by collecting image and depth information of the patient's face, and then registers the model onto this model. This accurately adapts abstract meridian maps from the standard facial space to the patient's 3D facial model, solving the problem of inaccurate positioning caused by individual anatomical differences. By displaying acupoint positioning and needle insertion path guidance information in the interactive interface, it provides visual guidance and improves the accuracy of the acupuncture process. During needle insertion, real-time collision detection is performed, and a safety warning is issued when a collision risk is detected, effectively improving safety.

[0028] The acquisition of image and depth information of the patient's face can be performed using a camera, which can be an RGB-D camera, simultaneously acquiring RGB color images of the patient's face (including skin color and texture information) and depth information (including the distance information of each pixel from the camera). Typically, several frames of data are acquired from multiple slightly different angles to ensure completeness.

[0029] For example, step S120 above, which generates an individualized 3D facial model based on image information and depth information, specifically includes the following steps: S121. Preprocess the image information and depth information. The preprocessing includes at least spatiotemporal alignment and noise reduction.

[0030] Spatiotemporal alignment refers to aligning consecutive frames of RGB images and depth maps spatiotemporally to ensure a one-to-one correspondence between color information and geometric position. Denoising refers to filtering the depth map (such as bilateral filtering or median filtering) to remove noise points caused by ambient light or surface reflection, resulting in smooth and accurate depth information.

[0031] S122. Generate a 3D point cloud based on the preprocessed depth information.

[0032] Each pixel in the preprocessed depth map is converted to 3D space using coordinate transformation formulas based on camera intrinsic parameters (focal length, optical center, etc.), generating a 3D point cloud representing the facial surface, consisting of countless 3D coordinate points (X, Y, Z). At this point, the point cloud is sparse and disordered.

[0033] S123. Detect facial key points using a deep learning algorithm and map the facial key points onto a 3D point cloud for mapping processing.

[0034] Using deep learning facial landmark detection algorithms (such as CNN-based models), RGB images are analyzed to automatically identify dozens or even hundreds of anatomically significant landmarks, such as the corners of the eyes, the tip of the nose, the wings of the nose, the corners of the mouth, and facial contour points. The facial landmark detection algorithm is deployed in the control module.

[0035] These 2D keypoints are mapped onto a 3D point cloud, becoming feature markers within the point cloud. The system then uses non-rigid iterative nearest-neighbor (ICP) registration or a similar algorithm with a pre-stored standard face model to drive deformation of the standard model, ensuring that its keypoints coincide with the keypoints of the actual scan. This step significantly improves the accuracy and reliability of the model.

[0036] S124. Convert the mapped 3D point cloud into a triangular mesh model according to the surface reconstruction algorithm.

[0037] The registered dense 3D point cloud obtained in the previous step is then used to generate a continuous mesh surface composed of triangular facets, also known as a triangular mesh model, through reconstruction algorithms such as Poisson Surface Reconstruction or Delaunay Triangulation. This triangular mesh model forms the geometric basis of the 3D facial model.

[0038] S125. Map the image information as a texture map onto the triangular mesh model to form a textured, individualized 3D facial model.

[0039] The RGB color image (i.e., image information) from the previous step is used as a texture map and precisely wrapped around the geometric mesh generated in the previous step. Each triangle is assigned realistic color information, resulting in a high-fidelity 3D model that includes both geometric shape and realistic skin tone, i.e., an individualized 3D facial model.

[0040] It's important to clarify the specific components of the model: the final generated "personalized 3D facial model" is not a single file, but a structured data set containing multiple layers of information, primarily including the following components: Geometric Mesh, Vertices (thousands of precisely distributed vertices in 3D space, each with its own coordinates (X, Y, Z) – this forms the model's 3D skeleton), Faces (triangles formed by connecting vertices, these faces combine to create continuous surfaces that express facial contours (such as the bridge of the nose and the hollows of the eyes), Texture Map (a 2D image file storing all visual appearance information of the patient's face, such as skin tone, pores, eyebrows, and lip color). Using UV mapping technology, this 2D image is accurately mapped and "wrapped" onto the 3D geometric mesh, giving the model a realistic appearance, and UV Coordinates (coordinate data defining how each point on the 2D texture map corresponds to each triangle on the 3D geometric mesh). This is the bridge connecting geometry and texture. Facial keypoints metadata is a set of labeled data associated with model vertices. It marks which vertices correspond to important anatomical landmarks (such as "left outer canthus," "tip of the nose," and "philtrum"). These keypoints serve as reference points for subsequent meridian and acupoint mapping algorithms to achieve precise localization.

[0041] The individualized 3D model obtained through image and depth information is the basis of the method in this application. Only by confirming the individualized 3D model can the meridian model in the database be matched with the patient's face, making it convenient to display the acupoint location and meridians on the patient's face.

[0042] For example, in step S130 above, the standard meridian map in the meridian database is a pre-constructed digital map, which at least includes: meridian pathways defined on a standard face model based on traditional Chinese medicine meridian theory; acupoints associated with the meridian pathways, each acupoint having three-dimensional coordinates on the standard face model; and a data structure defining the topological relationship between the meridian pathways and acupoints. The standard meridian map in the meridian database is pre-stored in a storage module and is retrieved from the stored data during use.

[0043] In step S130 above, the standard meridian map is registered onto the individualized facial 3D model through non-rigid deformation to obtain acupoint locations. An adaptive mapping algorithm is used in the calculation process. This algorithm can be configured in the control module, and the specific calculation process includes: S131. Based on the anatomical key points on the individualized facial 3D model and the corresponding standard key points on the standard meridian map, calculate the rigid body transformation matrix and initially align the standard meridian map to the individualized facial 3D model for initial global alignment.

[0044] The purpose of the above steps is to roughly align the template of the standard atlas from the "standard face" space to the "individual face" space. The method is based on two sets of corresponding anatomical key points (such as acupoints like Yintang, Jingming, Dicang, and Yingxiang). An optimal rigid transformation matrix (which may include rotation, translation, and uniform scaling) is calculated to make the key points on the template coincide as much as possible with the key points on the individual model. Common methods include least squares or variations of the Iterative Closest Point (ICP) algorithm. In some embodiments, hierarchical non-rigid deformation is used to handle complex deformations caused by differences in facial size, width, and contour.

[0045] S132. Based on the initial global alignment, perform deformation adjustments, including coarse-grained deformation based on thin plate spline transformation and fine-grained deformation based on Laplacian surface editing, so that the geometry of the standard meridian map fits the surface of the individualized facial 3D model.

[0046] The deformation adjustment process includes coarse-level deformation, treating the face as a whole elastic membrane. The algorithm calculates an affine transformation or Thin Plate Spline (TPS) transformation based on more corresponding feature points, stretching and bending the standard template holistically to make its contour basically match the individual facial contour. It also includes fine-level deformation, making local fine adjustments based on the coarse alignment. This often employs physically based deformation models or optical flow methods. The standard template is considered an elastic mesh, and the individual facial model is the mold. The "attraction" between each vertex on the template and the surface of the individual model is calculated, and elastic constraints are applied so that the entire mesh can both conform to the target surface and maintain its original continuity and smoothness (preventing excessive distortion). Examples include Laplacian surface editing or biomechanical simulations.

[0047] S133. During the deformation adjustment process, the curvature change of the meridian pathway is limited by the smoothness constraint term in order to maintain the physiological characteristics of the meridian pathway.

[0048] To ensure that the deformed result is not only geometrically accurate but also medically precise, the algorithm incorporates prior knowledge as a constraint, which is key to its differentiation from general computer graphics algorithms. Smoothness constraints refer to limiting the curvature changes of the meridian curves during deformation, preventing unnatural sharp angles or jitter, and maintaining their smooth physiological characteristics.

[0049] In step S140, needle insertion path guidance information is calculated and generated. This calculation is performed using a dynamic meridian and acupoint mapping algorithm, which is deployed in the control module. The calculation process includes: S141. Using the acupoint location as the target point, calculate the optimal needle insertion point from the skin surface to the target point.

[0050] S142. Based on medical standards and the tissue hierarchy model, determine the needle insertion direction vector and safe depth threshold.

[0051] The tissue hierarchy model is a three-dimensional model constructed based on anatomical data, containing hierarchical spatial distribution information of facial skin, muscles, blood vessels, and nerves. Dangerous tissue areas are marked in the tissue hierarchy model; these represent hazardous areas on the patient's face, and touching these areas during needle insertion may cause injury or danger.

[0052] S143. Generate a virtual guide line connecting the needle insertion point and the target point, extending to the safety depth threshold, as the needle insertion path guidance information.

[0053] Specifically, in step S150 above, during the needle insertion guidance process, real-time collision detection is performed based on the needle insertion path guidance information and the tissue hierarchy model, and a safety warning is issued when a collision risk is detected. This includes the following steps: S151. Construct a collision body in three-dimensional space based on the virtual guide line.

[0054] S152. Real-time calculation of the spatial distance relationship between the collider and the marked hazardous tissue areas in the tissue hierarchy model.

[0055] S153. When a collision object is detected to have intruded into the safe distance range of a dangerous area, it is determined that there is a collision risk.

[0056] In the above-described manner, during the needle insertion guidance process, the user follows the needle insertion path guidance information. If a collision object intrudes into the safe distance range of the dangerous tissue area during this process, a collision risk is confirmed, and a safety warning is issued.

[0057] Safety warnings can be issued through at least one of the following methods: altering the visual presentation of needle insertion path guidance information within the interactive interface; displaying a visual warning sign within the interactive interface; or triggering an audible and visual alarm. These methods, along with one or more of the above prompts, indicate to the operator that a potential risk has arisen and that timely adjustments are necessary.

[0058] This application also provides a cheek needle positioning and navigation system based on multimodal perception, including: a head-mounted display device for providing an interactive interface, and an image acquisition module, a control module and a collision warning module provided on the head-mounted display device.

[0059] The image acquisition module is used to acquire image and depth information of the patient's face. The control module generates an individualized 3D facial model based on the image and depth information; it also identifies standard meridian atlases from a meridian database and registers these atlases onto the individualized 3D facial model using non-rigid deformation to obtain acupoint locations. The control module further calculates and generates needle insertion path guidance information based on acupoint locations and a pre-stored tissue hierarchy model, and overlays and displays the acupoint locations and real-time needle insertion path guidance information on the interactive interface to execute the needle insertion guidance process. The collision warning module performs real-time collision detection between the needle insertion path guidance information and the tissue hierarchy model during the needle insertion guidance process and issues a safety warning when a collision risk is detected.

[0060] The head-mounted display device is equipped with AR glasses, which provide an AR interface as the interactive interface. When using it, the operator can use the various information superimposed on the interactive interface to assist in acupuncture treatment, thereby improving accuracy and safety.

[0061] For example, the system also includes a storage module and a remote communication module. The storage module is used to store data, including a meridian database, an organizational hierarchy model, and an individualized 3D facial model. The remote communication module is used to communicate with the control module and the cloud platform.

[0062] During use, communication is established between the control module and the remote platform, allowing real-time video streams and data of complex cases to be uploaded to the cloud for remote guidance. The cloud-based collaborative platform can push expert annotations, instructions, or updated algorithm models downwards, enabling continuous optimization and knowledge sharing.

[0063] For example, the system also includes a gesture recognition module, which receives and recognizes the user's gesture signals and transmits the recognized gesture commands to the control module. The control module controls various modules in the system, such as controlling the image acquisition module to turn on and off based on gesture commands, and controlling the communication module to transmit or download data.

[0064] The methods and systems described in this application have the following advantages compared to related technologies: This technology achieves high-precision, individualized acupoint localization, overcoming reliance on subjective experience. It acquires image and depth information of the patient's face using an image acquisition device to generate an individualized 3D facial model. Then, a standard meridian atlas is non-rigidly deformed and registered onto this individualized 3D facial model. This technique precisely adapts abstract meridian atlases from a standard face to the specific patient's 3D facial model, fundamentally solving the problem of inaccurate localization caused by individual anatomical differences, and improving localization accuracy to the objective, quantifiable millimeter level.

[0065] It provides intuitive visual operation guidance, lowering the operational threshold: by overlaying and displaying the acupoint location and needle insertion path guidance information in the interactive interface, it transforms the experiential knowledge that originally existed in the physician's mind into augmented reality (AR) information that is directly visible on the patient's real facial scene. This provides physicians with intuitive and immersive operation guidance, greatly reducing the learning difficulty for beginners and improving the standardization and confidence of all physicians' operations.

[0066] A proactive real-time safety warning mechanism was established to improve treatment safety: real-time collision detection was performed based on the needle insertion path guidance information and the tissue hierarchy model, and a safety warning was issued when a collision risk was detected. This step simulates the spatial relationship between the needle insertion path and key tissues in a virtual digital space, enabling risk identification and alarm issuance before physical contact occurs. This transforms safety protection from a passive reliance on experience to proactive technological safeguards, effectively preventing accidental injury to important nerves and blood vessels.

[0067] A complete digital diagnosis and treatment closed loop has been constructed, laying the foundation for standardization and datafication. The entire solution, from data collection, model building, algorithm registration to guidance and early warning, forms a complete technical closed loop. All operations are based on digital models and algorithms, enabling the recording and analysis of key information such as acupoint location and needle insertion parameters. This provides a solid technical foundation for the standardization and repeatability of acupuncture treatment, as well as for the accumulation of subsequent clinical research data.

[0068] Those skilled in the art will understand that the structure shown in the figure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0069] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0070] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0071] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0074] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A cheek needle localization and navigation method based on multimodal perception, characterized in that, The cheek needle localization and navigation method based on multimodal perception includes: Collect image and depth information of the patient's face; Based on the image information and the depth information, an individualized 3D facial model is generated; Based on the meridian database, a standard meridian map is confirmed, and the standard meridian map is registered onto the individualized facial 3D model through non-rigid deformation to obtain acupoint locations; Based on the acupoint location and the pre-stored tissue hierarchy model, the needle insertion path guidance information is calculated and generated, and the acupoint location and the needle insertion path guidance information are superimposed and displayed in the interactive interface to execute the needle insertion guidance process. During the needle insertion guidance process, real-time collision detection is performed between the needle insertion path guidance information and the tissue hierarchy model, and a safety warning is issued when a collision risk is detected.

2. The cheek needle localization and navigation method based on multimodal perception driving according to claim 1, characterized in that, Based on the image information and the depth information, a personalized 3D facial model is generated, including: The image information and depth information are preprocessed, and the preprocessing includes at least spatiotemporal alignment and noise reduction; A 3D point cloud is generated based on the preprocessed depth information; Facial key points are detected using a deep learning algorithm, and these key points are mapped onto the 3D point cloud for mapping processing. The mapped 3D point cloud is converted into a triangular mesh model according to the surface reconstruction algorithm; The image information is mapped onto the triangular mesh model as a texture map to form a textured, individualized 3D facial model.

3. The cheek needle localization and navigation method based on multimodal perception driving according to claim 1, characterized in that, The standard meridian atlas in the meridian database is a pre-constructed digital atlas, which includes at least: The meridian pathways defined on a standard human face model based on the theory of meridians in Traditional Chinese Medicine; Acupoints associated with the meridian pathways, each acupoint having three-dimensional coordinates on the standard human face model; Define a data structure for the topological relationship between the meridian pathway and the acupoint.

4. The cheek needle localization and navigation method based on multimodal perception driving according to claim 3, characterized in that, The standard meridian atlas is registered onto the individualized facial 3D model using non-rigid deformation to obtain acupoint locations, including: Based on the anatomical key points on the individualized facial 3D model and the corresponding standard key points on the standard meridian atlas, a rigid body transformation matrix is ​​calculated, and the standard meridian atlas is initially aligned to the individualized facial 3D model for initial global alignment. Based on the initial global alignment, deformation adjustments are performed, including coarse-grained deformation based on thin plate spline transformation and fine-grained deformation based on Laplacian surface editing, so that the geometry of the standard meridian map fits the surface of the individualized facial 3D model. During the deformation adjustment process, a smoothness constraint term is used to limit the curvature change of the meridian pathway in order to maintain the physiological characteristics of the meridian pathway.

5. The cheek needle localization and navigation method based on multimodal perception driving according to claim 1, characterized in that, The tissue hierarchy model is a three-dimensional model constructed based on anatomical data, which includes hierarchical spatial distribution information of facial skin, muscles, blood vessels and nerves.

6. The method according to claim 5, characterized in that, The calculation and generation of needle insertion path guidance information includes: Using the acupoint as the target point, calculate the optimal needle insertion point from the skin surface to the target point; Based on medical standards and the aforementioned tissue hierarchy model, the direction vector for needle insertion and the safe depth threshold are determined. A virtual guide line is generated connecting the needle insertion point and the target point, and extending to the safety depth threshold, to serve as the needle insertion path guidance information.

7. The method according to claim 6, characterized in that, The step of performing real-time collision detection based on the needle insertion path guidance information and the tissue hierarchy model includes: Construct a collision body in three-dimensional space based on the virtual guide lines; Real-time calculation of the spatial distance relationship between the collider and the marked hazardous tissue regions in the tissue hierarchy model; When a collision object is detected to have intruded into the safe distance range of the dangerous tissue area, it is determined that there is a collision risk.

8. The method according to any one of claims 1-7, characterized in that, The provision of a safety warning upon detection of a collision risk includes at least one of the following: In the interactive interface, the visual representation of the needle insertion path guidance information is changed; in the interactive interface, a visual warning sign pops up; and an audio-visual alarm is triggered.

9. A cheek needle positioning and navigation system based on multimodal perception, characterized in that, The cheek needle positioning and navigation system based on multimodal perception includes: a head-mounted display device for providing an interactive interface, and the head-mounted display device is equipped with an image acquisition module, a control module and a collision warning module; The image acquisition module is used to acquire image and depth information of the patient's face; The control module is used to generate an individualized three-dimensional facial model based on the image information and the depth information; and to confirm the standard meridian atlas based on the meridian database, and to register the standard meridian atlas onto the individualized three-dimensional facial model through non-rigid deformation to obtain acupoint locations; The control module is also used to calculate and generate needle insertion path guidance information based on the acupoint location and the pre-stored tissue hierarchy model, and to overlay and display the acupoint location and the needle insertion path guidance information in the interactive interface to execute the needle insertion guidance process. The collision warning module is used to perform real-time collision detection between the needle insertion path guidance information and the tissue hierarchy model during the needle insertion guidance process, and to issue a safety warning when a collision risk is detected.

10. The cheek needle positioning and navigation system based on multimodal perception driving according to claim 9, characterized in that, The system also includes a storage module and a remote communication module. The storage module is used to store data, including a meridian database, an organizational hierarchy model, and an individualized 3D facial model. The remote communication module is used to communicate with the control module and the cloud platform.

Images