Infant airway display system with electronic scale and early warning function

Abstract

The invention belongs to the field of mixed reality, and particularly relates to an infant airway display system with an electronic scale and an early warning function. The special visual laryngoscope for the infant is used for acquiring a real-time endoscopic image of the airway of the infant; the mixed reality three-dimensional model generation unit is used for constructing a mixed reality three-dimensional model containing a complete anatomical structure based on the infant chest CT data; an augmented reality model generation unit marks airway key data obtained by measuring key points of an important anatomical structure in an airway into the three-dimensional airway model, and generates an augmented reality model; the data processing unit is used for performing fusion processing on the airway key data and the real-time endoscopic image, generating key point position prompt information and dangerous operation early warning information, and sending the information to the AR glasses; and the AR glasses are used for displaying the fused data and superposing the key point position prompt and dangerous operation early warning information. The scheme can be used for auxiliary diagnosis and surgical planning.

Description

Technical Field

[0001] The present invention belongs to the field of mixed reality, and in particular relates to an infant airway display system with an electronic ruler and an early warning function. Background Art Failure of airway management in infants and young children can lead to delayed treatment, airway damage, and even endanger the life of the child. Airway management in infants and young children is currently both a clinical focus and a technical difficulty for operators and managers. Infants and young children have a relatively small mouth opening, and the tongue occupies a large proportion of the oral cavity, making it difficult to expose the glottis; the epiglottis is at an acute angle to the glottis, the glottis opening is relatively high, and the subglottis is physiologically narrow. Improper operation during airway surgery can easily cause subglottic damage, which can easily form scars after damage; in addition, infants and young children have a short tolerance to hypoxia. If difficult endotracheal intubation is encountered in infants and young children, it must be as fast and gentle as possible while ensuring the success rate, which is very difficult to implement.

[0002] In the existing technology, a real-time image of the airway can be seen using a video laryngoscope, but the information provided by a single video laryngoscope is limited. Currently, there are also some solutions that use a video laryngoscope in conjunction with VR and MR technology. VR and MR technology can generate airway models based on medical DICOM data, but such methods can only obtain static data on key parts, and cannot obtain relative information on the depth of intubation and the physiological position of subglottic stenosis in children. For doctors performing tracheal intubation or airway surgery, these key parts can only be obtained based on experience. In some complex situations, the displayed data is not accurate enough, which will increase the risk of operation. Summary of the invention

[0003] In view of this, the present invention provides an infant airway display system with an electronic ruler and an early warning function, comprising: a visual laryngoscope for infants, a mixed reality three-dimensional model generation unit, an augmented reality model generation unit, a data processing unit and AR glasses; The infant-specific visual laryngoscope is used to obtain a real-time endoscopic image of the infant's airway and send the real-time endoscopic image to the data processing unit; The mixed reality 3D model generation unit is used to construct a mixed reality 3D model containing a complete anatomical structure based on the infant chest CT data; The augmented reality model generation unit is used to separate the three-dimensional airway model from the mixed reality three-dimensional model; annotate the key airway data obtained by measuring the key points of the important anatomical structures in the airway into the three-dimensional airway model, generate the augmented reality model, and send the key airway data to the data processing unit; The data processing unit is used to fuse the key airway data and the real-time endoscopic image to generate key point location prompt information and dangerous operation warning information, and send them to the AR glasses; The AR glasses are used to display the mixed reality three-dimensional model, the augmented reality model, and the real-time endoscopic image superimposed with the key point position prompts and dangerous operation warning information, for auxiliary diagnosis and surgical planning.

[0004] In particular, the method for acquiring the key airway data includes: acquiring airway CT data based on chest CT data of infants and young children for 3D printing to obtain an airway entity model, and performing precise scale measurement on the airway entity model to obtain the key airway data.

[0005] In particular, the key points include the glottis, tracheal carina or lobe opening; the key data include the glottis diameter, the subglottic stenosis diameter, the distance from the glottis to the tracheal carina or the distance from the tracheal carina to the right upper lobe opening.

[0006] In particular, the data processing unit includes a glottis position prompt and dangerous operation warning module, which calculates the matching degree between the airway real-time image data sequence and the key point data sequence in the three-dimensional airway model through a self-attention mechanism, which is specifically implemented as follows:

[0007] in, Airway real-time image frame Key points in the three-dimensional airway model The matching weight of represents the i-th frame of image data in the airway real-time image data sequence, Represents the j-th key point data in the three-dimensional airway model; represents the linear transformation matrix of the query vector, which is used to map the input x to the space of the query vector, Q represents the query vector; The linear transformation matrix of the key vector, used to map the input y to the space of the key vector, represents the key vector, represents the dimension of the scaling factor, represents the index of the key point in the three-dimensional airway model, Represents the total number of key points in the 3D airway model.

[0008] In particular, the focus representation of the key points is calculated in combination with the matching weights and used for glottal position prompts and dangerous operation warnings, which is specifically implemented as follows:

[0009] in, Used to indicate the The focus display of key points of the real-time airway image data is used to indicate the glottis position and dangerous operation warning in real time. Represents a linear transformation matrix for transforming the j-th key point data in the three-dimensional airway model Mapped to the value vector space, Represents a vector of values.

[0010] In particular, the data processing unit also includes a navigation function module, which is used to calculate the weighted matching score between the catheter trajectory point and the key point in the three-dimensional airway model by a weighted distance calculation method based on parameterized similarity, which is specifically implemented as follows:

[0011] Where: p f represents the coordinate vector of the fth catheter trajectory point, y g represents the g-th key point in the three-dimensional airway model, ‖ represents the vector concatenation operation; W γ represents the learnable weight matrix and b γ represents the learnable bias vector, LeakyReLU is the leaky rectified linear unit activation function, γ fg represents the catheter trajectory point p f and the key point y in the three-dimensional airway model g The weighted matching score.

[0012] In particular, the weighted distance between the catheter trajectory point and the key point is calculated in combination with the weighted matching score, which is specifically implemented as follows:

[0013] Among them, d f represents the weighted distance between the fth catheter trajectory point and all key points in the three-dimensional airway model, which is used to calculate the distance between the catheter and the glottis or other key positions during navigation; p f represents the coordinate vector of the fth catheter trajectory point, y g represents the coordinate vector of the g-th key point in the three-dimensional airway model, ||p f -y g || represents the fth catheter trajectory point p f The key point y of the three-dimensional airway model g The Euclidean distance between .

[0014] In particular, the navigation function module further dynamically adjusts the optimal path for catheter insertion through a path planning algorithm, specifically including the following steps: based on the weighted distance between the catheter trajectory point and the key point, using the shortest path search algorithm to plan the optimal insertion path from the current position to the key point; dynamically monitoring the movement of the catheter, updating the distance between the current position and each of the key points in real time, and re-planning the optimal path to minimize the risk in the insertion path.

[0015] In particular, the mixed reality three-dimensional model is a common model shared by infants of different weights or a personalized model established for the symptoms of a single infant.

[0016] Beneficial effects: (1) Through the technical solution of the present invention, key positions in the airway can be accurately detected and located in real time, providing precise visual feedback for surgical navigation, thereby improving surgical accuracy and safety.

[0017] (2) Through the technical solution of the present invention, it is possible to monitor the dangerous area inside the airway in real time, timely warn of potential risk operations, effectively prevent accidental injuries during surgery, and ensure patient safety.

[0018] (3) Through the technical solution of the present invention, multimodal imaging data and prior three-dimensional model information can be integrated to establish a more accurate and comprehensive representation of the airway anatomical structure, providing a reliable basis for surgical planning and auxiliary decision-making.

[0019] (4) Through the technical solution of the present invention, it is possible to adaptively focus on key areas within the airway at different times and dynamically adjust attention allocation, thereby improving the robustness and adaptability of the system and being able to cope with complex and changing surgical scenarios.

[0020] (5) The technical solution of the present invention can significantly reduce the cognitive burden on doctors, lower the skill requirements for surgical operations, enable advanced surgical techniques to be adopted by more medical institutions and doctors, and improve the overall level of medical care. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the infant airway display system with electronic scale and early warning function in the present invention; Figure 2 is a schematic diagram of a visual laryngoscope in the system of the present invention; Figure 3 A schematic diagram of a mixed reality three-dimensional model in the present invention; Figure 4 is a schematic diagram of an augmented reality model in the present invention; Figure 5 When the system of the present invention is used, the user wears glasses to see the real-time image of the laryngoscope, the mixed reality three-dimensional model, and the schematic diagram of the augmented reality model. DETAILED DESCRIPTION

[0022] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0023] The present invention provides an infant airway display system with an electronic scale and an early warning function, such as Figure 1As shown, it includes: a visual laryngoscope for infants and young children, a mixed reality three-dimensional model generation unit, an augmented reality model generation unit, a data processing unit and AR glasses.

[0024] The infant-specific visual laryngoscope is as follows: Figure 2 As shown, it is used to obtain a real-time endoscopic image of the airway of an infant and send the real-time endoscopic image to the data processing unit; The mixed reality 3D model generation unit is used to construct a mixed reality 3D model containing a complete anatomical structure based on the infant chest CT data. The mixed reality 3D model is as follows: Figure 3 As shown, The augmented reality model generation unit is used to separate the three-dimensional airway model from the mixed reality three-dimensional model; annotate the key airway data obtained by measuring the key points of the important anatomical structures in the airway into the three-dimensional airway model, generate an augmented reality model, and send the key airway data to the data processing unit; the augmented reality model includes the annotated key measurement data (such as glottis diameter, etc.), such as Figure 4 As shown, it is used to help determine the catheter model and insertion depth in advance. For children who need to place a bronchial occluder and implement single-lung ventilation and lung isolation, it helps determine the position of the bronchial occluder cuff and reduces the difficulty of lung isolation in infants and young children.

[0025] The data processing unit is used to fuse the key airway data and the real-time endoscopic image to generate key point location prompt information and dangerous operation warning information, and send them to the AR glasses; The method for obtaining the key airway data includes: obtaining airway CT data based on the chest CT data of infants and young children for 3D printing to obtain an airway entity model, and performing precise ruler measurement on the airway entity model to obtain the key airway data. The key points include the glottis, tracheal carina or lobe opening; the key data include the glottis diameter, the subglottic stenosis diameter, the distance from the glottis to the tracheal carina or the distance from the tracheal carina to the right upper lobe opening.

[0026] The data processing unit includes a glottis position prompt and dangerous operation warning module, which calculates the matching degree between the real-time airway image data sequence and the key point data sequence in the three-dimensional airway model through a self-attention mechanism, which is specifically implemented as follows:

[0027] in, Airway real-time image frame Key points in the three-dimensional airway model The matching weight of represents the i-th frame of image data in the airway real-time image data sequence, Represents the j-th key point data in the three-dimensional airway model; represents the linear transformation matrix of the query vector, which is used to map the input x to the space of the query vector, Q represents the query vector; The linear transformation matrix of the key vector, used to map the input y to the space of the key vector, represents the key vector, represents the dimension of the scaling factor, represents the index of the key point in the three-dimensional airway model, Represents the total number of key points in the 3D airway model.

[0028] The focus representation of the key points is calculated in combination with the matching weights and used for glottal position prompts and dangerous operation warnings, which is specifically implemented as follows:

[0029] in, Used to indicate the The focus display of key points of the real-time airway image data is used to indicate the glottis position and dangerous operation warning in real time. Represents a linear transformation matrix for transforming the j-th key point data in the three-dimensional airway model Mapped to the value vector space, Represents a vector of values.

[0030] The data processing unit also includes a navigation function module, which is used to calculate the weighted matching score between the catheter trajectory point and the key point in the three-dimensional airway model by a weighted distance calculation method based on parameterized similarity, which is specifically implemented as follows:

[0031] Where: p f represents the coordinate vector of the fth catheter trajectory point, y g represents the g-th key point in the three-dimensional airway model, ‖ represents the vector concatenation operation; W γ represents the learnable weight matrix and b γ represents the learnable bias vector, LeakyReLU is the leaky rectified linear unit activation function, γ fg represents the catheter trajectory point p f and the key point y in the three-dimensional airway model g The weighted matching score.

[0032] This formula calculates the matching weights between the catheter trajectory point and the key point. It first concatenates the two vectors, then performs an inner product with the weight matrix, and then passes through the LeakyReLU activation function and softmax normalization to obtain the matching weights. This form introduces additional learnable parameters Wγ and bγ, and uses LeakyReLU instead of simple scaling operations, allowing for modeling of more complex matching patterns.

[0033] The weighted distance between the catheter trajectory point and the key point is calculated in combination with the weighted matching score, which is specifically implemented as follows:

[0034] Among them, d f represents the weighted distance between the fth catheter trajectory point and all key points in the three-dimensional airway model, which is used to calculate the distance between the catheter and the glottis or other key positions during navigation; p f represents the coordinate vector of the fth catheter trajectory point, y g represents the coordinate vector of the g-th key point in the three-dimensional airway model, ||p f -y g || represents the fth catheter trajectory point p f The key point y of the three-dimensional airway model g The Euclidean distance between .

[0035] The navigation function module further dynamically adjusts the optimal path for catheter insertion through a path planning algorithm, specifically including the following steps: based on the weighted distance between the catheter trajectory point and the key point, using the shortest path search algorithm to plan the optimal insertion path from the current position to the key point; dynamically monitoring the movement of the catheter, updating the distance between the current position and each of the key points in real time, and re-planning the optimal path to minimize the risk in the insertion path.

[0036] The mixed reality three-dimensional model is a common model shared by infants of different weights or a personalized model established for the symptoms of a single infant.

[0037] The AR glasses are used to display the mixed reality three-dimensional model, the augmented reality model, and the real-time endoscopic image superimposed with the key point position prompts and dangerous operation warning information, for auxiliary diagnosis and surgical planning.

[0038] The role of AR glasses in this system is to integrate all visual information into an augmented reality view, including: real-time endoscopic images; mixed reality 3D models that display subglottic anatomical structures that cannot be seen with a laryngoscope; and augmented reality models that include annotated key measurement data (such as glottis diameter, etc.). Among them, glottis position prompts and dangerous operation warning information are superimposed on the real-time endoscopic images.

[0039] All the above visual information is integrated into an augmented reality view to provide doctors with multi-dimensional and multi-angle airway visualization data. When the system of the present invention is used, the real-time endoscopic image of the laryngoscope, the mixed reality 3D model and the augmented reality model schematic diagram are shown in the figure. Figure 5 As shown in the figure, using these visualized data can assist clinicians in making more accurate diagnoses and surgical plans: key measurement data helps to assess the condition and select the appropriate operation method, glottal position prompts can guide intubation operations and improve accuracy, dangerous operation warnings can detect risks in time and avoid medical accidents, and three-dimensional anatomical views help to fully grasp the overall situation of the airway.

[0040] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0041] It is obvious to those skilled in the art that the embodiments of the present invention are not limited to the details of the exemplary embodiments described above, and that the embodiments of the present invention can be implemented in other specific forms without departing from the spirit or basic features of the embodiments of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the embodiments of the present invention is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the embodiments of the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units, modules or devices stated in the system, device or terminal claims can also be implemented by the same unit, module or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0042] Finally, it should be noted that the above implementation modes are only used to illustrate the technical solutions of the embodiments of the present invention and are not intended to limit them. Although the embodiments of the present invention have been described in detail with reference to the above preferred implementation modes, those skilled in the art should understand that the technical solutions of the embodiments of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An infant airway display system with an electronic scale and an early warning function, comprising: Infant-specific visual laryngoscope, mixed reality 3D model generation unit, augmented reality model generation unit, data processing unit and AR glasses; The infant-specific visual laryngoscope is used to obtain a real-time endoscopic image of the infant's airway and send the real-time endoscopic image to the data processing unit; The mixed reality 3D model generation unit is used to construct a mixed reality 3D model containing a complete anatomical structure based on the infant chest CT data; The augmented reality model generation unit is used to separate the three-dimensional airway model from the mixed reality three-dimensional model; annotate the key airway data obtained by measuring the key points of the important anatomical structures in the airway into the three-dimensional airway model, generate the augmented reality model, and send the key airway data to the data processing unit; The data processing unit is used to fuse the key airway data and the real-time endoscopic image to generate key point location prompt information and dangerous operation warning information, and send them to the AR glasses; The AR glasses are used to display the mixed reality three-dimensional model, the augmented reality model, and the real-time endoscopic image superimposed with the key point position prompts and dangerous operation warning information, for auxiliary diagnosis and surgical planning.

2. The infant airway display system according to claim 1, wherein the method for obtaining the key airway data comprises: Based on the chest CT data of infants and young children, airway CT data is obtained for 3D printing to obtain an airway entity model, and the airway entity model is precisely measured with a ruler to obtain the key airway data.

3. The infant airway display system according to claim 2, characterized in that: The key points include the glottis, tracheal carina or lung lobe opening; the key data include the glottis diameter, the subglottic stenosis diameter, the distance from the glottis to the tracheal carina or the distance from the tracheal carina to the right upper lobe opening.

4. The infant airway display system according to any one of claims 1 to 3, characterized in that: The data processing unit includes a glottis position prompt and dangerous operation warning module, which calculates the matching degree between the real-time airway image data sequence and the key point data sequence in the three-dimensional airway model through a self-attention mechanism, which is specifically implemented as follows: in, Airway real-time image frame Key points in the three-dimensional airway model The matching weight of represents the i-th frame of image data in the airway real-time image data sequence, Represents the j-th key point data in the three-dimensional airway model; represents the linear transformation matrix of the query vector, which is used to map the input x to the space of the query vector, Q represents the query vector; The linear transformation matrix of the key vector, used to map the input y to the space of the key vector, represents the key vector, represents the dimension of the scaling factor, represents the index of the key point in the three-dimensional airway model, Represents the total number of key points in the 3D airway model.

5. The infant airway display system according to claim 4, characterized in that: The focus representation of the key points is calculated in combination with the matching weights and used for glottal position prompts and dangerous operation warnings, which is specifically implemented as follows: in, Used to indicate the The focus display of key points of the real-time airway image data is used to indicate the glottis position and dangerous operation warning in real time. Represents a linear transformation matrix for transforming the j-th key point data in the three-dimensional airway model Mapped to the value vector space, Represents a vector of values.

6. The infant airway display system according to claim 5, characterized in that: The data processing unit includes a navigation function module, which is used to calculate the weighted matching score between the catheter trajectory point and the key point in the three-dimensional airway model by a weighted distance calculation method based on parameterized similarity, which is specifically implemented as follows: Where: p f represents the coordinate vector of the fth catheter trajectory point, y g represents the g-th key point in the three-dimensional airway model, ‖ represents the vector concatenation operation; W γ represents the learnable weight matrix and b γ represents the learnable bias vector, LeakyReLU is the leaky rectified linear unit activation function, γ fg represents the catheter trajectory point p f and the key point y in the three-dimensional airway model g The weighted matching score.

7. The infant airway display system according to claim 6, characterized in that: The weighted distance between the catheter trajectory point and the key point is calculated in combination with the weighted matching score, which is specifically implemented as follows: in, Indicates The weighted distance between each catheter trajectory point and all key points in the three-dimensional airway model is used to calculate the distance between the catheter and the glottis or other key positions during navigation; Indicates The coordinate vector of the catheter trajectory point, Indicates The coordinate vectors of the key points in the three-dimensional airway model, Indicates Catheter trajectory points Key points of the 3D airway model The Euclidean distance between .

8. The infant airway display system according to claim 7, characterized in that: The navigation function module further dynamically adjusts the optimal path for catheter insertion through a path planning algorithm, specifically including the following steps: based on the weighted distance between the catheter trajectory point and the key point, using the shortest path search algorithm to plan the optimal insertion path from the current position to the key point; dynamically monitoring the movement of the catheter, updating the distance between the current position and each of the key points in real time, and re-planning the optimal path to minimize the risk in the insertion path.

9. The infant airway display system according to any one of claims 5 to 8, characterized in that: The mixed reality three-dimensional model is a common model shared by infants of different weights or a personalized model established for the symptoms of a single infant.

Images