Human body posture estimation method, system and equipment based on multi-modal large language model and medium

Abstract

The invention relates to a human body posture estimation method, system and device based on a multi-modal large language model and a medium. The method comprises the following steps: acquiring language data of nursing personnel, a real-time image of a side-lying position of a surgical patient and basic body shape data of the surgical patient; converting the language data of the nursing personnel into text vectors through projection, and carrying out human body key point detection according to the side-lying position real-time image of the surgical patient and the basic body shape data of the surgical patient to obtain visual vectors; based on an LLAVA algorithm, performing multi-modal fusion on the text vector and the visual vector through the projection matrix to obtain a unified feature vector; and inputting the unified feature vector into a pre-trained multi-modal large language model to obtain a patient body position evaluation result and an interaction result. By means of the method, the human body posture of the surgical patient can be accurately recognized and evaluated, nursing personnel are assisted to complete lateral position placement of the patient, and lateral position related knowledge guidance inquiry of the nursing personnel is met in an interactive question and answer mode.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human posture estimation, and particularly relates to a human posture estimation method, system, device and medium based on a multi-modal large language model. BACKGROUND

[0002] With the increasing demand for precision in medical operations, surgical position management has become a key link to ensure the safety of operations and the comfort of patients. As a common position in thoracic surgery, neurosurgery and other departments, the lateral position has extremely high requirements for standardization. Improper placement of the lateral position can easily cause anatomical and physiological damage to the skin, respiratory, cardiovascular and nervous systems of the patient.

[0003] The existing surgical lateral position management mainly relies on the traditional manual operation mode. The nursing staff needs to observe the relative position of the patient's trunk, limbs and operating bed with the naked eye, and preliminarily judge whether the position meets the basic requirements. However, it is difficult to accurately judge key indicators such as shoulder joint abduction angle and hip joint flexion angle, and it is also difficult to determine whether it meets the clinical guideline standards. If there is any doubt about the position, the nursing staff needs to consult experienced medical staff or refer to paper materials to adjust the operation plan.

[0004] The above method is easily affected by surgical gown obstruction and visual angle deviation through the naked eye observation of the nursing staff, resulting in errors in position detail judgment. Whether the position meets the requirements depends on the experience of the operator, which is prone to differences. When encountering complex position problems, it is difficult to obtain professional guidance in a timely manner, which may delay the operation preparation process and increase the risk of position-related complications for patients. SUMMARY

[0005] Therefore, it is necessary to provide a human posture estimation method, system, device and medium based on a multi-modal large language model, which can complete surgical position evaluation through a computer system and help nursing staff in the operating room to accurately and efficiently place the lateral position.

[0006] In a first aspect, the application provides a human posture estimation method based on a multi-modal large language model, comprising:

[0007] Obtaining nursing staff language data, real-time images of a surgical patient in a lateral position, and basic body type data of the surgical patient. The nursing staff language data includes voice data and / or text data;

[0008] The nursing staff language data is converted into a text vector through projection, and a visual vector is obtained through human key point detection based on the real-time images of the surgical patient in the lateral position and the basic body type data of the surgical patient;

[0009] Based on the LLAVA algorithm, the text vector and the visual vector are multi-modal fused through a projection matrix to obtain a unified feature vector.

[0010] inputting the unified feature vector into a pre-trained multi-modal large language model to obtain a patient body position evaluation result and an interaction result; the multi-modal large language model performs compliance evaluation on a surgical patient lateral position and question and answer interaction on language data of nursing staff based on a pre-constructed nursing knowledge base; the nursing knowledge base includes Llam2-7B-based lateral position field knowledge question and answer pairs, lateral position standards and surgical body position rules.

[0011] In one of the embodiments, a visual vector is obtained by performing human key point detection on a real-time image of a surgical patient lateral position and basic body type data of the surgical patient, including:

[0012] 2D key point information is obtained by performing 2D-based human key point detection on the real-time image of the surgical patient lateral position.

[0013] 3D body position data of the surgical patient is completed based on the 2D key point information and the basic body type data of the surgical patient.

[0014] The 2D key point information and the 3D body position data are fused and encoded to obtain the visual vector.

[0015] In one of the embodiments, 2D key point information is obtained by performing 2D-based human key point detection on a real-time image of a surgical patient lateral position, including:

[0016] The real-time image of the surgical patient lateral position is segmented into fixed-size grid blocks to obtain a plurality of image units.

[0017] The image feature vector is obtained by inputting each image unit into a Transformer-based encoder; the Transformer-based encoder captures the association information between the image units through a multi-layer attention mechanism.

[0018] The 2D key point information is obtained by performing key point classification and joint coordinate regression decoding based on the image feature vector; the 2D key point information includes 2D joint coordinates.

[0019] In one of the embodiments, 3D body position data of the surgical patient is completed based on the 2D key point information and the basic body type data of the surgical patient, including:

[0020] Body type parameters are obtained based on the basic body type data of the surgical patient.

[0021] Posture parameters are determined based on the surgical patient lateral position.

[0022] A 3D basic template conforming to the body type and the standard lateral posture of the surgical patient is obtained based on the body type parameters and the posture parameters and a preset 3D body position template.

[0023] The 3D body position data is obtained based on a 2D-3D projection error objective function minimization according to the 2D joint information and the 3D base template.

[0024] In one embodiment, the 3D body position data is obtained based on a 2D-3D projection error objective function minimization according to the 2D joint information and the 3D base template, comprising:

[0025] The 2D-3D projection error objective function is:

[0026]

[0027] wherein, is a joint projection error; is a pose plausibility error; is a natural pose prior error; is a self-penetration error; is a body shape plausibility error; is a dynamic weight coefficient determined by the 2D joint information; , , and are weight coefficients respectively.

[0028] In one embodiment, the 3D body position data is obtained based on a 2D-3D projection error objective function minimization according to the 2D joint information and the 3D base template, comprising:

[0029] The body shape parameters and the pose parameters are optimized by a gradient descent algorithm until a target is met to obtain final body shape parameters and final pose parameters; the target is that a distance difference between 2D joint coordinates corresponding to 2D joint position coordinates after 2D projection of the 3D base template and the 2D joint position coordinates is minimum and a 2D-3D projection error objective function is minimum;

[0030] The 3D body position data is obtained according to the final body shape parameters and the final pose parameters.

[0031] In one embodiment, the multi-modal large language model obtains the patient body position evaluation result and the interaction result by the following method:

[0032] The body position information is extracted from the unified feature vector to obtain body parts and corresponding pose data;

[0033] The standard pose data of each body part is matched based on a nursing knowledge base, and the standard pose data is compared with the pose data to obtain a patient body position evaluation result;

[0034] The semantic analysis is performed on the unified feature vector to obtain a semantic vector;

[0035] Based on the nursing knowledge base, knowledge text corresponding to semantic vectors is matched, and based on preset prompt word engineering constraints, the knowledge text and patient positioning assessment results are transformed into natural language to obtain interactive results.

[0036] Secondly, this application also provides a human pose estimation system based on a multimodal large language model, including:

[0037] The data acquisition module is used to acquire language data of nursing staff, real-time images of surgical patients in lateral decubitus position, and basic body shape data of surgical patients; language data of nursing staff includes voice data and / or text data;

[0038] The feature vector module is used to convert nursing staff's language data into text vectors through projection, and to obtain visual vectors by detecting human key points based on real-time images of surgical patients in lateral decubitus position and basic body shape data of surgical patients.

[0039] The feature fusion module is used to perform multimodal fusion of text vectors and visual vectors through a projection matrix based on the LLAVA algorithm to obtain a unified feature vector;

[0040] The large language model module is used to input unified feature vectors into a pre-trained multimodal large language model to obtain patient positioning assessment results and interaction results. The multimodal large language model is based on a pre-built nursing knowledge base to conduct compliance assessment of surgical patients' lateral decubitus positions and to conduct question-and-answer interactions on nursing staff's language data. The nursing knowledge base includes Llam2-7B-based lateral decubitus position domain knowledge question-and-answer pairs, lateral decubitus position standards, and surgical positioning rules.

[0041] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above-described human pose estimation methods based on a multimodal large language model.

[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the above-described human pose estimation methods based on a multimodal large language model.

[0043] The aforementioned human pose estimation method, system, device, and medium based on a multimodal large language model acquires nursing staff's verbal data, real-time images of surgical patients in lateral decubitus positions, and basic body shape data of surgical patients. This encompasses both the communication of nursing staff's needs and the dynamic visual features of patient positioning and individual body shape differences. By using projection transformation to unify the nursing staff's verbal data into text vectors, and combining this with human keypoint detection to convert image and body shape data into visual vectors, the problem of heterogeneous formats and inability to directly fuse verbal semantic information and visual positioning information is solved. The LLAVA algorithm combined with a projection matrix achieves deep fusion of text and visual vectors, overcoming the limitations of independent text semantics and visual positioning, and avoiding the problem of mismatched evaluation criteria and surgical types leading to interactive responses that deviate from the patient's actual position. The system utilizes a multimodal large language model pre-trained with automation and clinical expertise to ensure efficient reasoning capabilities, handle positional assessment and question-and-answer interactions, and provides clinical compliance assurance for model output through a nursing knowledge base built on Llam2-7B. This enables self-service automated positional assessment, reducing subjective errors by nursing staff, and outputs interactive results that conform to clinical standards through the professional knowledge base, assisting nursing staff in efficiently and accurately placing patients in the lateral decubitus position. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the human pose estimation method based on a multimodal large language model according to the present invention.

[0046] Figure 2 This is a flowchart illustrating the steps of S12.

[0047] Figure 3 This is a flowchart illustrating the steps of step S204.

[0048] Figure 4 This is a structural diagram of the human pose estimation system based on a multimodal large language model according to the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In one embodiment, such as Figure 1 As shown, a human pose estimation method based on a multimodal large language model is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0051] S101. Acquire nursing staff's language data, real-time images of surgical patients in lateral decubitus position, and basic body shape data of surgical patients; nursing staff's language data includes voice data and / or text data.

[0052] In a schematic representation, nurses' verbal data serves as a vehicle for conveying needs, instructions, or questions, encompassing both voice and / or text data to transform their subjective requests into information that the system can process. Real-time images of surgical patients in the lateral decubitus position serve as a visual data source for capturing dynamic changes in patient posture; these can be images or image sequences converted from video, used to extract human posture features. Optionally, voice data can be collected in real-time via microphones or mobile terminals deployed in the operating room. The collection process supports noise reduction to filter out interference such as surgical instrument noise and environmental background noise. Furthermore, the collected voice data can be transcribed into text in real-time using a speech-to-text tool. Text data can be manually entered by nurses through the PC system interface or mobile app input boxes. Real-time images of surgical patients in the lateral decubitus position can be captured in real-time by high-definition cameras deployed around the operating table. The camera installation angle must cover the entire patient, focusing on the trunk and key joint areas of the limbs. The frame rate is set to 15-30 frames per second, and the image data can be directly transmitted to the system as an image sequence or video stream. The basic body data of surgical patients mainly includes core information such as the patient's height, weight, and gender. This data can be manually entered by nursing staff through the system input interface before surgery, or it can be automatically retrieved by connecting to the hospital's electronic medical record system. After the data is entered, the system will perform format verification to ensure the validity of the data.

[0053] S102. The language data of nursing staff is converted into text vectors by projection, and visual vectors are obtained by detecting human key points based on the real-time image of the surgical patient in the lateral decubitus position and the basic body shape data of the surgical patient.

[0054] In a schematic sense, projection refers to mapping language data in the form of natural language onto a pre-defined high-dimensional vector space. This involves extracting and quantifying the semantic features of language using natural language processing techniques. Specifically, the pre-processed language data, i.e., text or speech-to-text, is segmented into its smallest semantic units. A pre-trained text encoder, such as the BERT encoder based on Transformer, encodes these segmented semantic units, mapping each unit to a fixed-dimensional vector. Then, the vectors of all semantic units are integrated using methods such as mean pooling and max pooling to ultimately generate the text vector of the entire language data. This vector accurately represents the semantic information of the language, ensuring that the system understands the core needs of nursing staff.

[0055] The generation of visual vectors is achieved by taking real-time images of the surgical patient in the lateral decubitus position and basic body shape data as input, and by extracting human key points, completing body position information, and integrating and encoding features. Specifically, human key point detection technology in the field of computer vision is used to locate and extract the spatial feature information of the patient's key joints from the real-time images. Furthermore, combined with basic body shape data, information gaps caused by possible occlusion or viewpoint limitations in the images are filled in to form a complete human body position feature set. Optionally, this feature set is quantized and encoded by a visual encoder to be converted into a visual vector with the same dimension as the text vector. This vector covers the core spatial features of the patient's body position and provides data support for subsequent body position assessment.

[0056] S103. Based on the LLAVA algorithm, text vectors and visual vectors are fused in a multimodal manner through a projection matrix to obtain a unified feature vector.

[0057] Indicatively, the LLAVA (Large Language and Vision Assistant) algorithm is a multimodal fusion algorithm based on a large language model. It achieves deep alignment and semantic association between visual and textual information, solving the problem of incomplete information expression in a single modality. Specifically, a pre-set projection matrix is ​​loaded. This matrix, pre-trained on a large-scale vision-language dataset, maps visual vectors to the semantic space of text vectors. The visual vectors are then input into the projection matrix to complete the dimensionality transformation. Utilizing the built-in attention mechanism of the LLAVA algorithm, cross-modal attention calculations are performed on the transformed visual and text vectors to capture the semantic association between them. For example, when the text vector represents the evaluation of shoulder joint compliance, the attention mechanism strengthens the weights of shoulder joint-related features in the visual vector. Furthermore, through element-wise addition, weighted summation, or concatenation followed by normalization, the two attention-weighted vectors are integrated into a unified feature vector. This vector simultaneously contains semantic requirements and postural features, exhibiting good modal consistency, and can be directly input into the multimodal large language model for subsequent processing.

[0058] S104. Input the unified feature vector into the pre-trained multimodal large language model to obtain the patient positioning assessment results and interaction results. The multimodal large language model is based on a pre-built nursing knowledge base to conduct compliance assessment of the surgical patient's lateral decubitus position and to conduct question-and-answer interaction on the language data of nursing staff. The nursing knowledge base includes lateral decubitus position domain knowledge question-and-answer pairs based on Llam2-7B, lateral decubitus position standards, and surgical positioning rules.

[0059] In a schematic way, a multimodal large language model refers to a model that, based on a general large language model, is fine-tuned through a visual-language instruction set to possess cross-modal information understanding and reasoning capabilities. Based on a unified feature vector, it simultaneously achieves two major functions: patient positioning compliance assessment and interactive response to nursing staff needs. This replaces traditional manual assessment and consultation models, improving the accuracy and efficiency of surgical positioning management. Specifically, using general large language models, such as the LLaMA series and GPT series, as the base model, a visual-language instruction dataset specifically for lateral decubitus position nursing is constructed. This dataset includes images / videos related to the surgical lateral decubitus position, corresponding text descriptions (e.g., patient hip flexion at 25°), and instruction-response pairs (e.g., instruction assesses hip joint compliance, response meets standards, flexion angle is within the range of 15-30°). Furthermore, the dataset is combined with a pre-built nursing knowledge base to fine-tune the basic model. During training, the gradient descent algorithm is used to optimize the model parameters, enabling the model to accurately identify the body position information corresponding to visual features, and accurately match semantic requirements with the knowledge base content. In addition, the model's performance is validated using a validation set containing lateral decubitus data of patients with different surgical types and body types, as well as various nursing questions, to ensure that the model's evaluation accuracy is not less than 90% and the question-answering response accuracy is not less than 85%, meeting the requirements for clinical application.

[0060] The nursing knowledge base is based on the Llam2-7B model. It collects professional information related to surgical lateral decubitus positioning, as well as operational standards for lateral decubitus positioning in different departments and preventative measures for position-related complications. This information is organized into structured question-and-answer pairs. Further, the collected knowledge is deduplicated and categorized, clarifying lateral decubitus positioning standards. For example, these include the range of joint angles, the relative positional requirements between the body and the operating table, surgical positioning rules (placement procedures), and adjustment contraindications, forming the basic knowledge base. Optionally, the basic knowledge base can be linked to the visual-language instruction dataset generated during model training. Retrieval-enhanced generation (RAG) technology optimizes knowledge retrieval efficiency and supports dynamic updates, such as adding positioning standards for specific patient groups, ensuring the timeliness and comprehensiveness of the knowledge.

[0061] Specifically, after receiving a unified feature vector, the model processes it through feature parsing, knowledge base matching, and inference. This involves multimodal feature parsing of the unified feature vector to separate position-related visual features from the semantic needs of nursing staff. Further, based on the visual features, it extracts the patient's posture data for each body part and compares it with standard posture data in the nursing knowledge base to generate a patient posture assessment result. Simultaneously, based on the semantic needs features, it matches corresponding professional knowledge text in the nursing knowledge base. Through pre-defined prompts and engineering constraints, such as first outputting the assessment conclusion, then supplementing the standard basis, and finally providing operational suggestions, the assessment result and knowledge text are transformed into an interactive result in natural language.

[0062] The aforementioned human pose estimation method based on a multimodal large language model features both speech and text options, adapting to the diverse operational needs of nurses in surgical scenarios. By deriving key human points from real-time images of the surgical patient in lateral decubitus position and basic body shape data, the pose assessment becomes more individualized and dynamically accurate, avoiding bias from using a uniform visual standard to assess patients of different body types. Based on the LLAVA algorithm, text and visual vectors are fused and derived multimodally through a projection matrix, allowing for precise binding of semantic requirements in the text vectors with corresponding pose features in the visual vectors, improving the targeted effectiveness of assessment and interaction. The multimodal large language model can quickly output pose assessment results and interactive responses, avoiding the impact of excessive computation time on surgical pose placement efficiency and adapting to the real-time need for timely pose adjustments in surgical scenarios. The nursing knowledge base ensures clinical compliance in assessment and interaction, reducing operational risks. The standard lateral decubitus surgical pose rules in the knowledge base directly correspond to clinical nursing guidelines, ensuring that pose assessment results strictly adhere to clinical standards and avoiding misjudgments due to personal experience biases.

[0063] In one embodiment, such as Figure 2 As shown, visual vectors are obtained by detecting human key points based on real-time images of the surgical patient in the lateral decubitus position and the patient's basic body shape data. These vectors include:

[0064] S11. Based on the real-time image of the surgical patient in the lateral decubitus position, perform 2D-based human key point detection to obtain 2D joint information.

[0065] In a schematic manner, the input real-time image of a surgical patient in a lateral decubitus position is adapted. If it is video data, keyframes are extracted at a frequency of 10-15 frames per second. Then, the image / keyframe is normalized in size and pixel value to eliminate the interference of different device shooting resolution and light intensity differences on the detection results. For non-human areas such as surgical drapes and instruments, preliminary masking is performed through simple semantic segmentation to reduce the attention occupied by irrelevant areas on the algorithm. Optionally, VITPose is used for 2D human keypoint detection. Specifically, the preprocessed image enters the VITPose Encoder module. This module first divides the image into fixed-size grid blocks of 16×16 pixels, forming multiple independent image units. Further, the image units are converted into vector form and superimposed with position embedding vectors, which are then input to a multi-layer Transformer encoder. The encoder captures the spatial relationship between different grid blocks through a self-attention mechanism, and finally outputs an image feature vector containing global feature relationships. Furthermore, the image feature vectors are fed into VITPose's Decoder module, which performs keypoint detection through two parallel tasks: keypoint classification and coordinate regression, to obtain 2D keypoint information.

[0066] S12. Based on the 2D joint information as the basic framework, the 3D position of the surgical patient is completed according to the basic body shape data of the surgical patient to obtain the 3D position data.

[0067] In surgical settings, patients often fail to detect certain 2D joints due to surgical drapes or instrument obstruction, and 2D coordinates cannot reflect the spatial depth of these joints. This paper illustratively uses 2D joint information as skeletal anchor points, combined with the patient's basic body shape data, to complete the 3D body position data using the SMPL (Skinned Multi-Person Linear Model) 3D human motion capture model. This upgrades the dimensionality from planar to spatial form, providing key indicators such as spatial angles and relative heights for position compliance assessment. Specifically, the core parameters of the SMPL model are determined based on the patient's basic body shape data, including body shape parameters. Based on the patient's height, weight, and gender, initial β values ​​are matched from a pre-defined clinical body shape parameter library to determine the body shape characteristics of the 3D model, such as trunk width and limb length. Posture parameters. Based on the surgical lateral decubitus position, the θ value is initialized to ensure that the initial posture conforms to the basic physiological laws of the surgical position, reducing the number of subsequent optimization iterations. Combining the initial β and θ parameters, the template generation formula of the SMPL model is used. Where T is the average human template. This is a correction term for body shape deformation. For joint deformation correction, a 3D base template is generated that conforms to the patient's approximate body shape and basic lateral decubitus posture. Furthermore, the 3D base template is optimized based on 2D joint point information to ensure that the joint point coordinates projected onto the 2D plane are consistent with the 2D joint point information.

[0068] S13. The 2D joint information and 3D body position data are fused and encoded to obtain a visual vector.

[0069] In a schematic way, all 2D key point coordinates, i.e. pixel values, are mapped to the [0,1] range through coordinate / image side length, thus eliminating the influence of different image resolutions on coordinate values. The 3D joint coordinates are mapped to the [0,1] interval by the coordinate / human height to eliminate the influence of different body types on the coordinate values. An improved VITPose Block is selected as the visual encoder. This visual encoder adds a cross-dimensional attention layer on the basis of the traditional VITPose Block to achieve deep association between 2D and 3D features. For example, the standardized 2D joint features and 3D body position features are concatenated to form the initial fused feature vector. The cross-dimensional attention layer of the visual encoder calculates the association weight between 2D features and 3D features, such as the correlation between the x-coordinate of the hip joint in 2D and the Z-axis height of the hip joint in 3D. It strengthens mutually corroborating features. For example, if the x-coordinate of the hip joint in 2D is large and the Z-axis height of the hip joint in 3D is high, it indicates that the patient's lower limb may be externally rotated. The weight of this feature will be increased to weaken contradictory or noisy features. The fused feature vector after attention weighting is compressed to the same dimension as the text vector through a fully connected layer and normalized through a LayerNorm layer to finally generate a visual vector.

[0070] In one embodiment, 2D-based human keypoint detection is performed based on real-time images of the surgical patient in a lateral decubitus position to obtain 2D joint information, including:

[0071] S21. Divide the real-time image of the surgical patient in the lateral decubitus position into grid blocks of fixed size to obtain multiple image units.

[0072] To illustrate, the size of real-time images of surgical patients in the lateral decubitus position is normalized. Bilinear interpolation algorithm can be used to unify the target pixels, avoiding feature deformation caused by image stretching. The image pixel values ​​are mapped to the [0,1] interval to eliminate the influence of pixel value fluctuations under different lighting conditions. Furthermore, lightweight semantic segmentation models such as MobileNetV2+U-Net are used to quickly segment the image, that is, to identify non-human areas such as operating table, instruments, and surgical drapes, and set their mask values ​​to 0, and to perform equidistant segmentation on the processed image.

[0073] S22. Input each image unit into the Transformer-based encoder to obtain the image feature vector; the Transformer-based encoder captures the correlation information between each image unit through a multi-layer attention mechanism.

[0074] In a schematic manner, each image unit is transformed into a one-dimensional vector, and a position embedding vector is added to each vector to reflect its spatial position in the original image. The fused vector of the one-dimensional vector and the position embedding vector is input into a multi-layer Transformer encoder for feature association calculation. Specifically, a 12-layer Transformer encoder is preferred, with each layer containing a multi-head self-attention layer and a feedforward neural network layer. For example, the fused vector is split into 8 attention heads, and each attention head calculates the attention weight between all grid block vectors. The higher the weight value, the stronger the feature association between the two grid blocks. The calculation results of the 8 attention heads are concatenated and processed by linear transformation compression and layer normalization (LayerNorm) to eliminate the influence of the difference in feature distribution between different attention heads. Furthermore, the normalized feature vector is input into a feedforward neural network composed of two fully connected layers and a ReLU activation function to perform nonlinear transformation on the features, enhance key features, such as gradient features of joint edges, and suppress noise features, such as small fluctuations in skin texture. After processing by a 12-layer encoder, the vector of each grid block contains global spatial correlation information. Finally, the vectors of all grid blocks are integrated into a 768-dimensional image feature vector through mean pooling, which can fully characterize the global features of the patient's lateral decubitus image.

[0075] Optionally, confidence detection involving the human body is performed on each image unit. If a certain area grid block is occluded by a surgical drape, the self-attention layer will automatically increase the weight of its adjacent high-confidence grid blocks. For example, if the back sacrococcygeal joint grid block is occluded and its confidence is 0.2, the weight of the waist 18-12 grid block and the buttock 20-12 grid block will be increased, and the characteristics of the sacrococcygeal joint will be inferred through the relative position of the two.

[0076] S23. Perform joint point classification and joint coordinate regression decoding based on image feature vectors to obtain 2D joint point information; the 2D joint point information includes 2D joint coordinates.

[0077] In a schematic representation, the keypoint classification task uses a fully connected layer and a sigmoid activation function to determine whether each grid cell corresponding to an image feature vector contains key keypoints, such as the shoulder, hip, and ankle joints—preset core joints for the lateral decubitus position during surgery—and outputs the probability of each keypoint's presence. A probability threshold is set; if the probability of a keypoint's presence is greater than or equal to the threshold, it is considered present and proceeds to coordinate regression; if it is less than the threshold, it is considered absent or severely occluded, and coordinate calculation is temporarily suspended to avoid incorrect localization. The coordinate regression task calculates the precise pixel coordinates of the keypoint within the grid cell for each identified keypoint. This coordinates are then combined with the grid cell's position in the original image to obtain the keypoint's absolute 2D coordinates within the entire image. Finally, the results of the two tasks are fused to obtain 2D keypoint information containing the 2D coordinates of all core keypoints.

[0078] In one embodiment, such as Figure 2 As shown, based on 2D joint information as the basic architecture, the 3D position of the surgical patient is completed according to the patient's basic body shape data, resulting in 3D position data, including:

[0079] S201. Obtain body shape parameters based on the patient's basic body shape data.

[0080] To illustrate, the collected patient baseline body shape data is standardized to eliminate differences in units and value ranges. Specifically, the actual height is mapped to the [0,1] interval, and BMI is calculated by combining gender = weight (kg) / (height × height) (m). Then, BMI is mapped to the [0,1] interval, and gender is converted into a binary variable to match the statistical regularity of body shape parameters for different genders.

[0081] Optionally, a pre-constructed clinical surgical patient body shape parameter database can be built. This database collects body shape data from over 10,000 surgical patients of different genders, age groups, and BMIs, and obtains corresponding parameters by fitting the data using the SMPL model. Parameters, forming gender, standardized height, standardized BMI and A parameter mapping table. Illustratively, based on a subset of same-sex data in the patient gender screening database, the K-nearest neighbor algorithm (K=5) is used to find the 5 groups of samples that are closest to the patient's standardized height and standardized BMI. The 5 groups of samples... The average value of the parameters is taken as the patient's body shape parameter.

[0082] S202. Determine the posture parameters based on the lateral decubitus position of the surgical patient.

[0083] Indicative, The parameters are based on the clinical mapping of core joint angles in the lateral decubitus position, clearly identifying the joints requiring key control during surgery in this position and their standard angle ranges, such as the cervical joints (0° pitch, 0° yaw, and 0° lateral tilt) to avoid nerve compression. These standard angles are converted into rotational radian values ​​recognizable by the SMPL model, serving as... The parameters correspond to the initial values ​​of the joint dimensions. Optionally, different departments may have specific requirements for the lateral decubitus position during surgery, allowing for fine-tuning of the initial θ value.

[0084] S203. Based on the preset 3D body position template, a 3D basic template that conforms to the body shape and standard lateral decubitus posture of the surgical patient is obtained according to the body shape parameters and posture parameters.

[0085] The preset 3D posture template refers to the average 3D human body template T built into the SMPL model, derived from statistical data on the body shape and posture of a large number of normal adults. This template contains the initial spatial coordinates of over 2000 vertices of the human body, representing the basic form of a standard body shape and natural standing posture. Further, through body shape and posture correction, the average template T is transformed into a 3D basic template that conforms to the patient's body shape and aligns with a lateral decubitus posture. ,in, Bs(β), (as preset body shape basis vectors), each Corresponding to a body size basis vector For example, (If the torso is relatively wide), then It will stretch the top of the average template T to both sides, and the stretching amplitude is the same as Proportional. After body shape correction, template T is transformed into Furthermore, the attitude parameters Substitute into the joint deformation function Templates after body shape correction Adjusting the posture, among which, It consists of two parts: a joint rotation matrix and skin deformation weights. Specifically, through... The rotation matrix is ​​calculated for each joint's rotation dimension. The vertices around the joints are rotated by an angle. Based on preset skin deformation weights, the vertices at joint folds are fine-tuned. After pose correction, the template... Transform into This refers to the final 3D base template.

[0086] S204. Based on the objective function of minimizing the 2D-3D projection error, 3D body position data is obtained according to 2D joint information and 3D basic template.

[0087] Indicatively, the joints of the current 3D base template are projected onto a 2D plane. The distance difference between the projected 2D coordinates and the actual detected 2D joint coordinates is calculated, such as the Euclidean distance between the projected and actual coordinates of the shoulder joint. If the distance difference is greater than a preset threshold, the gradient descent algorithm is used for adjustment. and Parameters are used to regenerate the 3D template and calculate the projection error. This process is repeated until the projection error is less than the threshold and the objective function value is minimized. The result obtained at this point is... and It can accurately match the patient's actual body shape and position to obtain 3D position data, including the 3D spatial coordinates of all core joints, the spatial angles of key body parts, and the human body mesh model.

[0088] The above method can achieve global completion of body position information by back-deriving the 3D coordinates of occluded joints in 2D images through parameter optimization of the SMPL model.

[0089] In one embodiment, based on a 2D-3D projection error objective function, 3D body position data is obtained from 2D keypoint information and a 3D base template, including:

[0090] The objective function for the projection error in 2D-3D is given by the following formula:

[0091]

[0092] in, This refers to joint projection error; This is for attitude rationality error; This represents the prior error of the natural pose. This is the self-penetration error; Allow for reasonable error in body shape; These are dynamic weighting coefficients, determined by 2D joint information; , , and These are the weighting coefficients.

[0093] Indicative It is an index that quantifies the deviation between the coordinates of the joints of the 3D body positioning model projected onto the 2D plane and the coordinates of the actual detected 2D joints. The smaller the value, the closer the 3D model is to the actual body positioning in the image, which is key to ensuring that the 3D body positioning does not deviate from the image reference. If the error is too large, even if other error items are within acceptable limits, the 3D model will still have a significant deviation from the patient's actual position. Weighted Euclidean distance is used for calculation, taking into account the deviation of all core key points, and dynamic weights are also employed. To address occlusion issues in surgical settings, the specific formula is as follows: ,in, , For the first The pixel coordinates of each joint point projected from the 3D model onto the 2D plane can be calculated using the camera model, and then substituted with the camera's installation height, focal length, and horizontal angle in the surgical scene to simulate the real imaging process. , The first step in the actual detection of 2D joint information 2D pixel coordinates of each joint point The detection confidence level of 2D joints is determined by the probability that the joint exists is ≥0.8. Set to 1.0; moderate occlusion exists with a probability of 0.7-0.8. Set to 0.6; there exists a probability of complete occlusion < 0.7. Set to 0.2 to avoid low-confidence occluded joints interfering with the overall error calculation. N is the total number of joints involved in the calculation. It is an indicator that quantifies the deviation between the joint angle of the 3D model and the range of motion of the human physiological joints. It prevents postures that do not conform to human anatomy from appearing during the optimization process, such as the knee joint bending backward or the shoulder joint abducting beyond 180°. If the joint angle is within the preset tolerance range, the penalty value is 0; if it exceeds the range, the penalty value increases linearly with the deviation. It is an indicator that quantifies the similarity between the 3D model's posture and a database of natural lateral decubitus postures in clinical surgery. It avoids the appearance of abnormal postures in the 3D model that are physiologically possible but not used in clinical practice. It relies on a pre-built database of natural lateral decubitus postures in surgery, which contains 3D posture data of lateral decubitus positions for more than 5,000 patients with different surgical types and body types. Each data point is labeled as a natural posture. For example, the posture feature vector of the 3D model is extracted, and the similarity between the model and all natural posture feature vectors in the posture database is calculated using cosine similarity. The higher the similarity, the smaller the penalty value. It is an indicator that quantifies whether different parts of the human body in a 3D model have spatial penetration, avoiding physical contradictions such as an elbow penetrating the torso or a knee penetrating the thigh. The ball collision detection method simplifies the key parts of the human body into a sphere with a fixed radius. The Euclidean distance between the center joints of the two parts is calculated, which is the 3D spatial distance between the centers of the sphere. If the sphere does not collide, there is no penetration and the penalty value is 0; if the sphere collides, there is penetration and the penalty value is the difference between the sphere radius and the distance. Body shape parameters during the quantitative optimization process The variation range is compared with the index of the patient's actual body shape deviation, to prevent excessive distortion of the patient's body shape in order to fit the 2D projection. The deviation threshold of body shape parameters before and after optimization can be used to constrain the clinical body shape range.

[0094] In one embodiment, such as Figure 3 As shown, based on the objective function of minimizing the 2D-3D projection error, 3D body position data is obtained from 2D keypoint information and 3D basic template, including:

[0095] S301. Optimize the body shape parameters and posture parameters using the gradient descent algorithm until the target is met, and obtain the final body shape parameters and final posture parameters. The target is to minimize the distance difference between the 2D joint position coordinates and the corresponding 2D joint coordinates after 2D projection of the 3D basic template, and to minimize the 2D-3D projection error objective function.

[0096] This is illustrative, and the clinical body type database is used as a reference. Each dimension has upper and lower limits set to ensure the optimized result. The corresponding body type conforms to the normal physiological range of adults, and is... Each joint rotation dimension has a safety range set to avoid physiologically impossible postures. The gradient is the objective function. The partial derivative with respect to each parameter to be optimized reflects the effect of small changes in the parameter on... The direction and magnitude of the influence, the direction of the negative gradient is the direction that makes The direction of fastest reduction is the core basis for parameter updates, and this is used to... and Iterative updates are performed until a dual termination condition is met, for example, the average distance difference between the 3D projection and the 2D coordinates. The sum of all of them is the smallest.

[0097] S302. Obtain 3D body position data based on the final body shape parameters and final posture parameters.

[0098] Template generation formula based on SMPL model By substituting the final body shape parameters and final posture parameters, a 3D human body mesh composed of multiple vertices and triangular faces can be obtained. Key quantitative indicators required for surgical positioning assessment can be extracted from the mesh, including 3D joint coordinates, that is, the 3D spatial coordinates (X,Y,Z) of 23 core joints are extracted, as well as joint spatial angles and spatial relationships of body parts as 3D positioning data.

[0099] In one embodiment, the multimodal large language model obtains patient positioning assessment results and interaction results through the following method:

[0100] S31. Extract body position information from the unified feature vector to obtain body parts and their corresponding pose data.

[0101] Indicatively, each dimension of the unified feature vector corresponds to a semantic feature or a positional feature. During the training phase, the multimodal large language model has learned the attributes of the feature dimensions through a visual-language instruction set. For example, dimensions related to shoulder joint coordinates and hip joint angles are labeled as positional dimensions, and dimensions related to assessment and adjustment are labeled as semantic dimensions. During extraction, a preset positional feature mask is loaded, activating only the features labeled as positional dimensions and shielding the interference of semantic dimensions, ensuring that the extracted objects are only position-related data. Furthermore, the correspondence between feature dimension combinations and body parts is learned through the surgical lateral decubitus position dataset, ultimately outputting a list of body parts covering the core parts to be assessed in the surgical lateral decubitus position and their pose data. Among them, 3D spatial angles and 3D coordinates are extracted for joint parts, spatial relationships with the operating table are extracted for trunk parts, and 3D protrusion degree is extracted for pressure risk parts.

[0102] S32. Based on the nursing knowledge base, match the standard posture data of each body part, compare the standard posture data with the posture data, and obtain the patient's posture assessment results.

[0103] As an illustration, the nursing knowledge base adopts a hierarchical index structure. The first-level index is the surgical department, the second-level index is the surgical type, and the third-level index is the patient's special condition. During matching, the multimodal large language model will parse the surgical type input by the nurse from the semantic dimension of the unified feature vector and automatically locate the corresponding standard data entry. For example, compliance of each body part is calculated by comparing indicators item by item and quantifying deviations. Angle-type indicators calculate the deviation between the actual value and the standard range, spatial relationship-type indicators calculate the absolute difference between the actual value and the standard median value, and stress and risk-type indicators calculate the ratio of the actual value to the standard upper limit as the core data of the evaluation result. Optionally, if the deviation values ​​of all angle-type indicators are ≤2° and the stress-type risk coefficient is ≤1.0, it is judged as overall compliance; if 1-2 indicators are out of tolerance, it is judged as partial compliance, requiring fine-tuning; if ≥3 indicators are out of tolerance or 1 is severely out of tolerance, it is judged as non-compliance. Furthermore, the specific locations and deviations that exceeded the tolerances were clearly identified, such as a 7° lateral curvature angle of the neck (standard ≤ 5°, deviation 2°) and a 3cm height of the anterior superior iliac spine protrusion (standard ≤ 3cm, medium risk).

[0104] S33. Perform semantic analysis on the unified feature vector to obtain the semantic vector.

[0105] Similarly, a semantic feature mask is loaded, activating features marked as semantic dimensions in the unified feature vector while masking interference from the postural dimension. Then, a semantic encoder transforms the activated semantic features into semantic vectors. Further, a semantic classification sub-network, consisting of two fully connected layers and a Softmax activation function, maps the semantic vectors to preset demand types. These include assessment-type demands, with core keywords indicating compliance with standards and correctness (e.g., assessing whether the current posture conforms to neurosurgical standards). The classification sub-network outputs the probability of this type; if ≥0.8, it is classified as assessment. Demand types also include consultation-type demands, with core keywords indicating what and why needs to be understood (e.g., what postural knowledge should be considered during intracranial surgery); a probability ≥0.8 indicates consultation. Demand types also include operational demands, with core keywords indicating how to adjust and handle (e.g., how to adjust the lateral curvature of the neck); a probability ≥0.8 indicates operation. If the probability of a certain demand type is less than 0.8, attention weight analysis is used to determine the keyword with the highest weight in the semantic vector, prioritizing it as the demand type with the highest weight.

[0106] To further accurately match knowledge text, it is necessary to extract demand entities from semantic vectors. For example, Named Entity Recognition (NER) technology is used to identify entities such as surgical type, body part, and demand scenario in semantics. For instance, from how to adjust the scoliosis of a patient undergoing intracranial surgery, the entities are extracted as follows: surgical type = intracranial surgery, body part = neck, and demand scenario = adjustment. After entity extraction, it is associated with the postural assessment results.

[0107] S34. Based on the nursing knowledge base, match the knowledge text corresponding to the semantic vector, and based on the preset prompt word engineering constraints, convert the knowledge text and the patient's position assessment results into natural language to obtain the interactive results.

[0108] To illustrate, the knowledge texts stored in the nursing knowledge base are organized in a relational form of demand type-entity-knowledge content, such as operation category-neck-intracranial surgery neck adjustment plan. Specifically, the most relevant knowledge texts are retrieved from the knowledge base using demand type and demand entity as indexes. For example, if the operation category demand and entity are intracranial surgery and neck, the retrieved knowledge text is: "Neck adjustment for intracranial surgery requires maintaining a neutral position. During adjustment, the neck should be gently supported to avoid violent twisting; after adjustment, a neck pillow should be used for fixation. The height of the neck pillow should be selected according to the patient's body size, 5-8cm, to ensure that the lateral curvature angle of the neck is ≤5°." If multiple related knowledge texts are retrieved, the text similarity is calculated using cosine similarity, and the text with the highest similarity to the semantic vector is selected.

[0109] Furthermore, to avoid generating natural language with confusing logic or that does not conform to clinical operating habits, the output format is preset through the prompt word engineering. In surgical scenarios, scenario-based prompt word templates are mainly used, with different templates corresponding to different needs. For assessment-related needs, the template can include overall compliance conclusions, details of abnormal points, risk warnings, and standard basis. For operation-related needs, the template can include adjustment goals, operation steps, precautions, and verification methods. For consultation-related needs, the template can include core knowledge points, clinical significance, applicable scenarios, and related suggestions.

[0110] Optionally, the multimodal large language model adopts an autoregressive generation mode, which integrates knowledge text and postural assessment results into natural language. During generation, the model will first embed personalized data from the assessment results, and then combine it with the general rules of knowledge text to generate targeted content. For example, the neck pillow height of 5-8cm is refined to recommend an 8cm neck pillow based on the patient's obese body type, and the lateral curvature of the neck should be reduced to within 5° after adjustment. During the generation process, clinical logic verification will be used to eliminate contradictory content, and natural language polishing will be used to avoid rigid parameter listing.

[0111] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0112] Based on the same inventive concept, this application also provides a human pose estimation system based on a multimodal large language model for implementing the aforementioned human pose estimation method based on a multimodal large language model. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the human pose estimation system based on a multimodal large language model provided below can be found in the limitations of the human pose estimation method based on a multimodal large language model described above, and will not be repeated here.

[0113] In one exemplary embodiment, such as Figure 4 As shown, a human pose estimation system based on a multimodal large language model is provided, including:

[0114] Data acquisition module 401 is used to acquire nursing staff language data, real-time images of surgical patients in lateral decubitus position, and basic body shape data of surgical patients; nursing staff language data includes voice data and / or text data;

[0115] The feature vector module 402 is used to convert nursing staff's language data into text vectors through projection, and to obtain visual vectors by detecting human key points based on real-time images of surgical patients in lateral decubitus position and basic body shape data of surgical patients.

[0116] The feature fusion module 403 is used to perform multimodal fusion of text vectors and visual vectors through a projection matrix based on the LLAVA algorithm to obtain a unified feature vector;

[0117] The large language model module 404 is used to input a unified feature vector into a pre-trained multimodal large language model to obtain patient positioning assessment results and interaction results. The multimodal large language model is based on a pre-built nursing knowledge base to conduct compliance assessment of surgical patients' lateral decubitus positions and to conduct question-and-answer interactions on nursing staff's language data. The nursing knowledge base includes Llam2-7B-based lateral decubitus position domain knowledge question-and-answer pairs, lateral decubitus position standards, and surgical positioning rules.

[0118] In one embodiment, the feature vector module 402 is further configured to:

[0119] Based on real-time images of surgical patients in lateral decubitus position, 2D-based human key point detection is performed to obtain 2D joint information;

[0120] Based on 2D joint information, the 3D position of the surgical patient is completed according to the basic body shape data of the surgical patient to obtain 3D position data.

[0121] The 2D joint information and 3D body position data are fused and encoded to obtain a visual vector.

[0122] In one embodiment, a 2D module is also included for:

[0123] The real-time image of the surgical patient in the lateral decubitus position is segmented into grid blocks of fixed size to obtain multiple image units;

[0124] Each image unit is input into a Transformer-based encoder to obtain an image feature vector; the Transformer-based encoder captures the correlation information between each image unit through a multi-layer attention mechanism;

[0125] Joint point classification and joint coordinate regression decoding are performed based on image feature vectors to obtain 2D joint point information; the 2D joint point information includes 2D joint coordinates.

[0126] In one embodiment, a 3D module is also included for:

[0127] Body shape parameters are obtained based on the patient's baseline body shape data.

[0128] The posture parameters are determined based on the lateral decubitus position of the surgical patient.

[0129] Based on the preset 3D body position template, a 3D basic template that conforms to the body shape and standard lateral decubitus posture of the surgical patient is obtained according to the body shape parameters and posture parameters.

[0130] Based on the objective function of minimizing the 2D-3D projection error, 3D body position data is obtained from 2D joint information and 3D basic template.

[0131] In one embodiment, the 3D module is also used for:

[0132] The body shape and posture parameters are optimized using the gradient descent algorithm until the objective is met, resulting in the final body shape and posture parameters. The objective is to minimize the distance difference between the 2D joint coordinates after 2D projection of the 3D base template and the corresponding 2D joint coordinates, and to minimize the 2D-3D projection error objective function.

[0133] 3D body position data are obtained based on the final body shape parameters and final posture parameters.

[0134] In one embodiment, the large language model module 404 is further configured to:

[0135] Body position information is extracted from the unified feature vector to obtain body parts and their corresponding pose data;

[0136] Based on the nursing knowledge base, standard posture data of each body part is matched, and the standard posture data is compared with the posture data to obtain the patient's posture assessment results.

[0137] Semantic analysis is performed on the unified feature vector to obtain the semantic vector;

[0138] Based on the nursing knowledge base, knowledge text corresponding to semantic vectors is matched, and based on preset prompt word engineering constraints, the knowledge text and patient positioning assessment results are transformed into natural language to obtain interactive results.

[0139] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0140] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0141] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0142] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for human pose estimation based on a multimodal large language model, characterized in that, The method includes: Acquire nursing staff's language data, real-time images of surgical patients in lateral decubitus position, and basic body shape data of surgical patients; the nursing staff's language data includes voice data and / or text data; The language data of the nursing staff is converted into text vectors by projection, and visual vectors are obtained by detecting human key points based on the real-time image of the surgical patient in the lateral decubitus position and the basic body shape data of the surgical patient. Based on the LLAVA algorithm, the text vector and the visual vector are fused using a projection matrix to obtain a unified feature vector. The unified feature vector is input into a pre-trained multimodal large language model to obtain patient positioning assessment results and interaction results. The multimodal large language model is based on a pre-built nursing knowledge base to conduct compliance assessment of the lateral decubitus position of surgical patients and to conduct question-and-answer interaction on language data of nursing staff. The nursing knowledge base includes lateral decubitus position domain knowledge question-and-answer pairs based on Llam2-7B, lateral decubitus position standards, and surgical positioning rules.

2. The method according to claim 1, characterized in that, The step of detecting key human features and obtaining visual vectors based on real-time images of the surgical patient in a lateral decubitus position and the patient's basic body shape data includes: Based on the real-time images of the surgical patient in the lateral decubitus position, 2D-based human key point detection was performed to obtain 2D joint information; Based on the 2D joint information, the 3D position of the surgical patient is completed according to the basic body shape data of the surgical patient to obtain 3D position data. The 2D joint information and the 3D body position data are fused and encoded to obtain the visual vector.

3. The method according to claim 2, characterized in that, The step of performing 2D-based human key point detection based on the real-time image of the surgical patient in the lateral decubitus position to obtain 2D joint information includes: The real-time image of the surgical patient in the lateral decubitus position is divided into grid blocks of fixed size to obtain multiple image units; Each of the image units is input into a Transformer-based encoder to obtain an image feature vector; the Transformer-based encoder captures the correlation information between the image units through a multi-layer attention mechanism; The 2D joint information is obtained by classifying joints and regressing joint coordinates based on the image feature vectors; the 2D joint information includes 2D joint coordinates.

4. The method according to claim 3, characterized in that, The process involves using the 2D joint information as a basic framework, and supplementing the surgical patient's 3D position based on the patient's basic body shape data to obtain 3D position data, including: Body shape parameters were obtained based on the patient's baseline body shape data. The posture parameters are determined based on the lateral decubitus position of the surgical patient. Based on the preset 3D body position template, a 3D basic template that conforms to the body shape and standard lateral decubitus posture of the surgical patient is obtained according to the body shape parameters and the posture parameters. Based on the objective function of minimizing the 2D-3D projection error, the 3D body position data is obtained according to the 2D joint information and the 3D basic template.

5. The method according to claim 4, characterized in that, The objective function based on minimizing the 2D-3D projection error, which obtains the 3D body position data according to the 2D joint information and the 3D basic template, includes: The objective function for the projection error of the 2D-3D model is given by the following formula: in, This refers to joint projection error; This is for attitude rationality error; This represents the prior error of the natural pose. This is the self-penetration error; Allow for reasonable error in body shape; These are dynamic weighting coefficients, determined by 2D joint information; , , and These are the weighting coefficients.

6. The method according to claim 4, characterized in that, The objective function based on minimizing the 2D-3D projection error, which obtains the 3D body position data according to the 2D joint information and the 3D basic template, includes: The body shape parameters and the posture parameters are optimized using the gradient descent algorithm until the target is met, resulting in the final body shape parameters and the final posture parameters. The target is to minimize the distance difference between the 2D joint point position coordinates after 2D projection of the 3D base template and the corresponding 2D joint coordinates, and to minimize the objective function of the 2D-3D projection error. The 3D body position data is obtained based on the final body shape parameters and the final posture parameters.

7. The method according to claim 1, characterized in that, The multimodal large language model obtains the patient positioning assessment results and the interaction results through the following methods: Body position information is extracted from the unified feature vector to obtain body parts and their corresponding pose data; Based on the nursing knowledge base, standard posture data for each body part is matched, and the standard posture data is compared with the posture data to obtain the patient's posture assessment result. Semantic analysis is performed on the unified feature vector to obtain a semantic vector; Based on the nursing knowledge base, the knowledge text corresponding to the semantic vector is matched, and based on the preset prompt word engineering constraints, the knowledge text and the patient's postural assessment results are converted into natural language to obtain the interaction results.

8. A human pose estimation system based on a multimodal large language model, characterized in that, The system includes: The data acquisition module is used to acquire nursing staff language data, real-time images of surgical patients in lateral decubitus position, and basic body shape data of surgical patients; the nursing staff language data includes voice data and / or text data. The feature vector module is used to convert the nursing staff's language data into text vectors through projection, and to obtain visual vectors by detecting human key points based on the real-time image of the surgical patient in the lateral decubitus position and the basic body shape data of the surgical patient. The feature fusion module is used to perform multimodal fusion of the text vector and the visual vector through a projection matrix based on the LLAVA algorithm to obtain a unified feature vector; The large language model module is used to input the unified feature vector into a pre-trained multimodal large language model to obtain patient positioning assessment results and interaction results. The multimodal large language model is based on a pre-built nursing knowledge base to conduct compliance assessment of the lateral decubitus position of surgical patients and to conduct question-and-answer interaction on language data of nursing staff. The nursing knowledge base includes lateral decubitus position domain knowledge question-and-answer pairs based on Llam2-7B, lateral decubitus position standards, and surgical positioning rules.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images