A health risk assessment device based on multi-modal hand data

Abstract

The application discloses a health risk assessment device based on multi-modal hand data, comprising a data acquisition module, a prior construction module, a cross-modal interaction module, a prior guided fusion module and a multi-label prediction module. The device acquires double-hand RGB, near-infrared and 3D point cloud images, extracts the structured visual feature vector of the thenar eminence and hypothenar region, and constructs regional anatomy prior and age and gender global prior; through regional cross-modal interaction learning, the spectral complementary information is learned, and through the prior guided attention mechanism, the visual evidence is aggregated in the anatomically compatible region, and finally the double-task output node independently predicts the risk of coronary heart disease and hyperlipidemia. Through the collaborative modeling of multi-modal hand data and anatomy prior, the application realizes the precise, interpretable and non-invasive assessment of the risk of coronary heart disease and hyperlipidemia comorbidity, and provides an efficient, easy-to-promote intelligent solution for primary disease screening.

Description

Technical Field

[0001] This invention belongs to the field of major chronic disease risk prediction technology, specifically relating to a health risk assessment device based on multimodal hand data. Background Technology

[0002] Currently, top-tier hospitals primarily rely on sophisticated examinations such as coronary CTA or coronary angiography to diagnose coronary heart disease. While these methods are highly accurate, the equipment is expensive, the procedures are complex, and there are risks of radiation and invasiveness, making them difficult to popularize in primary healthcare institutions. Primary healthcare institutions, as the main battleground for the prevention and control of cardiovascular diseases and other chronic diseases, have long faced three core challenges in chronic disease management: First, "inaccurate diagnosis"—primary healthcare personnel are mostly general practitioners and lack systematic training in chronic disease specialties, making it difficult to accurately assess the combined risks of multiple risk factors; second, "incomplete management"—chronic disease patients often have multiple comorbidities, making it easy for primary healthcare management to overlook some aspects; and third, "inability to screen"—truly high-risk individuals are often hidden during routine examinations and are easily missed.

[0003] Currently, there is no specific comorbidity risk prediction model for coronary heart disease (CHD) combined with hyperlipidemia in clinical practice. In practice, comprehensive risk assessment systems for atherosclerotic cardiovascular disease (ASCVD) are mainly relied upon. While ASCVD risk assessment tools incorporate indicators such as blood lipids, they are not designed for the clinical scenario of managing both diseases simultaneously, and their application at the grassroots level generally faces the dilemma of data dependence and detachment from local realities. Some cutting-edge models, such as CT-FFR-based imaging assessments and SNP-based polygenic risk scoring, have high predictive accuracy, but they rely on large-scale equipment or special biomarkers, making them unavailable in primary healthcare institutions. Existing classic cardiovascular risk assessment models, while using conventional clinical indicators, require the inclusion of more than ten variables, including gender, age, current residence, region, waist circumference, total cholesterol, high-density lipoprotein cholesterol, current blood pressure level, whether taking antihypertensive medication, whether having diabetes, smoking status, and family history of cardiovascular disease. Doctors lack the time to meticulously measure physical indicators such as waist circumference and find it difficult to systematically collect complete risk factor information, making the models difficult to implement in practice. In addition, existing ASCVD risk scores underestimate or overestimate certain populations in practical applications, further limiting their clinical applicability.

[0004] In recent years, traditional Chinese medicine hand diagnosis, as a non-invasive and convenient auxiliary diagnostic method, has gradually gained attention in the identification of diseases such as coronary heart disease. However, current hand diagnosis research is still mainly based on traditional observation theory, with inconsistent hand diagnosis indicators, an incomplete indicator system, and a lack of objective and standardized quantitative assessment methods. Existing studies are mostly based on single images or single-modality classification methods, failing to fully utilize bilateral information from both hands, complementary perspectives of the palm and back of the hand, cross-modal information from multispectral images, and prior knowledge of hand anatomy, making it difficult to achieve stable and repeatable disease risk prediction.

[0005] In summary, the existing technologies have the following shortcomings: (1) There is a lack of non-invasive and lightweight screening tools specifically designed for the comorbid risk of coronary heart disease and hyperlipidemia; (2) Existing cardiovascular risk assessment models rely on a large number of clinical variables or large equipment, which are not very operable in grassroots settings; (3) Hand diagnosis research lacks multimodal, multi-perspective, and structured objective modeling methods, and has not yet formed an automated assessment scheme that can be promoted. Summary of the Invention

[0006] To address the above technical problems, this invention proposes a health risk assessment device based on multimodal hand data, comprising: The data acquisition and anatomical region localization module is used to acquire multimodal hand data of the patient under test. The multimodal hand data includes RGB images and near-infrared (NIR) images of the palms and backs of both hands, as well as 3D point cloud images of the palms of both hands. Based on the key anatomical points of the hand, the anatomical regions of the RGB and NIR images are localized, and the pixel regions of the thenar eminence, hypothenar eminence, and palm are extracted to generate structured image blocks for each region. The prior construction module is used to construct regional anatomical priors and global basic knowledge priors. The regional anatomical priors include thenar and hypothenar color priors extracted from RGB images, as well as thenar and hypothenar morphological priors, thenar 3D geometric quantization priors, and hypothenar 3D geometric quantization priors extracted from 3D point cloud images. The global basic knowledge priors include the age and gender coding features of the patients to be tested. The BPKER-Net network includes a cross-modal interaction module, a priori guided fusion module, and a multi-label prediction module. The cross-modal interaction module is used to input RGB and NIR image blocks of each anatomical region into the visual encoder to extract RGB and NIR visual feature vectors of each anatomical region. Then, it adaptively generates RGB and NIR modal weights for each anatomical region through a region-level gating network, and weighted fuses them to obtain the anatomical region-level cross-modal visual feature vectors of each anatomical region. Based on the anatomical region-level cross-modal visual feature vectors of the left and right hands from the same viewpoint and the same anatomical region, it constructs difference and related visual feature vectors to explicitly preserve the difference and common information of bilateral homologous anatomical regions. Finally, the anatomical region-level cross-modal visual feature vectors, difference and related visual feature vectors are input into the Transformer encoder to obtain the patient's global visual context information. The prior-guided fusion module encodes the anatomical priors of each region into query vectors and the global basic knowledge priors into basic knowledge embedding vectors. Using a prior-guided attention mechanism, it performs cross-modal attention calculation on a set of visual feature vectors compatible with the semantics of the regional anatomical priors, retrieving local visual evidence for the corresponding anatomical region to obtain regional prior retrieval evidence. The regional prior retrieval evidence, patient-level global visual context information, and patient-level global basic knowledge embedding vectors are then fused to obtain patient-level fusion features. The multi-label prediction module is used to input the patient-level fusion features into the independent coronary heart disease classification head and hyperlipidemia classification head, respectively, and after mapping by the Sigmoid activation function, independently output the risk probability of coronary heart disease and the risk probability of hyperlipidemia. The total training loss function of the BPKER-Net network is a weighted sum of the basic training objective loss and the bilateral consistency constraint loss. The basic training objective loss is the sum of the binary cross-entropy loss of the coronary heart disease label and the hyperlipidemia label. The bilateral consistency constraint loss is used to perform L2 distance constraints on the cross-modal visual feature vectors of the left and right homologous anatomical regions in the common phenotypic subspace. The BPKER-Net network employs a two-stage training strategy: the first stage is an RGB single-modal warm-up stage, in which the visual encoder is trained using only biphasic RGB images and a basic training target loss; the second stage is an end-to-end joint optimization stage, in which near-infrared (NIR) images, regional anatomy prior-guided attention mechanisms, global basic knowledge embedding vectors, and bilateral consistency constraint loss are introduced to jointly optimize the parameters of the cross-modal interaction module, prior-guided fusion module, and multi-label prediction module.

[0007] Furthermore, in the data acquisition and anatomical region localization module, the anatomical region localization based on the anatomical key points of the hand for the RGB and NIR images specifically involves inputting the RGB and NIR images into the MediaPipe Hands model, locating the hand bounding box and initial orientation in the entire image using the BlazePalm detector, cropping the entire hand, and then inputting it into the HandLandmark model to output 21 normalized 3D key points; constructing a bounding box based on the wrist key points, the base of the thumb key points, and the base of the index finger key points to extract the thenar eminence region; fitting a rotation rectangle with orientation adaptability based on the wrist key points, the base of the ring finger key points, and the base of the little finger key points to extract the hypothenar eminence region; and setting a fixed-scale window based on the geometric center of the wrist key points and the bases of the index, middle, ring, and little fingers to extract the palm region.

[0008] Furthermore, in the prior construction module, the thenar and hypothenar morphology priors extracted from 3D point cloud images specifically include preprocessing the 3D point cloud image, setting 15% of the height along the Z-axis as a threshold, and deleting background interference points below this height; calculating the bounding box of the point cloud on the XY plane, retaining the middle 80% of the central region based on the geometric center of the bounding box, and removing stray points around the central region; assuming the palm is facing upwards, fingers are pointing towards the positive Y-axis, and thumb is pointing towards the positive X-axis when acquiring the 3D point cloud image, with the positive X-axis side as the thenar side and the negative X-axis side as the hypothenar side, calculating the median of the point cloud along the X-axis to divide the palm region into left and right parts. The horizontal distance from each point to the centerline is calculated and normalized using the centerline as a reference. The thenar and hypothenar regions are then divided according to the normalization ratio. Within the divided thenar and hypothenar regions, a local palmar plane is fitted using principal component analysis. The centroids of the thenar and hypothenar regions are used as reference points. The vertical distance from each point in each region to the local palmar plane is calculated. The points are then sorted from largest to smallest according to their vertical distance. The points with the highest vertical distance values ​​are selected, and the number of these points accounts for 10% of the total number of points in the region. The average vertical distance of these selected points is used as a fullness evaluation index. Based on this index, the shape of the region is determined to be full, flat, or concave.

[0009] Furthermore, the prior-guided fusion module encodes the anatomical priors of each region into query vectors in the following way: For each regional anatomy prior, its prior value, prior category, corresponding palmar region, and corresponding anatomical attribute are independently embedded, and then the embedding vectors are added together to construct a query vector. The prior categories include morphological priors, color priors, and 3D geometric quantization priors; the palm region includes the thenar eminence and hypothenar eminence; and the anatomical attributes include morphological attributes, color attributes, and geometric quantization attributes. The dimensions of each embedding vector are consistent with the feature dimensions of the visual feature vector.

[0010] Furthermore, the specific method by which the region-level gating network in the cross-modal interaction module generates RGB modal weights and NIR modal weights is as follows: the region-level gating network takes the RGB visual feature vector and NIR visual feature vector of the same anatomical region as input, and after passing through two fully connected layers and the nonlinear activation function ReLU mapping, it outputs the unnormalized scores of the RGB modality and NIR modality. Then, the unnormalized scores are normalized to the range [0,1] by the Softmax function to obtain the RGB modal weights and NIR modal weights, and the sum of the RGB modal weights and NIR modal weights is 1.

[0011] Furthermore, the set of visual feature vectors compatible with the prior semantics of regional anatomy in the prior-guided fusion module is predefined by an anatomical compatibility rule, which is: each regional anatomy prior corresponds to a predefined set of visual feature vector indices, and the set of visual feature vector indices contains only visual feature vectors consistent with the prior semantics of regional anatomy. Among them, the compatible set of the thenar color prior and the thenar morphology prior only includes the visual feature vectors of the thenar regions on the left and right palms, the compatible set of the hypothenar color prior and the hypothenar morphology prior only includes the visual feature vectors of the hypothenar regions on the left and right palms and the back of the hand, and the compatible set of the 3D geometric quantization prior includes the visual feature vectors of the corresponding regions and their left and right homologous regions.

[0012] Furthermore, in the data acquisition and anatomical region localization module, patient-level multimodal datasets are aggregated by patient ID. Each patient's multimodal dataset includes palm and hand photos and back of hand photos. All viewpoint images, 3D point cloud features, and prior information of the same patient only appear in one of the training or validation sets.

[0013] Furthermore, the specific method for constructing differential visual feature vectors and related visual feature vectors in the cross-modal interaction module is as follows: calculate the left and right hand anatomical region-level cross-modal visual feature vectors for the same viewpoint and the same anatomical region. and The absolute value of the difference and the element-wise product are used to obtain the difference visual feature vectors: And obtain the relevant visual feature vectors: .

[0014] Furthermore, in the multi-label prediction module, the coronary heart disease classification head and the hyperlipidemia classification head are independent of each other. The risk probabilities of coronary heart disease and hyperlipidemia output by the BPKER-Net network constitute four patient status prediction results, including: positive for coronary heart disease only [1,0], positive for hyperlipidemia only [0,1], positive for coronary heart disease combined with hyperlipidemia [1,1], and negative for both [0,0]. The four statuses correspond to different risk stratifications in the co-occurrence screening scenario of coronary heart disease and hyperlipidemia.

[0015] Furthermore, the bilateral consistency constraint loss The calculation formula is:

[0016] In the formula, This represents a set of perspectives and anatomical regions that can establish a left-right homology relationship. This represents a single-layer linear projection function that maps the visual feature vectors of the left and right homologous anatomical regions to a common phenotypic subspace. Represents the square of the L2 norm; This represents the cross-modal visual feature vector at the anatomical region level of the left hand. The anatomical region-level cross-modal visual feature vector of the right hand; Based on perspective categories, including palm, back of hand, The anatomical area includes the entire hand, the thenar eminence, the hypothenar eminence, and the palm. The total training loss function is:

[0017] In the formula, The loss is the basic training objective. The weights represent the bilateral consistency constraint weights.

[0018] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention represents coronary heart disease and hyperlipidemia as two-dimensional independent label outputs, allowing the model to predict the risk of both related diseases simultaneously and independently, which is more in line with the real-world screening scenario of "co-existence of diseases". Simultaneously, this invention adopts a patient-level modeling paradigm, modeling the left and right hands, palms and backs of hands, multispectral images, and multiple types of prior information of the same patient as a whole, rather than splitting multiple images into independent samples. This avoids sample-level data leakage and improves the stability and clinical consistency of predictions.

[0019] 2. This invention, for the first time, encodes the shape, color, and 3D geometric description of the thenar eminence, hypothenar eminence, and palm into query vectors. Through a prior-guided cross-modal attention mechanism, the model can directionally aggregate visual evidence within anatomically relevant local regions such as the thenar and hypothenar eminences. Compared to traditional feature stitching methods, this mechanism effectively avoids the problem of low-dimensional priors being overwhelmed by high-dimensional visual features, enabling prior knowledge to actively participate in the modality fusion process.

[0020] 3. Existing methods for processing multispectral images mostly involve global feature stitching, which cannot distinguish the effectiveness differences of different modalities in different anatomical regions. This invention learns the complementary relationship between RGB and near-infrared images at the anatomical region level, estimates modal reliability weights for each anatomical region, and allows the model to adaptively adjust the contribution ratio of each modality based on the patient, region, and image quality.

[0021] 4. This invention explicitly models the consistency and asymmetry of left and right homologous regions within a patient-level framework. It preserves bilateral control information by constructing differential visual feature vectors and related visual feature vectors, and stabilizes the homologous region representation in the common phenotypic subspace through bilateral consistency constraints. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the BPKER-Net model structure of the present invention; Figure 2 This is a schematic diagram of the prior construction module of the present invention; Figure 3 This is a schematic diagram of the cross-modal interaction module of the present invention; Figure 4 This is a schematic diagram of the prior art fusion module of the present invention; Figure 5 This is a schematic diagram of the structure of the multi-label prediction module of the present invention; Figure 6 This is an example of visual analysis of comorbid patients in this invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Example This embodiment proposes a health risk assessment method based on multimodal hand data, including the following steps: S1. Data Acquisition and Anatomical Region Localization This invention uses the traditional Chinese medicine hand diagnosis instrument described in CN202511354234.0 or ZL202510578954.9 to collect RGB images of the palms and backs of the hands of the patient. and near-infrared (NIR) images , and 3D point cloud images of the palm sides of both hands.

[0025] To achieve precise localization of the anatomical regions of the palm, RGB images are used. and NIR images The BlazePalm detector from MediaPipeHands is used to locate the bounding box and initial orientation of the hand in the entire image; then the entire hand is cropped and fed into the Hand Landmark model, outputting 21 normalized 3D keypoints (x, y, z); the wrist keypoints ( Key points at the base of the thumb ( ), and key points at the base of the index finger ( Construct a bounding box based on the reference point (), and define this bounding box region as the thenar eminence region; use the wrist point ( Key points at the base of the ring finger ( ), and key points at the base of the little finger ( Using the wrist point as a reference, a rotational rectangle with orientation adaptability is fitted, and this region is defined as the hypothenar eminence region; ) and index finger ( ), middle finger ( ), ring finger ( ), little finger ( Construct a bounding box based on the root of the ), and define this bounding box region as the palm region.

[0026] The MediaPipe Hands consists of a BlazePalm detector and a Hand Landmark model. The BlazePalm detector is used to locate the palm region and output the palm bounding box with the hand's orientation. The HandLandmark model is the first keypoint regression model, which receives the output of the BlazePalm detector and outputs the coordinates of 21 keypoints of the hand.

[0027] S2, Extracting Regional Anatomical Priors This invention constructs two types of regional anatomy priors: color feature priors and 3D morphological quantification priors.

[0028] The thenar and hypothenar color priors are extracted from RGB images. Specifically, they are based on RGB images. The pixel color values ​​of the thenar and hypothenar eminences are extracted from the 'a' channel in the LAB color space. The color categories of the thenar and hypothenar eminences are labeled based on color category thresholds, including "rosy" and "non-rosy". A region is considered rosy if its average chromaticity 'a' ≥ 20; otherwise, it is considered non-rosy.

[0029] Priorities for extracting the thenar eminence shape from 3D point cloud images include: priors for the thenar eminence shape, priors for the thenar eminence 3D geometric quantization, and priors for the thenar eminence 3D geometric quantization. First, image preprocessing is performed. The total range of all points on the Z-axis is calculated. The height of the point cloud in the Z-axis direction is set as 15% as the threshold, and background interference points below this height are deleted. Then, a minimum rectangle that can enclose all points is calculated on the XY plane. Only the points in the middle region are kept, and the stray points around the middle region are removed. The middle region is a rectangle with the geometric center of the minimum rectangle as its geometric center. The length and width of the middle region are 80% of the length and width of the minimum rectangle.

[0030] Then, the thenar region and hypothenar region are segmented, and the specific process includes: ① calculating the median of the point cloud on the X-axis, and dividing the palm region into left and right sides with the median as the midline; ② based on the midline, calculating the horizontal distance from each point to the midline and normalizing the distance; ③ screening the hypothenar region and thenar region according to the normalization ratio, wherein it is assumed that when collecting 3D point cloud images, the palm faces upward, the fingers face the positive direction of the Y-axis, and the thumb faces the positive direction of the X-axis, the side in the positive direction of the X-axis corresponds to the thenar region, and the side in the negative direction of the X-axis corresponds to the hypothenar region, and the above screening results accurately calibrate the positions of the two regions on the premise of retaining the original spatial structure of the palm; ④ in the thenar region and the hypothenar region, the local palm planes of the thenar region and the hypothenar region are respectively fitted by principal component analysis (PCA), taking the centroids in the thenar region and the hypothenar region as reference points respectively, calculating the vertical distance from each point in each region to the local palm plane of the corresponding region, then sorting the distances from largest to smallest, selecting a part of points with top-ranked vertical distance values, the number of the part of points accounts for 10% of the total number of points in the region, and calculating the average value of the vertical distances of the part of points (this average value is recorded as Top10 in the following steps) as the plumpness evaluation index, that is, the quantified average thickness.

[0031] Mark the 3D geometric quantitative priors of the thenar region and hypothenar region based on the plumpness state determination criteria: in the thenar region, if Top10≥33mm, it is marked as plump; if 29mm<Top10<33mm, it is marked as flat; if Top10≤29mm, it is marked as sunken; in the hypothenar region, if Top10≥29mm, it is marked as plump; if 23mm<Top10<29mm, it is marked as flat; if Top10≤23mm, it is marked as sunken. The consistency between the shapes of thenar and hypothenar calculated based on this method and manual annotation exceeds 93%, indicating that the Top10 index can stably reflect the thenar eminence characteristics.

[0032] S3, Construction of patient-level multimodal dataset By patient number ( ) as the unit to collect the multimodal dataset of each patient, only retaining the multimodal datasets of patients who have both front and back double-view data, and a total of 755 patients meeting the pairing criteria are included. The multimodal dataset of the same patient includes RGB images and NIR images of palms and backs of both hands, point cloud images of both palms and prior variables. The health label of each patient adopts a dual-task multi-label form, including the coronary heart disease label ( ) and the hyperlipidemia label ( Each label is a binary variable (1 indicates disease, 0 indicates no disease). Based on the combination of the two disease labels, the dataset is divided into four patient states: no coronary heart disease and no hyperlipidemia [0,0] (226 cases), hyperlipidemia only [0,1] (258 cases), coronary heart disease only [1,0] (209 cases), and coronary heart disease combined with hyperlipidemia [1,1] (62 cases). Photographs of both palms or backs of both hands together ensure that both palms and backs of hands can be observed simultaneously during model training and testing, and the coexistence of both hands in the same image makes it possible to model bilateral relationships. RGB and NIR images provide cross-spectral complementary information.

[0033] Each patient's prior variables consist of two categories: the first category is anatomical priors of the hand regions, including the thenar eminence morphology, color, hypothenar eminence morphology, color, and the quantified mean thickness of the thenar and hypothenar eminences extracted based on 3D point clouds; the second category is global basic knowledge priors, including age and gender. Age is standardized to an age feature based on the age mean and standard deviation, and the subject's gender is mapped to two binary coded features: male and female. When the subject is male, the male feature is set to 1 and the female feature to 0; when the subject is female, the male feature is set to 0 and the female feature to 1. Ultimately, each patient's prior variables consist of four categorical anatomical priors of the hand regions and five continuous priors, where the continuous priors include the quantified mean thickness of the thenar and hypothenar regions, one age feature, and two gender coded features.

[0034] When partitioning the training and validation sets, a patient-level five-fold cross-validation strategy was used to partition the multimodal datasets of all patients. Specifically, the multimodal datasets corresponding to all patients were divided into five non-overlapping subsets, and one subset was selected sequentially as the validation set, while the other four subsets were used as the training set. This resulted in five independent model training and validation iterations. To avoid the same patient's multimodal data appearing in both the training and validation sets, the data partitioning was performed on a patient-by-patient basis, meaning that all multimodal data corresponding to the same patient was assigned to only one of the training or validation sets.

[0035] To maintain the consistency of the distribution of different disease combinations among patients in each compromise, before performing the five-fold split, the merged encoded value of the health label of each patient's multimodal dataset is calculated using the following formula. :

[0036] in, Indicates coronary heart disease label, This indicates a hypertension label. Both can be binary labels, with values ​​of 0 or 1.

[0037] Subsequently, the sample proportions of these four combined coding values ​​in the multimodal datasets of all patients were statistically analyzed, and stratified sampling was performed based on these proportions to ensure that the distribution of different disease combinations in the five subsets remained relatively consistent. Finally, the mean and standard deviation of the performance metrics obtained from five-fold cross-validation were used as the model performance evaluation results.

[0038] S4. Train the BPKER-Net health risk assessment model based on multimodal hand data. The BPKER-Net health risk assessment model based on multimodal hand data in this embodiment includes a cross-modal interaction module, a priori guided fusion module, and a multi-label prediction module.

[0039] In the cross-modal interaction module, firstly, the RGB and NIR image blocks of each anatomical region are input into the visual encoder to extract the RGB and NIR visual feature vectors of each anatomical region. For each patient, each anatomical region includes four regions (r): the whole hand, thenar eminence, hypothenar eminence, and palm. The visual feature vector of each anatomical region is represented as follows:

[0040] In the formula, m represents the RGB or NIR mode. This represents the visual encoder for the corresponding modality, responsible for converting the cropped image patch into a high-dimensional visual feature vector (Token), preserving the overall shape and color distribution of the cropped image patch. This indicates an image cropping operation, specifically based on the type of hand. Perspective Category Anatomical area From the input raw image The corresponding local image patch is cropped, with hand categories including left hand and right hand, and viewpoint categories including palm and back of hand. and These represent learnable vectors for hand type, viewpoint type, anatomical region, and modality, respectively. The dimension of the learnable vectors is consistent with the dimension of the visual feature vectors, representing the source information of the image. They are incorporated into the visual feature vectors (tokens) of each anatomical region through addition.

[0041] Preferably, the visual encoder Backbone networks such as ResNet, EfficientNet, or VisionTransformer can be used to extract visual feature vectors from RGB and NIR image blocks of each anatomical region.

[0042] The aforementioned visual feature vectorization method establishes a unified coordinate reference system for multimodal interaction: RGB and NIR features can be precisely aligned within the same palm anatomical region; homologous anatomical regions of the left and right hands can be directly compared; the palm and back of the hand can serve as complementary perspectives to collaboratively participate in modeling; and hand anatomical priors can also be semantically located to the corresponding visual feature vectors based on the anatomical region. Without this structured step, subsequent prior-guided attention would struggle to determine which visual evidence each prior should retrieve.

[0043] Subsequently, the cross-modal interaction module estimates adaptive modal weights for RGB and NIR for each anatomical region, and generates anatomical region-level cross-modal visual feature vectors accordingly.

[0044]

[0045] In the formula, This represents the modal weights at the anatomical region level. These weights do not assume that any particular modality is always superior, but rather allow the BPKER-Net model to adaptively adjust the contributions of RGB and NIR based on patient, anatomical region, and image quality differences. When NIR images provide stable vascular texture information, the BPKER-Net model can utilize NIR to supplement evidence more; when NIR is affected by acquisition distance or ambient temperature, the BPKER-Net model can rely more on RGB color texture information.

[0046] Among them, the anatomical region-level modal weights and Generated using a lightweight gating network. This gating network takes the RGB and NIR visual feature vectors of the corresponding anatomical region as input, passes through two fully connected layers and a non-linear activation function ReLU, and outputs unnormalized scores for the two modalities. These scores are then normalized to the [0,1] range using a Softmax function, satisfying... . and These are the visual feature vectors and anatomical origin information carried by the RGB or NIR images, respectively.

[0047] After completing cross-modal interactions within anatomical regions, the cross-modal interaction module of the BPKER-Net model further models the relationships between anatomical regions. Homologous anatomical regions of the left and right hands can reflect bilateral consistency or asymmetry of body features; different perspectives of the palm and back of the hand can provide complementary information; and synergistic changes in adjacent anatomical regions within the palm may collectively reveal systemic phenotypic alterations. To explicitly preserve bilateral relationships, the cross-modal interaction module constructs left-right hand difference visual feature vectors for the same perspective and the same anatomical region. With related visual feature vectors :

[0048]

[0049] in, Visual feature vectors describing the differences in anatomical regions of the left and right hands. This describes the common visual feature vectors of the homologous anatomical regions of the left and right hands. The differential visual feature vectors prevent the BPKER-Net model from over-assuming complete bilateral consistency, while the related visual feature vectors preserve the common phenotypes of the homologous anatomical regions, enabling the left and right comparisons to be incorporated into subsequent fusion in a learnable manner.

[0050] The cross-modal interaction module ultimately combines all anatomical region-level cross-modal visual feature vectors ( ), visual feature vectors of differences in anatomical regions ( ) and related visual feature vectors ( Input together with the patient-level visual Transformer While preserving local visual feature vectors, bilateral and multi-view contexts are learned to obtain the global visual context information of the i-th patient. :

[0051] Patient-grade visual Transformer A multi-layer Transformer encoder structure is adopted, with each layer containing a multi-head self-attention mechanism and a feedforward network. All regional anatomical cross-modal visual feature vectors ( ), visual feature vectors of differences in anatomical regions ( ) and related visual feature vectors ( Before input, the visual feature vectors are linearly projected to a unified feature dimension. The Transformer performs global self-attention computation between visual feature vectors, allowing information to be fully propagated between the left and right hands, between the palm and back of the hand, and between different anatomical regions. Through this design, the BPKER-Net model not only learns the visual feature vectors of a single local anatomical region, but also captures the global correlation and bilateral symmetry of the patient-level palm structure, forming visual feature vectors with patient-level context awareness.

[0052] The prior-guided fusion module first encodes the anatomical priors of each region into query vectors. These regional anatomical priors are divided into two categories and processed separately: one category includes regional anatomical priors such as thenar eminence morphology, thenar eminence color, hypothenar eminence morphology, hypothenar eminence color, and related 3D geometric quantification priors; the other category includes global basic knowledge priors, such as age and gender. Each query vector is constructed as follows:

[0053] in, Index representing prior features. Indicates the prior value, Represents the prior category (shape / color / geometric quantification). This indicates the corresponding palm area (thenar eminence / hypothenar eminence). This represents morphology, color, or geometric quantization. The dimensions of each embedding vector are consistent with the dimensions of the visual feature vector to ensure feature alignment during subsequent cross-modal attention computation. Through this design, anatomical priors for each region are no longer passive inputs but actively participate in the multimodal fusion process, guiding the BPKER-Net model to search for visual evidence in relevant anatomical regions.

[0054] To avoid unconstrained matching of regional anatomical priors globally, the prior-guided fusion module utilizes a prior-guided attention mechanism to perform cross-modal attention calculation on a set of visual feature vectors that are semantically compatible with the regional anatomical prior for each query vector. This retrieves local visual evidence that matches the regional anatomical prior, thereby obtaining regional prior retrieval evidence.

[0055] For example, the compatible set of thenar color prior. The prior set only includes visual feature vectors of the thenar eminence region on the palms of both hands. The hypothenar eminence morphology prior only includes visual feature vectors of the hypothenar eminence region on the palms / backs of both hands. The 3D geometric quantization prior can simultaneously focus on the corresponding anatomical region and its left and right homologous anatomical regions. The prior-guided attention after anatomical region constraints is represented as follows:

[0056] in, This represents the set of visual feature vectors compatible with the k-th prior feature, used to limit the range of visual regions that the prior features can focus on during attention computation. This represents the cross-modal region representation in the set. Indicated by As a query vector, Query As key vectors (Key) and value vectors (Value), the information most relevant to the prior is extracted from compatible visual region features, i.e., the first... Region prior retrieval evidence is obtained by aggregating prior features from their compatible visual regions.

[0057] Furthermore, age and gender can also be used as prior information in model building, but they belong to global basic knowledge priors and do not correspond to specific anatomical regions of the palm. Therefore, this embodiment does not use the aforementioned anatomical compatibility rules for local visual retrieval of age and gender priors. Instead, it sets up an independent basic knowledge prior encoding branch to encode age and gender features, and then concatenates and fuses them with patient-level features to obtain a patient-level global basic knowledge embedding vector. In this way, the auxiliary role of global basic knowledge priors such as age and gender in disease discrimination or state assessment can be fully utilized, while avoiding interference from such global priors in the semantic correspondence between regional anatomical priors such as thenar and hypothenar eminences and local visual evidence, thereby improving the anatomical consistency of the prior fusion process and the stability of model discrimination.

[0058] Finally, the priori-guided fusion module will integrate the patient's... Global visual context information Regional prior retrieval evidence obtained based on regional anatomy prior retrieval, and patient... Global foundational knowledge is embedded into vector fusion to obtain patient-level fusion features. .

[0059] Subsequently, the multi-label prediction module accepts patient-level fusion features. Furthermore, separate disease classification heads are used to generate the risk probability of coronary heart disease or hyperlipidemia, respectively:

[0060] in, This represents the output function for the t-th disease task. Indicates coronary heart disease. This represents hyperlipidemia, specifically implemented as a three-layer MLP structure. The two classifiers are independent of each other; therefore, the BPKER-Net model outputs two independent risk probabilities, rather than a mutually exclusive class. Activation function... This is used to map the output to the risk probability of the corresponding disease.

[0061] During training, the BPKER-Net model calculates the loss separately for the output coronary heart disease label and hyperlipidemia label, and then optimizes the entire network together. The basic training objective uses multi-label binary cross-entropy.

[0062] in, and These represent the true labels for coronary heart disease and hyperlipidemia, respectively. and These represent the probabilities of coronary heart disease and hyperlipidemia output by the BPKER-Net model, respectively. This represents the binary cross-entropy loss function. Since both labels can be positive simultaneously, the BPKER-Net model does not treat coronary heart disease and hyperlipidemia as competing conditions, but rather learns their coexistence patterns in the same patient.

[0063] In addition to the task loss, the BPKER-Net model incorporates a bilateral consistency constraint in the common phenotypic subspace. This constraint only applies to the shared projective representations of homologous anatomical regions on the left and right sides, without erasing the asymmetric information retained in the differing visual feature vectors, resulting in the bilateral consistency loss. Total loss :

[0064]

[0065] in, Let represent the set of viewpoints and anatomical regions from which left-right homology can be established, and let represent the number of elements in the set. The weights for the bilateral consistency constraint (ranging from 0.01 to 0.1) Denotes the square of the L2 norm. This represents a single-layer linear projection function that maps the visual feature vectors of the left and right homologous anatomical regions to a common phenotypic subspace. This represents the weight of the bilateral consistency constraint, and its value is preferably in the range of 0.01 to 0.1.

[0066] The BPKER-Net model employs a two-stage training strategy: the first stage is an RGB-only warm-up training stage, which uses only RGB images and the basic training objective loss. The first stage trains the visual encoder, enabling the BPKER-Net model to acquire initial discriminative ability on a relatively stable RGB phenotype. The second stage is the joint training of the complete model, which introduces NIR images, a region anatomy prior-guided attention mechanism, global basic knowledge embedding vectors, and bilateral consistency constraints, and is based on the total loss function. The model is jointly optimized end-to-end. This two-stage training strategy reduces the instability in the early stages of multimodal joint training and improves the model's convergence and generalization capabilities.

[0067] To evaluate the model, this embodiment assesses the performance of the BPKER-Net model using a patient-level five-fold cross-validation protocol. Evaluation metrics include Accuracy (characterizing overall prediction accuracy), Macro-F1 (characterizing average classification ability on the two disease labels of coronary heart disease and hyperlipidemia), Macro-AUROC (characterizing the model's ability to rank and discriminate samples at different risk levels), and Macro-AUPRC (characterizing the model's quality of identifying positive samples under class imbalance).

[0068] S5. The patient's multimodal dataset will be obtained using steps S1-S2. This dataset will be input into the trained health risk assessment model BPKER-Net based on multimodal hand data to obtain the patient's health risk assessment results. The multimodal dataset for the same patient includes RGB images and NIR images of both palms and backs of hands, point cloud images of both palms, and prior variables.

[0069] The comparison methods cover three types of models: the first type is prior-only models that use only prior information, such as XGBoost and TabNet, used to evaluate the performance of prior information alone for risk prediction; the second type is RGB-only visual models, such as ResNet-50, EfficientNet-B0, Swin-T, and ConvNeXt-Tiny, used to evaluate the predictive ability of single-modal visual representations in this task; the third type is multimodal fusion and image-table joint learning methods, such as MCB, TFN, MFB, LMF, MMTM, DAFT, MulT, MMCL, TIP, and CGMCL, used to evaluate the applicability and performance of general multimodal fusion methods in the multi-label comorbidity risk prediction task described in this invention. The performance comparison of different methods on the patient-level multi-label screening task is shown in Table 1. Table 1

[0070] As shown in Table 1, BPKER-Net achieved the best performance across all four core evaluation metrics, with Accuracy, Macro-F1, Macro-AUROC, and Macro-AUPRC scores of 0.8893±0.0031, 0.8611±0.0085, 0.9210±0.0097, and 0.8904±0.0056, respectively. Compared to the other top-performing models in Accuracy, Macro-F1, Macro-AUROC, and Macro-AUPRC, the BPKER-Net model improved performance by 0.0513, 0.0641, 0.0350, and 0.0219, respectively. The prior-only model, which only uses prior information, performed significantly worse than all visual models, indicating that while anatomical priors and basic knowledge priors for the hand region carry some discriminative information, they are insufficient to independently complete stable patient-level screening. The RGB-only visual model outperforms the prior-only model across the board, demonstrating that hand images contain disease-related visual phenotypic information that can be effectively extracted through deep learning. The general multimodal approach shows improvement over the unimodal model, but still lags behind BPKER-Net, indicating that simply providing multimodal input does not automatically translate into a performance advantage; the key to performance improvement lies in whether the model can model the inherent, structured correspondences between different information sources.

[0071] To further analyze the specific contributions of each module in BPKER-Net, a systematic ablation experiment was conducted in this embodiment. The experimental results are shown in Table 2: Table 2 Ablation experiments of key components of BPKER-Net

[0072] As shown in Table 2, the BPKER-Net model experienced the most significant overall performance decline after removing prior knowledge of hand region anatomy and basic information. The Macro-F1 score dropped from 0.8611 to 0.7890, and the Macro-AUROC score decreased from 0.9210 to 0.8650. This indicates that prior information is not simply an additional input feature, but plays a crucial role in the aggregation of patient-level evidence, effectively guiding the model to retrieve structured evidence related to disease risk from multimodal visual features.

[0073] Further comparison of the effects of different priors: After removing the hand prior, Macro-F1 decreased to 0.7890; after removing the 3D geometric quantization prior, Macro-F1 decreased to 0.8220; after removing the age and gender priors, Accuracy, Macro-F1, Macro-AUROC, and Macro-AUPRC decreased to 0.8715, 0.8425, 0.9007, and 0.8872, respectively. This indicates that color, shape, and 3D geometric quantization features can provide complementary disease-related discriminative information from different perspectives, and that the various prior information types are not simply substitutable. Age and gender, as global basic knowledge priors, mainly play a role in calibrating the baseline risk levels of different populations, but the main discriminative ability of the BPKER-Net model still comes from patient-level fusion features.

[0074] Regarding the prior fusion method, when replacing the region anatomy prior and basic knowledge prior with ordinary vector concatenation, the Macro-F1 score decreased by 0.0591; when retaining the region anatomy prior and basic knowledge prior but canceling cross-modal attention calculation, the Macro-F1 score decreased by 0.0466. This indicates that the performance improvement does not simply come from increasing the input dimension, but from the region anatomy prior and basic knowledge prior participating in cross-modal attention calculation as query vectors, thereby enabling the proactive retrieval of corresponding local visual evidence under anatomical semantic constraints.

[0075] Regarding multispectral interaction, when replacing the anatomical region-level cross-modal visual feature vectors obtained from RGB-NIR interaction with feature concatenation, the Macro-F1 score decreased to 0.7800, and the Macro-AUROC score decreased to 0.8580. Removing the acquisition of left-hand difference visual feature vectors and left-hand related visual feature vectors, or removing the acquisition of global visual context information, resulted in a decrease in Macro-F1 of 0.0496 and 0.0621, respectively.

[0076] In terms of bilateral modeling, removing the bilateral consistency loss function or the RGB-only warm-up training phase also leads to performance degradation. Applying consistency constraints to homologous anatomical regions in the common phenotypic subspace helps stabilize the shared phenotypic representation of homologous regions; the latter shows that establishing a relatively stable initial visual phenotypic representation using RGB images before full multimodal joint training can improve the stability and convergence of subsequent multimodal fusion training.

[0077] To verify whether BPKER-Net can form a discriminative path with clinical semantic consistency, this invention conducts case-level interpretability experiments. Unlike traditional heatmaps that only show which pixel regions in the image the model focuses on, this invention focuses on analyzing three types of structured evidence within BPKER-Net: first, the retrieval attention from regional anatomical priors to visual feature vectors of anatomical regions; second, the task-specific attention of query vectors for coronary heart disease and hyperlipidemia to different local visual evidence; and third, the reliability weights of RGB and NIR spectral modalities in different anatomical regions.

[0078] This invention selects two representative cases for analysis based on the extra-fold validation prediction results. The first case is a comorbid case with a true label of [1,1], and the BPKER-Net model output probabilities are CHD=0.9999 and Hyper=0.4563. The retrieval attention from regional anatomical priors to visual feature vectors of anatomical regions shows that the BPKER-Net model can map visual feature vectors of different anatomical regions to semantically consistent palm regions. The thenar eminence morphology prior primarily retrieves [left palm_thenar eminence], the thenar eminence color prior primarily retrieves [right hand back_thenar eminence], and the thenar eminence 3D geometric quantization prior primarily retrieves [right hand back_thenar eminence]. The hypothenar eminence morphology, hypothenar eminence color, and hypothenar eminence 3D geometric quantization priors primarily retrieve regions such as [left hand back_hypothenar eminence] or [right palm_hypothenar eminence]. The age and gender priors primarily retrieve visual feature vectors of the entire hand region. The coronary artery disease branch primarily focuses on the thenar eminence color visual feature vector (attention weight 0.802), followed by the thenar eminence morphological visual feature vector and gender encoding. The hyperlipidemia branch exhibits a more dispersed attention distribution, primarily focusing on thenar eminence color, thenar eminence 3D geometric quantization, thenar eminence morphology, and local whole-hand visual feature vectors. RGB / NIR reliability analysis indicates that the BPKER-Net model significantly activates the NIR modality in this comorbid case. Regions with high NIR weights include [right palmar surface_thenar eminence], [right palmar surface_palm center], [left palmar surface_thenar eminence], [right hand dorsum_thenar eminence], and [right hand dorsum_hypothenar eminence], corresponding to NIR weights of approximately 0.987.

[0079] The second case was a single-disease case, with both the true and predicted labels being [0,1]. The BPKER-Net model output probabilities were CHD=0.00005 and Hyper=0.8084. Prior retrieval results showed that the thenar eminence color and morphology priors primarily retrieved [left palm_thenar eminence], the hypothenar eminence morphology and 3D geometric quantization priors primarily retrieved [left back of hand_hypothenar eminence], and the hypothenar eminence color priors primarily retrieved [right palm_hypothenar eminence]. Attention results for the disease query vectors showed that the hyperlipidemia branch primarily focused on local visual evidence such as hypothenar eminence color, thenar eminence 3D geometric quantization, and [left palm_whole hand], while the coronary heart disease branch, although also focusing on some thenar eminence color visual feature vectors and gender encoding, ultimately output a very low probability. This indicates that the BPKER-Net model does not simply map all abnormal palmar phenotypes to two disease positivity, but rather selects different task-related evidence from a shared evidence base through disease-specific queries.

[0080] The above interpretability analysis shows that BPKER-Net has three interpretability advantages: First, rigorous prior knowledge enables the retrieval of corresponding visual regions according to anatomical semantics, with the thenar, hypothenar, and whole-hand priors forming stable associations with their corresponding anatomical regions; second, the coronary heart disease and hyperlipidemia branches can select different task-related information from the same set of visual region features, demonstrating disease-specific discrimination ability; third, the RGB / NIR reliability weights reveal how the BPKER-Net model selects cross-spectral information in different regions, enabling the BPKER-Net model to form a structured evidence chain of "prior guidance, region retrieval, modality selection, and disease decoding." This experiment does not prove a causal relationship between the regions of interest to the BPKER-Net model and the disease, but rather shows that the internal discrimination path of the BPKER-Net model has good consistency with the pre-set palm anatomical priors and multimodal design logic, thereby enhancing the examineability and interpretability of the BPKER-Net model in clinical screening scenarios.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A health risk assessment device based on multimodal hand data, characterized in that, include: The data acquisition and anatomical region localization module is used to acquire multimodal hand data of the patient under test. The multimodal hand data includes RGB images and near-infrared (NIR) images of the palms and backs of both hands, as well as 3D point cloud images of the palms of both hands. Based on the key anatomical points of the hand, the anatomical regions of the RGB and NIR images are localized, and the pixel regions of the thenar eminence, hypothenar eminence, and palm are extracted to generate structured image blocks for each region. The prior construction module is used to construct regional anatomical priors and global basic knowledge priors. The regional anatomical priors include thenar and hypothenar color priors extracted from RGB images, as well as thenar and hypothenar morphological priors, thenar 3D geometric quantization priors, and hypothenar 3D geometric quantization priors extracted from 3D point cloud images. The global basic knowledge priors include the age and gender coding features of the patients to be tested. The BPKER-Net network includes a cross-modal interaction module, a priori guided fusion module, and a multi-label prediction module. The cross-modal interaction module is used to acquire anatomical region-level cross-modal visual feature vectors of RGB and NIR image blocks of each anatomical region; and construct difference visual feature vectors and related visual feature vectors based on the anatomical region-level cross-modal visual feature vectors of the left and right hands from the same viewpoint and the same anatomical region, so as to explicitly preserve the difference and common information of bilateral homologous anatomical regions; then input the anatomical region-level cross-modal visual feature vectors, difference visual feature vectors and related visual feature vectors into the Transformer encoder to obtain the patient's global visual context information; The prior-guided fusion module is used to encode the anatomical priors of each region into query vectors, and to encode the global basic knowledge priors into basic knowledge embedding vectors; it retrieves local visual evidence of the corresponding anatomical region for each region's anatomical prior query to obtain the region's prior retrieval evidence. The patient-level fusion feature is obtained by fusing the region prior retrieval evidence, patient-level global visual context information, and patient-level global basic knowledge embedding vector. The multi-label prediction module is used to output the probability of coronary heart disease risk and the probability of hyperlipidemia risk.

2. The health risk assessment device based on multimodal hand data according to claim 1, characterized in that, In the data acquisition and anatomical region localization module, the anatomical region localization of the RGB image and NIR image based on the anatomical key points of the hand is specifically performed by inputting the RGB image and NIR image into the MediaPipe Hands model, locating the hand bounding box and initial orientation in the whole image using the BlazePalm detector, cropping the entire hand and inputting it into the Hand Landmark model, and outputting 21 normalized 3D key points. Construct a bounding box based on the key points of the wrist, the base of the thumb, and the base of the index finger, and extract the thenar eminence region. Using the wrist key points, the key points at the base of the ring finger and the key points at the base of the little finger as references, a rotation rectangle with orientation adaptability is fitted to extract the hypothenar eminence region. Using the geometric center of the wrist key points and the base of the index, middle, ring, and little fingers as a reference, a window of fixed size is set to extract the palm area.

3. The health risk assessment device based on multimodal hand data according to claim 1, characterized in that, In the prior construction module, the thenar and hypothenar eminence morphology priors extracted from 3D point cloud images specifically include: preprocessing the 3D point cloud image, setting 15% of the height along the Z-axis as a threshold, and deleting background interference points below this height; calculating the bounding box of the point cloud on the XY plane, retaining the middle 80% of the central region based on the geometric center of the bounding box, and removing stray points around the central region; assuming the palm is facing upwards, fingers are pointing towards the positive Y-axis, and thumb is pointing towards the positive X-axis when acquiring the 3D point cloud image, with the positive X-axis side as the thenar eminence side and the negative X-axis side as the hypothenar eminence side, calculating the median of the point cloud along the X-axis to divide the palm region into left and right parts, and using... The horizontal distance from each point to the centerline is calculated and normalized using the centerline as a reference. The thenar and hypothenar regions are then divided according to the normalization ratio. Within the divided thenar and hypothenar regions, a local palmar plane is fitted using principal component analysis. The centroids of the thenar and hypothenar regions are used as reference points. The vertical distance from each point in each region to the local palmar plane is calculated. The points are then sorted from largest to smallest according to their distance. A subset of points with the highest vertical distance values ​​is selected, and these subsets, comprising 10% of the total number of points in the region, are used as the average vertical distance of the subsets. This average vertical distance is then used as a fullness assessment index. Based on this index, the shape of the region is determined to be full, flat, or concave.

4. The health risk assessment device based on multimodal hand data according to claim 1, characterized in that, Retrieving local visual evidence for each anatomical prior query in a region includes: using a prior-guided attention mechanism, performing cross-modal attention calculation on a set of visual feature vectors compatible with the semantics of the anatomical prior for each region, retrieving local visual evidence for the corresponding anatomical region, and obtaining the region prior retrieval evidence; A set of visual feature vectors compatible with the prior semantics of regional anatomy is predefined by an anatomical compatibility rule, which is that each regional anatomy prior corresponds to a predefined set of visual feature vector indices, and the set of visual feature vector indices contains only visual feature vectors that are consistent with the prior semantics of regional anatomy. Among them, the compatible set of the thenar color prior and the thenar morphology prior only includes the visual feature vectors of the thenar regions on the left and right palms, the compatible set of the hypothenar color prior and the hypothenar morphology prior only includes the visual feature vectors of the hypothenar regions on the left and right palms and the back of the hand, and the compatible set of the 3D geometric quantization prior includes the visual feature vectors of the corresponding regions and their left and right homologous regions.

5. The health risk assessment device based on multimodal hand data according to claim 1, characterized in that, The cross-modal interaction module inputs RGB and NIR image blocks of each anatomical region into the visual encoder to extract RGB and NIR visual feature vectors of each anatomical region. Then, through a region-level gating network, the RGB and NIR visual feature vectors of the same anatomical region are used as input. After passing through two fully connected layers and the nonlinear activation function ReLU, the unnormalized scores of the RGB and NIR modes are output. The unnormalized scores are then normalized to the range [0,1] by the Softmax function to obtain the RGB and NIR mode weights, and the sum of the RGB and NIR mode weights is 1. Finally, the weighted fusion is used to obtain the anatomical region-level cross-modal visual feature vectors of each anatomical region.

6. The health risk assessment device based on multimodal hand data according to claim 4, characterized in that, In the prior-guided fusion module, the anatomical priors of each region are encoded into query vectors in the following way: For each regional anatomy prior, its prior value, prior category, corresponding palmar region, and corresponding anatomical attribute are independently embedded, and then the embedding vectors are added together to construct a query vector. The prior categories include morphological priors, color priors, and 3D geometric quantization priors; the palm region includes the thenar eminence and hypothenar eminence; and the anatomical attributes include morphological attributes, color attributes, and geometric quantization attributes. The dimensions of each embedding vector are consistent with the feature dimensions of the visual feature vector.

7. The health risk assessment device based on multimodal hand data according to claim 1, characterized in that, In the data acquisition and anatomical region localization module, patient-level multimodal datasets are aggregated by patient ID. Each patient's multimodal dataset includes palm and hand photos and back and hand photos. All viewpoint images, 3D point cloud features and prior information of the same patient only appear in one of the training set or validation set.

8. The health risk assessment device based on multimodal hand data according to claim 1, characterized in that, The specific method for constructing differential and related visual feature vectors in the cross-modal interaction module is as follows: calculate the left and right hand anatomical region-level cross-modal visual feature vectors for the same viewpoint and the same anatomical region. and The absolute value of the difference and the element-wise product are used to obtain the difference visual feature vectors: And obtain the relevant visual feature vectors: .

9. The health risk assessment device based on multimodal hand data according to claim 1, characterized in that, In the multi-label prediction module, the patient-level fusion features are input into the independent coronary heart disease (CHD) classification head and hyperlipidemia classification head, respectively. After mapping by the Sigmoid activation function, the CHD risk probability and hyperlipidemia risk probability are output independently. The CHD risk probability and hyperlipidemia risk probability output by the BPKER-Net network constitute four patient status prediction results, including: CHD only positive [1,0], hyperlipidemia only positive [0,1], CHD combined with hyperlipidemia positive [1,1], and both negative [0,0]. The four statuses correspond to different risk stratifications in the CHD and hyperlipidemia comorbidity screening scenario.

10. The health risk assessment device based on multimodal hand data according to claim 1, characterized in that, The total training loss function of the BPKER-Net network is a weighted sum of the basic training objective loss and the bilateral consistency constraint loss. The basic training objective loss is the sum of the binary cross-entropy loss of the coronary heart disease label and the hyperlipidemia label. The bilateral consistency constraint loss is used to perform L2 distance constraints on the cross-modal visual feature vectors of the left and right homologous anatomical regions in the common phenotypic subspace. The BPKER-Net network adopts a two-stage training strategy: the first stage is the RGB single-modal warm-up stage, which uses only two-handed RGB images and the basic training target loss to train the visual encoder. The second stage is the end-to-end joint optimization stage, which introduces near-infrared (NIR) images, regional anatomy prior-guided attention mechanism, global basic knowledge embedding vector, and bilateral consistency constraint loss to jointly optimize the parameters of the cross-modal interaction module, prior-guided fusion module, and multi-label prediction module. The bilateral consistency constraint loss The calculation formula is: In the formula, This represents a set of perspectives and anatomical regions that can establish a left-right homology relationship. This represents a single-layer linear projection function that maps the visual feature vectors of the left and right homologous anatomical regions to a common phenotypic subspace. Represents the square of the L2 norm; This represents the cross-modal visual feature vector at the anatomical region level of the left hand. The anatomical region-level cross-modal visual feature vector of the right hand; Based on perspective categories, including palm, back of hand, The anatomical area includes the entire hand, the thenar eminence, the hypothenar eminence, and the palm. The total training loss function is: In the formula, The loss is the basic training objective. The weights represent the bilateral consistency constraint weights.

Images