A method and system for reconstruction of a missing hand and assessment of disability level

Abstract

This invention discloses a method and system for reconstructing a missing hand and assessing disability levels. It improves the Pix2Pix generator by embedding a CBAM (Concentrated Hand Amplifier) ​​that focuses on the details of the truncated edges of the hand, adding a discriminator with spectral normalization to enhance training stability, and incorporating a loss function that integrates multiple loss terms to balance reconstruction fidelity and preservation of healthy regions. After preprocessing to obtain a standardized missing hand image, the improved Pix2Pix is ​​trained until the loss is minimized to obtain the optimal reconstruction model. In application, the model generates a predicted image of the complete hand, extracts the hand mask, and calculates the disability index. This index is then mapped to the disability level according to the "Disability Level Assessment of Employees Due to Work-Related Injuries and Occupational Diseases" (GB / T 16180). This solves the problems of subjectivity and texture distortion in traditional assessments, achieving high-fidelity reconstruction of the missing hand and quantitative assessment of disability levels, applicable to scenarios such as work-related injury assessment.

Description

Technical Field

[0001] This invention relates to the technical field of deep learning and medical image analysis, and in particular to a method and system for reconstructing a missing hand and assessing the degree of disability. Background Technology

[0002] Hand disabilities are a common type of disability in work-related injuries and accidents. The degree of finger loss is an important basis for assessing work capacity, workers' compensation, and developing rehabilitation plans. Currently, the assessment of hand disability levels mainly relies on the subjective evaluation of medical experts. Traditional assessment methods require experts to combine imaging data and clinical examinations, comparing each item against disability standards to estimate the degree of functional loss, which has many drawbacks: First, it is highly subjective; different experts may have different assessment results for the same case, and the same expert may have inconsistent assessments at different times. Second, it is inefficient; each assessment requires a long time, making it difficult to meet the needs of large-scale disability assessments. Third, standardization is difficult; the forms of hand disabilities are diverse, and experts cannot accurately quantify the proportion of loss, resulting in poor repeatability of assessment results.

[0003] In recent years, deep learning technology has made groundbreaking progress in the field of medical image analysis. Generative Adversarial Networks (GANs) have demonstrated powerful capabilities in image generation, image completion, and image transformation, providing a new technical path for the automated quantitative assessment of hand defects. Among them, Pix2Pix, as a classic framework of conditional GANs, uses a U-Net generator and a PatchGAN discriminator to achieve the transformation of paired images. However, the original Pix2Pix has significant shortcomings: the U-Net generator treats all spatial locations and channels equally, lacking the ability to selectively focus on key regions and making it difficult to focus on the texture details of the truncated edges of the hand; the loss function only includes adversarial loss and L1 reconstruction loss, which can easily tamper with the texture of the original healthy finger when generating the missing part; at the same time, the training data is mostly standardized images with a pure black background, which has a domain difference from real clinical photographs, and direct application will lead to a significant decrease in reconstruction results. In addition, existing GAN-based image completion technologies only stay at the level of visual restoration and do not combine the generated results with the quantitative assessment of disability levels, so they cannot provide quantifiable reference for clinical identification.

[0004] In summary, designing a scheme that can accurately reconstruct images of incomplete hands, effectively preserve the texture of healthy areas, and achieve disability level assessment based on the reconstruction results is a key research focus in the field of hand disability assessment, and is of great significance for improving the objectivity, standardization, and efficiency of disability assessment. Summary of the Invention

[0005] The primary objective of this invention is to address the problems of strong subjectivity and difficulty in standardization in existing hand disability assessment methods, as well as the shortcomings of the original Pix2Pix method in hand reconstruction, such as lack of attention, easy text tampering, and domain discrepancies. This invention provides a method for hand reconstruction and disability assessment that achieves accurate reconstruction of hand images. Using the reconstructed complete hand image as a reference standard, a disability index is calculated through pixel-level area comparison, which is then mapped to a standardized disability level. This effectively improves the objectivity, accuracy, and automation of hand disability assessment.

[0006] The second objective of this invention is to provide a system for the reconstruction of a missing hand and the assessment of disability levels.

[0007] The first objective of this invention is achieved through the following technical solution: a method for reconstructing a missing hand and assessing the level of disability. This method uses an improved conditional generative adversarial network (Pix2Pix) as its core to achieve complete reconstruction of a missing hand image and quantitative assessment of the level of disability. The Pix2Pix method improves the generator, discriminator, and loss function of the original Pix2Pix. The improvement to the generator is achieved by embedding a convolutional block attention module (CBAM) in the U-Net architecture, enabling the network to focus on the texture details of the truncated edges of the hand. The improvement to the discriminator is achieved by adding spectral normalization to the convolutional layers of the PatchGAN architecture to improve training stability. The improvement to the loss function is achieved by adding VGG perceptual loss and content preservation loss to balance the realism of the reconstructed texture with the preservation of pixels in healthy regions. The content preservation loss is achieved by automatically extracting a mask of the existing region from the missing hand image, performing a dilation operation on the mask to smooth the edges, and then calculating the L1 distance between the reconstructed image and the original missing hand image within the mask region. This constrains the generator to generate only in the missing region, ensuring that the healthy region of the missing hand image and the reconstructed image are completely consistent.

[0008] The specific implementation of this method includes the following steps:

[0009] 1) Preprocessing stage: The Otsu thresholding method and GrabCut algorithm are used to extract the hand region and remove the background from the incomplete hand image. Both the Otsu thresholding method and GrabCut algorithm include maximum connected component extraction and morphological post-processing steps, and finally obtain a standardized incomplete hand image with a pure black background and uniform resolution, thus eliminating the domain difference between the training data and the real data.

[0010] 2) The preprocessed incomplete hand image is used as a feature map and fed into the improved Pix2Pix for training. The improved generator reconstructs the complete hand image from the incomplete hand image. The improved discriminator determines whether the reconstructed image is real or fake compared with the real complete hand image. In backpropagation, the loss value between the reconstruction result and the real label is calculated through the improved loss function. After multiple iterations until the loss value is minimized, the trained Pix2Pix is ​​obtained as the optimal reconstruction model.

[0011] 3) After preprocessing the image of the missing hand to be evaluated in step 1), input it into the optimal reconstruction model to generate the corresponding predicted image of the complete hand. Extract the binary mask of the hand from the preprocessed image of the missing hand and the predicted image of the complete hand respectively. During extraction, perform morphological closing and opening operations on the binary mask of the hand to eliminate noise. Extract and count the pixels of the hand according to the rule that the gray level threshold is greater than the preset value. Then calculate the disability index by pixel-level area difference. At the same time, re-use the binary mask of the hand to identify the fingers of the complete missing hand and determine the missing fingers. Finally, map the disability index to the standardized disability level with reference to the "Disability Grading of Employees with Work-Related Injuries and Occupational Diseases" (GB / T 16180) and output the disability index percentage and disability level determination result.

[0012] Furthermore, the Convolutional Block Attention (CBAM) module sequentially performs channel attention processing and spatial attention processing on the input feature map, and adds the processed feature map to the original input feature map through residual connections. The channel attention processing involves performing global average pooling and global max pooling on the feature map to obtain two channel-dimensional feature vectors, which are then added together through two shared multilayer perceptron layers, and then the channel weights are obtained through the Sigmoid activation function. The input feature map is multiplied by the channel weights to complete the channel dimension recalibration. The spatial attention processing involves taking the average and maximum values ​​of the channel-recalibrated feature map along the channel dimension, concatenating them, and then performing a 7×7 convolution and a Sigmoid activation function to obtain spatial weights. The channel-recalibrated feature map is multiplied by the spatial weights to complete the spatial dimension recalibration. Through dual channel and spatial attention weighting, the generator can accurately focus on the edges and texture details of the hand incompleteness, suppress background noise interference, and significantly improve the texture clarity and edge connection naturalness of the reconstructed incomplete hand.

[0013] Furthermore, the spectral normalization improves all convolutional layers of the discriminator in the PatchGAN architecture by sequentially performing spectral norm calculation and weight normalization on the weight matrix of the convolutional layers, and then applying the normalized weight matrix to the feature mapping of the convolutional layers. The spectral norm calculation involves solving for the maximum singular value of the weight matrix of the convolutional layers to obtain the spectral norm that represents the numerical range of the weight matrix. The weight normalization process involves dividing the original weight matrix of the convolutional layers by its corresponding spectral norm, constraining the spectral norm of the normalized weight matrix, limiting the Lipschitz norm of the discriminator, and using the normalized weight matrix for feature extraction and mapping of the convolutional layers. This completes the spectral normalization improvement of the discriminator, which can effectively avoid the overconfidence problem caused by overly strong discriminator training, improve the overall training stability of the generative adversarial network, allow the generator to learn better image structural features, and improve the generation effect of reconstructed images of missing hands.

[0014] Furthermore, the total loss function of the improved generator is:

[0015] ;

[0016] In the formula, Represents a generator. This represents the total loss of the generator. The adversarial loss is represented and calculated using the standard binary cross-entropy loss. Represents pixel level Reconstruction losses, Represents pixel level Weighting coefficients for reconstruction loss; The VGG perceptual loss is represented by the extraction of two high-level semantic features (ReLU3_4 and ReLU5_4) from a pre-trained VGG-19 network. The L1 distance between the reconstructed image and the real image is calculated in the feature space. The weighting coefficients represent the VGG perceptual loss. This indicates that the content has been preserved. The weighting coefficient represents the loss in content retention.

[0017] Furthermore, the calculation process for the content retention loss is as follows:

[0018] a) Automatically extract the mask M of the existing region from the input image of the missing hand. The specific extraction process is as follows: First, convert the input image of the missing hand into a grayscale image and determine the pixel threshold as -0.9. This threshold is obtained by actual measurement. The actual measurement shows that the grayscale value 12 / 255 is a clear finger region. After normalizing the grayscale value to the range of [-1,1], the corresponding pixel threshold -0.9 is obtained. The region of pixel value greater than -0.9 in the grayscale image is marked as 1, and this region is the region where a healthy finger exists; the other regions with pixel values ​​less than or equal to -0.9 are marked as 0, and this region is the region where the finger is missing or the background region.

[0019] b. Perform a 3×3 max pooling dilation operation on the mask M to expand the boundary pixels and achieve a smooth transition at the image edges;

[0020] c. Calculate the L1 distance between the reconstructed image and the input image within the mask region to obtain the content preservation loss, so that the generator can freely generate the missing parts only in the mask 0 region, and output pixels that are completely consistent with the input in the mask 1 region.

[0021] Furthermore, the formula for calculating the incompleteness index is as follows:

[0022] ;

[0023] In the formula, The defect index, Predict the number of hand pixels in a complete hand image; This represents the number of pixels in the hand portion of the preprocessed image of the incomplete hand.

[0024] Furthermore, for static hand injury or missing finger recognition, the specific steps are as follows:

[0025] Step 1: Reuse the extracted hand binary mask. Directly use the hand binary mask generated during the defect index calculation stage of the preprocessed defective hand image and the complete hand prediction image for subsequent finger segmentation and recognition.

[0026] Step 2: Skeletonize the binary mask of the hand in the complete hand prediction image, obtain the hand midline, extract the finger mask by fingertip detection and metacarpophalangeal joint localization, identify the thumb and sort the other four fingers according to the horizontal coordinate, and output the mask of the five fingers, fingertip coordinates or centroid coordinates and corresponding names.

[0027] Step 3: Extract the remaining finger regions from the binary mask of the hand in the preprocessed image of the missing hand using connected component labeling, calculate the centroid or fingertip coordinates of each region, and output the mask and coordinate information of the remaining fingers of the missing hand.

[0028] Step 4: Match missing fingers with complete fingers. Coordinate matching is performed based on Euclidean distance. The overlap is calculated by combining the mask intersection-union ratio (IoU). If IoU ≈ 0, the finger is considered missing. If IoU is greater than 25%, the finger is considered present. The 25% threshold can avoid misjudging background noise and also avoid missing severely missing fingers.

[0029] Step 5: Based on the area percentage of the Disability Index (DI), and combined with the IoU to determine the name and number of missing fingers, a joint assessment is made: DI between 0-5% indicates no significant functional loss; DI between 5-15% indicates a level 10 disability, corresponding to the loss of a single fingertip; DI between 15-25% indicates a level 9 disability, corresponding to the partial or complete loss of a single finger; DI between 25-40% indicates a level 8 disability, corresponding to the complete loss of 1-2 fingers; DI between 40-60% indicates a level 7 disability, corresponding to the loss of multiple fingers; DI between 60-80% indicates a level 6 disability, corresponding to the loss of most fingers; DI between 80-100% indicates a level 5 disability, corresponding to near-total loss of the hand.

[0030] The second objective of this invention is achieved through the following technical solution: a system for reconstructing a missing hand and assessing a disability level, used to implement the aforementioned method for reconstructing a missing hand and assessing a disability level, comprising:

[0031] The preprocessing module is used to perform image segmentation, background removal, and resolution normalization on the original image of the missing hand, and outputs a 256×256 normalized image of the missing hand with a pure black background.

[0032] The model calling module is used to load the pre-trained optimal reconstruction model, receive the incomplete hand image output by the preprocessing module, complete model inference, and generate a complete hand prediction image.

[0033] The evaluation output module is used to extract the binary mask of the hand and count pixels from the preprocessed image of the missing hand and the predicted image of the complete hand. It calculates the Disability Index (DI) and maps it to the disability level according to the national standard. Then, it performs skeletonization, fingertip detection, metacarpophalangeal joint localization and finger sorting on the predicted image of the complete hand to obtain the mask and coordinates of the five fingers. Then, it performs connected component labeling on the image of the missing hand to obtain the mask of the remaining fingers. It completes the matching of missing fingers and complete fingers by Euclidean distance matching and IoU overlap calculation to identify missing fingers. It performs joint judgment by combining DI and IoU and finally outputs the percentage of the Disability Index, the disability level judgment result and a visual comparison map of the missing hand and the complete hand.

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] 1. This invention makes multi-dimensional targeted improvements to the original Pix2Pix, embedding the attention module CBAM to enable the network to focus on the details of the truncated edge of the hand, improving training stability through spectral normalization, and merging multiple loss functions to balance reconstruction fidelity and preservation of healthy regions. This solves the problems of texture distortion and unnatural edge connection in traditional reconstruction methods, and achieves high-fidelity complete reconstruction of images of incomplete hands, providing accurate visual evidence for disability level assessment.

[0036] 2. This invention constructs a pixel-level disability index quantification index and a finger loss determination method for disabled hands. It achieves objective quantification of the degree of disability by counting the number of pixels in the hand and the difference in area. It also accurately maps the disability index to the "Disability Grading Standard for Work-Related Injuries and Occupational Diseases" (GB / T 16180) and formulates standardized rules for disability grading. This solves the problems of subjectivity and low standardization in traditional assessment methods and improves the objectivity, authority and repeatability of the assessment results.

[0037] 3. This invention is based on deep learning, and the model has strong generalization ability. It can adapt to images of incomplete hands from different shooting angles and backgrounds. At the same time, the quantitative evaluation rules can be flexibly adjusted according to actual assessment needs. It is not only applicable to the assessment of hand disability level in work injury assessment, but can also be extended to related scenarios such as traffic accident disability assessment and limb disability rating, and has broad application prospects. Attached Figure Description

[0038] Figure 1 A schematic diagram of the improved Pix2Pix training and overall implementation process.

[0039] Figure 2This is a schematic diagram of the improved Pix2Pix overall architecture; in the diagram, the true / false probabilities represent the discriminator's judgment result on the true / false classification probability of the input data.

[0040] Figure 3 This is a schematic diagram of a generator architecture with a Convolutional Block Attention Module (CBAM). In the diagram, Conv represents a regular convolution operation, ConvT represents a transposed convolution operation, LeakyReLU and ReLU are different activation functions, and CBAM is the attention enhancement unit for the feature map.

[0041] Figure 4 This is a schematic diagram of the structure of the Convolutional Block Attention Module (CBAM).

[0042] Figure 5 This is a schematic diagram of the channel attention layer structure of the Convolutional Block Attention Module (CBAM).

[0043] Figure 6 This is a schematic diagram of the spatial attention layer structure of the Convolutional Block Attention Module (CBAM).

[0044] Figure 7 This is a diagram illustrating the calculation of the disability index and disability level.

[0045] Figure 8 This is an architecture diagram of the system of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0047] Example 1

[0048] like Figures 1 to 7 As shown in the figure, this embodiment discloses a method for reconstructing a missing hand and assessing the degree of disability. This method uses an improved attention-based Pix2Pix conditional generative adversarial network as its core. By improving the generator, discriminator, and loss function, it achieves high-fidelity reconstruction of the missing hand image. Furthermore, it achieves standardized assessment of the degree of disability through pixel-level quantization and mapping to national standards. Simultaneously, a corresponding modular system is built to ensure the method's implementation. The specific implementation steps are as follows:

[0049] 1) Preprocessing stage: This includes collecting a dataset of original images of mutilated hands, containing images of hands with different types of hand mutilation, different shooting angles, and different backgrounds. Each original image is binarized using the Otsu thresholding method to segment the hand region and the background region. Based on the binarization results, the GrabCut algorithm is used for refined foreground extraction to complete accurate segmentation of the hand region and background removal. Maximum connected component extraction is performed on the segmentation results to remove isolated noise regions in the image. Morphological opening and closing operations are then performed for post-processing to optimize the edge morphology of the hand region. The processed hand images are uniformly scaled to a resolution of 256×256, and the background is filled with pure black to obtain standardized mutilated hand images. After eliminating domain differences, a standardized dataset for model training and evaluation is constructed, which is finally used as the augmented dataset for training.

[0050] 2) Model training phase, such as Figure 2 As shown, the preprocessed augmented dataset is fed into the improved Pix2Pix for iterative training. During training, multi-strategy data augmentation is used to enrich the diversity of input samples. The improved generator and discriminator complete the reconstruction and real / fake determination of the incomplete hand image. A multi-loss term fusion method is used to calculate the loss value between the reconstruction result and the real label. After multiple iterations of optimization until the loss value converges to the minimum, the optimal reconstruction model with strong generalization ability and excellent reconstruction effect is finally obtained. The specific process is as follows:

[0051] The preprocessed augmented dataset is input into the improved Pix2Pix generator, such as... Figure 3 As shown, the classic downsampling-upsampling symmetric architecture of U-Net is adopted, performing 8 downsampling and upsampling operations respectively; a convolutional attention module (CBAM) is embedded in both the downsampling and upsampling stages of the U-Net architecture, as shown in the figure. Figure 4 , Figure 5 , Figure 6 As shown, this module includes a channel attention layer and a spatial attention layer. The channel attention layer achieves channel dimension feature recalibration through global average pooling and global max pooling, while the spatial attention layer achieves spatial dimension feature recalibration through channel dimension pooling, 7×7 convolution and Sigmoid activation. This enables the generator to have stronger foreground and background scene differentiation and global modeling capabilities, accurately extracting texture features of the truncated edges of the incomplete hand, while effectively suppressing background noise interference.

[0052] The discriminator adopts the PatchGAN architecture, which concatenates the input image of the incomplete hand with the reconstructed image output by the generator in the channel dimension to obtain a comprehensive feature map containing the input conditions and the reconstruction results. After downsampling layer by layer, feature information at different scales is extracted, and the result of judging the authenticity of the reconstructed image is output, providing a basis for subsequent loss calculation.

[0053] After obtaining the discriminator's judgment result and the generator's reconstruction result, the improved Pix2Pix uses a multi-loss term fusion method to calculate the loss value between the reconstruction result and the true complete hand label. The total loss function of the improved generator is:

[0054] ;

[0055] In the formula, Represents a generator. This represents the total loss of the generator. The adversarial loss is represented and calculated using the standard binary cross-entropy loss. Represents pixel level Reconstruction losses, Represents pixel level The weighting coefficient for reconstruction loss is set to 100. The VGG perceptual loss is represented by the extraction of two high-level semantic features (ReLU3_4 and ReLU5_4) from a pre-trained VGG-19 network. The L1 distance between the reconstructed image and the real image is calculated in the feature space. The weighting coefficient representing the VGG perceptual loss has a value of 10. This indicates that the content has been preserved. The weighting coefficient representing the content retention loss has a value of 10.

[0056] The loss value is fed back to the generator and discriminator through the backpropagation algorithm to iteratively optimize the network parameters. The loss value is evaluated on a preset validation set after a fixed number of training rounds. When the loss value on the validation set converges to the minimum value and no longer decreases, training is stopped and the current model parameters are saved. Finally, the optimal missing hand reconstruction model is obtained, which is called the optimal reconstruction model.

[0057] 3) Evaluation and application phase, such as Figure 7 As shown, the image of the amputated hand to be evaluated is preprocessed and then input into the optimal reconstruction model. Through image reconstruction, mask extraction, pixel counting, amputation index calculation, amputation finger recognition and disability level mapping, a standardized evaluation result is output. The specific process is as follows:

[0058] For the original image of the mutilated hand to be evaluated, repeat all operations in step 1) to generate a standardized mutilated hand image with a pure black background of 256×256. Input the standardized mutilated hand image into the optimal reconstruction model. The model generates a corresponding 256×256 complete hand prediction image through forward inference. Extract the hand binary mask from both the preprocessed mutilated hand image and the complete hand prediction image output by the model: First, convert both images to grayscale. Extract the hand pixel region according to the rule that the grayscale threshold is greater than 15. Then, perform morphological closing and opening operations on the extracted mask to eliminate noise and holes in the mask, and obtain a clean and complete hand binary mask. Count the pixels of the two hand binary masks to obtain the number of hand pixels in the preprocessed mutilated hand image and the number of hand pixels in the complete hand prediction image. Calculate the mutilation index DI of the hand to be evaluated according to the pixel-level area difference formula. The formula for calculating the mutilation index is:

[0059] ;

[0060] In the formula, The defect index, Predict the number of hand pixels in a complete hand image; This represents the number of pixels in the hand portion of the preprocessed image of the incomplete hand.

[0061] For static hand missing finger recognition, the aforementioned hand binary mask is reused for missing finger recognition. The hand binary mask of the complete hand prediction image is skeletonized to obtain the hand central axis with a single pixel width. The endpoints on the skeleton are detected as fingertips. The metacarpophalangeal joints are located according to the skeleton branches and the centroid of the palm. The finger regions are extracted along the skeleton from the metacarpophalangeal joints to the fingertips and a five-finger mask is generated. The thumb is identified based on angular features and the other four fingers are sorted from left to right according to the horizontal coordinates to complete the naming of the five fingers. For the hand binary mask of the missing hand image, connected component labeling is used to extract the independent regions of the remaining fingers. The centroid and fingertip coordinates of each connected region are calculated. The remaining fingers of the missing hand are matched with the five-finger mask of the complete hand using Euclidean distance, and the intersection-union ratio (IoU) of the mask is calculated. The presence or absence of the thumb, index finger, middle finger, ring finger, and little finger is determined based on the matching results and overlap.

[0062] The calculated Disability Index (DI) is combined with the missing finger to determine the disability level. A standardized disability level mapping is performed according to the "Disability Grading Standards for Work-Related Injuries and Occupational Diseases" (GB / T 16180). Specific rules are as follows: DI between 0-5% indicates no significant functional loss; DI between 5-15% indicates a level 10 disability; DI between 15-25% indicates a level 9 disability; DI between 25-40% indicates a level 8 disability; DI between 40-60% indicates a level 7 disability; DI between 60-80% indicates a level 6 disability; and DI between 80-100% indicates a level 5 disability. The final output is the percentage of the Disability Index of the hand to be assessed and the standardized disability level determination result. Simultaneously, a visual comparison image of the original image of the missing hand and the predicted image of the complete hand is output, providing visual evidence for manual review.

[0063] Example 2

[0064] This embodiment discloses a system for reconstructing a missing hand and assessing a disability level, used to implement the system for reconstructing a missing hand and assessing a disability level described in Embodiment 1, such as... Figure 8 As shown, it includes the following functional modules:

[0065] The preprocessing module is used to perform image segmentation, background removal, and resolution normalization on the original image of the missing hand, and output a normalized image of the missing hand with a pure black background.

[0066] The model calling module is used to load the pre-trained optimal reconstruction model, receive the incomplete hand image output by the preprocessing module, complete model inference, and generate a complete hand prediction image.

[0067] The evaluation output module is used to extract the binary mask of the hand and count pixels from the preprocessed image of the missing hand and the predicted image of the complete hand. It calculates the Disability Index (DI) and maps it to the disability level according to the national standard. Then, it performs skeletonization, fingertip detection, metacarpophalangeal joint localization and finger sorting on the predicted image of the complete hand to obtain the mask and coordinates of the five fingers. Then, it performs connected component labeling on the image of the missing hand to obtain the mask of the remaining fingers. It completes the matching of missing fingers and complete fingers by Euclidean distance matching and IoU overlap calculation to identify missing fingers. It performs joint judgment by combining DI and IoU and finally outputs the percentage of the Disability Index, the disability level judgment result and a visual comparison map of the missing hand and the complete hand.

[0068] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for reconstructing a missing hand and assessing the degree of disability, characterized in that, This method uses an improved conditional generative adversarial network, Pix2Pix, as its core to achieve complete reconstruction of images of missing hands and quantitative assessment of disability levels. Pix2Pix improves upon the original Pix2Pix's generator, discriminator, and loss function. The generator is improved by embedding a convolutional block attention module (CBAM) into the U-Net architecture, allowing the network to focus on the texture details of the truncated edges of the hand. The discriminator is improved by adding spectral normalization to the convolutional layers of the PatchGAN architecture to enhance training stability. The loss function is improved by incorporating VGG perceptual loss and content preservation loss to balance the realism of the reconstructed texture with the preservation of pixels in healthy regions. Specifically, the content preservation loss involves automatically extracting a mask of the existing region from the image of the missing hand, performing dilation on the mask to smooth the edges, and then calculating the L1 distance between the reconstructed image and the original image of the missing hand within the mask region. This constrains the generator to generate only in the missing regions, ensuring that the healthy regions of the image of the missing hand are completely consistent with the reconstructed image. The specific implementation of this method includes the following steps: 1) Preprocessing stage: The Otsu thresholding method and GrabCut algorithm are used to extract the hand region and remove the background from the incomplete hand image. Both the Otsu thresholding method and GrabCut algorithm include maximum connected component extraction and morphological post-processing steps, and finally obtain a standardized incomplete hand image with a pure black background and uniform resolution, thus eliminating the domain difference between the training data and the real data. 2) The preprocessed incomplete hand image is used as a feature map and fed into the improved Pix2Pix for training. The improved generator reconstructs the complete hand image from the incomplete hand image. The improved discriminator determines whether the reconstructed image is real or fake compared with the real complete hand image. In backpropagation, the loss value between the reconstruction result and the real label is calculated through the improved loss function. After multiple iterations until the loss value is minimized, the trained Pix2Pix is ​​obtained as the optimal reconstruction model. 3) After preprocessing the image of the missing hand to be evaluated in step 1), input it into the optimal reconstruction model to generate the corresponding predicted image of the complete hand. Extract the binary mask of the hand from the preprocessed image of the missing hand and the predicted image of the complete hand respectively. During extraction, perform morphological closing and opening operations on the binary mask of the hand to eliminate noise. Extract and count the pixels of the hand according to the rule that the gray level threshold is greater than the preset value. Then calculate the disability index by pixel-level area difference. At the same time, re-use the binary mask of the hand to identify the fingers of the complete missing hand and determine the missing fingers. Finally, map the disability index to the standardized disability level with reference to the "Disability Grading of Employees with Work-Related Injuries and Occupational Diseases" (GB / T 16180) and output the disability index percentage and disability level determination result.

2. The method for reconstructing a missing hand and assessing the degree of disability according to claim 1, characterized in that, The Convolutional Block Attention (CBAM) module sequentially performs channel attention and spatial attention processing on the input feature map, and adds the processed feature map to the original input feature map through residual connections. The channel attention processing involves performing global average pooling and global max pooling on the feature map to obtain two channel-dimensional feature vectors, which are then added together through a shared two-layer multilayer perceptron, and then the channel weights are obtained through a sigmoid activation function. The input feature map is multiplied by the channel weights to complete the channel dimension recalibration. The spatial attention processing involves taking the average and maximum values ​​along the channel dimension of the channel-recalibrated feature map, concatenating them, and then performing a 7×7 convolution and a sigmoid activation function to obtain spatial weights. The channel-recalibrated feature map is multiplied by the spatial weights to complete the spatial dimension recalibration.

3. The method for hand reconstruction and disability assessment according to claim 1, characterized in that, The spectral normalization improves all convolutional layers of the discriminator in the PatchGAN architecture by sequentially performing spectral norm calculation and weight normalization on the weight matrix of the convolutional layers, and then applying the normalized weight matrix to the feature mapping of the convolutional layers. The spectral norm calculation involves solving for the maximum singular value of the weight matrix of the convolutional layers to obtain the spectral norm that represents the numerical range of the weight matrix. The weight normalization process involves dividing the original weight matrix of the convolutional layers by its corresponding spectral norm, constraining the spectral norm of the normalized weight matrix, limiting the Lipschitz norm of the discriminator, and using the normalized weight matrix for feature extraction and mapping of the convolutional layers, thus completing the spectral normalization improvement of the discriminator.

4. The method for reconstructing a missing hand and assessing the degree of disability according to claim 1, characterized in that, The improved generator's total loss function is: ； In the formula, Represents a generator. This represents the total loss of the generator. The adversarial loss is represented and calculated using the standard binary cross-entropy loss. Represents pixel level Reconstruction losses, Represents pixel level Weighting coefficients for reconstruction loss; The VGG perceptual loss is represented by the extraction of two high-level semantic features (ReLU3_4 and ReLU5_4) from a pre-trained VGG-19 network. The L1 distance between the reconstructed image and the real image is calculated in the feature space. The weighting coefficients represent the VGG perceptual loss. This indicates that the content has been preserved. The weighting coefficient represents the loss in content retention.

5. The method for hand reconstruction and disability assessment according to claim 1, characterized in that, The calculation process for the content retention loss is as follows: a) Automatically extract the mask M of the existing region from the input image of the missing hand. The specific extraction process is as follows: First, convert the input image of the missing hand into a grayscale image and determine the pixel threshold as -0.

9. This threshold is obtained by actual measurement. The actual measurement shows that the grayscale value 12 / 255 is a clear finger region. After normalizing the grayscale value to the range of [-1,1], the corresponding pixel threshold -0.9 is obtained. Mark the region in the grayscale image with a pixel value greater than -0.9 as 1. This region is the region where a healthy finger exists. Areas with pixel values ​​less than or equal to -0.9 are marked as 0, and these areas are either missing fingers or background areas; b. Perform a 3×3 max pooling dilation operation on the mask M to expand the boundary pixels and achieve a smooth transition at the image edges; c. Calculate the L1 distance between the reconstructed image and the input image within the mask region to obtain the content preservation loss, so that the generator can freely generate the missing parts only in the mask 0 region, and output pixels that are completely consistent with the input in the mask 1 region.

6. The method for hand reconstruction and disability assessment according to claim 1, characterized in that, The formula for calculating the incompleteness index is as follows: ； In the formula, The defect index, Predict the number of hand pixels in a complete hand image; This represents the number of pixels in the hand portion of the preprocessed image of the incomplete hand.

7. The method for hand reconstruction and disability assessment according to claim 6, characterized in that, The specific steps for recognizing missing fingers in a static hand are as follows: Step 1: Reuse the extracted hand binary mask. Directly use the hand binary mask generated during the defect index calculation stage of the preprocessed defective hand image and the complete hand prediction image for subsequent finger segmentation and recognition. Step 2: Skeletonize the binary mask of the hand in the complete hand prediction image, obtain the hand midline, extract the finger mask by fingertip detection and metacarpophalangeal joint localization, identify the thumb and sort the other four fingers according to the horizontal coordinate, and output the mask of the five fingers, fingertip coordinates or centroid coordinates and corresponding names. Step 3: Extract the remaining finger regions from the binary mask of the hand in the preprocessed image of the missing hand using connected component labeling, calculate the centroid or fingertip coordinates of each region, and output the mask and coordinate information of the remaining fingers of the missing hand. Step 4: Match missing fingers with complete fingers. Coordinate matching is performed based on Euclidean distance. The overlap is calculated by combining the mask intersection-union ratio (IoU). If IoU ≈ 0, the finger is considered missing. If IoU is greater than 25%, the finger is considered present. The 25% threshold can avoid misjudging background noise and also avoid missing severely missing fingers. Step 5: Based on the area percentage of the Disability Index (DI), and combined with the IoU to determine the name and number of missing fingers, a joint assessment is made: DI between 0-5% indicates no significant functional loss; DI between 5-15% indicates a level 10 disability, corresponding to the loss of a single fingertip; DI between 15-25% indicates a level 9 disability, corresponding to the partial or complete loss of a single finger; DI between 25-40% indicates a level 8 disability, corresponding to the complete loss of 1-2 fingers; DI between 40-60% indicates a level 7 disability, corresponding to the loss of multiple fingers; DI between 60-80% indicates a level 6 disability, corresponding to the loss of most fingers; DI between 80-100% indicates a level 5 disability, corresponding to near-total loss of the hand.

8. A system for reconstructing a missing hand and assessing disability levels, characterized in that, The method for reconstructing a missing hand and assessing a disability level according to any one of claims 1 to 7 includes: The preprocessing module is used to perform image segmentation, background removal, and resolution normalization on the original image of the missing hand, and output a normalized image of the missing hand with a pure black background. The model calling module is used to load the pre-trained optimal reconstruction model, receive the incomplete hand image output by the preprocessing module, complete model inference, and generate a complete hand prediction image. The evaluation output module is used to extract the binary mask of the hand and count pixels from the preprocessed image of the missing hand and the predicted image of the complete hand. It calculates the Disability Index (DI) and maps it to the disability level according to the national standard. Then, it performs skeletonization, fingertip detection, metacarpophalangeal joint localization and finger sorting on the predicted image of the complete hand to obtain the mask and coordinates of the five fingers. Then, it performs connected component labeling on the image of the missing hand to obtain the mask of the remaining fingers. It completes the matching of missing fingers and complete fingers by Euclidean distance matching and IoU overlap calculation to identify missing fingers. It performs joint judgment by combining DI and IoU and finally outputs the percentage of the Disability Index, the disability level judgment result and a visual comparison map of the missing hand and the complete hand.

Images