Static plantar pressure image data processing method based on improved YOLOV5 model

Abstract

The invention relates to the technical field of image processing, in particular to a static plantar pressure image data processing method based on an improved YOLOV5 model. Compared with traditional floor table type equipment and desktop level equipment, the static plantar pressure diagnosis system for auxiliary diagnosis of the flatfoot based on the YOLOV5 model can remarkably improve the flatfoot diagnosis effect. The improved YOLOV5 model added with the attention mechanism module can further improve the flatfoot diagnosis effect, early diagnosis and intervention are carried out on the patient, and good prognosis is achieved. Based on an advanced deep learning model and a comprehensive data analysis method, the accuracy and efficiency of flatfoot evaluation are remarkably improved through automatic auxiliary diagnosis. Through a highly automatic process, manual intervention is not needed in the whole process from data acquisition, preprocessing to diagnosis output, personal errors are reduced, and the reliability and consistency of diagnosis are improved. A non-professional user can easily use the foot sole health management system through a friendly user interface and simplified operation steps, the foot sole health management system is widely suitable for various scenes such as families, clinics and hospitals, and intelligentization and popularization of foot sole health management are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a static plantar pressure image data processing method based on an improved YOLOV5 model. Background Art

[0002] There are two types of static plantar pressure measurement systems on the market:

[0003] One is a relatively large floor-standing desktop device, which consists of a workstation and a force platform. It uses a high-density, high-precision pressure sensor matrix to provide a detailed plantar pressure distribution map, equipped with advanced data analysis and visualization, and can provide detailed pressure distribution maps, center of gravity trajectory analysis, gait analysis, etc. It is used in hospital departments by professional physicians and requires patients to go to the hospital regularly for diagnosis. For example, the flat-plate plantar pressure test system of Jiangsu Dechang Medical Technology Co., Ltd.

[0004] For larger floor-standing desktop devices: the operation is complicated and requires professional personnel to operate and maintain (Note 1). At the same time, the visual interface is not user-friendly and cannot perform automated diagnosis. It needs to be operated by a professional physician and analyzed and evaluated (Note 2) for diagnosis.

[0005] Although the diagnostic range is wide, including flat feet, high arches, bunions and other foot structure abnormalities and plantar ulcer risk assessment, due to the wide range, the traditional image processing algorithms generally used cannot improve the accuracy and speed of detecting a specific foot abnormality.

[0006] The other is a smaller portable interval-level device that can provide basic plantar pressure distribution diagrams and simple data analysis functions. It is quick to install, has a user-friendly interface, and is easy for non-professionals to use. It is generally used by professional physicians in hospital departments, and can also be extended to home scenes for patients to use themselves. For example, the SENNOSCAN-21A foot pressure detector from Shenzhen Chuanggan Technology Co., Ltd. is a desktop plantar pressure detector.

[0007] For smaller desktop devices: Compared with large systems, the accuracy and resolution of traditional foot image processing are significantly insufficient. Some desktop devices may lack sufficient clinical validation, making it difficult to guarantee the accuracy and reliability of their diagnostic results. Because the output of the plantar pressure detector is the pressure of each area of ​​the foot, the data processing capacity is limited and may not be able to meet professional medical needs, and the diagnosis and evaluation capabilities of complex conditions are limited. At the same time, there is also the disadvantage that large systems cannot achieve automated diagnosis, and need to be judged by experienced staff or professional clinicians, or patients need to receive professional training, which limits their use. Summary of the invention

[0008] The purpose of the present invention is to provide a static plantar pressure image data processing method based on an improved YOLOV5 model in order to solve the above problems.

[0009] The technical solution of the present invention is as follows:

[0010] The static plantar pressure image data processing method based on the improved YOLOV5 model includes the following steps:

[0011] Step (1) Plantar pressure image acquisition: Use a high-precision plantar pressure sensor array to obtain the pressure distribution data of the plantar in a static standing state; the sensor array covers the entire plantar area and can accurately measure the pressure value of each pressure point; the real-time plantar pressure image acquisition is realized, and the pressure data collected by the pressure sensor matrix is ​​P,

[0012] P=PressureSensorMatrix(t);

[0013] Step (2) Plantar image data acquisition: Use a high-resolution camera to capture plantar images to ensure image clarity and detail integrity. The camera is installed in a fixed position to ensure consistency in shooting angle and distance; the image data captured by the high-resolution camera is I: I = HighResCamera(t);

[0014] Step (3) Plantar pressure data preprocessing: normalize the collected plantar pressure data to eliminate the pressure differences between different individuals. Use filtering technology to remove noise in the data to ensure the accuracy and consistency of the pressure data.

[0015] Step (4) Plantar image data preprocessing: crop, rotate and scale the plantar image to ensure that the foot area in the image is located at the center of the image and has a consistent size. Image enhancement technology is applied to improve the image clarity and feature visibility.

[0016] Step (5) Data alignment: Align the plantar pressure data and the plantar image data using the SIFT image registration algorithm to ensure that the two types of data correspond to the same time point and foot area, and the two are aligned and output on the same image;

[0017] Step (6) extracting fusion features: extracting aligned plantar pressure features and plantar images based on the improved YOLOV5 model, and performing feature fusion;

[0018] Step (7) high-precision image recognition: The YOLOV5 model uses deep learning technology to achieve high-precision recognition of plantar images. It is known for its efficient architecture that can achieve target detection in a single forward pass, can process input images in a very short time, and can identify important features in the image with high precision. Compared with traditional image processing methods, the YOLOV5 model can maintain excellent recognition performance in complex backgrounds and under different lighting conditions.

[0019] Step (8) obtains a confidence score based on the YOLO target detection algorithm: The YOLO series is a single-stage target detection algorithm, and YOLOV1 divides the input image into grids of uniform size; regards the prediction of the target and target bounding box in the grid as a regression problem; and obtains the location, confidence, and category of the object.

[0020] The step (3) of pre-processing the plantar pressure data specifically comprises the following steps:

[0021] Step (3.1) Develop an automated image pixel normalization and noise filtering algorithm to standardize the collected pressure data and eliminate differences between individuals. The algorithm can scale and adjust the data according to the preset standard pressure range. Let the normalized data be P norm :

[0022]

[0023] Then Gaussian filtering is performed to remove noise from the data to ensure the accuracy and consistency of the data. The filtered data is P filtered :

[0024] P filtered =P norm *G(x,y,σ)

[0025] Among them, G(x,y,σ) is a Gaussian kernel with scale σ, and * represents the convolution operation.

[0026] The step (6) of extracting fusion features specifically includes the following steps:

[0027] The aligned plantar pressure features and plantar images are extracted based on the improved YOLOv5 model, and feature fusion is performed; real-time recognition, for the plantar image I, the feature map F is extracted by the YOLOv5 model I , plantar pressure map F P :

[0028] F I =YOLOv5(I)

[0029] F P =YOLOv5(P)

[0030] The foot sole image feature F I and plantar pressure characteristics F P Alignment, the aligned features are recorded as F' I and F' P :

[0031] F′ I = SIFT(F I );

[0032] F′ P =SIFT(F P );

[0033] The aligned plantar image feature F' I and plantar pressure characteristics F' P Fusion is performed to obtain the fusion feature F F . Use the weighted summation method for fusion, with weights α and β respectively:

[0034] F F =αF′ I +βF′ P ;

[0035] α+β=1;

[0036] Based on the fusion feature F F Perform real-time recognition and use the classifier to obtain the recognition result

[0037]

[0038] The step (7) of high-precision image recognition specifically includes the following steps:

[0039] Step (3.1) Compression operation: Compress the feature map output by the C3 block from the size of H×W×C to the feature map Z of 1×1×C through global average pooling. The formula is as follows:

[0040]

[0041] Step (3.2) Excitation operation: Channel recalibration is performed by learning the channel weight vector S. σ represents the sigmoid activation function, W1 and W2 are learnable parameter matrices, δ represents the ReLU activation function, and the excitation operation formula is as follows:

[0042] S=σ(E1δE2(Z))

[0043] After being processed by the SE module, channel attention improves the saliency of important features while suppressing irrelevant features.

[0044] CBAM module (Convolutional Block Attention Module):

[0045] Step (3.3) Channel attention branch:

[0046] The size of the input feature map U is H×W×C. Through global maximum pooling and average pooling operations, U is converted into two one-dimensional descriptor vectors. These vectors are then processed and merged through a multi-layer perceptron, and finally a channel weight from 0 to 1 is generated through sigmoid activation. The learned channel weight coefficient is multiplied by the input feature map uX to obtain the channel-focused output feature map ZC, as follows:

[0047] Z C =σ(MLP(F max (u X ))+MLP(F avg (u X )))×u X ;

[0048] Spatial attention branch:

[0049] Through global maximum pooling and average pooling operations, the input feature map uX is compressed in the channel dimension to generate two two-dimensional feature maps. These feature maps are then concatenated on the channel to form a tensor with two channels, which is further convolved through a 7×7 convolution kernel to reduce it to one channel. The spatial attention coefficients are then generated through sigmoid activation. Finally, the input feature map uX is multiplied by these coefficients to obtain the spatially focused output feature map ZX, as follows:

[0050] Z X =σ(Conv 7×7 (F max '(u X )),Conv 7×7 (F avg '(u X )))×u X

[0051] By integrating attention mechanism modules such as SE and CBAM into the C3 module of the YOLOv5 model, the detection accuracy and generalization ability of the model are significantly improved; these improvements enable the model to better focus on information-rich areas and suppress irrelevant areas, thereby achieving more accurate static plantar pressure diagnosis;

[0052] Through training and learning of a large number of plantar images, the improved YOLOV5 model can accurately identify key features such as the height and shape of the arch. These features include but are not limited to the contour, arch, and morphological changes of the arch. The model can accurately locate the position and shape of these features in the image through detailed pixel-level analysis.

[0053] The step (8) of obtaining a confidence score based on the YOLO target detection algorithm specifically includes the following steps:

[0054] In YOLOV3, the parameters of the center position are logistically regressed to limit them to the range of 0-1, which greatly improves the detection accuracy. In addition, multiple independent logistic regression classifiers are used to replace SoftMax to improve the accuracy of multi-target classification. Based on YOLOV3, YOLOV4 combines advanced technologies such as data enhancement, regularization and loss function improvement and enhancement of model sensitivity field, introduction of attention mechanism, feature integration and post-processing methods to further improve detection efficiency. The prior anchor frame of YOLOV5 is similar to that of YOLOV3 and YOLOV4. However, YOLOV5 incorporates the training prediction anchor frame into the network. During the training process, the optimal anchor frame value of different training sets is adaptively calculated to make the model suitable for various data sets, which greatly improves the positioning accuracy of different tasks and data sets.

[0055] The YOLOv5 model has four variants, namely YOLOv5s, YOLOv5 m, YOLOv5 l and YOLOv5x. Among them, the model size of YOLOv5s is relatively small, so it is suitable for embedded applications. The model architecture can be divided into three parts: CSPDarkNet53 for feature extraction is an improved DarkNet53 network structure that uses the Cross Stage Partial (CSP) module for feature extraction. Let the input image be I and the output feature map be F, then:

[0056] F=CSPDarkNet53(I)

[0057] FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) are used to fuse and transmit feature maps. The feature map F after FPN and PAN modules fpn and F pan :

[0058] F fpn =FPN(F);

[0059] F pan =PAN(F);

[0060] The category and position prediction modules are based on the feature map F fpn Perform target detection;

[0061] Assume the predicted bounding box is (including center coordinates, width, height), category is

[0062]

[0063] By leveraging YOLOv5s, we aim to achieve real-time foot type detection on low-power devices; during inference, when there is an object in a given cell, the IoU between the bounding box and the ground-truth object can be calculated:

[0064]

[0065] in represents the intersection area of ​​the predicted bounding box and the true bounding box, Represents the union area of ​​the predicted bounding box and the true bounding box;

[0066] The predicted category information and target box confidence Multiply them together to get the category confidence score of each target box

[0067] The beneficial effects of the present invention are as follows:

[0068] Traditional floor-standing devices and desktop devices have limited data processing capabilities, usually relying on relatively basic image processing algorithms and low computing power. The features processed are such as the color saturation of the image. Simple pixel-level processing makes it difficult to process large-scale high-resolution data. This project uses the K-means clustering algorithm to obtain accurate prior boxes and divide the feature map into grids. On this basis, the grid division and anchor box prediction methods reduce false detections and missed detections, and improve training efficiency and detection accuracy. At the same time, random affine transformation, hybrid enhancement and HSV enhancement techniques are used to enrich the data set, which can improve the robustness and generalization ability of subsequent models.

[0069] The system's high-precision image recognition technology effects:

[0070] Compared with traditional floor-standing devices and desktop devices, the static plantar pressure diagnosis system based on the YOLOV5 model to assist in the diagnosis of flat feet can significantly improve the diagnosis effect of flat feet. Figure 6 As shown in the figure, the improved YOLOV5 model with the added attention mechanism module can further improve the diagnosis of flat feet, provide early diagnosis and intervention for patients, and achieve a good prognosis.

[0071] Comprehensive data analysis of diagnosis and treatment effects:

[0072] A single data source (such as plantar pressure data or plantar image data) is difficult to fully reflect the health status of the plantar. The diagnosis method based on a single data source tends to ignore some key features, resulting in insufficient accuracy and comprehensiveness of the diagnosis results. The fusion data comprehensive analysis can integrate the information of the two data sources, such as Figure 7 As shown in the heat map, the model has learned more features to more comprehensively and accurately reflect the health status of the sole of the foot, thereby improving the diagnosis effect.

[0073] This project system is based on advanced deep learning models and comprehensive data analysis methods. Automated auxiliary diagnosis significantly improves the accuracy and efficiency of flat foot assessment. Through a highly automated process, from data collection, preprocessing to diagnostic output, the entire process does not require human intervention, reducing human errors and improving the reliability and consistency of diagnosis. Figure 8 The friendly user interface and simplified operation steps make it easy for non-professional users to use it. It is widely used in various scenarios such as home, clinics and hospitals, realizing the intelligent and popularization of plantar health management.

[0074] Advanced data preprocessing methods to optimize images (obtaining prior boxes + data enhancement)

[0075] Plantar image and plantar pressure data fusion alignment procedure: Through image registration, the SIFT image registration algorithm based on feature points is used to find the key points in the plantar image, calculate the mapping relationship between the plantar image and the pressure sensor array, and align the key points on the image with the sensor data. During the data acquisition process, the timestamps of the plantar pressure data and the plantar image are recorded synchronously, and the data are aligned according to the timestamps to ensure that they correspond to the same time point.

[0076] The improved YOLOV5 model system based on the attention mechanism: controlled by the host buttons + screen display, and controlled autonomously by the patient; after the host is connected to the plantar detector, the patient's plantar data is collected and input into the embedded system for diagnosis by the image acquisition card, and the results are visualized on the host screen.

[0077] The system realizes: fully automated collection, fusion, recognition and output of patient diagnosis (including but not limited to diagnostic information, plantar pressure distribution ratio, proportional distribution of pressure in various areas of the plantar, and plantar images and identification frames, etc.), which is convenient for patients, their families and medical staff to monitor, track and review and analyze, and determine whether the patient has flat feet or whether the intervention after diagnosis is effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1Schematic diagram of data enhancement of the present invention. We randomly change the hue, saturation and value of the image by 15%, 70% and 40% respectively. Horizontal enhancement with 50% probability, mosaic enhancement with 100% probability and mixed enhancement with 10% probability are used for the image.

[0079] Figure 2 Schematic diagram of the basic modules of YOLO-v5 and improved YOLO-v5 of the present invention. Figure 2 Middle: (A) YOLO-v5 core network. It consists of two 6×6 convolutions and three CSPDarkNet53 network modules. For each CSP module, the feature map size is reduced to half of the original size and the feature channels are doubled. (B) Neck part of YOLO-v5. Using the classic feature pyramid structure, all scales are connected and fused from top to bottom to construct a high-level semantic feature map. (C) Multi-scale prediction head of YOLO-v5. It predicts three prediction graphs with different aspect ratios. Each scale prediction graph will predict three target anchor frames at the same time. The target anchor frame needs to predict four positions, the confidence and classification information of the anchor frame. (D) SE attention structure. By squeezing the global pooling operation, the spatial dimension is compressed into a feature map of size 1×1×C to extract global and weight information. (E) CBAM attention structure. It can not only learn the importance of each feature channel independently, but also add the maximum scaling operation, which can simultaneously obtain important information of each feature channel and feature space.

[0080] Figure 3 Schematic diagram of features extracted from different model network layers of the present invention. Figure 3 In the figure: a, b, c, d, e, and f represent the features extracted by different model network layers.

[0081] Figure 4 This is the visualized plantar pressure image of the present invention. Figure 4 Middle: (A) flatfoot patient, (B) healthy person, (C) patient with unilateral flat foot.

[0082] Figure 5 It is the overall automation flow chart of the present invention.

[0083] Figure 6 PR curves are identified for the three different classes of models of the present invention. Figure 6 Middle: (A) Regular YOLO-v5 model. (B) YOLO-v5 model with SE attention mechanism. (C) YOLO-v5 model with CBAM attention mechanism.

[0084] Figure 7 This is the Grad-CAM heat map of the present invention. Figure 7Middle: (A) The high activation area of ​​the pixel sensitivity of the visualization prediction image is located in the midfoot area, (B) The high activation area of ​​the pixel sensitivity of the visualization prediction image is located in the metatarsal area. (C) The high activation area of ​​the pixel sensitivity of the visualization prediction image is located in the midfoot area, (D) The high activation area of ​​the pixel sensitivity of the visualization prediction image is located in the metatarsal area.

[0085] Figure 8 It is the user interface of the present invention. Figure 8 Middle: (A) Foot mapping plantar system. (B) Plantar pressure image. (C) Plantar pressure diagnosis output image, 1 is defined as flat foot, 0 is defined as healthy foot. (D) Image pressure distribution ratio. (E) Proportional distribution of pressure in each area of ​​the plantar.

[0086] Fig. 9 This is the plantar pressure image and annotation of the present invention. Fig. 9 Middle: (A) Flat foot, (C) Healthy foot, (E) Patient with one flat foot. The red dashed box indicates the ground truth. The following three images are visualization results manually labeled by doctors using labelImg. (B) Flat foot, (D) Healthy foot, (F) Patient has one flat foot. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] Example 1: Plantar pressure image acquisition and advanced data preprocessing method implementation

[0089] The original plantar pressure image can be obtained through the static plantar pressure detection platform and the image acquisition device. The existing plantar pressure system has some limitations in the processing of raw image data, such as poor generalization ability, information loss and noise interference, which will affect the detection accuracy and robustness of the static plantar pressure diagnosis system output based on the improved YOLOV5 model to assist in the diagnosis of flat feet. In order to solve these problems, the present invention integrates multiple data preprocessing methods, performs a series of preprocessing on the plantar image, ensures data quality, and improves the recognition accuracy and generalization ability of the subsequent diagnosis model.

[0090] 1.2 Technical implementation plan:

[0091] Obtaining the prior box:

[0092] We use the K-means clustering algorithm to obtain 9 prior boxes. When the sizes are (w k ,h k ), each prior box can be expressed as:

[0093] B k =(w k ,h k )

[0094] where k∈{1, 2, 3, 4, 5, 6, 7, 8, 9}, w k and h k Represent the width and height of the kth prior box respectively.

[0095] Use these prior boxes to divide the feature map. When the input image size is (W, H), the feature map is divided into grids of the same size as the prior box according to the size of the prior box:

[0096]

[0097] Object detection and box prediction:

[0098] When the center of the object is within the grid, let the target box be (x, y, w, h), where x, y are the coordinates of the box center, and w, h are the width and height of the box. Then this specific grid unit is responsible for object detection. The model is based on the initial anchor box B k Output prediction box The difference is calculated and the network parameters are updated iteratively.

[0099] The loss function can be expressed as:

[0100] Loss total =Loss coord +Loss conf +Loss class

[0101] The coordinate loss is:

[0102]

[0103] Confidence loss:

[0104]

[0105] Classification loss:

[0106]

[0107] Data augmentation techniques:

[0108] Random affine transformation: Randomly rotate, scale, translate, and flip images to increase data diversity.

[0109] Hybrid enhancement (Mosaic data enhancement): The hybrid enhancement technology proposed in YOLOv4 and YOLOX is used to stitch four images into one to enrich the background and target of the training data.

[0110] HSV Enhancement: Adjust the hue, saturation, and brightness of an image to enhance the color diversity of the image.

[0111] These data preprocessing methods can effectively improve the detection accuracy in the YOLOV5 framework and enhance model generalization. Figure 1 shown.

[0112] Example 2: High-precision image recognition technology implementation

[0113] The YOLOV5 model is a model with superior performance in the field of target detection. Through deep learning technology, it has achieved high-precision recognition of plantar images. It is known for its efficient architecture that can achieve target detection in a single forward pass. It can process input images in a very short time and identify important features in the image with high precision. Compared with traditional image processing methods, the YOLOV5 model can maintain excellent recognition performance in complex backgrounds and different lighting conditions.

[0114] 2.2 Technical implementation plan:

[0115] In the application of plantar images, the introduction of the improved YOLOV5 model significantly improves the diagnostic ability of the plantar pressure system. The present invention enhances the feature representation capability by integrating the attention mechanism module into the YOLOv5 model, especially into the C3 module (the third residual block of the CSPDarknet53 backbone network). The specific implementation includes the following attention mechanism modules: Squeeze-and-Excitation Networks (SENet) and Convolutional BlockAttention Module (CBAM). SE module and CBAM module are commonly used attention mechanism modules. SENet improves feature learning ability by modeling channel relationships, while CBAM focuses on information-rich areas and suppresses irrelevant areas by inferring spatial and channel attention in turn. The simple and efficient design of SE, CBAM and other attention modules makes it easy to integrate into different levels of YOLOv5s, thereby significantly improving performance and achieving high-precision image recognition.

[0116] SE module (Squeeze-and-Excitation):

[0117] Compression operation: The feature map output by the C3 block is compressed from the size of H×W×C to the feature map Z of 1×1×C through global average pooling. The formula is as follows:

[0118]

[0119] Excitation operation: Channel recalibration is performed by learning the channel weight vector S. σ represents the sigmoid activation function, W1 and W2 are learnable parameter matrices, δ represents the ReLU activation function, and the excitation operation formula is as follows:

[0120] S=σ(E1δE2(Z))

[0121] After being processed by the SE module, channel attention improves the saliency of important features while suppressing irrelevant features.

[0122] CBAM module (Convolutional Block Attention Module):

[0123] Channel attention branch:

[0124] The size of the input feature map U is H×W×C. Through global maximum pooling and average pooling operations, U is converted into two one-dimensional descriptor vectors. These vectors are then processed and merged through a multi-layer perceptron, and finally a channel weight from 0 to 1 is generated through sigmoid activation. The learned channel weight coefficient is multiplied by the input feature map uX to obtain the channel-focused output feature map ZC, as follows:

[0125] Z C =σ(MLP(F max (u X ))+MLP(F avg (u X )))×u X

[0126] Spatial attention branch:

[0127] Through global maximum pooling and average pooling operations, the input feature map uX is compressed in the channel dimension to generate two two-dimensional feature maps. These feature maps are then concatenated on the channel to form a tensor with two channels, which is further convolved through a 7×7 convolution kernel to reduce it to one channel. The spatial attention coefficients are then generated through sigmoid activation. Finally, the input feature map uX is multiplied by these coefficients to obtain the spatially focused output feature map ZX, as follows:

[0128] Z X=σ(Conv 7×7 (F max '(u X )),Conv 7×7 (F avg '(u X )))×u X

[0129] like Figure 2 As shown in the figure, by integrating attention mechanism modules such as SE and CBAM into the C3 module of the YOLOv5 model, the present invention significantly improves the detection accuracy and generalization ability of the model. These improvements enable the model to better focus on information-rich areas and suppress irrelevant areas, thereby achieving more accurate static plantar pressure diagnosis.

[0130] Through training and learning of a large number of plantar images, the improved YOLOV5 model can accurately identify key features such as the height and shape of the arch. Figure 3 As shown, these features include but are not limited to the contour, camber, and morphological changes of the arch of the foot. The model can accurately locate the position and morphology of these features in the image through detailed pixel-level analysis.

[0131] The overall process of the static plantar pressure diagnosis automation system for flatfoot assisted diagnosis based on the improved YOLOV5 model is as follows: Figure 5 shown.

[0132] The YOLO series is a single-stage target detection algorithm. YOLOV1 divides the input image into grids of uniform size; treats the prediction of targets and target bounding boxes in the grid as a regression problem; and obtains the location, confidence, and category of the object. However, YOLO-v1 poses challenges in accurately locating objects and has a low recall rate. Therefore, k-means clustering is added to YOLOV2, and prior anchor frames are used to perform migration constraints and confidence prediction on the target location; this further improves the performance of YOLOV1. However, during the target detection process, the regression parameters of the center position of the prior anchor frame in YOLOV2 are not effectively constrained, which may cause the predicted anchor frame to appear at any position in the original image, resulting in low prediction accuracy. In addition, SoftMax activation classification is only applicable to a single target and not to multiple classifications.

[0133] Therefore, in YOLOV3, a logistic regression is performed on the parameters of the center position to limit it to the range of 0–1, which greatly improves the detection accuracy. In addition, multiple independent logistic regression classifiers are used to replace SoftMax to improve the accuracy of multi-target classification. Based on YOLOV3, YOLOV4 combines advanced technologies such as Bag-of-FreeBies (data enhancement, regularization, and loss function improvement) and Bag-of-Specials (enhancing the model sensitivity field, introducing attention mechanism, feature integration, and post-processing methods) to further improve detection efficiency. The prior anchor frame of YOLOV5 is similar to that of YOLOV3 and YOLOV4; however, YOLOV5 incorporates the training prediction anchor frame into the network. During the training process, the optimal anchor frame values ​​of different training sets are adaptively calculated, making the model applicable to various data sets, greatly improving the positioning accuracy of different tasks and data sets.

[0134] The YOLOv5 model has four variants, namely YOLOv5s, YOLOv5 m, YOLOv5 l and YOLOv5x; among them, the model size of YOLOv5s is relatively small, so it is suitable for embedded applications. The model architecture can be divided into three parts: CSPDarkNet53 for feature extraction, which is an improved DarkNet53 network structure that uses the Cross Stage Partial (CSP) module for feature extraction. Let the input image be I and the output feature map be F, then:

[0135] F=CSPDarkNet53(I)

[0136] FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) are used to fuse and transmit feature maps. The feature map F after FPN and PAN modules fpn and F fpn :

[0137] F fpn =FPN(F)

[0138] F pan =PAN(F)

[0139] The category and position prediction modules are based on the feature map F fpn Perform target detection. Assume that the predicted bounding box is (including center coordinates, width, height), category is

[0140]

[0141] By leveraging YOLOv5s, our goal is to achieve real-time foot type detection on low-power devices. During inference, when there is an object in a given cell, the IoU between the bounding box and the ground-truth object can be calculated:

[0142]

[0143] in represents the intersection area of ​​the predicted bounding box and the true bounding box, Represents the union area of ​​the predicted bounding box and the true bounding box.

[0144] The predicted category information and target box confidence Multiply them together to get the category confidence score of each target box

[0145]

[0146] However, YOLOv5s has limited learning capabilities. The attention mechanism enhances the object detection model by improving its learned representation capabilities. The attention module requires only a few additional parameters, thereby enhancing functionality without significantly increasing the model complexity.

[0147] Example 3: Comprehensive data analysis to improve diagnosis and treatment efficiency

[0148] Most existing auxiliary diagnosis systems rely on only a single data source, such as plantar pressure data or plantar image data, which makes it difficult to fully reflect the health status of the plantar. Based on the improved YOLOV5 model, the plantar pressure data and plantar image data are integrated and analyzed to improve the response speed of the system, which can achieve a more comprehensive and rapid diagnosis.

[0149] 3.2 Technical implementation plan:

[0150] Plantar pressure data collection

[0151] Use a high-precision plantar pressure sensor array to obtain pressure distribution data on the sole of the foot in a static standing state. The sensor array covers the entire plantar area and can accurately measure the pressure value of each pressure point.

[0152] Plantar image data acquisition

[0153] A high-resolution camera is used to capture images of the sole of the foot, ensuring image clarity and detail integrity. The camera is installed in a fixed position to ensure consistency in shooting angle and distance.

[0154] Plantar pressure data preprocessing

[0155] The collected plantar pressure data is normalized to eliminate the pressure differences between different individuals. Filtering technology is used to remove noise in the data to ensure the accuracy and consistency of the pressure data.

[0156] Plantar image data preprocessing

[0157] The plantar images were cropped, rotated, and scaled to ensure that the foot area in the image was located in the center of the image and had a consistent size. Image enhancement techniques were applied to improve image clarity and feature visibility.

[0158] Data alignment

[0159] The plantar pressure data and plantar image data are aligned using the SIFT image registration algorithm, such as Figure 4 As shown, it is ensured that the two types of data correspond to the same time point and foot area, and the two are aligned and output on the same picture.

[0160] SIFT Image Registration Algorithm

[0161] SIFT (Scale-Invariant Feature Transform) is an algorithm for extracting and describing image features. It has scale invariance and rotation invariance and is widely used in image registration, object recognition, image stitching and other fields. The following is a detailed introduction to the SIFT algorithm and related formulas.

[0162] The main steps of SIFT algorithm are:

[0163] (1) Scale space extreme value detection

[0164] (2) Key point positioning

[0165] (3) Direction allocation

[0166] (4) Key point description

[0167] (5) Feature point matching

[0168] Scale space extrema detection

[0169] Goal: Find potential key points of an image in different scale spaces.

[0170] Method: Construct a Gaussian pyramid and perform Gaussian blur on the image at different scales. Then, calculate the Difference of Gaussian (DoG) pyramid in each scale image to detect extreme points.

[0171] formula:

[0172] Gaussian Blur:

[0173] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0174] Where G(x,y,σ) is a Gaussian kernel with scale σ, I(x,y) is the original image, and * represents the convolution operation.

[0175] Difference of Gaussians:

[0176] D(x,y,σ)=L(x,y,kσ)-L(x,y,σ)

[0177] Here, k is a constant factor, usually k=√2.

[0178] Key point positioning

[0179] Goal: Accurately locate key points detected at different scales and remove low-contrast points and edge response points.

[0180] Method: Perform quadratic surface fitting on each extreme point to accurately determine the position and scale of the key points.

[0181] formula:

[0182] Sub-pixel position of extreme points: where H is the Hessian matrix, is the gradient of the Difference of Gaussian function.

[0183] Hessian matrix and gradient:

[0184] Direction allocation

[0185] Goal: Assign one or more orientations to each keypoint to achieve rotation invariance.

[0186] Method: Calculate the gradient direction histogram of the key point neighborhood and select the direction of the maximum peak as the main direction of the key point.

[0187] formula:

[0188] Gradient magnitude and direction:

[0189] m(x,y)=√(L(x+1,y)-L(x-1,y)) 2 +(L(x,y+1)-L(x,y-1)) 2

[0190]

[0191] Key point description

[0192] Goal: Generate a descriptor for each keypoint for feature matching.

[0193] Method: Calculate the gradient direction and magnitude in the neighborhood of the key point and generate a gradient direction histogram (usually 4

[0194] × 4 grids, each with 8 directions, for a total of 128-dimensional descriptors).

[0195] formula:

[0196] Descriptor generation:

[0197] Descriptor = [H1, H2, ..., H128]

[0198] Where Hi is the count value of the gradient direction histogram.

[0199] Feature point matching

[0200] Goal: Find similar key points in two images.

[0201] Method: Compare descriptors using Euclidean distance or other metrics to find the best matching keypoint pairs.

[0202] formula:

[0203] Euclidean distance:

[0204]

[0205] Among them, d1 and d2 are the descriptors of two key points.

[0206] Extract fusion features

[0207] The aligned plantar pressure features and plantar images are extracted based on the improved YOLOv5 model, and feature fusion is performed. Real-time recognition, for the plantar image I, the feature map F is extracted by the YOLOv5 model I , plantar pressure map F P :

[0208] F I =YOLOv5(I)

[0209] F P =YOLOv5(P)

[0210] The foot sole image feature F I and plantar pressure characteristics F P Alignment, the aligned features are recorded as F' I and F' P :

[0211] F′ I =SIFT(F I )

[0212] F′ P=SIFT(F P )

[0213] The aligned plantar image feature F' I and plantar pressure characteristics F' P Fusion is performed to obtain the fusion feature F F . Use the weighted summation method for fusion, with weights α and β respectively:

[0214] F F =αF′ I +βF′ P

[0215] α+β=1

[0216] Based on the fusion feature F F Perform real-time recognition and use the classifier Classfier to get the recognition result

[0217]

[0218] The model uses its efficient neural network architecture to learn the fused features and complete the detection task with only one forward pass, which greatly shortens the processing time and makes real-time detection and diagnosis possible. This real-time performance is particularly important for clinical applications, as it can provide diagnostic results immediately during patient examinations, improving diagnostic efficiency.

[0219] Embodiment 4: Realization of automated auxiliary diagnosis system

[0220] 4.1 Brief Introduction

[0221] Many systems require a lot of manual intervention, including data annotation and result analysis, which are inefficient and easily affected by human factors. This auxiliary diagnosis system realizes a highly automated diagnosis process, from data acquisition, preprocessing to diagnosis output, all of which are automatically completed by the system. It can improve diagnostic efficiency, reduce manual operation time and human errors. It is easy for users to operate and can be easily used by non-professional users.

[0222] 4.2 Technical implementation plan:

[0223] Data collection automation

[0224] Through the plantar pressure sensor matrix and the high-resolution camera, the real-time plantar pressure image acquisition is realized. The pressure data collected by the pressure sensor matrix is ​​P, and the image data collected by the high-resolution camera is I:

[0225] P = PressureSensorMatrix(t)

[0226] I = HighResCamera(t)

[0227] Automatic preprocessing of plantar pressure image data

[0228] Develop an automated image pixel normalization and noise filtering algorithm to standardize the collected pressure data and eliminate differences between individuals. The algorithm can scale and adjust the data according to the preset standard pressure range. Let the normalized data be P norm :

[0229]

[0230] Then Gaussian filtering is performed to remove noise from the data to ensure the accuracy and consistency of the data. The filtered data is P filtered :

[0231] P filtered =P norm *G(x,y,σ)

[0232] Among them, G(x,y,σ) is a Gaussian kernel with scale σ, and * represents the convolution operation.

[0233] Automatic feature fusion and extraction

[0234] Train and deploy the improved YOLOv5 model to achieve automatic fusion feature extraction. The model can automatically extract key features such as arch height and arch shape from the plantar image. The prediction results are

[0235]

[0236] Diagnosis and output automation

[0237] Use the fused feature data to train the improved YOLOV5 comprehensive diagnosis model. Automated model training is achieved while using automatic grid search parameter adjustment technology to find the best model parameters and optimize the model. The best parameter is θ * :

[0238] θ * =argminθL(YOLOv5(F F ;θ),y)

[0239] Where L is the loss function and y is the true label.

[0240] Real-time diagnosis

[0241] Deploy a real-time diagnosis system, input the real-time collected plantar pressure image data into the trained comprehensive diagnosis model for real-time diagnosis. The system can automatically output the diagnosis results, including healthy feet, flat feet, and flat feet.

[0242] Automatic result presentation and report generation

[0243] Develop an automatic report generation module to generate detailed diagnostic reports, including pressure distribution diagrams and related diagnostic results and presentation boxes. The results are visualized in the user interface to display diagnostic results and suggestions. The interface should be simple and easy to use, so that non-professional users can easily operate and understand the diagnostic results.

[0244] Diagnostic criteria for flat feet

[0245] MLA collapse, physical examination, and X-rays are used to diagnose flatfoot. The midpoint of the calcaneus, the midpoints of the medial and lateral malleolus, and the midpoint of the lower third of the calf are traced and connected. The normal range of the angle obtained is -5° to +5°, and an angle less than -5° is a deformity of inversion of the foot. In addition, a drop of 10 mm or more in the navicular bone would be interpreted as flat. Therefore, further X-rays are required. The X-ray film draws a straight line from the lowest point of the calcaneus to the lowest point of the talus, and then draws a straight line from the lowest point of the talus to the lowest point of the first metatarsal head. The angle between the two lines normal ranges from 113°–130°, and more than 130° is diagnosed as flatfoot.

[0246] During the development of this system, the training dataset based on the improved YOLOV5 model strictly followed the flatfoot inclusion and exclusion criteria:

[0247] Inclusion criteria: (1) Age range: 5-60 years old; (2) Patients with normal gait and no musculoskeletal disease; (3) No history of foot trauma or surgery; (4) All arches were confirmed by X-ray examination.

[0248] Exclusion criteria: (1) obvious foot deformity; (2) presence of high arches; (3) history of lower limb and foot fractures or ankle sprains leading to ligament or articular cartilage damage; (4) leg length discrepancy > 2 cm; (5) patients with cognitive impairment who were unable to cooperate with the test; and (6) missing data.

[0249] For healthy subjects, the inclusion and exclusion criteria were consistent, except for not meeting the diagnostic criteria for flat feet.

[0250] Training annotation of plantar pressure images

[0251] After the collected plantar pressure images were anonymized, two professionally trained rehabilitation therapists labeled them; labelImg (1.8.6) was used to select the midfoot of the image for labeling classification ( Fig. 9). A text file containing the category, object center coordinates, object bounding box, and aspect ratio information of the whole image corresponding to the original image is generated. The images are labeled as flat feet (grade 0) or healthy feet (grade 1), represented by values ​​0 and 1, respectively. In each image, the four corners of the middle foot area are selected as the center points and recorded in a txt file using (x, y) coordinates. The total dataset is divided into training and test sets in a ratio of 8:2 using a random seed 0).

[0252] Evaluation indicators of model systems

[0253] The system of the present invention uses accuracy, precision, recall, F1 score, average precision (AP) and mean average precision (mAP) to evaluate the merits of the model (the formula is as follows):

[0254]

[0255] True positives (TP) are correctly classified positive samples, false positives (FP) are misclassified positives, and false negatives (FN) are misclassified negative samples. Accuracy is the proportion of correctly classified samples. Precision is the ratio of TP to the total number of predicted positives. Recall is the ratio of TP to the total number of actual positives. The F1 score is the harmonic mean of precision and recall. AP and mAP are commonly used in object detection to evaluate the detection effectiveness and performance of a model. AP is calculated by calculating the area under the precision-recall curve, integrating the precision P for each discrete recall R point from 0 to 1. The higher the AP value, the better the detection performance of a certain class. Let C represent the total number of classes in the detection model, c represent each class, and AP(c) represent the AP for each class c. mAP is calculated by first calculating the AP for each class and then averaging the AP for all classes to obtain a comprehensive evaluation indicator that reflects the detection effect of the model on all classes.

[0256] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and scope of the elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0257] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A static plantar pressure image data processing method based on an improved YOLOV5 model, characterized in that: The method comprises the following steps: Step (1) Plantar pressure image acquisition: using a high-precision plantar pressure sensor array to obtain pressure distribution data of the plantar in a static standing state; the sensor array covers the entire plantar area and can accurately measure the pressure value of each pressure point; the real-time plantar pressure image acquisition is realized, and the pressure data collected by the pressure sensor matrix is ​​P. P=PressureSensorMatrix(t); Step (2) Plantar image data acquisition: Use a high-resolution camera to capture plantar images to ensure image clarity and detail integrity. The camera is installed in a fixed position to ensure consistency in shooting angle and distance; the image data captured by the high-resolution camera is I: I = HighResCamera(t); Step (3) Plantar pressure data preprocessing: normalize the collected plantar pressure data to eliminate the pressure differences between different individuals. Use filtering technology to remove noise in the data to ensure the accuracy and consistency of the pressure data. Step (4) Plantar image data preprocessing: crop, rotate and scale the plantar image to ensure that the foot area in the image is located at the center of the image and has a consistent size. Image enhancement technology is applied to improve the image clarity and feature visibility. Step (5) Data alignment: Align the plantar pressure data and the plantar image data using the SIFT image registration algorithm to ensure that the two types of data correspond to the same time point and foot area, and the two are aligned and output on the same image; Step (6) extracting fusion features: extracting aligned plantar pressure features and plantar images based on the improved YOLOV5 model, and performing feature fusion; Step (7) high-precision image recognition: The YOLOV5 model uses deep learning technology to achieve high-precision recognition of plantar images. It is known for its efficient architecture that can achieve target detection in a single forward pass, can process input images in a very short time, and can identify important features in the image with high precision. Compared with traditional image processing methods, the YOLOV5 model can maintain excellent recognition performance under complex backgrounds and different lighting conditions; Step (8) obtains a confidence score based on the YOLO target detection algorithm: The YOLO series is a single-stage target detection algorithm, and YOLOV1 divides the input image into grids of uniform size; regards the prediction of the target and target bounding box in the grid as a regression problem; and obtains the location, confidence, and category of the object.

2. The static plantar pressure image data processing method based on the improved YOLOV5 model according to claim 1, characterized in that: The step (3) of pre-processing the plantar pressure data specifically comprises the following steps: Step (3.1) Develop an automated image pixel normalization and noise filtering algorithm to standardize the collected pressure data and eliminate differences between individuals. The algorithm can scale and adjust the data according to the preset standard pressure range. Let the normalized data be P norm : Then Gaussian filtering is performed to remove noise from the data to ensure the accuracy and consistency of the data. The filtered data is P filtered : P filtered =P norm *G(x,y,σ) Among them, G(x,y,σ) is a Gaussian kernel with scale σ, and * represents the convolution operation.

3. The static plantar pressure image data processing method based on the improved YOLOV5 model according to claim 1, characterized in that: The step (6) of extracting fusion features specifically includes the following steps: The aligned plantar pressure features and plantar images are extracted based on the improved YOLOV5 model, and feature fusion is performed. Real-time recognition, for the plantar image I, the feature map F is extracted by the YOLOv5 model I , plantar pressure map F P : F I =YOLOv5(I) F P =YOLOv5(P) The foot sole image feature F I and plantar pressure characteristics F P Alignment, the aligned features are recorded as F' I and F' P : F′ I =SIFT(F I ); F′ P =SIFT(F P ); The aligned plantar image feature F' I and plantar pressure characteristics F' P Fusion is performed to obtain the fusion feature F F . Use the weighted summation method for fusion, with weights α and β respectively: F T =αF′ I +βF′ P ; α+β＝1； Based on the fusion feature F F Perform real-time recognition and use the classifier to obtain the recognition result 4. The static plantar pressure image data processing method based on the improved YOLOV5 model according to claim 1, characterized in that: The step (7) of high-precision image recognition specifically includes the following steps: Step (3.1) Compression operation: Compress the feature map output by the C3 block from the size of H×W×C to the feature map Z of 1×1×C through global average pooling. The formula is as follows: Step (3.2) Excitation operation: Channel recalibration is performed by learning the channel weight vector S. σ represents the sigmoid activation function, W1 and W2 are learnable parameter matrices, δ represents the ReLU activation function, and the excitation operation formula is as follows: S=σ(E1δE2(Z)) After processing by the SE module, channel attention increases the saliency of important features while suppressing irrelevant features. CBAM module (Convolutional Block Attention Module): Step (3.3) Channel attention branch: The size of the input feature map U is H×W×C. Through global maximum pooling and average pooling operations, U is converted into two one-dimensional descriptor vectors. These vectors are then processed and merged through a multi-layer perceptron, and finally a channel weight from 0 to 1 is generated through sigmoid activation. The learned channel weight coefficient is multiplied by the input feature map uX to obtain the channel-focused output feature map ZC, as follows: WITH C =σ(MLP(F max (at X ))+MLP(F avg (at X )))×u X ; Spatial attention branch: Through global maximum pooling and average pooling operations, the input feature map uX is compressed in the channel dimension to generate two two-dimensional feature maps. These feature maps are then concatenated on the channel to form a tensor with two channels, which is further convolved through a 7×7 convolution kernel to reduce it to one channel. The spatial attention coefficients are then generated through sigmoid activation. Finally, the input feature map uX is multiplied by these coefficients to obtain the spatially focused output feature map ZX, as follows: Z X =σ(Conv 7×7 (F max '(u X )),Conv 7×7 (F avg '(u X )))×u X By integrating attention mechanism modules such as SE and CBAM into the C3 module of the YOLOv5 model, the detection accuracy and generalization ability of the model are significantly improved; these improvements enable the model to better focus on information-rich areas and suppress irrelevant areas, thereby achieving more accurate static plantar pressure diagnosis; Through training and learning of a large number of plantar images, the improved YOLOV5 model can accurately identify key features such as the height and shape of the arch. These features include but are not limited to the contour, arch, and morphological changes of the arch. The model can accurately locate the position and shape of these features in the image through detailed pixel-level analysis.

5. The static plantar pressure image data processing method based on the improved YOLOV5 model according to claim 1, characterized in that: The step (8) of obtaining a confidence score based on the YOLO target detection algorithm specifically includes the following steps: In YOLOV3, the parameters of the center position are logistically regressed to limit them to the range of 0-1, which greatly improves the detection accuracy. In addition, multiple independent logistic regression classifiers are used to replace SoftMax to improve the accuracy of multi-target classification. Based on YOLOV3, YOLOV4 combines advanced technologies such as data enhancement, regularization and loss function improvement and enhancement of model sensitivity field, introduction of attention mechanism, feature integration and post-processing methods to further improve detection efficiency. The prior anchor frame of YOLOV5 is similar to that of YOLOV3 and YOLOV4; however, YOLOV5 incorporates the training prediction anchor frame into the network. During the training process, the optimal anchor frame values ​​of different training sets are adaptively calculated, making the model applicable to various data sets, greatly improving the positioning accuracy of different tasks and data sets; The YOLOv5 model has four variants, namely YOLOv5s, YOLOv5 m, YOLOv5 l and YOLOv5x. Among them, the model size of YOLOv5s is relatively small, so it is suitable for embedded applications. The model architecture can be divided into three parts: CSPDarkNet53 for feature extraction is an improved DarkNet53 network structure that uses the Cross Stage Partial (CSP) module for feature extraction. Let the input image be I and the output feature map be F, then: F=CSPDarkNet53(I) FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) are used to fuse and transmit feature maps. The feature map F after FPN and PAN modules fpn and F pan : F fpn =FPN(F); F pan =PAN(F); The category and position prediction modules are based on the feature map F fpn Perform target detection; Assume the predicted bounding box is (including center coordinates, width, height), category is By leveraging YOLOv5s, we aim to achieve real-time foot type detection on low-power devices; during inference, when there is an object in a given cell, the IoU between the bounding box and the ground-truth object can be calculated: in represents the intersection area of ​​the predicted bounding box and the true bounding box, Represents the union area of ​​the predicted bounding box and the true bounding box; The predicted category information and target box confidence Multiply them together to get the category confidence score of each target box

Images