Method for processing static plantar pressure image data based on improved yolov5 model

Abstract

Provided is a method for processing static plantar pressure image data based on an improved YOLOV5 model, which comprises the following steps: step (1) plantar pressure image acquisition; step (2) plantar image data acquisition; step (3) plantar pressure data preprocessing; step (4) plantar image data preprocessing; step (5) data alignment; step (6) extraction of fusion features; step (7) high-precision image recognition; and step (8) obtaining a confidence score on the basis of a YOLO target detection algorithm.

Description

Static plantar pressure image data processing method based on improved YOLOV5 model TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a static plantar pressure image data processing method based on an improved YOLOV5 model. BACKGROUND

[0002] The static plantar pressure measurement systems on the market at present are divided into two forms:

[0003] One is a large floor type device composed of a workstation and a force platform, which uses a high-density and high-precision pressure sensor matrix to provide a detailed plantar pressure distribution map, is equipped with advanced data analysis and visualization, and can provide a detailed pressure distribution map, center of gravity trajectory analysis, gait analysis, etc. The use scene is in a hospital department and is operated by a professional physician, and the patient needs to regularly go to the hospital for diagnosis. For example, the flat plate type plantar pressure test system of Jiangsu Dechang Medical Technology Co., Ltd.

[0004] For the large floor type device, the operation is complex, and professional personnel are needed for operation and maintenance (Note 1), and the visualization interface is not user-friendly, and the diagnosis cannot be automatically diagnosed, and needs to be operated by a professional physician for analysis and evaluation (Note 2) for diagnosis.

[0005] Although the diagnosis range is wide, it can include foot structure abnormalities such as flat feet, high arch feet, hallux valgus, and foot ulcer risk assessment, but because the range is wide, a traditional image processing algorithm is generally used, and the detection accuracy and speed for a certain foot abnormality cannot be improved.

[0006] The other is a small volume portable interval level device, which can provide a basic plantar pressure distribution map and simple data analysis function, is installed quickly, has a user-friendly operation interface, is easy for non-professionals to use, and the use scene is generally in a hospital department and is operated by a professional physician, and can also be extended to a family scene and is used by the patient himself. For example, the SENNOSCAN-21A foot pressure detector of Shenzhen Chuanggan Technology Co., Ltd. is a desktop foot pressure detector

[0007] For the small volume desktop level device, compared with the large system, the traditional foot image processing accuracy and resolution are obviously insufficient, some desktop level devices may lack sufficient clinical verification, and it is difficult to ensure the accuracy and reliability of the diagnosis results. Because the output of the foot pressure detector is the pressure of each region of the foot, the data processing capability is limited, and it may not be able to meet the professional medical needs, and the diagnosis and evaluation ability for complex conditions is limited. At the same time, the large system also has the disadvantage that it cannot realize automatic diagnosis, and needs to be judged by experienced staff or professional clinicians, or needs to be professionally trained for the patient, thereby causing limited use. SUMMARY

[0008] The purpose of the present application is to provide a static plantar pressure image data processing method based on an improved YOLOV5 model to solve the above problems.

[0009] The technical solution of the present application is as follows:

[0010] The static plantar pressure image data processing method based on the improved YOLOV5 model comprises the following steps:

[0011] Step (1) Plantar pressure image acquisition: Use a high-precision plantar pressure sensor array to obtain the pressure distribution data of the plantar surface in a static standing state; the sensor array covers the entire plantar surface area and can accurately measure the pressure value of each pressure point; realize real-time plantar pressure image acquisition, and the pressure data collected by the pressure sensor matrix is P, P=PressureSensorMatrix(t);

[0012] Step (2) Plantar image data acquisition: Use a high-resolution camera to shoot a plantar image to ensure the clarity and detail integrity of the image. The camera is installed in a fixed position to ensure the consistency of the shooting angle and distance; the image data collected by the high-resolution camera is I: I=HighResCamera(t);

[0013] Step (3) Plantar pressure data preprocessing: normalize the collected plantar pressure data to eliminate the pressure differences between different individuals. Use filtering technology to remove noise in the data to ensure the accuracy and consistency of the pressure data.

[0014] Step (4) Plantar image data preprocessing: crop, rotate and scale the plantar image to ensure that the foot area in the image is located at the center of the image and has consistent size, and apply image enhancement technology to improve the clarity and feature visibility of the image.

[0015] Step (5) Data alignment: align the plantar pressure data and the plantar image data using the SIFT image registration algorithm to ensure that the two types of data correspond to the same time point and foot area, and the alignment of the two is output on the same picture.

[0016] Step (6) Extract and fuse features: extract the aligned plantar pressure features and plantar image based on the improved YOLOV5 model and perform feature fusion;

[0017] Step (7) high-precision image recognition; the YOLOV5 model realizes high-precision recognition of the plantar image through deep learning technology; it is known for its efficient architecture for target detection through single forward transmission, and can process input images in a very short time and identify important features in the image with high precision; compared with traditional image processing methods, the YOLOV5 model can maintain excellent recognition performance under complex background and different lighting conditions;

[0018] Step (8) obtaining a confidence score based on the YOLO target detection algorithm: the YOLO series is a single-stage target detection algorithm, YOLOV1 divides the input image into uniform grids; the prediction of the target and the target bounding box in the grid is regarded as a regression problem; and the position, confidence and category of the object are obtained.

[0019] The step (3) plantar pressure data preprocessing specifically includes the following steps:

[0020] Step (3.1) develop an automatic image pixel normalization and noise filtering algorithm to standardize the collected pressure data and eliminate differences between different individuals. The algorithm can scale and adjust the data according to the preset standard pressure range, and the normalized data is P norm :

[0021] Then Gaussian filtering is performed to remove noise in the data and ensure the accuracy and consistency of the data, and the filtered data is P filtered : P filtered =P norm *G(x,y,σ)

[0022] Where G(x,y,σ) is a Gaussian kernel with scale σ, and * represents convolution operation.

[0023] The step (6) of extracting fused features specifically includes the following steps:

[0024] Align the plantar pressure features and plantar image based on the improved YOLOV5 model and perform feature fusion; real-time recognition, for the plantar image I, the feature map F I is extracted through the YOLOv5 model, the plantar pressure map F P : F I = YOLOv5(I) F P = YOLOv5(P)

[0025] Align the plantar image features F I and the plantar pressure features F P , and the aligned features are denoted as F′ I and F′ P : F′I = SIFT (F I ) ; F' P = SIFT (F P ) ;

[0026] The aligned foot bottom image features F' I and the foot bottom pressure features F' P are fused to obtain fused features F F . The fusion is performed by weighted summation method, and the weights are a and β respectively: F F = aF' I + βF' P ; a + β = 1;

[0027] Based on the fused features F F , real-time recognition is performed, and the recognition result

[0028] The step (7) high-precision image recognition specifically comprises the following steps:

[0029] Step (3.1) compression operation: the feature map output by the C3 block is compressed from the size of HxWxC to the size of 1x1xC through global average pooling, and the formula is as follows:

[0030] Step (3.2) excitation operation: channel recalibration is performed through learning channel weight vector S. σ represents the sigmoid activation function, W1 and W2 are learnable parameter matrices, and δ represents the ReLU activation function. The excitation operation formula is as follows: S = σ (E1δE2 (Z))

[0031] After processing by the SE module, the channel attention improves the saliency of important features while suppressing irrelevant features.

[0032] CBAM module (Convolutional Block Attention Module):

[0033] Step (3.3) channel attention branch:

[0034] The size of the input feature map U is HxWxC. Through global maximum pooling and average pooling operations, U is converted into two one-dimensional descriptor vectors. Then these vectors are processed by a multilayer perceptron respectively and combined, and finally a channel weight from 0 to 1 is generated through sigmoid activation. The learned channel weight coefficient is multiplied by the input feature map uX to obtain the output feature map ZC of channel attention, and the formula is as follows: Z C = σ (MLP (F max (u X))+MLP(F avg (u X )))×u X ;

[0035] Spatial attention branch:

[0036] The input feature map uX is compressed in the channel dimension by global max pooling and average pooling operations, generating two two-dimensional feature maps. These feature maps are then concatenated in the channel dimension to form a tensor with two channels, which is further convolved by a 7x7 convolution kernel to reduce to one channel. Then the sigmoid activation is used to generate spatial attention coefficients. Finally, the input feature map uX is multiplied by these coefficients to obtain the spatial attention output feature map ZX, as follows: Z X =σ(Conv 7×7 (F max ′(u X )),Conv 7×7 (F avg ′(u X )))×u X

[0037] By integrating attention mechanism modules such as SE and CBAM into the C3 modules of the YOLOv5 model, the detection accuracy and generalization ability of the model are significantly improved. These improvements enable the model to better focus on information-rich areas and suppress irrelevant areas, thereby achieving more accurate static plantar pressure diagnosis.

[0038] Through training and learning on a large number of plantar images, the improved YOLOV5 model can accurately identify key features such as arch height and morphology. These features include but are not limited to arch contour, arch degree, and morphological changes. The model can accurately locate the position and morphology of these features in the image through detailed analysis at the pixel level.

[0039] The step (8) based on the YOLO target detection algorithm to obtain the confidence score specifically includes the following steps:

[0040] The parameters of the center position in YOLOV3 are subjected to logistic regression, which is limited to the range of 0-1, thereby greatly improving the detection accuracy; in addition, multiple independent logistic regression classifiers are used to replace SoftMax to improve the accuracy of multi-target classification; based on YOLOV3, YOLOV4 combines advanced technologies such as data enhancement, regularization, and loss function improvement and enhanced model sensitivity field, introduction of attention mechanism, feature integration, and post-processing method to further improve detection efficiency; the prior anchor frame of YOLOV5 is similar to YOLOV3 and YOLOV4; however, YOLOV5 merges the training prediction anchor frame into the network; during the training process, the optimal anchor frame value of different training sets is adaptively calculated, making the model applicable to various data sets, greatly improving the positioning accuracy of different tasks and data sets;

[0041] The YOLOv5 model has four variants, namely YOLOv5s, YOLOv5 m, YOLOv5 1, and YOLOv5x; among them, the model size of YOLOv5s is relatively small, so it is suitable for embedded applications; the model architecture can be divided into three parts: CSPDarkNet53 for feature extraction, which is an improved DarkNet53 network structure that uses the Cross Stage Partial (CSP) module for feature extraction; let the input image be I and the output feature map be F, then:

[0042] FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) are used to fuse and transmit feature maps; after passing through the FPN and PAN modules, the feature map F fpn and F pan : F fpn = FPN(F); F pan =PAN(F);

[0043] The while-class and position prediction module is based on the feature map F fpn for target detection;

[0044] Assuming that the predicted bounding box is (including center coordinates, width, and height), and the class is

[0045] By using YOLOv5s, our goal is to achieve real-time foot type detection on low-power devices; during inference, when there is a target in the specified cell, the IoU between the bounding box and the real target can be calculated:

[0046] where represents the intersection region of the predicted bounding box and the real bounding box, represents the union region of the predicted bounding box and the real bounding box;

[0047] The predicted class information and the target frame confidence are multiplied to obtain the class confidence score of each target frame

[0048] The beneficial effects of the present application are as follows:

[0049] The traditional floor-standing device and desktop device have limited data processing capability, usually relying on basic image processing algorithms and low computing power, and the processed features such as image color saturation, simple pixel-level processing, are difficult to process large-scale high-resolution data. Through the K-means clustering algorithm, the present project obtains accurate prior boxes, divides the feature map into grids, and on this basis, the grid division and anchor box prediction method reduces false positives and misses, improves training efficiency and detection accuracy. At the same time, random affine transformation, mixed enhancement and HSV enhancement technology are adopted, which enriches the data set and can improve the robustness and generalization ability of the subsequent model.

[0050] Effect of high-precision image recognition technology of the system:

[0051] Compared with the traditional floor-standing device and desktop device, the static plantar pressure diagnosis system based on YOLOV5 model auxiliary diagnosis of flat foot can significantly improve the diagnosis effect of flat foot. As shown in FIG. 6, the improved YOLOV5 model with an attention mechanism module can further improve the diagnosis effect of flat foot, which is conducive to early diagnosis and intervention of patients and achieves good prognosis.

[0052] Comprehensive data analysis diagnosis and treatment effect:

[0053] Single data source (such as plantar pressure data or plantar image data) is difficult to fully reflect the health status of the plantar. The diagnosis method of single data source is easy to ignore some key features, which leads to the lack of accuracy and comprehensiveness of the diagnosis result. The fusion data comprehensive analysis can integrate the information of two data sources, as shown in FIG. 7, the model learns more features, and more comprehensively and accurately reflects the health status of the plantar, thereby improving the diagnosis effect.

[0054] The project system is based on advanced deep learning models and comprehensive data analysis methods, and the automated auxiliary diagnosis significantly improves the accuracy and efficiency of flat foot evaluation. Through a highly automated process, from data collection, preprocessing to diagnosis output, the whole process does not require human intervention, reducing human error and improving the reliability and consistency of diagnosis. At the same time, as shown in Figure 8, the friendly user interface and simplified operation steps make it easy for non-professional users to use, widely applicable to various scenarios such as families, clinics and hospitals, realizing the intelligentization and popularization of foot health management.

[0055] Advanced data preprocessing method optimizes image (acquisition of prior box + data enhancement)

[0056] Foot image and foot pressure data fusion alignment program: through image registration, using the SIFT image registration algorithm based on feature points, find the key points in the foot image, calculate the mapping relationship between the foot image and the pressure sensor array, align the key points on the image with the sensor data. In the data collection process, the timestamps of the foot pressure data and the foot image are recorded synchronously, and the data are aligned according to the timestamps to ensure that they correspond to the same time point.

[0057] YOLOV5 model system based on attention mechanism improvement: controlled by host button + screen display, patient self-control; After connecting the foot detector to the host, the patient's foot data is collected, input into the embedded system by the image acquisition card for diagnosis, and the results are visualized on the host screen.

[0058] The system realizes full automation of collecting, fusing, identifying and outputting the diagnosis of patients (including but not limited to diagnosis information, foot pressure distribution, foot pressure distribution ratio, foot pressure distribution ratio of each region, and foot image and identification box, etc.), which is convenient for patients, family members, medical staff to monitor, track and review analysis, and to judge whether the patient has flat feet or whether the intervention after diagnosis is effective. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a schematic diagram of data enhancement of the present application. We randomly change the hue, saturation and value of the image by 15%, 70% and 40% respectively. The horizontal with a probability of 50%, the mosaic enhancement with a probability of 100% and the mixed enhancement with a probability of 10% are used for the image.

[0060] Figure 2 is a schematic diagram of the YOLO-v5 and improved YOLO-v5 base module of the present application. In Figure 2: (A) YOLO-v5 core network. It is composed of two 6x6 convolution and three CSPDarkNet53 network modules. For each CSP module, the feature map size is reduced to half of the original size, and the feature channel is doubled. (B) Neck part of YOLO-v5. The classic feature pyramid structure is adopted, and all scale connections are fused from top to bottom to build a high-level semantic feature map. (C) Multi-scale prediction head of YOLO-v5. It predicts three prediction graphs with different aspect ratios. Each scale prediction graph will simultaneously predict three target anchor frames. The target anchor frame needs to predict four positions, the confidence of the anchor frame and the classification information. (D) SE attention structure. Through the squeeze global pooling operation, the spatial dimension is compressed into a feature map with a size of 1x1xC, and the global and weight information is extracted. (E) CBAM attention structure. It can not only independently learn the importance of each feature channel, but also increase the maximum scaling operation, which can simultaneously obtain important information of each feature channel and feature space.

[0061] Figure 3 is a schematic diagram of the features extracted by different model network layers of the present application. In Figure 3: a, b, c, d, e, f respectively represent the features extracted by different model network layers.

[0062] Figure 4 is a visual plantar pressure image of the present application. In Figure 4: (A) flat foot patient, (B) healthy person, (C) single flat foot patient.

[0063] Figure 5 is a whole automatic flowchart of the present application.

[0064] Figure 6 is a PR curve of three different types of model identification of the present application. In Figure 6: (A) ordinary YOLO-v5 model. (B) YOLO-v5 model with added SE attention mechanism. (C) YOLO-v5 model with added CBAM attention mechanism.

[0065] Figure 7 is a Grad-CAM heat map of the present application. In Figure 7: (A) The high activation area of the visualized prediction image pixel sensitivity is located in the midfoot region, (B) The high activation area of the visualized prediction image pixel sensitivity is located in the metatarsal region. (C) The high activation area of the visualized prediction image pixel sensitivity is located in the midfoot region, (D) The high activation area of the visualized prediction image pixel sensitivity is located in the metatarsal region.

[0066] Figure 8 is a user interface of the present application. In Figure 8: (A) Foot mapping plantar system. (B) Plantar pressure image. (C) Plantar pressure diagnosis output image, 1 is defined as flat foot, 0 is defined as healthy foot. (D) Image pressure distribution ratio. (E) Proportional distribution of pressure in each region of the plantar.

[0067] Fig. 9 is a plantar pressure image and labeling of the present application. In Fig. 9: (A) flat foot, (C) healthy foot, (E) single flat foot patient. The red dashed box represents the basic fact. The three pictures below are the visualization results of manual labeling by doctors using labelImg. (B) flat foot, (D) healthy foot, (F) patient has one flat foot. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0069] Example 1: Plantar pressure image acquisition and advanced data preprocessing method implementation

[0070] Raw plantar pressure images can be obtained through a static plantar pressure detection platform and an image acquisition device. Existing plantar pressure systems have some limitations in raw image data processing, such as poor generalization ability, information loss, and noise interference, which will affect the detection accuracy and robustness of the static plantar pressure diagnosis system based on the improved YOLOV5 model for auxiliary diagnosis of flat feet. In order to solve these problems, the present application integrates multiple data preprocessing methods to ensure data quality and improve the recognition accuracy and generalization ability of the subsequent diagnosis model by performing a series of preprocessing on the plantar images.

[0071] 1.2 Technical implementation scheme:

[0072] Acquisition of prior boxes:

[0073] Nine prior boxes are obtained through the K-means clustering algorithm, with sizes of 10x13, 16x30, 33x23, 30x61, 62x45, 59x119, 116x90, 156x198, and 373x326. We use the K-means clustering algorithm to obtain nine prior boxes. When the size is (w k ,h k ), each prior box can be represented as: B k =(w k ,h k )

[0074] where k∈{1,2,3,4,5,6,7,8,9}, w k and h k represent the width and height of the kth prior box, respectively.

[0075] Using these prior box partition features, when the input image size is (W, H), according to the size of the prior box, the feature map is divided into a grid with the same size as the prior box:

[0076] Target detection and box prediction:

[0077] When the target center is located in the grid, let the target box be (x, y, w, h), where x and y are the coordinates of the box center, and w and h are the width and height of the box. Then the specific grid unit is responsible for target detection. The model is based on the initial anchor box B k Output prediction box Calculate the difference and iteratively update the network parameters.

[0078] The loss function can be represented as: Loss total = Loss coord + Loss conf + Loss class

[0079] Where the coordinate loss is:

[0080] Confidence loss:

[0081] Classification loss:

[0082] Data augmentation techniques:

[0083] Random affine transformation: Randomly rotate, scale, translate and flip the image to increase the diversity of the data.

[0084] Mixed enhancement (Mosaic data enhancement): Use the mixed enhancement technique proposed in YOLOv4 and YOLOX to splice four images into one, enrich the background and target of the training data.

[0085] HSV enhancement: Adjust the hue, saturation and brightness of the image to enhance the color diversity of the image.

[0086] These data preprocessing methods can effectively improve the detection accuracy in the YOLOV5 framework and enhance the model generalization. After data preprocessing, as shown in FIG. 1.

[0087] Example 2: High-precision image recognition technology implementation

[0088] The YOLOV5 model is a superior model in the field of target detection, which realizes high-precision recognition of plantar images through deep learning technology. It is known for its efficient architecture that can achieve target detection in a single forward pass, capable of processing input images in a very short time and identifying important features in the image with high precision. Compared with traditional image processing methods, the YOLOV5 model can maintain excellent recognition performance in complex backgrounds and different lighting conditions.

[0089] 2.2 Technical implementation scheme:

[0090] In the application of plantar images, the introduction of the improved YOLOV5 model significantly improves the diagnostic ability of the plantar pressure system. The present application integrates attention mechanism modules into the YOLOv5 model, especially into the C3 module (the third residual block of the CSPDarknet53 backbone network), to enhance the feature representation capability. The specific implementation includes the following attention mechanism modules: Squeeze-and-Excitation Networks (SENet) and Convolutional Block Attention Module (CBAM). The SE module and the CBAM module are commonly used attention mechanism modules. SENet improves feature learning ability by modeling channel relationships, while CBAM focuses on information-rich areas and suppresses irrelevant areas by sequentially inferring spatial and channel attention. The simple and efficient design of SE, CBAM and other attention modules makes it easy to integrate into different levels of YOLOv5s, thereby significantly improving performance and achieving high-precision image recognition.

[0091] SE module (Squeeze-and-Excitation):

[0092] Compression operation: The feature map output by the C3 block is compressed from HxWxC size to 1x1xC feature map Z through global average pooling, as follows:

[0093] Excitation operation: Channel recalibration is performed through learning of channel weight vector S. σ represents the sigmoid activation function, W1 and W2 are learnable parameter matrices, and δ represents the ReLU activation function. The excitation operation formula is as follows: S = σ(E1δE2(Z))

[0094] After processing by the SE module, channel attention improves the saliency of important features while suppressing irrelevant features. CBAM module (Convolutional Block Attention Module):

[0095] Channel attention branch:

[0096] The size of the input feature map U is H x W x C. U is converted into two one-dimensional descriptor vectors through global max-pooling and average-pooling operations. These vectors are then processed by multi-layer perceptrons respectively and merged, and finally a channel weight from 0 to 1 is generated through sigmoid activation. The learned channel weight coefficient is multiplied by the input feature map uX to obtain the channel attention output feature map ZC, as follows: Z C = σ(MLP(F max (u X ))+MLP(F avg (u X ))) x u X

[0097] Spatial attention branch:

[0098] The input feature map uX is compressed in the channel dimension through global max-pooling and average-pooling operations to generate two two-dimensional feature maps. These feature maps are then concatenated in the channel to form a tensor with two channels, which is further convolved by a 7x7 convolution kernel to reduce to one channel. Then a spatial attention coefficient is generated through sigmoid activation. Finally, the input feature map uX is multiplied by these coefficients to obtain the spatial attention output feature map ZX, as follows: Z X = σ(Conv 7×7 (F max ′(u X )), Conv 7×7 (F avg ′(u X ))) x u X

[0099] As shown in FIG. 2, by integrating attention mechanism modules such as SE and CBAM into the C3 module of the YOLOv5 model, the present application significantly improves the detection accuracy and generalization ability of the model. These improvements enable the model to better focus on information-rich areas and suppress irrelevant areas, thereby achieving more accurate static plantar pressure diagnosis.

[0100] Through training and learning on a large number of plantar images, the improved YOLOV5 model can accurately identify key features such as arch height and morphology. As shown in FIG. 3, these features include but are not limited to arch contour, arch degree, and morphological changes, and the model can accurately locate the position and morphology of these features in the image through detailed analysis at the pixel level.

[0101] The overall process of the static plantar pressure diagnosis automation system assisted by the improved YOLOV5 model for flat foot diagnosis is shown in FIG. 5.

[0102] The YOLO series is a single-stage object detection algorithm. YOLO-v1 divides the input image into uniform-sized grids; predicts the object and the object bounding box in the grid as a regression problem; and obtains the object location, confidence, and class. However, YOLO-v1 poses challenges in accurately locating objects and has a low recall rate. Therefore, k-means clustering is added in YOLO-v2, and the prior anchor frame is used to constrain the object location and predict the confidence; this further improves the performance of YOLO-v1. However, in the object detection process, the regression parameters of the center position of the prior anchor frame in YOLO-v2 are not effectively constrained, which can cause the predicted anchor frame to appear at any position in the original image, resulting in low prediction accuracy. In addition, SoftMax activation classification is only suitable for a single target and is not suitable for multiple classifications.

[0103] Therefore, in YOLO-v3, the parameters of the center position are subjected to logistic regression, which limits them to the range of 0-1, thereby greatly improving the detection accuracy. In addition, multiple independent logistic regression classifiers are used to replace SoftMax to improve the accuracy of multi-target classification. Based on YOLO-v3, YOLO-v4 combines advanced techniques such as Bag-of-FreeBies (data augmentation, regularization, and loss function improvement) and Bag-of-Specials (enhanced model sensitivity field, introduction of attention mechanism, feature integration, and post-processing method) to further improve detection efficiency. The prior anchor frame of YOLO-v5 is similar to that of YOLO-v3 and YOLO-v4; however, YOLO-v5 merges the training prediction anchor frame into the network. During training, the optimal anchor frame value for different training sets is adaptively calculated, making the model suitable for various data sets and greatly improving the positioning accuracy for different tasks and data sets.

[0104] The YOLOv5 model has four variants, namely YOLOv5s, YOLOv5m, YOLOv51, and YOLOv5x; among them, the model size of YOLOv5s is relatively small, so it is suitable for embedded applications. The model architecture can be divided into three parts: CSPDarkNet53 for feature extraction, which is an improved DarkNet53 network structure using the Cross Stage Partial (CSP) module for feature extraction, where the input image is I and the output feature map is F, then: F = CSPDarkNet53(I)

[0105] FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) are used to fuse and transmit feature maps, and the feature maps F fpn and F pan : Ffpn = FPN(F) pan = PAN(F)

[0106] Meanwhile, the category and position prediction module is based on the feature map F fpn to perform target detection. Assuming the predicted bounding box is (including center coordinates, width, height), and the category is

[0107] By utilizing YOLOv5s, our goal is to achieve real-time foot type detection on low-power devices. During inference, when there is a target in the specified cell, the IoU between the bounding box and the real target can be calculated:

[0108] where represents the intersection area of the predicted bounding box and the real bounding box, represents the union area of the predicted bounding box and the real bounding box.

[0109] Multiply the predicted category information and the target frame confidence to get the category confidence score of each target frame

[0110] However, the learning ability of YOLOv5s is limited. Attention mechanisms enhance object detection models by improving their learning representation capabilities. Attention modules only require a small number of additional parameters, thereby enhancing functionality without significantly increasing model complexity.

[0111] Example 3: Comprehensive data analysis to improve diagnosis and treatment efficiency

[0112] Most existing auxiliary diagnosis systems rely solely on a single data source, such as plantar pressure data or plantar image data, making it difficult to fully reflect the plantar health status. Based on the improved YOLOV5 model, by integrating plantar pressure data and plantar image data, comprehensive analysis of the two types of data improves the response speed of the system, enabling more comprehensive and rapid diagnosis.

[0113] 3.2 Technical implementation scheme:

[0114] Plantar pressure data acquisition

[0115] A high-precision plantar pressure sensor array is used to obtain the pressure distribution data of the foot under static standing conditions. The sensor array covers the entire plantar area and can accurately measure the pressure values of each pressure point.

[0116] Plantar image data acquisition

[0117] High-resolution cameras are used to capture foot pressure data, ensuring the clarity and completeness of the data. The cameras are installed in fixed positions to ensure consistency in the angle and distance of the data.

[0118] Foot pressure data preprocessing

[0119] The collected foot pressure data is normalized to eliminate differences in pressure between different individuals. Filtering techniques are used to remove noise from the data, ensuring the accuracy and consistency of the pressure data.

[0120] Foot image data preprocessing

[0121] The foot image is cropped, rotated and scaled to ensure that the foot region in the image is centered and has consistent dimensions. Image enhancement techniques are applied to improve the clarity and feature visibility of the image.

[0122] Data alignment

[0123] The foot pressure data and foot image data are aligned using the SIFT image registration algorithm, as shown in Figure 4, to ensure that the two types of data correspond to the same time point and foot region. The alignment output of the two is on the same picture.

[0124] SIFT image registration algorithm

[0125] SIFT (Scale-Invariant Feature Transform) is an algorithm for extracting and describing image features, which has scale invariance and rotation invariance, and is widely used in image registration, object recognition and image stitching fields. The following is a detailed introduction to the SIFT algorithm and related formulas.

[0126] Main steps of SIFT algorithm:

[0127] (1) Scale space extreme value detection

[0128] (2) Key point positioning

[0129] (3) Direction assignment

[0130] (4) Key point description

[0131] (5) Feature point matching

[0132] Scale space extreme value detection

[0133] Objective: Find potential key points of the image in different scale spaces.

[0134] Method: Construct a Gaussian pyramid and apply Gaussian blur at different scales. Then, compute the Difference of Gaussian (DoG) pyramid in each scale to detect extreme points.

[0135] Equation:

[0136] Gaussian blur: L(x, y, σ) = G(x, y, σ) * I(x, y)

[0137] where G(x, y, σ) is the Gaussian kernel with scale σ, I(x, y) is the original image, and * denotes convolution operation.

[0138] Difference of Gaussians: D(x, y, σ) = L(x, y, kσ) - L(x, y, σ)

[0139] where k is a constant factor, usually k = √2.

[0140] Key point localization

[0141] Objective: Accurately locate the key points detected at different scales and remove low-contrast points and edge response points.

[0142] Method: Perform quadratic surface fitting for each extreme point to accurately determine the location and scale of the key point.

[0143] Equation:

[0144] Sub-pixel location of extreme points: where H is the Hessian matrix, is the gradient of the Gaussian difference function.

[0145] Hessian matrix and gradient:

[0146] Orientation assignment

[0147] Objective: Assign one or more orientations to each key point to achieve rotation invariance.

[0148] Method: Compute the gradient direction histogram of the key point neighborhood and select the maximum peak direction as the main direction of the key point.

[0149] Equation:

[0150] Gradient magnitude and direction: m(x, y) = √(L(x+1, y) - L(x-1, y)) 2 + (L(x, y+1) - L(x, y-1)) 2

[0151] Key point description

[0152] Goal: Generate a descriptor for each keypoint for feature matching.

[0153] Method: Compute gradient direction and magnitude in the neighborhood of the keypoint and generate a gradient direction histogram (usually a 4x4 grid with 8 directions per grid, resulting in a 128-dimensional descriptor).

[0154] Formula:

[0155] Descriptor generation: Descriptor = [H1, H2, …, H128]

[0156] where Hi is the count value of the gradient direction histogram.

[0157] Feature point matching

[0158] Goal: Find similar key points in two images.

[0159] Method: Compare descriptors using Euclidean distance or other metric methods to find the best matching keypoint pair.

[0160] Formula:

[0161] Euclidean distance:

[0162] where d1 and d2 are the descriptors of two key points.

[0163] Extraction of fusion features

[0164] By improving the YOLOV5 model, the aligned plantar pressure features and plantar images are extracted and the features are fused. Real-time recognition, for the plantar image I, the feature map F I is extracted by YOLOv5 model P F I = YOLOv5(I) F P = YOLOv5(P)

[0165] Align the plantar image features F I and the plantar pressure features F P , and the aligned features are F′ I and F′ P : F′ I = SIFT(F I ) F′ P = SIFT(F P )

[0166] Align the plantar image features F′ I and the plantar pressure features F′ PFusion is performed to obtain fusion features F F . Fusion is performed using weighted summation method, and the weights are a and b respectively: F F = aF' I + bF' P a + b = 1

[0167] Based on the fusion features F F Real-time recognition is performed, and the recognition result is obtained using the classifier Classifier

[0168] The model utilizes its efficient neural network architecture, learns the fused features, and only needs one forward pass to complete the detection task, greatly shortening the processing time, making real-time detection and diagnosis possible. This real-time performance is particularly important for clinical applications, as it can provide immediate diagnostic results during patient examination, improving diagnostic efficiency.

[0169] Example 4: Implementation of an automated auxiliary diagnosis system

[0170] 4.1 Brief description

[0171] Many systems require a lot of manual intervention, including data labeling and result analysis, which is inefficient and easily affected by human factors. This auxiliary diagnosis system realizes a highly automated diagnosis process, from data acquisition, preprocessing to diagnosis output, which is automatically completed by the system. It can improve the efficiency of diagnosis, reduce manual operation time and human error. User operation is simple, and non-professional users can easily use it.

[0172] 4.2 Technical implementation scheme:

[0173] Data acquisition automation

[0174] Through the foot pressure sensor matrix and high-resolution camera, real-time foot pressure image acquisition is realized. The pressure data collected by the pressure sensor matrix is P, and the image data collected by the high-resolution camera is I: P = PressureSensorMatrix(t) I = HighResCamera(t)

[0175] Automatic preprocessing of foot pressure image data

[0176] Develop automatic image pixel normalization and noise filtering algorithm, standardize the collected pressure data, eliminate the differences between different individuals. The algorithm can scale and adjust the data according to the preset standard pressure range, and the normalized data is P norm :

[0177] Then Gaussian filtering is performed to remove noise in the data, ensuring data accuracy and consistency. The filtered data is P filtered : P filtered = P norm * G(x, y, σ)

[0178] where G(x, y, σ) is a Gaussian kernel with scale σ, and * denotes convolution operation.

[0179] Automatic feature fusion and extraction

[0180] Train and deploy the improved YOLOv5 model to achieve automatic feature fusion and extraction. The model can automatically extract key features such as arch height and arch shape from the plantar image, and the prediction result is

[0181] Diagnosis and output automation

[0182] Using the fused feature data, train the improved YOLOV5 comprehensive diagnosis model. Realize automatic model training and use automatic grid search tuning technology to find the best model parameters, optimize the model, and the best parameters are θ * : θ * = argminθL(YOLOv5(F F ; θ), y)

[0183] where L is the loss function and y is the true label.

[0184] Real-time diagnosis

[0185] Deploy the real-time diagnosis system, input the real-time collected plantar pressure image data into the trained comprehensive diagnosis model, and perform real-time diagnosis. The system can automatically output the diagnosis results, including healthy feet, two feet flat, and single foot flat.

[0186] Automatic result display and report generation

[0187] Develop an automatic report generation module to generate detailed diagnosis reports, including pressure distribution maps and related diagnosis results and bounding boxes. The results are visualized on the user interface, showing the diagnosis results and suggestions. The interface should be simple and easy to use, so that non-professional users can easily operate and understand the diagnosis results.

[0188] Diagnosis criteria for flat feet

[0189] MLA collapse, physical examination and X-ray examination are used for the diagnosis of flat feet. The midpoint of the calcaneus, the midpoint of the medial malleolus and the midpoint of the lateral malleolus and the midpoint of the lower third of the calf are tracked and connected. The normal range of the angle obtained is -5° to +5°, and an angle less than -5° is a deformity of foot varus. In addition, a drop of 10 mm or more in the scaphoid bone will be interpreted as flat. Therefore, further X-ray examination is required. The X-ray film is a straight line from the lowest point of the calcaneus to the lowest point of the talus, and then a straight line from the lowest point of the talus to the lowest point of the first metatarsal head, and the normal range of the angle between the two lines is 113°-130°, and more than 130° will be diagnosed as flat feet.

[0190] During the development of the system, the training data set based on the improved YOLOV5 model strictly follows the inclusion and exclusion criteria for flat feet:

[0191] Inclusion criteria: (1) Age range: 5-60 years old; (2) Patients with normal gait and no motor system disease; (3) No history of foot trauma or surgery; (4) All foot arches are determined by X-ray examination.

[0192] Exclusion criteria: (1) obvious foot deformity; (2) presence of high arch; (3) history of lower extremity and foot fracture or ankle sprain leading to ligament injury or joint cartilage injury; (4) leg length difference >2 cm; (5) patients with cognitive dysfunction who cannot cooperate with the test; (6) lack of data.

[0193] For healthy subjects, the inclusion and exclusion criteria are consistent, except that they do not meet the diagnostic criteria for flat feet.

[0194] Training labeling of plantar pressure images

[0195] After the collected plantar pressure images were anonymized, two rehabilitation therapists with professional training labeled them; labelImg (1.8.6) was used to select the midfoot of the image for labeling classification (Figure 9). A text file containing the class, target center coordinates, target bounding box and aspect ratio information of the overall image corresponding to the original image was generated. The images were labeled as flat feet (level 0) or healthy feet (level 1), represented by values 0 and 1 respectively. In each image, the four corners of the middle foot region were selected as the center points, and the (x, y) coordinates were recorded in the txt file. The total data set was divided into training set and test set in the ratio of 8:2 using random seed 0.

[0196] Evaluation indicators of the model system

[0197] The system uses accuracy, precision, recall, F1 score, average precision (AP) and mean average precision (mAP) to evaluate the advantages of the model (formulas as follows):

[0198] True positives (TP) are correctly classified positive samples, false positives (FP) are misclassified positives, and false negatives (FN) are misclassified negative samples. The accuracy is the proportion of correctly classified samples. The precision is the ratio of TP to the total number of predicted positives. The recall is the ratio of TP to the total number of actual positives. The F1 score is the harmonic mean of precision and recall. AP and mAP are usually used for object detection to evaluate the detection effectiveness and performance of the model. AP is calculated by integrating the precision P at each discrete recall R point from 0 to 1 by calculating the area under the precision-recall curve. The higher the AP value, the better the detection performance of a certain class. Let C represent the total number of classes in the detection model, c represent each class, and AP(c) represent the AP of each class c. The calculation method of mAP is to first calculate the AP of each class, and then average the AP of all classes, thereby obtaining a comprehensive evaluation index reflecting the detection effect of the model on all classes.

[0199] It is obvious to a person skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended to encompass all variations falling within the meaning and range of the essential elements of the claims. Any reference signs in the claims should not be considered as limiting the claims to which they relate.

[0200] Furthermore, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the present specification is described in this way only for the sake of clarity, and a person skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that a person skilled in the art can understand.

Claims

1. A static plantar pressure image data processing method based on an improved YOLOV5 model, characterized in that: Comprising the following steps: Step (1) Plantar pressure image acquisition: using high-precision plantar pressure sensor array, obtaining the pressure distribution data of the plantar in the static standing state; the sensor array covers the entire plantar area, which can accurately measure the pressure value of each pressure point; realizing the collection of real-time plantar pressure image, the pressure data collected by the pressure sensor matrix is P, P = PressureSensorMatrix(t); Step (2) Plantar image data acquisition: using high-resolution camera to shoot plantar image, ensuring the definition and detail integrity of the image. The camera is installed in a fixed position to ensure the consistency of shooting angle and distance; the image data collected by the high-resolution camera is I: I = HighResCamera(t); Step (3) Plantar pressure data preprocessing: normalizing the collected plantar pressure data to eliminate the pressure difference between different individuals. Using filtering technology to remove noise in the data, ensuring the accuracy and consistency of the pressure data. Step (4) Plantar image data preprocessing: cropping, rotating and scaling the plantar image to ensure that the foot area in the image is located at the center of the image and has consistent size, applying image enhancement technology to improve the definition and feature visibility of the image. Step (5) Data alignment: aligning the plantar pressure data and the plantar image data using SIFT image registration algorithm to ensure that the two types of data correspond to the same time point and foot area, and the alignment output of the two is on the same picture; Step (6) Extracting and fusing features: extracting the aligned plantar pressure features and plantar image based on the improved YOLOV5 model and fusing the features; Step (7) High-precision image recognition: YOLOV5 model realizes high-precision recognition of plantar image through deep learning technology; it is known for its efficient architecture for target detection through single forward transmission, which can process input images in a very short time and identify important features in the image with high precision; Compared with traditional image processing methods, YOLOV5 model can maintain excellent recognition performance in complex background and different lighting conditions; Step (8) Obtain confidence score based on YOLO target detection algorithm: YOLO series is a single-stage target detection algorithm, YOLOV1 divides the input image into uniform grids; the prediction of the target and the target bounding box in the grid is regarded as a regression problem; and the position, confidence and category of the object are obtained.

2. The static plantar pressure image data processing method based on the improved YOLOV5 model according to claim 1, characterized in that: The step (3) plantar pressure data preprocessing specifically comprises the following steps: Step (3.1) develops an automatic image pixel normalization and noise filtering algorithm to standardize the collected pressure data and eliminate differences between different individuals. The algorithm can scale and adjust the data according to the preset standard pressure range, and set the normalized data as P norm : Then Gaussian filtering is performed to remove noise in the data, ensure the accuracy and consistency of the data, and the filtered data is P filtered : P filtered = P norm *G(x,y,σ) Wherein, G(x,y,σ) is a Gaussian kernel with scale σ, * represents convolution operation.

3. The static plantar pressure image data processing method based on the improved YOLOV5 model according to claim 1, characterized in that: The step (6) extracting and fusing features specifically comprises the following steps: Extracting the aligned plantar pressure features and plantar image based on the improved YOLOV5 model and fusing the features; Real-time recognition, for the plantar image I, extracting a feature map F by a YOLOv5 model I , a plantar pressure map F P : F I = YOLOv5(I) F P = YOLOv5(P) aligning the plantar image features F I and the plantar pressure features F P , the aligned features are denoted as F' I and F' P , respectively. F′ I = SIFT(F I ); F′ P = SIFT(F P ); The aligned foot bottom image features F' I and the foot bottom pressure features F' P are fused to obtain fused features F F . The fusion is performed using a weighted sum method, and the weights are α and β, respectively: F F = aF' + βF' I + βF' P ; α+β＝1； based on the fusion feature F F Real-time recognition is performed, and a recognition result is obtained using a classifier Classifier 4. The static plantar pressure image data processing method based on the improved YOLOV5 model according to claim 1, characterized in that: The step (7) high-precision image recognition specifically comprises the following steps: Step (3.1) compression operation: the feature map output by the C3 block is compressed from the size of HxWxC to a feature map Z of 1x1xC by global average pooling, as follows: Step (3.2) Excitation operation: Channel recalibration is performed by learning the channel weight vector S. σ represents the sigmoid activation function, W1 and W2 are learnable parameter matrices, and δ represents the ReLU activation function. The excitation operation formula is as follows: S = σ (E1δE2 (Z)) After processing by the SE module, the channel attention improves the saliency of important features while suppressing irrelevant features. CBAM module (Convolutional Block Attention Module): Step (3.3) Channel attention branch: The size of the input feature map U is HxWxC. Through global max pooling and average pooling operations, U is converted into two one-dimensional descriptor vectors. Then these vectors are processed by multi-layer perceptron respectively and merged, and finally the channel weight is generated from 0 to 1 through sigmoid activation. The learned channel weight coefficient is multiplied by the input feature map uX to obtain the output feature map ZC of channel attention, and the formula is as follows: Z C = σ(MLP(F max (u X ))+ MLP(F avg (u X ))) x u X ; Spatial attention branch: Through global max pooling and average pooling operations, the input feature map uX is compressed in the channel dimension to generate two two-dimensional feature maps. Then these feature maps are concatenated in the channel to form a tensor with two channels, which is further convolved by a 7x7 convolution kernel to reduce to one channel. Then the spatial attention coefficient is generated through sigmoid activation. Finally, the input feature map uX is multiplied by these coefficients to obtain the output feature map ZX of spatial attention, and the formula is as follows: Z X = σ(Conv 7×7 (F max ′(u X )),Conv 7×7 (F avg ′(u X ))) × u X By integrating attention mechanism modules such as SE and CBAM into the C3 module of the YOLOv5 model, the detection accuracy and generalization ability of the model are significantly improved; these improvements enable the model to better focus on information-rich areas and suppress irrelevant areas, thereby achieving more accurate static plantar pressure diagnosis; Through training and learning on a large number of plantar images, the improved YOLOV5 model can accurately identify key features such as arch height and shape, including but not limited to arch contour, arch degree, and shape changes. The model can accurately locate the position and shape of these features in the image through detailed analysis at the pixel level.

5. The static plantar pressure image data processing method based on the improved YOLOV5 model according to claim 1, characterized in that: The step (8) based on the YOLO target detection algorithm to obtain the confidence score specifically includes the following steps: In YOLOV3, the parameters of the center position are subjected to logistic regression, which is limited to the range of 0-1, thereby greatly improving the detection accuracy; in addition, multiple independent logistic regression classifiers are used instead of SoftMax to improve the accuracy of multi-target classification; based on YOLOV3, YOLOV4 combines advanced technologies such as data augmentation, regularization, and loss function improvement and enhanced model sensitivity field, introduction of attention mechanism, feature integration, and post-processing method to further improve detection efficiency; the prior anchor frame of YOLOV5 is similar to YOLOV3 and YOLOV4; however, YOLOV5 merges the training prediction anchor frame into the network; In the training process, the optimal anchor frame value of different training sets is calculated adaptively, so that the model is suitable for various data sets, and the positioning accuracy of different tasks and data sets is greatly improved; YOLOv5 model has four variants, namely YOLOv5s, YOLOv5 m, YOLOv5 1 and YOLOv5x; Among them, the model size of YOLOv5s is relatively small, so it is suitable for embedded applications; The model architecture can be divided into three parts: CSPDarkNet53 for feature extraction, which is an improved DarkNet53 network structure, which uses Cross Stage Partial (CSP) module for feature extraction, and sets the input image as I and the output feature map as F, then: F = CSPDarkNet53 (I) FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) are used to fuse and transmit feature maps. The feature maps F fpn and F pan : F fpn = FPN(F); F pan = PAN(F); The while class and position prediction module is based on the feature map F fpn Performs object detection; Assume the predicted bounding box is (including center coordinates, width, height), category is By utilizing YOLOv5s, our goal is to achieve real-time foot type detection on low-power devices; during inference, when there is a target in the specified cell, the IoU between the bounding box and the real target can be calculated: wherein representing an intersection region of the predicted bounding box and the real bounding box, Indicates the union area of the predicted bounding box and the real bounding box; predicting the class information and target box confidence Multiplying, to obtain the class confidence score of each target frame

Images