A method and system for identifying scoliosis of gait

Abstract

The present invention discloses a method and system for identifying scoliosis for gait, and relates to the technical field of computer vision, including: step 1: collecting picture data sets; step 2: constructing a CNN network model introducing an attention mechanism; the CNN network model includes a multi-scale feature extraction module, a time pooling module, an aggregation module, a space pooling module, a fully connected layer and an output layer connected in sequence; step 3: using the picture data set to train the CNN network model to obtain a gait recognition model; step 4: collecting video files, extracting a number of silhouette pictures from the video files, and processing all the silhouette pictures into pictures centered on portraits, converting the processed pictures into Numpy arrays, and constructing gait data to be identified; step 5: inputting the gait data to be identified into the gait recognition model to obtain a scoliosis recognition result. The present invention uses the gait posture of walking in normal times to determine whether there is scoliosis, thereby improving recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly to a method and system for identifying scoliosis of the gait. Background Art

[0002] Currently, the screening and diagnosis methods for scoliosis mainly rely on physical examinations and X-ray imaging. Physical examinations usually include observing the symmetry of the shoulders, scapulae, waist, and the curve of the spine. A simple bending and flexion test can be used to preliminarily determine whether scoliosis exists, but this type of examination method has low accuracy and relies on the doctor's experience. It cannot accurately quantify the degree of curvature of the spine, and this type of contact examination also has the risk of privacy leakage. X-ray imaging can accurately measure the curvature angle of the spine (Cobb angle), but its radiation poses potential risks to health. In addition, X-ray imaging usually requires patients to go to the hospital, which is costly and cumbersome, affecting the popularity of early screening.

[0003] In recent years, with the development of computer vision, machine learning, deep learning and other technologies, image-based automatic scoliosis recognition methods have gradually attracted attention. Such methods usually use non-invasive means (such as infrared, 3D scanning, optical images, etc.) to obtain spinal surface morphology information, and combine artificial intelligence algorithms to automatically analyze the scoliosis. These systems and methods can quickly and inexpensively screen for scoliosis by constructing a three-dimensional model of the patient's spine or directly analyzing the image. Compared with traditional methods, recognition methods based on computer vision and machine learning have the advantages of no radiation, convenience, and high repeatability, and can be used for daily initial screening and follow-up observations. However, current image recognition of scoliosis usually uses standing static images to identify the position of human body parts, thereby automatically identifying the scoliosis situation. When the degree of scoliosis is low, people may be able to adjust their posture while standing, resulting in inaccurate recognition.

[0004] Therefore, how to improve the accuracy of scoliosis identification while protecting patient privacy is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a method and system for identifying scoliosis in gait, which is based on computer vision recognition technology and uses the gait posture of daily walking to determine whether scoliosis exists, thereby improving recognition accuracy and solving the problems of high cost, cumbersome process and possible privacy leakage in checking scoliosis in adolescents.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for identifying scoliosis in gait comprises the following steps:

[0008] Step 1: Collect image dataset;

[0009] Step 2: Construct a CNN network model with an attention mechanism. The CNN network model includes a multi-scale feature extraction module with an attention mechanism, a temporal pooling module, an aggregation module, a spatial pooling module, a fully connected layer, and an output layer.

[0010] Step 3: Use the image dataset to train the CNN network model to obtain a gait recognition model;

[0011] Step 4: Collect video files, extract several silhouette images from the video files, and process all silhouette images into images centered on the portrait. Convert the processed images into Numpy arrays to construct the gait data to be identified.

[0012] Step 5: Input the gait data to be recognized into the gait recognition model to obtain the scoliosis recognition result.

[0013] Preferably, the multi-scale feature extraction module includes a convolution layer and four improved basic block layers connected in sequence, each improved basic block layer outputs a feature of one scale, and outputs four features of different scales f1, f2, f3 and f4 in sequence, and the scale feature output by the previous improved basic block layer is used as the input of the next improved basic block layer; the improved basic block layer includes a first convolution layer, a second convolution layer, an MCA attention layer and a residual connection layer connected in sequence, and the residual connection layer performs a residual connection on the output of the first convolution layer and the output of the MCA attention layer, and outputs the feature f x , x=1,2,3,4.

[0014] Preferably, the improved basic block layer adopts the BasicBlock of the residual network Resnet as the basis, and adds an optional MCA attention mechanism.

[0015] Preferably, the MCA attention layer performs feature rearrangement and fusion processing on the C×H×W feature map output by the second convolutional layer, where C represents the number of channels, H represents the height, and W represents the width. The specific steps include:

[0016] Step 311: Rearrange the C×H×W feature map F into a H×C×W feature map F W , H×1×1 features are obtained through average pooling and standard pooling, and the H×1×1 features are rearranged into 1×1×H features. Then 1×K Conv2d convolutions are used to extract 1×1×H features, which are then rearranged into H×1×1 features. After Sigmoid activation function, they are combined with H×C×W feature map F W Multiply them to obtain the rearranged feature Y1, the shape of Y1 is C×H×W;

[0017] Step 312: Rearrange the C×H×W feature map X into a W×H×C feature map F H , get W×1×1 features through average pooling and standard pooling, rearrange the W×1×1 features into 1×1×W features, extract 1×1×W features through 1×K Conv2d convolutions, and then rearrange them into W×1×1 features, pass the Sigmoid activation function and combine them with the W×H×C feature map F H Multiply them to obtain the rearranged feature Y2, the shape of Y2 is C×H×W;

[0018] Step 313: Use the C×H×W feature map F as the C×H×W feature map F C , C×1×1 features are obtained through average pooling and standard pooling, and the C×1×1 features are rearranged into 1×1×C features. Then 1×K Conv2d convolutions are used to extract 1×1×C features, which are then rearranged into C×1×1 features. After Sigmoid activation function, they are combined with C×H×W feature map F C Multiply them to obtain the rearranged feature Y3, the shape of Y3 is C×H×W;

[0019] Step 314: Add the rearranged feature Y1, the rearranged feature Y2, and the rearranged feature Y3 and take the average value to obtain the attention feature Y.

[0020] Preferably, the shape of the four features f1, f2, f3, and f4 output by the multi-scale feature extraction module is n×C×H×W, where n represents the gait frame, C represents the number of channels, H represents the height, and W represents the width; the time pooling module includes four time pooling layers, and the four scale features f1, f2, f3, and f4 are respectively input into a corresponding time pooling layer for feature concentration, and four basic features z1, z2, z3, and z4 are obtained respectively. The expression for feature concentration of the time pooling module is:

[0021]

[0022] Among them, z represents the basic features, f represents the features of different scales, and TP() represents the time pooling operation.

[0023] Preferably, the aggregation module includes three groups of fusion blocks connected in sequence, each group of fusion blocks includes a basic block layer (BasicBlock) without an MCA attention layer and a feature fusion layer; the basic block layer includes a first convolution layer, a second convolution layer and a residual connection layer connected in sequence, and the residual connection layer performs a residual connection on the output of the first convolution layer and the output of the second convolution layer; the process of feature fusion of the aggregation module is:

[0024] Step 321: Take the basic feature z2 as the first feature F raw, the basic feature z1 is input into the basic block layer of the first group of fusion blocks, and the second feature F is extracted 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the first group of fusion blocks to obtain the first fusion feature ;

[0025] Step 322: Take the basic feature z3 as the first feature F raw , the first fusion feature The features extracted after inputting into the basic block layer of the second group of fusion blocks are used as the second features F 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the second group of fusion blocks to obtain the second fusion feature ;

[0026] Step 323: Take the basic feature z4 as the first feature F raw , the second fusion feature The features extracted after inputting into the basic block layer of the third group of fusion blocks are used as the second feature F 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the third group of fusion blocks to obtain the aggregated features .

[0027] Preferably, the expression for feature fusion performed by the aggregation module is:

[0028]

[0029] Among them, F() represents the BasicBlock operation, and g[] represents the feature fusion operation.

[0030] Preferably, the feature fusion layer includes a feature splicing layer, an information channel aggregation layer, a feature segmentation layer, a GELU nonlinear activation layer and a fusion layer connected in sequence; the process of the feature fusion layer performing feature fusion processing in step 321 includes:

[0031] Step 3211: The first feature F raw and the second feature F 2nd Input to the feature merging layer, the first feature F with size C×H×W raw and the second feature F with dimensions C×H×W 2nd Combined into 2C×H×W mixed features along the channel dimension ;

[0032] Step 3212: Mix the features Input to the information channel aggregation layer, and then pass through a point-by-point convolution and a depth-wise convolution to obtain the aggregated mixed features. ;

[0033] Step 3213: Mixed features after aggregation Input to the feature segmentation layer, and aggregate the mixed features Split into the first mixed feature along the channel dimension and the second mixed feature , the first mixed feature Input to the GELU nonlinear activation layer;

[0034] Step 3214: GELU nonlinear activation layer activates the first mixed feature ;

[0035] Step 3215: Activate the first blend feature With the second mixed feature Perform element-by-element multiplication in the fusion layer to obtain the multiplied features, then perform point-by-point convolution on the multiplied features to obtain the fused features, and then add the fused features to the first feature F initially input. raw After element-by-element addition, point-by-point convolution is performed to obtain the first fusion feature. .

[0036] Preferably, the expression for feature fusion performed by the feature fusion layer is:

[0037]

[0038] in, The first mixed feature representing the mixed feature after the l-th layer aggregation, The second mixed feature represents the mixed feature after the l-th layer aggregation, l=1,2,…,L, L represents the number of feature layers; Represents the mixed features of the lth layer; DConv3 represents the depth convolution with a kernel size of 3×3; PConv represents the point-by-point convolution with a kernel size of 3×3; represents Hadamard product; GELU represents GELU nonlinear activation; Represents the first feature of the lth layer.

[0039] Preferably, the spatial pooling module performs horizontal pooling on the aggregated features to obtain comprehensive features , expressed as:

[0040]

[0041] Among them, maxpool() represents the maximum pooling operation; avgpool() represents the average pooling operation.

[0042] Preferably, the fully connected layer maps the comprehensive features to the metric space; the output layer includes a BN neck layer (BNNeck layer) and a classification layer.

[0043] Preferably, in the process of CNN network model training, a hybrid loss is constructed by calculating triplet loss and cross entropy loss, and the model is optimized according to the hybrid loss; wherein, the comprehensive features output by the spatial pooling module are mapped in the fully connected layer, and the triplet loss is calculated. It is expressed as:

[0044]

[0045] in, Represents the anchor point sequence, i.e., the target sample; Represents a sequence of positive samples belonging to the same category as the anchor point; Represents a sequence of negative samples that belong to different categories from the anchor point; N represents the number of samples in a training batch; represents the embedding function that converts each sequence into an embedding space, realizing the mapping of comprehensive features to the metric space in the fully connected layer; || ||2 represents the Euclidean norm; Represents the distance between the anchor sequence and the positive sample sequence. Since they are of the same category, the distance should be as small as possible; Represents the distance between the anchor sequence and the negative sample sequence. Due to different categories, this distance is greater than the distance of the positive sample sequence and is at least one more boundary value than the distance of the positive sample sequence. ; Represents the boundary used to establish separation between positive and negative matches; the triplet loss enhances the model's ability to effectively distinguish between classes; the above triplet loss can enhance the model's ability to distinguish different categories;

[0046] The features mapped to high-dimensional space by the fully connected layer are sent to the BNNeck layer of the output layer for normalization. The normalized features are output through the classification layer of the output layer to predict the scoliosis recognition results and calculate the cross entropy loss. , the BNNeck layer refines the feature space before the final classification stage. The classification layer uses the softmax function to implement feature classification and uses the cross entropy loss to measure the fidelity between the predicted label and the actual label, as follows:

[0047]

[0048] Where N represents the number of samples in a training batch; represents the scoliosis recognition result predicted by the model; Represents the real scoliosis recognition results annotated in the image dataset;

[0049] The mixed loss is It is expressed as: .

[0050] Preferably, the specific process of collecting and processing the video file in step 4 to obtain the gait data to be identified includes:

[0051] Step 41: Convert the video file into a silhouette file, and extract a preset number of frame images through a Python algorithm;

[0052] Step 42: using the PP-HumanSeg method to obtain a human outline from the frame image, converting the image into a grayscale image, taking the human outline as the center, cutting out the black part of the human outline in the grayscale image, and filling the blank part outside the human outline with black to obtain a processed image;

[0053] Step 43: Save the processed images corresponding to all frame images in pkl format, convert them into Numpy arrays, and form the gait data to be identified.

[0054] Preferably, the distance between the video acquisition device and the person is between 1.4 meters and 4.2 meters, so that the person is located in the center of the video as much as possible, and a video file is generated; the video file is framed into 720p through the Python algorithm, and 300 frame images are fixedly extracted according to the length of the video, and the size of the input file is compressed in this way; the image size after processing is 64×64; the pkl format file can greatly increase the calculation speed of the model.

[0055] Preferably, the scoliosis results include positive, neutral and negative. Positive, neutral and negative are divided according to the degree of scoliosis.

[0056] A scoliosis recognition system for gait, comprising an image acquisition module, a model training module, a video processing module and a recognition module;

[0057] The image acquisition module collects picture data sets and video files, and sends the picture data sets to the model training module and sends the video files to the video processing module;

[0058] The model training module builds a CNN network model with an attention mechanism, and uses a picture training set to train the CNN network model to obtain a gait recognition model that is sent to the recognition module.

[0059] The video processing module extracts several silhouette images from the video file, processes all the silhouette images into images centered on the portrait, converts the processed images into Numpy arrays, constructs the gait data to be recognized, and sends them to the recognition module;

[0060] The recognition module inputs the gait data to be recognized into the gait recognition model to obtain the scoliosis recognition result.

[0061] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for identifying scoliosis in gait, which is an emerging non-invasive detection method that determines the abnormality of the spine by analyzing the individual's walking posture and movement characteristics. The present invention designs a new recognition model by setting a multi-scale feature extraction module that introduces an attention mechanism, thereby improving the accuracy and versatility of scoliosis recognition in gait, thereby solving the problem of high cost and cumbersome process of checking scoliosis in adolescents. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0063] Figure 1 A flow chart of a method for identifying scoliosis in gait provided by the present invention;

[0064] Figure 2 A schematic diagram of the image processing process of the CNN network model provided by the present invention;

[0065] Figure 3 A schematic diagram of the feature processing process of the MCA attention layer provided by the present invention;

[0066] Figure 4 A schematic diagram of the feature fusion processing process of the feature fusion layer provided by the present invention;

[0067] Figure 5 A schematic diagram of the feature processing process of the improved basic block layer provided by the present invention;

[0068] Figure 6 A schematic diagram of the feature processing process of the basic block layer provided by the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The embodiment of the present invention discloses a method for identifying scoliosis of the gait, such as Figure 1 As shown, the following steps are included:

[0071] S1: collect image dataset;

[0072] S2: Construct a CNN network model with an attention mechanism. The CNN network model includes a multi-scale feature extraction module with an attention mechanism, a temporal pooling module, an aggregation module, a spatial pooling module, a fully connected layer, and an output layer.

[0073] S3: Use the image dataset to train the CNN network model to obtain the gait recognition model;

[0074] S4: Collect video files, extract several silhouette images from the video files, process all the silhouette images into images centered on the portrait, convert the processed images into Numpy arrays, and construct the gait data to be identified;

[0075] S5: Input the gait data to be recognized into the gait recognition model to obtain the scoliosis recognition result.

[0076] In a specific embodiment, the method for identifying scoliosis in gait is specifically implemented as follows:

[0077] S1: Collect image datasets; the collected data for gait recognition of scoliosis comes from the public dataset Scoliosis1K; preprocess the image dataset, select training videos, and select 30 frames of images from 300 frames extracted from the training videos for training. The data is reshaped, and the black parts on both sides of the image data with a height and width of 64×64 are adaptively removed, and it is reshaped into an image with a height and width of 64×44;

[0078] S2: Send the training set to the constructed CNN network model; the model structure is as follows Figure 2 As shown;

[0079] S21: The multi-scale feature extraction module of the CNN network model can be divided into five layers. The first layer is a convolution transformation, which transforms the channels of the input image from 1 to 64, followed by regularization and activation function. The activation function uniformly uses ReLu, and then passes through four improved basic block layers (using the improved Resnet's BasicBlock layer). Each improved basic block layer (improved BasicBlock layer) is equipped with an optional MCA attention layer. The output features of each improved basic block layer are saved to obtain four scale features f1, f2, f3 and f4; in the improved basic block layer, an MCA attention mechanism is added to the output of the second convolution layer, and a residual connection is performed to enhance the feature extraction capability of the module; the improved basic block layer includes the first convolution layer, the second convolution layer, the MCA attention layer and the residual connection layer connected in sequence. The residual connection layer performs a residual connection on the output of the first convolution layer and the output of the MCA attention layer, and outputs the feature f x, x=1,2,3,4, such as Figure 5 As shown;

[0080] The MCA attention layer performs feature rearrangement and fusion processing on the C×H×W feature map output by the second convolutional layer, where C represents the number of channels, H represents the height, and W represents the width. Figure 3 As shown, the specific steps include:

[0081] S211: C×H×W feature map F, Rearrange into H×C×W feature map F W , H×1×1 features are obtained through average pooling and standard pooling, and the H×1×1 features are rearranged into 1×1×H features. Then 1×K Conv2d convolutions are used to extract 1×1×H features, which are then rearranged into H×1×1 features. After Sigmoid activation function, they are combined with H×C×W feature map F W Multiply them to obtain the rearranged feature Y1, the shape of Y1 is C×H×W;

[0082] S212: Rearrange the C×H×W feature map X into a W×H×C feature map F H , get W×1×1 features through average pooling and standard pooling, rearrange the W×1×1 features into 1×1×W features, extract 1×1×W features through 1×K Conv2d convolutions, and then rearrange them into W×1×1 features, pass the Sigmoid activation function and combine them with the W×H×C feature map F H Multiply them to obtain the rearranged feature Y2, the shape of Y2 is C×H×W;

[0083] S213: Taking the C×H×W feature map F as the C×H×W feature map F C , C×1×1 features are obtained through average pooling and standard pooling, and the C×1×1 features are rearranged into 1×1×C features. Then 1×K Conv2d convolutions are used to extract 1×1×C features, which are then rearranged into C×1×1 features. After Sigmoid activation function, they are combined with C×H×W feature map F C Multiply them to obtain the rearranged feature Y3, the shape of Y3 is C×H×W;

[0084] S214: Add the rearranged feature Y1, the rearranged feature Y2, and the rearranged feature Y3 and take the average value to obtain the attention feature Y;

[0085] S22: The four extracted scale features f1, f2, f3, and f4 of shape n×C×H×W are sent to the temporal pooling module, where n, C, H, and W represent the gait frame, channel, height, and width dimensions, respectively; the processing of the temporal pooling module is as follows:

[0086]

[0087] Temporal pooling condenses the features into basic features z, and the four scale features f1, f2, f3, and f4 obtain four corresponding basic features z1, z2, z3, and z4;

[0088] S23: Input the basic features z1, z2, z3, and z4 into the aggregation module for fusion to obtain the aggregated features , the expression is:

[0089]

[0090] Among them, F represents the operation of improving the BasicBlock layer in S21 by removing the MCA attention layer; [] represents the feature fusion operation of the feature fusion module GFM;

[0091] The aggregation module includes three groups of fusion blocks connected in sequence. Each group of fusion blocks includes a basic block layer (BasicBlock layer) without an MCA attention layer and a feature fusion layer. The basic block layer includes a first convolutional layer, a second convolutional layer, and a residual connection layer connected in sequence. The residual connection layer performs a residual connection on the output of the first convolutional layer and the output of the second convolutional layer, such as Figure 6 As shown in the figure, the process of feature fusion by the aggregation module is as follows:

[0092] S231: Take the basic feature z2 as the first feature F raw , the basic feature z1 is input into the basic block layer of the first group of fusion blocks, and the second feature F is extracted 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the first group of fusion blocks to obtain the first fusion feature ;

[0093] The feature fusion layer includes a feature splicing layer, an information channel aggregation layer, a feature segmentation layer, a GELU nonlinear activation layer and a fusion layer connected in sequence; the process of the feature fusion layer performing feature fusion processing in step 321 is as follows: Figure 4 As shown, including:

[0094] S2311: Combine the first feature Fraw and the second feature F 2nd Input to the feature merging layer, the first feature Fraw with size C×H×W and the second feature F with size C×H×W 2nd Combined into 2C×H×W mixed features along the channel dimension ;

[0095] S2312: Mixing features Input to the information channel aggregation layer, and then pass through a point-by-point convolution and a depth-wise convolution to obtain the aggregated mixed features. ;

[0096] S2313: Mixed features after aggregation Input to the feature segmentation layer, and aggregate the mixed features Split into the first mixed feature along the channel dimension and the second mixed feature , the first mixed feature Input to the GELU nonlinear activation layer;

[0097] S2314: GELU nonlinear activation layer activates the first mixed feature ;

[0098] S2315: First hybrid feature after activation With the second mixed feature Perform element-by-element multiplication in the fusion layer to obtain the multiplied features, then perform point-by-point convolution on the multiplied features to obtain the fused features, add the fused features to the first feature Fraw initially input element-by-element, and perform point-by-point convolution to obtain the first fused feature ;

[0099] The expression for feature fusion in the feature fusion layer is:

[0100]

[0101] in, The first mixed feature representing the mixed feature after the l-th layer aggregation, The second mixed feature represents the mixed feature after the l-th layer aggregation, l=1,2,…,L, L represents the number of feature layers; Represents the mixed features of the lth layer; DConv3 represents the depth convolution with a kernel size of 3×3; PConv represents the point-by-point convolution with a kernel size of 3×3; represents Hadamard product; GELU represents GELU nonlinear activation; represents the first feature of layer l;

[0102] S232: Take the basic feature z3 as the first feature F raw , the first fusion feature The features extracted after inputting into the basic block layer of the second group of fusion blocks are used as the second features F 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the second group of fusion blocks to obtain the second fusion feature ; The process of feature fusion in the feature fusion layer is the same as S2311-S2315;

[0103] S233: Take the basic feature z4 as the first feature F raw , the second fusion feature The features extracted after inputting into the basic block layer of the third group of fusion blocks are used as the second feature F 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the third group of fusion blocks to obtain the aggregated features ; The process of feature fusion in the feature fusion layer is the same as S2311-S2315;

[0104] S24: Aggregate Features Input to the spatial pooling module for horizontal pooling to obtain comprehensive features , the expression is:

[0105]

[0106] Among them, maxpool() represents the maximum pooling operation; avgpool() represents the average pooling operation; that is, global pooling is performed on 16 horizontal segments to extract comprehensive features;

[0107] S25: Input the comprehensive features into the fully connected layer, map these vectors into the metric space, and calculate the triplet loss , specifically:

[0108]

[0109] in, Represents the anchor point sequence, i.e., the target sample; Represents a sequence of positive samples belonging to the same category as the anchor point; Represents a sequence of negative samples that belong to different categories from the anchor point; N represents the number of samples in a training batch; represents the embedding function that transforms each sequence into an embedding space; || ||2 represents the Euclidean norm; Represents the distance between the anchor sequence and the positive sample sequence. Since they are of the same category, the distance should be as small as possible; Represents the distance between the anchor sequence and the negative sample sequence. Due to different categories, this distance is greater than the distance of the positive sample sequence and is at least one more boundary value than the distance of the positive sample sequence. ; Represents the boundary used to establish separation between positive and negative matches; this configuration enhances the model’s ability to effectively distinguish between classes;

[0110] S26: The features after the fully connected layer are sent to the BN neck layer (BNNeck layer) of the output layer for normalization. The feature space is refined before the final classification stage. The normalized features are output through the classification layer of the output layer to predict the scoliosis recognition results, and the cross entropy loss is calculated. , cross entropy loss is used to measure the fidelity between the predicted label and the actual label, and the expression is:

[0111]

[0112] Where N represents the number of samples in a training batch; represents the scoliosis recognition result predicted by the model; Represents the real scoliosis recognition results annotated in the image dataset;

[0113] S27: Based on triple loss and cross entropy loss Calculating Mixing Loss , expressed as:

[0114]

[0115] The network model is optimized according to the mixed loss to obtain the gait recognition model;

[0116] S3: Collect video files and process them to obtain gait data to be identified;

[0117] The user is required to upload a 10-15 second gait video. The video suggests that the subject walks along the corridor. The camera is taken at a distance of 1.4 to 4.2 meters from the participant, ensuring that the participant is in the center of the video as much as possible. For the video file, the video is framed into 720p through a python script, and 300 frames are fixedly extracted according to the length of the video. The size of the input file is compressed in this way. The processed video file is obtained through the PP-HumanSeg algorithm to obtain the character outline. The image is converted into a grayscale image, and the redundant black part is cropped with the portrait as the center, and the insufficient part is supplemented with black. The image size is 64×64. The extracted set of frame images is processed into 300 frames of 64×64 image data and saved in pkl format as the gait data to be identified.

[0118] S4: The gait data to be recognized is sent to the trained gait recognition model, which will determine the scoliosis condition and divide it into positive, neutral and negative according to the degree of curvature.

[0119] On the other hand, in a specific embodiment, a recognition system based on the above-mentioned scoliosis recognition method for gait includes an image acquisition module, a model training module, a video processing module and a recognition module;

[0120] The image acquisition module collects picture data sets and video files, and sends the picture data sets to the model training module and sends the video files to the video processing module;

[0121] The model training module builds a CNN network model with an attention mechanism, and uses a picture training set to train the CNN network model to obtain a gait recognition model that is sent to the recognition module.

[0122] The video processing module extracts several silhouette images from the video file, processes all the silhouette images into images centered on the portrait, converts the processed images into Numpy arrays, constructs the gait data to be recognized, and sends them to the recognition module;

[0123] The recognition module inputs the gait data to be recognized into the gait recognition model to obtain the scoliosis recognition result.

[0124] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0125] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying scoliosis in gait, characterized in that: The following steps are involved: Step 1: Collect image dataset; Step 2: Construct a CNN network model with an attention mechanism. The CNN network model includes a multi-scale feature extraction module with an attention mechanism, a temporal pooling module, an aggregation module, a spatial pooling module, a fully connected layer, and an output layer. The multi-scale feature extraction module includes a sequentially connected convolution layer and four improved basic block layers. Each improved basic block layer outputs a feature of one scale, and sequentially outputs features f1, f2, f3 and f4 of four different scales. The feature output by the previous improved basic block layer is used as the input of the next improved basic block layer. The improved basic block layer includes a first convolution layer, a second convolution layer, an MCA attention layer and a residual connection layer connected in sequence. The residual connection layer performs a residual connection on the output of the first convolution layer and the output of the MCA attention layer, and outputs the feature f x , x=1,2,3,4; Step 3: Use the image dataset to train the CNN network model to obtain a gait recognition model; Step 4: Collect video files, extract several silhouette images from the video files, and process all silhouette images into images centered on the portrait. Convert the processed images into Numpy arrays to construct the gait data to be identified. Step 5: Input the gait data to be recognized into the gait recognition model to obtain the scoliosis recognition result.

2. A method for identifying scoliosis in gait according to claim 1, characterized in that: The MCA attention layer performs feature rearrangement and fusion processing on the C×H×W feature map output by the second convolutional layer, where C represents the number of channels, H represents the height, and W represents the width. The specific steps include: Step 311: Rearrange the C×H×W feature map F into a H×C×W feature map F W , H×1×1 features are obtained through average pooling and standard pooling, and the H×1×1 features are rearranged into 1×1×H features. Then 1×K Conv2d convolutions are used to extract 1×1×H features, which are then rearranged into H×1×1 features. After Sigmoid activation function, they are combined with H×C×W feature map F W Multiply them to obtain the rearranged feature Y1, the shape of Y1 is C×H×W; Step 312: Rearrange the C×H×W feature map X into a W×H×C feature map F H , get W×1×1 features through average pooling and standard pooling, rearrange the W×1×1 features into 1×1×W features, extract 1×1×W features through 1×K Conv2d convolutions, and then rearrange them into W×1×1 features, pass the Sigmoid activation function and combine them with the W×H×C feature map F H Multiply them to obtain the rearranged feature Y2, the shape of Y2 is C×H×W; Step 313: The C×H×W feature map F is subjected to average pooling and standard pooling to obtain C×1×1 features, the C×1×1 features are rearranged into 1×1×C features, and then 1×K Conv2d convolutions are used to extract 1×1×C features, which are then rearranged into C×1×1 features, and the Sigmoid activation function is used to multiply the C×H×W feature map F to obtain rearranged features Y3, the shape of which is C×H×W; Step 314: Add the rearranged feature Y1, the rearranged feature Y2, and the rearranged feature Y3 and take the average value to obtain the attention feature Y.

3. The method for identifying scoliosis in gait according to claim 1, characterized in that: The temporal pooling module includes four temporal pooling layers. The four scale features f1, f2, f3 and f4 output by the multi-scale feature extraction module are respectively input into a corresponding temporal pooling layer for feature concentration to obtain four basic features z1, z2, z3 and z4 respectively.

4. The method for identifying scoliosis in gait according to claim 3, characterized in that: The aggregation module includes three groups of fusion blocks connected in sequence, each group of fusion blocks includes a basic block layer and a feature fusion layer; the basic block layer includes a first convolutional layer, a second convolutional layer and a residual connection layer connected in sequence, and the residual connection layer performs a residual connection on the output of the first convolutional layer and the output of the second convolutional layer; the process of feature fusion in the aggregation module is as follows: Step 321: Take the basic feature z2 as the first feature F raw , the basic feature z1 is input into the basic block layer of the first group of fusion blocks, and the second feature F is extracted 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the first group of fusion blocks to obtain the first fusion feature z′1; Step 322: Take the basic feature z3 as the first feature F raw , the first fusion feature z′1 is input to the basic block layer of the second group of fusion blocks, and the extracted feature is used as the second feature F 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the second group of fusion blocks to obtain the second fusion feature z′2; Step 323: Take the basic feature z4 as the first feature F raw The second fusion feature z′2 is input to the basic block layer of the third group of fusion blocks, and the extracted feature is used as the second feature F 2nd , the first feature F raw and the second feature F 2nd At the same time, it is input into the feature fusion layer of the third group of fusion blocks to obtain the aggregated feature z′.

5. The method for identifying scoliosis in gait according to claim 4, characterized in that: The feature fusion layer includes a feature splicing layer, an information channel aggregation layer, a feature segmentation layer, a GELU nonlinear activation layer and a fusion layer connected in sequence; the process of the feature fusion layer performing feature fusion processing in step 321 includes: Step 3211: First Feature F raw and the second feature F 2nd Input to the feature merging layer, the first feature F with size C×H×W raw and the second feature F with dimensions C×H×W 2nd Combined into a 2C×H×W mixed feature F along the channel dimension fuse ; Step 3212: Mix the feature F fuse Input to the information channel aggregation layer, through point-by-point convolution and depth convolution in turn, to obtain the aggregated mixed feature F′ fuse ; Step 3213: Aggregated hybrid feature F′ fuse Input to the feature segmentation layer, and aggregate the mixed feature F′ fuse Split along the channel dimension into the first mixed feature F gate and the second mixed feature F con , the first mixed feature F gate Input to the GELU nonlinear activation layer; Step 3214: GELU nonlinear activation layer activates the first mixed feature F gate ; Step 3215: Activated first hybrid feature F gate With the second mixed feature F con Perform element-by-element multiplication in the fusion layer to obtain the multiplied features, then perform point-by-point convolution on the multiplied features to obtain the fused features, and then add the fused features to the first feature F raw After element-by-element addition, point-by-point convolution is performed to obtain the first fusion feature z′1.

6. The method for identifying scoliosis in gait according to claim 4, characterized in that: The spatial pooling module performs horizontal pooling on the aggregated feature Z′ to obtain the comprehensive feature f′, which is expressed as: f=maxpool(z′)+avgpool(z′) Among them, maxpool() represents the maximum pooling operation; avgpool() represents the average pooling operation; z′ represents the aggregation feature.

7. The method for identifying scoliosis in gait according to claim 1, characterized in that: In the process of CNN network model training, the hybrid loss is constructed by calculating the triplet loss and the cross entropy loss, and the model is optimized according to the hybrid loss; Triplet loss It is expressed as: in, Represents the anchor point sequence, which is the target sample; Represents a sequence of positive samples belonging to the same category as the anchor point; represents a negative sample sequence that belongs to a different category from the anchor point; N represents the number of samples in a training batch; f represents the embedding function that converts each sequence into an embedding space; || ||2 represents the Euclidean norm; Represents the distance between the anchor sequence and the positive sample sequence; represents the distance between the anchor sequence and the negative sample sequence; α represents the boundary used to establish separation between positive and negative matches; The features of the fully connected layer are sent to the BN neck layer of the output layer for normalization. The normalized features are output through the classification layer of the output layer to output the predicted scoliosis recognition results and calculate the cross entropy loss. Cross Entropy Loss It is expressed as follows: Where N represents the number of samples in a training batch; Represents the scoliosis recognition result predicted by the model; y i Represents the real scoliosis recognition results annotated in the image dataset; The mixed loss is It is expressed as:

8. The method for identifying scoliosis in gait according to claim 1, characterized in that: The specific process of collecting and processing the video file in step 4 to obtain the gait data to be identified includes: Step 41: Convert the video file into a silhouette file, and extract a preset number of frame images through a Python algorithm; Step 42: using the PP-HumanSeg method to obtain a human outline from the frame image, converting the image into a grayscale image, taking the human outline as the center, cutting out the black part of the human outline in the grayscale image, and filling the blank part outside the human outline with black to obtain a processed image; Step 43: Save the processed images corresponding to all frame images in pkl format, convert them into Numpy arrays, and form the gait data to be identified.

9. A scoliosis recognition system for gait, characterized in that: A method for identifying scoliosis in gait according to any one of claims 1 to 8, comprising an image acquisition module, a model training module, a video processing module and a recognition module; The image acquisition module collects image data sets and video files, and sends the image data sets to the model training module and sends the video files to the video processing module; The model training module builds a CNN network model with an attention mechanism, and uses a picture training set to train the CNN network model to obtain a gait recognition model that is sent to the recognition module. The video processing module extracts several silhouette images from the video file, processes all the silhouette images into images centered on the portrait, converts the processed images into Numpy arrays, constructs the gait data to be recognized, and sends them to the recognition module; The recognition module inputs the gait data to be recognized into the gait recognition model to obtain the scoliosis recognition result.

Images