A gait image segmentation method based on human contour

Abstract

The present invention discloses a gait image segmentation method based on human body contour. The method comprises the following steps: first, a gait dataset CI‑Gait for the elderly with corresponding cognitive impairment classification labels is produced. Secondly, based on the instance segmentation network BlendMask, a new instance segmentation network C‑BlendMask is constructed by combining BottleneckTransformer and CoordConv structures, and applied to a self-built dataset. Segmentation software is used to annotate some images in the self-built dataset, and a corresponding manually segmented dataset is constructed. Finally, the accuracy of the neural network segmentation images is verified by comparing the structural similarity between the manually segmented images and the neural network segmentation images. Compared with the existing technology, the method proposed in the present invention improves the segmentation accuracy without increasing the training time, and has broad application prospects in gait recognition, health monitoring of the elderly, and the like.

Description

Technical Field

[0001] The present invention relates to a gait image segmentation method based on human body contour, belonging to the field of computer vision and intelligent information technology. Background Art

[0002] Cognitive impairment is characterized by a decreased ability to perceive and understand the world around them. In clinical medicine, patients with cognitive impairment are categorized by specific symptoms and signs. Syndromes such as dementia, mental retardation, aphasia, amnesia, and delirium can all be characterized by cognitive impairment. Dementia refers to the comprehensive decline in intellectual function while conscious. Depending on the severity of the condition, cognitive impairment progresses through three stages: from normal aging to mild cognitive impairment (MCI), and finally to severe cognitive impairment (early dementia). The diagnosis of cognitive impairment is often limited by environmental factors and requires comprehensive evaluation and analysis by a physician. Currently, there are no targeted treatments or effective diagnostic and treatment options for this condition. Therefore, timely interventions and targeted prevention strategies for patients with early cognitive impairment can effectively mitigate symptoms and delay onset. They can also enable patients already suffering from cognitive impairment to receive timely diagnosis, effective treatment, and appropriate care, thereby delaying the onset of early dementia and reducing its prevalence.

[0003] Instance segmentation is the inevitable result of the evolution of object detection and semantic segmentation from coarse-grained to fine-grained reasoning, and is a research hotspot in the field of computer vision. Object detection or localization refers to the gradual development of digital image reasoning (i.e., from coarse-grained to fine-grained reasoning). When detecting a target object, it provides the category and location information of the target object in the image, with the specific location information provided in the form of center coordinates or boundary coordinates. The task of semantic segmentation is to assign each pixel of all instance objects in the target image to the class to which it belongs. Objects of the same class have the same class label. With the in-depth study of segmentation technology, it has been found that instance segmentation can distinguish different individuals of the same class in more detail and provide them with different labels. Therefore, instance segmentation technology is equivalent to the combination of object detection technology and semantic segmentation technology, solving two problems at the same time.

[0004] Instance segmentation methods can be divided into two categories: candidate box-based methods and candidate box-free methods. The former first coarsely locates candidate boxes in the image, then performs pixel-level template segmentation to obtain bounding box-based object instances. Faster-RCNN is a typical candidate box-based instance segmentation method. Candidate box-free methods first use a multi-layer neural network to predict instance object information at the pixel level, and then obtain instance results through pixel clustering. However, how to obtain appropriate instance object information has always been a research focus and difficulty. In addition, on datasets with simple scenes, candidate box-free methods have higher segmentation accuracy than candidate box-based methods. However, in scenes with a large number of object categories and complex datasets, candidate box-free methods have significant shortcomings. Therefore, how to learn more robust image features on data with limited space and complex objects is a worthy research problem. Furthermore, while convolutional neural networks (CNNs) have been able to meet the requirements of most instance segmentation tasks, ensuring the effectiveness of network training when instances of the same class are close together within the scene is a worthy research problem. Summary of the Invention

[0005] This paper proposes a human body contour-based gait image segmentation method. The goal is to create a gait dataset for elderly individuals with different cognitive impairment classification labels, build an instance segmentation network using the Bottleneck Transformer architecture, and then perform instance segmentation on images in the self-constructed dataset.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] (1) Capture and create a CI-Gait dataset for elderly people's gait, which includes multiple perspectives and is labeled with corresponding cognitive impairment levels after doctors assess their cognitive status;

[0008] (2) Combine the Bottleneck Transformer structure, CoordConv structure and instance segmentation network BlendMask to construct an instance segmentation network C-BlendMask with higher segmentation accuracy.

[0009] (3) Label Me software was used to manually segment 1250 images in the CI-Gait dataset to form the standard dataset Manual-Gait. By comparing the structural similarity (SSIM) between manually segmented images and neural network segmented images, the accuracy of the images segmented by the neural network was quantitatively analyzed.

[0010] (4) The new instance segmentation network C-BlendMask is tested on the public dataset COCO dataset and the self-built dataset CI-Gait to verify the effectiveness of the improved network. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of the shooting process of the CI-Gait dataset produced by the present invention.

[0012] Figure 2 This is an example segmentation network diagram based on the Bottleneck Transformer structure and CoodConv convolutional layer of the present invention.

[0013] Figure 3 This is a module structure diagram based on Bottleneck Transformer of the present invention.

[0014] Figure 4 This is the structural diagram of the CoordConv convolutional layer module based on the present invention.

[0015] Figure 5 This is a diagram of the similarity calculation results based on the SSIM index of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with the accompanying drawings:

[0017] The BlendMask network uses a similar structure to other candidate-box-based instance segmentation networks, but adds a bottom module for score prediction. The bottom module has a shape of N×K×H / s×W / s, where N is the batch size, K is the number of bottom-level score maps, H×W is the input size, and s is the stride of the score map output. The network uses DeepLab V3+ as the decoder. The bottom module can be fed with backbone features such as traditional semantic segmentation networks, or feature pyramid structures such as YOLACT and Panoramic FPN. The top-level attention information is predicted by independent convolutional layers in each detection tower. The Blender module, the core module of the BlendMask network, takes as input the output of the bottom-level output maps, the top-level attention information, and the candidate boxes in the image. It fuses the attention with positional information to output the final prediction.

[0018] First, use the cropping module in Mask R-CNN to crop the score map P for each candidate box d , and then rescale the region to a fixed size R×R feature map r d , r d is defined as follows:

[0019]

[0020] After performing ROIAlign operation on the top attention layer, we get the interpolation a d , using interpolation a d , change the size of the attention map from the original M×M to R×R, where R={r d |d=1,...,D}.

[0021]

[0022] Then, use the softmax function to transform a' along the Kth dimension d Normalize it to form a score graph set s d .

[0023]

[0024] Finally, for each entity r in region R and score S d , s d Use element-wise multiplication between them and sum along the Kth dimension to get the mask logit m d .

[0025]

[0026] Here, k is the index of the bias.

[0027] Introducing the Transformer global information aggregation capability, the multi-head self-attention module (MHSA) is used on the basis of the convolutional neural network to replace the simple 3×3 spatial convolution layer in the original segmentation network backbone structure. MSHA treats position encoding as spatial attention. All attention is performed on the two-dimensional feature map. The relative position encoding R of the two height and width in the feature map is h 、R w As spatial attention of different dimensions, relative position encoding R h 、R w are two learnable vectors. After adding them together to the same dimension and size as the spatial vector q, they are multiplied by the vector q and the resulting qr T and qk T Adding these together yields the attention logic. This allows MHSA to allocate more attention to the desired area, making it easier to converge. MHSA only introduces a multi-head structure in the blue area shown in the figure.

[0028] The CoordConv layer can be implemented as a simple extension of the standard convolution, where additional channels are instantiated and filled with constant, untrained coordinate information, after which they are concatenated channel-by-channel to the input representation and a standard convolution layer is applied. Specifically, the i-coordinate channel is a matrix of size h×w and rank-1, with the first row filled with 0, the second row filled with 1, the third row filled with 2, and so on. The j-coordinate channel is similar, but its columns are filled with constant values ​​and the rows are not padded. In all experiments, the i and j coordinate values ​​are finally linearly scaled so that their values ​​fall in the range [-1, 1]. For two-dimensional convolution, two (i, j) coordinates are sufficient to fully specify the input pixel, but if necessary, more channels can be added to bias the model to learn a specific solution. If more experimental requirements are required, a third channel can be provided for the r coordinate, which can be expressed as:

[0029]

[0030] The CoordConv convolution layer is added after the input module of the segmentation network C-BlendMask to better segment the gait images.

[0031] Because the self-built CI-Gait dataset does not have corresponding instance segmentation labels, the comparative experiments in this paper were conducted on the public dataset COCO. The data in Table 1 show the segmentation experimental results of each model on the COCO dataset. Mask-RCNN is a flexible and efficient classic instance segmentation model, BlendMask is the basic instance segmentation model used in this paper, and C-BlendMask is an improved model of this paper.

[0032] Table 1 COCO dataset segmentation results (%)

[0033]

[0034] From Table 1, we can see that the improved network C-BlendMask of the present invention has the following characteristics: 50 Compared with the basic network BlendMask, the AP and AP 75 、AP S 、AP M 、AP L Compared with Mask-RCNN network and basic network BlendMask, they have different degrees of improvement, especially in the segmentation effect of large-area objects, which is significantly better than Mask-RCNN segmentation network and basic segmentation network BlendMask. When using BoT-50 backbone network, the indicator AP LCompared with the basic network BlendMask, it is improved by about 2%. When using the BoT-101 backbone network, the indicator AP L Compared with the basic network BlendMask, there is a 3.6% improvement.

[0035] In order to quantitatively verify the segmentation effect of the improved network model C-BlendMask on the self-built dataset CI-Gait, the present invention adopts the method of calculating the structural similarity SSIM between the images in the manual segmentation dataset Manual-Gait and the images segmented by the corresponding neural network to verify the accuracy of the network model segmentation. Figure 5 The results show that the segmentation similarity between each image in the manual segmentation dataset Manual-Gait and its corresponding neural network segmentation image reached more than 95%. It can be seen that the improved model C-BlendMask also has a good segmentation effect on gait images in the self-built dataset CI-Gait, which proves the effectiveness and good generalization of the improved network.

Claims

1. A gait image segmentation method based on human body contour, characterized in that The following steps are involved: (1) We filmed and produced the CI-Gait dataset for elderly people's gait, which includes multiple perspectives and gives corresponding cognitive impairment classification labels after doctors assess their cognitive status; (2) Based on the fully convolutional instance segmentation network BlendMask, the Bottleneck Transformer structure and the CoordConv convolution layer are combined. On the basis of the segmentation network BlendMask, the Transformer global information aggregation capability is introduced. On the basis of the convolutional neural network, the multi-head self-attention module is used to replace the simple 3×3 spatial convolution layer in the original segmentation network backbone structure; at the same time, the CoordConv convolution layer is added after the input module of the segmentation network; and a new instance segmentation network C-BlendMask is successfully constructed; (3) LabelMe software was used to manually annotate some images in the CI-Gait dataset. A total of 1,250 images of three different cognitive impairment classification levels, namely normal cognitive function, mild cognitive impairment, and severe cognitive impairment, were selected from the self-built gait dataset CI-Gait. Among them, 450 images were normal cognition, 450 images were MCI status, and 350 images were severe cognitive impairment, for the production of the Manual-Gait dataset. The 1,250 images were segmented and annotated using the manual segmentation and annotation tool LabelMe software. The dataset consisting of the 1,250 manually segmented binary contour images was recorded as the dataset Manual-Gait. The structural similarity index between the image segmented by the neural network and its corresponding manually segmented image was calculated to quantitatively analyze the effectiveness of the segmentation network. The structural similarity index is used to judge the similarity between two given images. SSIM calculates the similarity between two images through three measurement indices: brightness contrast, contrast contrast, and structural contrast. Its definition is as follows: SSIM(x,y)=f(l(x,y),c(x,y),s(x,y))=[l(x,y)] α ,[c(x,y)] β ,[s(x,y)] γ (1) Among them, l(x,y), c(x,y), and s(x,y) are brightness contrast function, contrast contrast function, and structure contrast function, respectively; α, β, and γ are all greater than 0, which are used to adjust the proportion of these three modules in the structural similarity index respectively; the values ​​of the three parameters are all set to 1; the value range of the SSIM function is [0,1]. The larger the value, the less distortion between the images, and the more similar the two images are.

2. The method according to claim 1, characterized in that (1) produced the elderly gait dataset CI-Gait, and the construction method is as follows: The video collection site is arranged as a solid color background. The subjects walk from the baseline at one end to the baseline at the other end in the specified direction. While walking, the cameras fixed at 45 degrees, 90 degrees, and 135 degrees are synchronously filmed, and the subjects need to maintain basically the same size in the frame throughout the whole process; the camera used for filming needs to be placed in a fixed position and assisted by instruments to keep it stable; during the filming process, the subjects need to ensure that their whole body is in the frame when walking within the baseline and always maintain a moderate distance from the camera; because this dataset is a classification study on the cognitive function of the elderly, The subjects should be elderly people over 65 years old, who can independently complete the filming of walking videos and cooperate with medical staff to complete the simple mental state questionnaire for the assessment of cognitive impairment level; at the same time, the subjects' basic hearing and vision functions should be maintained at normal levels, and the doctor should assess and exclude the factors of other major diseases; finally, the subjects should voluntarily participate in this study and be fully informed of the video shooting process in which they participate; after multiple rounds of collection, a gait video database of elderly people with cognitive impairment was obtained; after manual screening and verification, damaged videos and uncontrolled videos caused by uncontrollable factors during the shooting process were removed. After the videos that met the shooting criteria were collected, there were 156 cases of elderly gait data that met the standards. Each subject's video was required to walk from one end of the baseline to the other end, and back and forth three times. The cameras at 45°, 90°, and 135° respectively took 6 videos. Therefore, each subject had a total of 3×6=18 video sequences, which were stored in mp4 format; the shooting time of each video sequence was 5s, the walking stride length of the subject captured in the video was 4 meters, and each video sequence contained two or more complete gait walking cycles; the frame rate of each video sequence was 30fps, The resolution of each frame is 1920×1080; the subjects were surveyed according to the Simple Mental State Questionnaire; the degree of cognitive impairment of each subject was determined based on the questionnaire results; among them, 59 cases had normal cognitive function, 58 cases had mild cognitive impairment, and 39 cases had severe cognitive impairment; the video data was processed frame by frame to ensure the stability of the gait data; incomplete segments of the characters in the video were removed, and after processing, an average of 50 gait images were obtained for each gait video, and all gait images totaled 140,000, successfully constructing the CI-Gait gait dataset for the elderly.

Images