A method for extracting and analyzing barefoot footprints on site based on semantic segmentation

Through deep learning semantic segmentation method and Unet network automatic extraction of barefoot footprints, combined with scale information, the problems of low efficiency and poor accuracy of barefoot footprint extraction in the existing technology are solved, and efficient and accurate analysis under shadow and fuzzy conditions are achieved.

CN118196435BActive Publication Date: 2024-11-29ANHUI UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410323240.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-21
Publication Date
2024-11-29
Estimated Expiration
2044-03-21

AI Technical Summary

Technical Problem

In the prior art, the barefoot footprint extraction method relies on manual drawing of contours, is inefficient and is susceptible to shadows and image blur, resulting in inaccurate extraction results.

Method used

The semantic segmentation method based on deep learning is used to segment the barefoot footprint image through the Unet network and the region relationship extraction module, and the barefoot outline is automatically extracted and physiological characteristics are inferred based on the ruler information.

Benefits of technology

Improves the efficiency and accuracy of barefoot footprint extraction, reduces manual intervention, and can accurately analyze barefoot characteristics under shadow and fuzzy conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118196435B_ABST
    Figure CN118196435B_ABST
Patent Text Reader

Abstract

The present invention discloses a field barefoot footprint extraction and analysis system based on semantic segmentation, comprising the following steps: obtaining a field barefoot footprint image to be extracted, wherein a ruler exists in the image; sending the acquired image to a processor for processing; using the image segmentation function in the system to segment the barefoot footprint image with a ruler to obtain a barefoot footprint segmentation result; using the analysis function in the system to preliminarily infer the shoe size, height, gender, weight and other physiological characteristics of the footprint owner; and finally printing out the barefoot extraction result image and feature analysis result in a display window. The present invention relates to the field of image processing. The field barefoot footprint extraction and analysis system based on semantic segmentation can automatically extract the field footprint and give a preliminary analysis, especially for the situation where objective factors such as shadows and image blur affect the extraction of artificial footprints, thereby improving the efficiency and accuracy of footprint image extraction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of footprint image analysis, and in particular to an on-site barefoot footprint extraction and analysis method based on semantic segmentation. Background Art

[0002] In the field of criminal investigation, footprints are one of the most common clues and are of great significance to the detection of cases. Barefoot footprints are a major type of footprints at the scene. Experts can obtain relevant physiological characteristics such as height and weight of the person who left the footprints by analyzing the barefoot footprint images collected at the scene.

[0003] The premise of barefoot analysis is to extract barefoot from barefoot images to prevent complex background information from affecting the analysis of footprints. The traditional barefoot footprint extraction method is to manually draw the barefoot outline on the footprint image to extract the footprint, manually mark the barefoot key points, measure the barefoot length and width, and infer the shoe size, height and other information of the person. Compared with the semantic segmentation method based on deep learning to segment barefoot images to automatically extract barefoot outlines, the manual outline drawing by experts is labor-intensive and time-consuming, and the extracted outline is easily affected by human subjective factors. In addition, in the on-site collection environment, due to the uneven ground on the scene and the uneven lighting on the footprint surface, shadows will appear on the collected footprint image, which will have a great impact on the manual identification of the outline. Therefore, the automatic extraction of footprints from on-site barefoot images plays an important role in the analysis and comparison of footprints. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the shortcomings of the prior art, the present invention provides a method for on-site barefoot footprint extraction and analysis based on semantic segmentation, which automatically extracts on-site footprints and gives a preliminary analysis, especially for situations where objective factors such as shadows and image blur affect the extraction of artificial footprints, thereby improving the efficiency and accuracy of footprint image extraction.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented by the following technical scheme: a method for extracting and analyzing barefoot footprints on site based on semantic segmentation, comprising the following steps:

[0008] S1: Obtain the barefoot footprint image to be extracted, with a ruler in the image;

[0009] S2: Send the collected image to the CPU processor for image processing;

[0010] S3 uses the image segmentation function in the system to segment the barefoot footprint image with a ruler to obtain the barefoot footprint segmentation result;

[0011] S4: Use the analysis function in the system to preliminarily infer the shoe size, height, gender, weight and other physiological characteristics of the footprint owner;

[0012] S5: Finally, the barefoot extraction result image and feature analysis result are printed out in the display window.

[0013] The semantic segmentation network training used for the segmentation function in step S3 includes the following steps:

[0014] The first step is to collect training images: walk on the floor of the simulated on-site environment to leave dust footprints, place a ruler next to the footprints, take photos to obtain on-site barefoot images, manually annotate the acquired images, and obtain the training labels corresponding to the images. The annotated objects are the ruler and barefoot.

[0015] The second step is optimal network training: the obtained barefoot dataset of at least 1,000 images is divided into training set, validation set and test set. Before training, the training set is augmented and all image shapes are filled with squares. After scaling to 256×256, the augmented and preprocessed training set is used to train the segmentation network. The divided validation set and test set are used to validate and test the network to evaluate the performance of the network.

[0016] The specific steps for training the segmentation network are as follows:

[0017] (1) Construct a backbone network for extracting footprint features: The backbone network uses the Unet network. The Unet network structure is an encoder-decoder structure. The encoder structure consists of five stages. It downsamples the input original image, obtains feature maps of different scales, and extracts high-level semantic information of the segmentation target. In the 0th stage, the original image is upgraded to a multi-dimensional feature map. The input is {X in ∈R 3×256×256}, after the convolutional layer, it reaches {X 1 ∈R 64×256×256}; In the last four stages, the feature maps will go through the pooling layer, convolution layer, BN layer and ReLU activation function to obtain feature maps of four different scales. The output feature of the last stage is {X 5 ∈R 1024×16×16 The decoder structure upsamples the high-level feature map generated by the encoder and fuses the feature information of the corresponding feature layer in the encoder through skip connections to map the feature back to the original image size. It can be divided into four stages. The output feature of the last stage is {Y∈R 256×256×256}, which is the output feature of the backbone network;

[0018] (2) Construct a regional relationship extraction module: The extracted regional relationship is the association information between the input image region and the feature map region in the backbone network encoder; in ∈R 3×H×W} and feature map Convolution and spatial transformation operations are performed, where C is the number of feature channels, H and W represent the height and width of the original image respectively, and H 1 and W 1 Represent the height and width of the feature respectively; first convolve the original image and the feature, convert its channel number to 256, and get {X in ∈R 256×H×W}and right Divide the patch into blocks with a fixed step size s, flatten each patch block in the spatial dimension, and then splice the patches in order in the 0th dimension to obtain F i The same operation is used to obtain Will and X i Perform matrix multiplication on the two-dimensional tensor on the corresponding channel to obtain F R Represents the relationship between the input image area and the feature map area in the backbone network encoder. When the network is trained, X in The corresponding ground truth generated monitor it, Represents the regional relationship between the ground truth after pooling and itself;

[0019] (3) Construct a regional relationship modulation module: Use the extracted regional relationship to spatially modulate the features extracted by the backbone network to enhance the overall barefoot features. 256×H×W}Convolution and spatial transformation operations are performed, and after convolution and downsampling, F R and Y 1 Perform matrix multiplication on the two-dimensional tensor on the corresponding channel to obtain Y 2 Perform inverse spatial transformation to obtain {Y 3 ∈R 256×H×W}, change Y 3 With Y 1 Add element by element to get the enhanced feature {Y en ∈R 256×H×W}, and finally the features are mapped into a predicted mask map through the segmentation head to obtain the segmentation result.

[0020] As an optimization, in step S3, the image segmentation function is to input the barefoot footprint image to be extracted into the trained semantic segmentation network model, and use the network model to segment the image to obtain the barefoot footprint mask, thereby obtaining the segmentation result.

[0021] As an optimization, in step S4, the analysis function system preliminarily infers the shoe size, height, gender, weight and other physiological characteristics of the footprint owner through the segmented footprint. In step S3, the barefoot is segmented and the mask of the ruler is segmented, and the pixel distance between each part of the barefoot and the ruler is calculated. The real distance of the barefoot is calculated using the scale between the ruler pixel distance and the real distance, and the shoe size of the footprint owner is converted through the data of foot length and foot width, and the physiological information such as height, weight and gender are preliminarily estimated.

[0022] As an optimization, in the second step, the data is divided into training set, validation set and test set in a ratio of 7:1:2. The methods for data augmentation of the training set include rotation and upside-down flipping, with rotation angles of 90 degrees, 180 degrees and 270 degrees respectively.

[0023] (III) Beneficial effects

[0024] The present invention provides a method for extracting and analyzing barefoot footprints on site based on semantic segmentation, which has the following beneficial effects:

[0025] The present invention uses a semantic segmentation method based on deep learning to make barefoot footprint extraction intelligent. By learning the regional relationship between the original image and the feature map, the backbone network footprint features are enhanced to obtain better segmentation results, reducing the impact of objective factors such as image shadows and image blur on footprint extraction. Compared with manual calculation and analysis of manual contour extraction, it can not only reduce manpower and do not require relevant professional knowledge, but also has a faster extraction and analysis efficiency. With the continuous development of artificial intelligence, society has entered an era of informatization and intelligence. This invention can reduce the analysis pressure for criminal investigation experts, improve efficiency, and play a positive role in criminal investigation work. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the barefoot extraction and analysis system of the present invention;

[0027] Figure 2 This is a framework diagram for model training and testing of the present invention;

[0028] Figure 3 It is a framework diagram of the regional relationship extraction and modulation module of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1-3 , the present invention provides a technical solution:

[0031] like Figure 2 As shown, model training and testing are completed in the following steps:

[0032] The first step is to collect training images: walk on the floor of a simulated on-site environment to leave dust footprints, place a ruler next to the footprints, and take photos to obtain on-site barefoot images. The acquired images are manually annotated to obtain the training labels corresponding to the images. The annotated objects are the ruler and barefoot.

[0033] The second step is optimal network training: the obtained barefoot dataset of at least 1,000 images is divided into training set, validation set and test set. Before training, the training set is augmented, and all image shapes are filled with squares, and then scaled to 256×256. The augmented and preprocessed training set is used to train the segmentation network, and the divided validation set and test set are used to verify and test the network to evaluate the network performance. The specific steps for training the segmentation network are as follows:

[0034] (1) Construct a backbone network for extracting footprint features: The backbone network uses the Unet network. The Unet network structure is an encoder-decoder structure. The encoder structure consists of five stages. It downsamples the input original image, obtains feature maps of different scales, and extracts high-level semantic information of the segmentation target. In the 0th stage, the original image is upgraded to a multi-dimensional feature map. The input is {X in ∈R 3×256×256}, after the convolutional layer, it reaches {X 1 ∈R 64×256×256}; In the last four stages, the feature maps will go through the pooling layer, convolution layer, BN layer and ReLU activation function to obtain feature maps of four different scales. The output feature of the last stage is {X 5 ∈R 1024×16×16 The decoder structure upsamples the high-level feature map generated by the encoder and fuses the feature information of the corresponding feature layer in the encoder through skip connections to map the feature back to the original image size. It can be divided into four stages. The output feature of the last stage is {Y∈R 256×256×256}, which is the output feature of the backbone network.

[0035] (2) Construct a regional relationship extraction module: The extracted regional relationship is the association information between the input image region and the feature map region in the backbone network encoder. in ∈R 3×H×W} and feature map Convolution and spatial transformation operations are performed, where C is the number of feature channels, H and W represent the height and width of the original image respectively, and H 1 and W 1 Represent the height and width of the feature respectively. First, convolve the original image and the feature, convert its channel number to 256, and get {X in ∈R 256×H×W}and right Divide the patch into blocks with a fixed step size s, flatten each patch block in the spatial dimension, and then splice the patches in order in the 0th dimension to obtain F i The same operation is used to obtain Will and X i Perform matrix multiplication on the two-dimensional tensor on the corresponding channel to obtain F R Represents the relationship between the input image area and the feature map area in the backbone network encoder. When the network is trained, X in The corresponding ground truth generated monitor it, Represents the regional relationship between the ground truth after pooling and itself.

[0036] (3) Construct a regional relationship modulation module: Use the extracted regional relationship to spatially modulate the features extracted by the backbone network to enhance the overall barefoot features. 256×H×W}Convolution and spatial transformation operations are performed, and after convolution and downsampling, F R and Y 1 Perform matrix multiplication on the two-dimensional tensor on the corresponding channel to obtain Y 2 Perform inverse spatial transformation to obtain {Y 3 ∈R 256×H×W}, change Y 3 With Y 1 Add element by element to get the enhanced feature {Y en ∈R 256×H×W}, and finally the features are mapped into a predicted mask map through the segmentation head to obtain the segmentation result.

[0037] The third step is the barefoot image segmentation function: the barefoot footprint image to be extracted is input into the trained semantic segmentation network model, and the network model is used to segment the image to obtain the barefoot footprint mask and the ruler mask. The barefoot footprint mask is used to separate the barefoot and background of the image and extract the on-site footprints. The ruler mask is used to analyze the barefoot footprints.

[0038] The fourth step is the barefoot footprint analysis function: the semantic segmentation network segments the barefoot and the ruler mask at the same time, calculates the pixel distance between each part of the barefoot and the ruler, and uses the scale between the ruler pixel distance and the true distance to calculate the true distance of the barefoot. The shoe size of the footprint owner is calculated through data such as foot length and foot width, and physiological information such as birth height, weight and gender is preliminarily estimated.

[0039] Based on the above steps, Figure 1 As shown in FIG. 1 , a method for extracting and analyzing barefoot footprints on site based on semantic segmentation is described. The specific steps are as follows:

[0040] S1: Obtain the barefoot footprint image to be extracted, with a ruler in the image;

[0041] S2: Send the collected image to the CPU processor for image processing;

[0042] S3 uses the image segmentation function in the system to segment the barefoot footprint image with a ruler to obtain the barefoot footprint segmentation result;

[0043] S4: Use the analysis function in the system to preliminarily infer the shoe size, height, gender, weight and other physiological characteristics of the footprint owner;

[0044] S5: Finally, the barefoot extraction result image and feature analysis result are printed out in the display window.

[0045] In the optimization process of the entire semantic segmentation network of the present invention, the network gradually learns the multi-scale semantic features and global context information of different barefoot objects. When the model finds the best network model through continuous optimization, the entire network can be regarded as an expert with rich experience in footprint extraction. At this point, the model has the ability to quickly segment barefoot footprints on site.

[0046] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for extracting and analyzing barefoot footprints on site based on semantic segmentation, characterized in that: The following steps are involved: S1: Obtain the barefoot footprint image to be extracted, with a ruler in the image; S2: Send the collected image to the CPU processor for image processing; S3 uses the image segmentation function in the system to segment the barefoot footprint image with a ruler to obtain the barefoot footprint segmentation result; S4: Use the analysis function in the system to preliminarily infer the shoe size, height, gender, weight and other physiological characteristics of the footprint owner; S5: finally printing out the barefoot extraction result image and feature analysis result in the display window; The semantic segmentation network training used for the segmentation function in step S3 includes the following steps: The first step is to collect training images: walk on the floor of the simulated on-site environment to leave dust footprints, place a ruler next to the footprints, take photos to obtain on-site barefoot images, manually annotate the acquired images, and obtain the training labels corresponding to the images. The annotated objects are the ruler and barefoot. The second step is optimal network training: the obtained barefoot data set of at least 1,000 images is divided into training set, validation set and test set; before training, the training set is augmented and all image shapes are filled with squares, and then the size is scaled to 256×256; the segmentation network is trained with the augmented and preprocessed training set, and the network is verified and tested using the divided validation set and test set to evaluate the network performance; The specific steps for training the segmentation network are as follows: (1) Construct a backbone network for extracting footprint features: The backbone network uses the Unet network. The Unet network structure is an encoder-decoder structure. The encoder structure consists of five stages. It downsamples the input original image, obtains feature maps of different scales, and extracts high-level semantic information of the segmentation target. In the 0th stage, the original image is upgraded to a multi-dimensional feature map. The input is {X in ∈R 3×256×256 }, after the convolutional layer, we get {X1∈R 64×256×256 }; In the last four stages, the feature maps will go through the pooling layer, convolution layer, BN layer and ReLU activation function to obtain feature maps of four different scales. The output feature of the last stage is {X5∈R 1024×16×16 The decoder structure upsamples the high-level feature map generated by the encoder and fuses the feature information of the corresponding feature layer in the encoder through skip connections to map the feature back to the original image size. It can be divided into four stages. The output feature of the last stage is {Y∈R 256×256×256 }, which is the output feature of the backbone network; (2) Construct a regional relationship extraction module: The extracted regional relationship is the association information between the input image region and the feature map region in the backbone network encoder; in ∈R 3×H×W } and feature map Convolution and spatial transformation operations are performed, where C is the number of feature channels, H and W represent the height and width of the original image, and H1 and W1 represent the height and width of the feature. First, convolution is performed on the original image and the feature, and the number of channels is converted to 256 to obtain {X in ∈R 256×H×W }and right Divide the patch into blocks with a fixed step size s, flatten each patch block in the spatial dimension, and then splice the patches in order in the 0th dimension to obtain F i The same operation is used to obtain Will and X i Perform matrix multiplication on the two-dimensional tensor on the corresponding channel to obtain F R Represents the relationship between the input image area and the feature map area in the backbone network encoder; when the network is trained, it will be represented by X in The corresponding ground truth generated monitor it, Represents the regional relationship between the ground truth after pooling and itself; (3) Construct a regional relationship modulation module: use the extracted regional relationship to spatially modulate the features extracted by the backbone network to enhance the overall features of barefoot; 256×H×W }Convolution and spatial transformation operations are performed, and after convolution and downsampling, F R and Y 1 Perform matrix multiplication on the two-dimensional tensor on the corresponding channel to obtain Y 2 Perform inverse spatial transformation to obtain {Y 3 ∈R 256×H×W }, change Y 3 With Y 1 Add element by element to get the enhanced feature {Y en ∈R 256×H×W }, and finally the features are mapped into a predicted mask map through the segmentation head to obtain the segmentation result.

2. The on-site barefoot footprint extraction and analysis method based on semantic segmentation according to claim 1, characterized in that: In step S3, the image segmentation function is to input the barefoot footprint image to be extracted into the trained semantic segmentation network model, and use the network model to segment the image to obtain the barefoot footprint mask, thereby obtaining the segmentation result.

3. The on-site barefoot footprint extraction and analysis method based on semantic segmentation according to claim 1, characterized in that: In step S4, the analysis function system preliminarily infers the shoe size, height, gender, and weight physiological characteristics of the footprint owner through the segmented footprints; in step S3, the barefoot is segmented while the ruler mask is also segmented, and the pixel distance between each part of the barefoot and the ruler is calculated. The real distance of the barefoot is calculated using the scale between the ruler pixel distance and the real distance, and the shoe size of the footprint owner is converted through data such as foot length and foot width, and the height, weight and gender physiological information are preliminarily estimated.

4. The on-site barefoot footprint extraction and analysis method based on semantic segmentation according to claim 1, characterized in that: In the second step, the data is divided into training set, validation set and test set in a ratio of 7:1:

2. The methods for data augmentation of the training set include rotation and upside-down flipping, with rotation angles of 90 degrees, 180 degrees and 270 degrees respectively.

Citation Information

Patent Citations

  • Cross-modal footprint image retrieval system based on weakening modal difference

    CN116401395A

  • Remote sensing semantic segmentation method based on U-Net3 +

    CN117152435A