A posture recognition method based on two-stage pooling S2E module

Abstract

The present invention relates to the field of gesture recognition technology, and provides a method for a gesture recognition model based on a two-stage pooling S2E module. The gesture recognition model is based on a progressive two-stage pooling S2E module. The technical solution is a backbone network based on the ResNet‑101 network structure, and a new S2E module with feature compression and excitation functions is proposed to form a new model. The S2E module accelerates model convergence by establishing a better channel attention mechanism, which can help the network extract features from a large number of channel graphs more quickly and filter out more critical feature information, so that the network model can recognize data more accurately. Experimental results show that the new model can balance the problems of prediction accuracy, computing resource consumption and running speed. At the same time, in the qualitative comparison experiment of gesture recognition, our model can focus on more detailed areas and avoid misrecognition due to gesture intersection and occlusion.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a posture and action recognition method based on an attention mechanism of a progressive two-stage pooling S2E module. Background Art

[0002] With the rapid development of artificial intelligence technology and the doubling of massive data in the big data era, pedestrian action recognition has been increasingly widely used in video surveillance, human-computer interaction, security, and fine action analysis of sports events. However, due to problems such as target occlusion, lighting changes, and complex backgrounds, the effect of pedestrian action recognition is not ideal. To address the problem of action recognition accuracy in complex scenes, posture recognition technology based on the attention mechanism has solved this technical problem well. Traditional attention mechanisms are usually used to evaluate the dependencies between information, so they have been widely used in natural language processing tasks.

[0003] At present, most gesture recognition algorithms based on neural networks focus on increasing the width, depth and resolution of the neural network to increase the consumption of computing resources and the training speed, thereby improving the prediction accuracy; while the SE-based compression and excitation model is a small additional module of the lightweight network. The SE model realizes the excitation or suppression of intra-channel features in a lightweight and efficient manner, and embeds it into various networks, achieving good results. However, this attention mechanism is not accurate enough in compressing and extracting deep semantic features, and cannot accurately extract unevenly distributed feature map information.

[0004] Therefore, in view of the above shortcomings, it is necessary to provide a model that can solve the balance problem between prediction accuracy and computing resource consumption and running speed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that it is difficult to balance the prediction accuracy, computing resource consumption and running speed in the process of posture estimation, and in the qualitative comparison experiment of chaotic posture recognition, there is the problem of wrong recognition caused by posture intersection and occlusion. In view of the defects in the prior art, a technical solution is provided that can improve the accuracy of posture recognition in a chaotic state and control the consumption of computing resources and time consumption.

[0006] In order to solve the above technical problems, the present invention provides a posture recognition method based on a two-stage pooling S2E module, comprising the following steps:

[0007] Step 1: Detect the human body bounding box of the input image, select YOLOv3-spp for target detection, and combine the pyramid structure two-step framework model of FPN to detect the human body frame posture of various scales;

[0008] Step 2: Perform redundant judgment calculations on the bounding boxes of human postures and delete or merge redundant detection boxes.

[0009] Step 3: Scale a series of human detection boxes to a size of 320x256, fill the blank parts with 0, and send the processed human detection boxes to the sppe (single-person pose estimator) model combined with the S2E-ResNet-101 module in turn, and independently perform pose estimation on each human box; then use DUC to upsample the 10x8 resolution feature map output by the backbone network to 80x64; Step 4: Generate a key point heat map for pose prediction, use a convolution with an output channel of 17 to replace the fully connected layer, and then detect the maximum value point in the heat map to directly generate the key point prediction of the human body, and use the value directly as the confidence for the next step of processing;

[0010] Step 5: Pose-nms is used to evaluate the possible repeated poses. Pose-nms uses the estimated pose with the highest confidence as the standard each time to evaluate the similarity between the remaining poses and the original pose. When evaluating the similarity, it includes the similarity of the key points in the same part of the two poses, as well as the similarity of the key points in different parts but similar in space. The final similarity score is obtained by multiplying the two by their respective weights. If the evaluation value is higher than the set threshold, they are put into a set. Then, the repeated poses in a set are weighted based on the confidence of each point, fused and output. This process is repeated until all poses are output or fused.

[0011] Step 6: Posture recognition visualization output. Each set of predicted key points is connected to generate the human body posture. In this process, the key points with low confidence will be discarded. The generated human body posture is added to the original image and finally visualized.

[0012] Preferably, the specific process of step 3 is:

[0013] S2E (Two-step Squeeze and Excitation) attention mechanism with two layers of compression and an excitation module;

[0014] The first layer of compression operation of S2E is the local average pooling operation, which extracts the local information of the feature map and extracts the scattered feature information effectively and fairly. The second layer of compression operation is a global maximum pooling operation, which extracts the most prominent features in the previous local average pooling. The following is the formula of the compression process used by the S2E module:

[0015]

[0016] I c =max(p c ) (2);

[0017] Among them, u c represents the cth feature map in the output channel U, I c The channel length is described as c tensor information, s represents the index of the feature map, a and b represent the positions of the two compression layers of S2E in the correlation matrix, and l represents the number of network layers.

[0018] The compression module can compress the information of the entire graph into one unit through global average pooling, and then compress the entire channel feature map into a c×1 tensor I used to describe the channel, where c is the length of the channel. The excitation module is used to capture the relationship between channel feature maps. The author of the SE module chose two layers of full connection to achieve the above functions;

[0019] A = σ(W2*ReLu(W1*I)) (3);

[0020] Where W1 is the first c×c / r fully connected layer, W2 is the second c×c / r fully connected layer, r (reduction ratio) is a dimensionality reduction factor, σ represents the activation function sigmoid. In order to make the finally learned attention not mutually exclusive, the sigmoid function is selected to implement the activation. Finally, the weight of the attention A generated by the entire operator is multiplied by the initial entire channel feature by feature layer.

[0021] Preferably, in the S2E-ResNet-101 module used in step 3, the stacking numbers of the residual structures corresponding to the 4 layers are 3, 4, 23, and 3. After the 4 layers, the input image has been scaled to one thirty-second of the input image, and then a simple 3-layer upsampling is performed on the feature map with small resolution, the predicted feature map is enlarged to 8 times, and finally compressed to 17 channels and the features are output;

[0022] The attention mechanism is embedded in the tail of the first residual structure of the layer in the S2E-ResNet-101 module, which is used to adjust the channel. It is embedded after the output of the pointwise convolution, and the final attention weight is multiplied by the output of the previous pointwise convolution for the corresponding channel to implement the attention mechanism.

[0023] The last two layers of upsampling use the DUC (Dense Upsampling Convolution) structure. First, a 3x3 convolution with a stride of 1 is performed to double the input channel. After regularization and activation by the activation function, upsampling is performed by a pixel shuffle with a scale of 2. The pixel shuffle method is to fuse pixels at the same position based on adjacent scale square channels.

[0024] Preferably, in order to achieve lightweight network resource consumption, the h-sigmoid function is used to replace the sigmoid function at the end of the two fully connected layers in the excitation operation.

[0025] Preferably, in the S2E model and the ResNet-101 model, the S2E module is embedded after the bottleneck of the 2nd, 3rd, and 4th feature extraction layers in ResNet-101, and the size of the average pooling in the first step of the compression operation is set to half the width and the step size is set to half the size to achieve the best effect.

[0026] Preferably, MSEloss is used to calculate the loss function, and the rmsprop optimizer is used to optimize the training parameters.

[0027] Preferably, in the verification procedure, the model predicts both the heat map of the original image and the heat map of the flipped image, and then takes the average of these two heat maps as the prediction data input for verification.

[0028] The beneficial effects of the present invention are:

[0029] 1. The posture recognition model based on the two-level pooling S2E module proposed in the present invention can significantly improve the prediction accuracy of the entire posture recognition model while considering the computing speed and computing resources, making the algorithm suitable for real-time applications and deployment on hardware with limited resources.

[0030] 2. Accelerate model convergence by establishing a better channel attention mechanism, which can help the network extract features faster from a large number of channel maps and filter out more key feature information.

[0031] 3. In the qualitative comparison experiment of chaotic posture recognition, our model can focus on more detailed areas and avoid misrecognition due to posture intersection and occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the overall improvement idea diagram of this embodiment.

[0033] Figure 2This is the network model diagram of S2E+ResNet101 for posture detection in this embodiment.

[0034] Figure 3 Flow chart of the posture recognition model method based on the progressive two-stage pooling S2E module of this embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the embodiment of the present invention clearer, the following will provide a detailed implementation method and specific operation process of the technical scheme in the embodiment of the present invention in combination with the drawings in the embodiment of the present invention, but the protection scope of the present invention is not limited to this embodiment. The data set used in this experiment is the COCO 2014 data set, which adopts a top-down framework to calculate the position of the key points and the confidence score (confidence score) by estimating the output heat map. The COCO 2014 data set has more than 130k images, in which each person is annotated with 17 key points. More than 2k of the images are used as validation sets for verification, and the remaining images are used as training sets for model training. 500 images are selected for data enhancement through graphic processing, expanded to 10k equivalent images and tested.

[0036] The posture recognition model method based on the progressive two-stage pooling S2E module provided in an embodiment of the present invention first constructs a neural network model based on ResNet-101. The constructed neural network model includes a convolutional pooling processing layer, a residual block, an Instance Normalization layer, a spatial attention module, etc.

[0037] Combination Figure 1 The overall improvement idea diagram of this embodiment is as follows: Figure 2 The network model diagram of S2E+ResNet101 for posture detection in this embodiment is as follows: Figure 3 The flowchart of the posture recognition model method based on the progressive two-stage pooling S2E module of this embodiment, the specific steps of this embodiment are as follows:

[0038] Step 1: Detection of human bounding boxes. YOLOv3-spp is used for target detection, and the pyramid structure two-step framework model of FPN is combined to accurately detect human frame postures of various scales and detect human bounding boxes of frame images in the dataset.

[0039] Step 2: Bounding box redundancy processing. Redundant judgment calculations are performed on the bounding boxes of human postures, and redundant detection boxes are deleted or merged. The size of the boxes for human detection is extracted from the real labels of the coco dataset, and the height and width are corrected to a size of 5:4 based on experience. Then, the boxes of the human body are cropped from the corresponding image, and the resolution of the image is corrected to 320x256, and the redundant parts are filled with 0. In addition, the dataset is expanded through data enhancement operations, including random rotation, random size scaling, and random up and down or left and right flipping of the image, to reduce the risk of overfitting of the model.

[0040] Step 3: Model training. The human body detection frames after removing redundancy are sent to the sppe (single-person pose estimator) model combined with the S2E-ResNet-101 module in turn, and the pose estimation of each human body frame is detected independently; then the 10x8 resolution feature map output by the backbone network is upsampled to 80x64 using DUC. Compared with simple bilinear upsampling, DUC can use channel information to repair more detailed information that is usually missing in the bilinear method.

[0041] Step 4: Generate a key point heat map for posture prediction. Use a convolution with an output channel of 17 to replace the fully connected layer to obtain the key point heat map, then detect the maximum value point in the heat map to directly generate the key point prediction of the human body, and use the value directly as the confidence for the next step of processing.

[0042] In the ResNet101 network structure, the stacking numbers of the residual structures corresponding to the 4 layers are 3, 4, 23, and 3. After the 4 layers, the input image has been scaled to 1 / 32 of the input image, and then a simple 3-layer upsampling is performed on the feature map with small resolution, the predicted feature map is enlarged to 8 times, and finally compressed to 17 channels and the features are output.

[0043] The first layer of compression operation of S2E is a local average pooling operation, which extracts local information of the feature map, so that the scattered feature information can be extracted effectively and fairly. The second layer of compression operation is a global maximum pooling, which can extract the most prominent features in the previous local average pooling. The following is the formula of the compression process used by the S2E module:

[0044]

[0045] I c =max(p c ) (2);

[0046] Among them, u crepresents the cth feature map in the output channel U, I c The channel length is described as c tensor information, s represents the index of the feature map, and a and b represent the positions of the two compression layers of S2E in the correlation matrix respectively.

[0047] The compression module can compress the information of the entire graph into one unit through global average pooling, and then compress the entire channel feature map into a c×1 tensor I used to describe the channel, where c is the length of the channel. The excitation module is used to capture the relationship between channel feature maps. The author of the SE module chose two layers of full connection to achieve the above function.

[0048] A = σ(W2*ReLu(W1*I)) (3);

[0049] Where W1 is the first c×c / r fully connected layer, W2 is the second c×c / r fully connected layer, r (reduction ratio) is a dimensionality reduction factor, and σ represents the activation function sigmoid. In order to make the final learned attention not mutually exclusive, the sigmoid function is selected for activation.

[0050] The attention mechanism is embedded in the tail of the first residual structure of the ResNet network layer, which is mainly used to adjust the channel. That is, it is embedded after the output of the point-wise convolution, and the final attention weight operator is multiplied by the output of the previous point-wise convolution on the corresponding channel feature-by-feature layer to implement the attention mechanism.

[0051] The best effect can be achieved by embedding the S2E module after the bottleneck of the 2nd, 3rd, and 4th feature extraction layers in ResNet-101 and setting the average pooling size of the first step of the compression operation to half the width and the stride to half the size.

[0052] In the first sampling process, pixel shuffle with scale 2 is selected. The DUC (Dense Upsampling Convolution) structure is used for the next two upsampling layers. First, a 3x3 convolution with a step size of 1 is performed to double the input channel. After regularization and activation by the activation function, upsampling is performed by pixel shuffle with scale 2. The pixelshuffle method is to fuse adjacent scale square channels based on pixels in the same position. Compared with general bilinear upsampling, it can fill the lack of resolution information through channel information, so the effect will be better.

[0053] MSEloss is used to calculate the loss function, and the rmsprop optimizer is selected to optimize the training parameters. The rmsprop optimizer uses a gradient inheritance idea to improve the problem of gradient oscillation, and adaptively adjusts the size of the learning rate during the learning process, learning a learning process that first increases and then decreases.

[0054] A convolution with 17 output channels is used to replace the fully connected layer to generate a key point heat map for posture prediction. Then, the maximum value point in the heat map is detected to directly generate the key point prediction of the human body and obtain the confidence value. In the verification procedure, the model predicts the heat map of the original image and the heat map of the flipped image at the same time, and then takes the average of these two heat maps as the prediction data input for verification.

[0055] Step 5: Model evaluation. Use pose-nms

[18] to evaluate possible repeated poses. Pose-nms uses the estimated pose with the highest confidence as the standard each time and evaluates the similarity between the remaining poses and the original pose. Here, not only the similarity of the key points in the same part of the two poses is evaluated, but also the similarity of the key points in different parts but spatially similar. The final similarity score is obtained by multiplying the two by their respective weights. If the evaluation value is higher than the set threshold, they are put into a set. Then, the repeated poses in the set are assigned weights based on the confidence of each point, fused and output, and the process is repeated until all poses are output or fused.

[0056] The general standard for posture estimation is used on the COCO dataset, namely the mAP metric based on OKS, where OKS (object keypoint similarity) defines the similarity between different human postures. The results mainly report mAP (mean Average Precision) and mAR (mean Average Recall) based on different IOU standards, namely: AP (the average of the precision under 10 IOU standards, these 10 values ​​are evenly obtained between 0.50 and 0.95), AP50 (AP score when IOU = 0.50), AP75, AR (calculation standard is similar to AP, calculation of recall) and AR50.

[0057]

[0058] Table 1 Comparison of predictions of SE and S2E models

[0059] Where h and w are the height and width of the input feature map, respectively. It should be noted that the size of the pooling kernel refers to the size of the pooling kernel used in the first compression of the two modules. In fact, the S2E module has a second compression pooling, namely the global maximum pooling. In addition, in S2E-ResNet101, the S2E module is added to the 2nd, 3rd, and 4th layers.

[0060] From the comparison of the above tables, we can find that since the S2E module has a better channel attention mechanism than the SE module, the various data indicators of S2E-ResNet have been greatly improved compared with SE-ResNet. Its mAP has reached 68.2, which is 4.1 mAP percentage points higher than SE-ResNet. Even in the standard indicator of mAP50 with a higher fault tolerance rate, the new model has improved by 4.7 mAP percentage points and mAR by 8.3 percentage points compared with the original model.

[0061] Furthermore, the applicant conducted a comparative experiment:

[0062]

[0063]

[0064] Table 2 Comparative experiment

[0065] From Table 2, we can see that when the input scale is the same, a larger network structure will perform better than a smaller network, but the number of parameters is larger. At the same time, the difference in FLOPs size of different network structures also determines the pros and cons of an algorithm model. Based on the comparative experiments in Table 2, our algorithm can achieve a relatively balanced effect in terms of complexity and accuracy. Compared with large network structures with a large number of parameters, our algorithm occupies less memory and has lower performance requirements for hardware. At the same time, compared with small networks with a small number of parameters, the advantage of our algorithm is reflected in recognition accuracy. Therefore, we can see that our algorithm can balance both accuracy and speed.

[0066] Step 6: Posture recognition visualization output. Each set of predicted key points is connected to generate the human body posture. In this process, key points with low confidence will be discarded. The generated human body posture is added to the original image and finally visualized.

[0067] Furthermore, in order to prove that the present invention has a better effect, the applicant conducted an ablation experiment on the method of the present invention, and compared the final results with mean average precision (mAP) and mean recall (mAR). The experimental description and data are as follows:

[0068]

[0069] Table 3 Ablation experiment

[0070] Baseline represents the SE-ResNet model, and S2E-1, S2E-2, S2E-3, S2E-4, and S2E-5 are based on the S2E-ResNet model, respectively. Different pooling kernel sizes, strides, activation functions, and S2E modules are selected to add to different layers for experimental comparison. The final result is that we can get the most outstanding effect by embedding the S2E module after the bottleneck of the 2nd, 3rd, and 4th feature extraction layers in ResNet-101, and embedding a simple SE module only in the 1st layer. Among them, the average pooling size of the first step of the compression operation is set to half the width and the stride is set to half the size to achieve the best effect.

[0071] In order to apply it on a lightweight network, considering that the general mobile device has a large resource overhead on the SIGMOID activation function, the end of the two fully connected layers in the excitation operation uses a similar H-SIGMOID function to replace the original SIGMOID function, and observe the final effect. The fifth model in the above table is the effect of enabling H-SIGMOID. It can be seen that compared with the previous SIGMOID activation function, H-SIGMOID as the final activation function can still maintain a relatively good level in the end, and can be applied in lightweight networks that are more demanding on resource overhead. It can be seen that the posture recognition model based on the progressive two-level pooling S2E module proposed in this paper can help the network extract features from a large number of channel graphs more quickly and filter out more critical feature information, so that the network model can correctly recognize posture information.

[0072] In order to further verify the effectiveness of the posture recognition method, SE-ResNet-101 and S2E-ResNet-101 are selected as the comparative experiment of single-person posture estimation predictor. The comparison results of this embodiment and the real-time monitoring of human skeleton points in the image in the data set based on the SE attention module can be seen in the qualitative comparison experiment of chaotic posture recognition. Due to the crossing action of the legs of the running human body, the two legs in the image block each other, and there are visual errors in the thigh and calf. Based on the method of the present invention, it is possible to focus on more detail areas and avoid misidentification caused by reasons such as posture crossing and occlusion. For images with multiple actions, the skeleton point detection of the present invention is more accurate.

[0073] As described above, the posture recognition method based on the S2E attention mechanism provided by the present invention is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all disclosed features, or all steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A posture recognition method based on a two-stage pooling S2E module, characterized in that: The following steps are involved: Step 1: Detect the human body bounding box of the input image, select YOLOv3-spp for target detection, and combine the pyramid structure two-step framework model of FPN to detect the human body frame posture of various scales; Step 2: Perform redundant judgment calculations on the bounding boxes of human postures and delete or merge redundant detection boxes. Step 3: Scale a series of human detection boxes to 320x256, fill the blank parts with 0, and send the processed human detection boxes to the sppe (single-person poseestimator) model combined with the S2E-ResNet-101 module in turn to independently perform pose estimation on each human box; then use DUC to upsample the 10x8 resolution feature map output by the backbone network to 80x64; S2E (Two-step Squeeze and Excitation) attention mechanism with two layers of compression and an excitation module; The first layer of compression operation of S2E is the local average pooling operation, which extracts the local information of the feature map and extracts the scattered feature information effectively and fairly. The second layer of compression operation is a global maximum pooling operation, which extracts the most prominent features in the previous local average pooling. The following is the formula of the compression process used by the S2E module: I c = max(p c ) (2); Among them, u c represents the cth feature map in the output channel U, I c The channel length is described as c tensor information, s represents the index of the feature map, a and b represent the positions of the two compression layers of S2E in the correlation matrix, and l represents the number of network layers; The compression module can compress the information of the entire graph into one unit through global average pooling, and then compress the entire channel feature map into a c×1 tensor I used to describe the channel, where c is the length of the channel. The excitation module is used to capture the relationship between channel feature maps. The SE module uses two layers of full connection to achieve this function; A = σ(W2*ReLu(W1*I)) (3); Where W1 is the first c×c / r fully connected layer, W2 is the second c×c / r fully connected layer, r (reduction ratio) is a dimensionality reduction factor, σ represents the activation function sigmoid. In order to make the finally learned attention not mutually exclusive, the sigmoid function is selected to realize the activation. Finally, the weight of the attention A generated by the entire operator is multiplied by the initial entire channel feature by feature layer. Step 4: Generate a key point heat map for posture prediction. Use a convolution with an output channel of 17 to replace the fully connected layer to obtain the key point heat map. Then detect the maximum value point in the heat map to directly generate the key point prediction of the human body, and use the value directly as the confidence for the next step of processing. Step 5: Pose-nms is used to evaluate the possible repeated poses. Pose-nms uses the estimated pose with the highest confidence as the standard each time to evaluate the similarity between the remaining poses and the original pose. When evaluating the similarity, it includes the similarity of the key points in the same part of the two poses, as well as the similarity of the key points in different parts but similar in space. The final similarity score is obtained by multiplying the two by their respective weights. If the evaluation value is higher than the set threshold, they are put into a set. Then, the repeated poses in a set are weighted based on the confidence of each point, fused and output. This process is repeated until all poses are output or fused. Step 6: Posture recognition visualization output. Each set of predicted key points is connected to generate the human body posture. In this process, the key points with low confidence will be discarded. The generated human body posture is added to the original image and finally visualized.

2. The posture recognition method based on the two-stage pooling S2E module according to claim 1 is characterized in that: In the S2E-ResNet-101 module used in step 3, the stacking numbers of the residual structures corresponding to the 4 layers are 3, 4, 23, and 3. After the 4 layers, the input image has been scaled to one thirty-second of the input image, and then a simple 3-layer upsampling is performed on the feature map with small resolution, the predicted feature map is enlarged to 8 times, and finally compressed to 17 channels and the features are output; The attention mechanism is embedded in the tail of the first residual structure of the layer in the S2E-ResNet-101 module, which is used to adjust the channel. It is embedded after the output of the pointwise convolution, and the final attention weight is multiplied by the output of the previous pointwise convolution for the corresponding channel to implement the attention mechanism. The last two layers of upsampling use the DUC (Dense Upsampling Convolution) structure. First, a 3x3 convolution with a stride of 1 is performed to double the input channel. After regularization and activation by the activation function, upsampling is performed by a pixel shuffle with a scale of 2. The pixel shuffle method is to fuse pixels at the same position based on adjacent scale square channels.

3. The posture recognition method based on the two-stage pooling S2E module according to claim 2 is characterized in that: In order to achieve lightweight network resource consumption, the h-sigmoid function is used to replace the sigmoid function at the end of the two fully connected layers in the excitation operation.

4. The posture recognition method based on the two-stage pooling S2E module according to claim 3 is characterized in that: In the S2E model and ResNet-101 model, the S2E module is embedded after the bottleneck of the 2nd, 3rd, and 4th feature extraction layers in ResNet-101. At the same time, the average pooling size of the first step of the compression operation is set to half the width and the step size is set to half the size to achieve the best effect.

5. The posture recognition method based on the two-stage pooling S2E module according to claim 4 is characterized in that: MSEloss is used to calculate the loss function, and the rmsprop optimizer is used to optimize the training parameters.

6. The posture recognition method based on the two-stage pooling S2E module according to claim 5 is characterized in that: In the validation procedure, the model predicted the heatmap of the original image and the heatmap of the flipped image at the same time, and then took the average of these two heatmaps as the prediction data input for validation.

Images