A video target state recognition model training method and a recognition method

Abstract

This invention discloses a training method and recognition method for a video target state recognition model. The model training method includes: inputting image frames into a feature extraction module to obtain high-level features; upsampling the high-level features through a spatial inference module and training them against saliency labels, calculating a first loss function, and training until the first loss function converges to a preset level; sequentially inputting the high-level features into a ConvLSTM network, a fully connected layer, and a Softmax layer in a temporal inference module to obtain the predicted state of the current target, comparing it against state labels, training, calculating a second loss function, and training until the second loss function converges to a preset level, thereby obtaining a video target state recognition model. This recognition model enables the recognition of target states in video images. The method combining temporal and spatial inference achieves high state recognition accuracy and realizes automated feature event detection of video image sequences.

Description

Technical Field

[0001] This invention belongs to the field of video image data processing technology, and more specifically, relates to a method for training a recognition model and a recognition method for video target states. Background Technology

[0002] Image sequences are records of a target's actions and states over a continuous period of time. They possess continuity in both time and space, and the target exhibits different spatiotemporal characteristics in different states. Leveraging this characteristic of images, artificial intelligence techniques can be used to construct deep learning models for images, enabling the detection and recognition of keyframe states within image sequences. This approach facilitates automated processing and analysis of image data and can be extended to various near-real-time applications to achieve rapid and automated target state assessment.

[0003] However, current video keyframe state recognition primarily focuses on motion state recognition in videos containing people, with less research on object state recognition. In videos containing people, human pose detection models are often used to capture human motion information, such as in applications like pedestrian gait recognition, where the pedestrian target occupies almost the entire image frame. However, when recognizing object keyframe states, keypoint information cannot be used as an aid, and in some video images, the target only occupies a small proportion of the image area. Therefore, the accuracy of keyframe state detection is not high for image sequences where the target occupies a small proportion and the background area is large. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a video target state recognition model training method and recognition method, the purpose of which is to improve the accuracy of video target state recognition.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for training a video target state recognition model is provided, comprising:

[0006] Input consecutive image frames from the training set into the feature extraction module to obtain high-level features;

[0007] The high-level features are upsampled by the spatial reasoning module and compared with the saliency labels for training. The first loss function is calculated and trained to converge the first loss function to a preset degree.

[0008] The high-level features are input into the temporal inference module, which includes a ConvLSTM network, a fully connected layer, and a Softmax layer. After the high-level features are sequentially input into the ConvLSTM network, the fully connected layer, and the Softmax layer, the predicted state of the current target is obtained and compared with the state label for training. A second loss function is calculated and trained to converge the second loss function to a preset degree to obtain a video target state recognition model.

[0009] Preferably, the image frame input to the feature extraction module is a preprocessed image frame, and the preprocessing process is as follows:

[0010] Save the bitmap data entity of the bmp format file as a 16-bit unsigned integer x;

[0011] Using the first conversion formula The 16-bit unsigned integer x is normalized to obtain a normalized integer x', where max(x) represents the maximum value of the 16-bit unsigned integer x;

[0012] The normalized integer x' is converted into an 8-bit unsigned integer y, and the second conversion formula is used. Perform a nonlinear transformation on the 8-bit unsigned integer y, and save the transformation result z as a PNG file.

[0013] Preferably, the feature extraction module includes five convolutional layers of a VGG-16 network. After the input image frame passes through the five convolutional layers of the VGG-16 network, two high-level features are output by the fifth layer of the VGG-16 network.

[0014] Preferably, the spatial reasoning module includes four deconvolutional layers and one convolutional layer. The high-level features input to the spatial reasoning module are first upsampled through four deconvolutional layers, and then passed through one convolutional layer to obtain a salient pixel image before being output. The output image is then compared with the salient labels for training.

[0015] Preferably, the spatial reasoning module further includes a Sigmoid activation layer, and after obtaining a salient pixel map through one convolutional layer, it further includes performing Sigmoid activation on the salient pixel map and then outputting it.

[0016] Preferably, the ConvLSTM network includes memory gates, forget gates, and output gates.

[0017] Preferably, the probability of each state is returned after passing through the Softmax layer, and the state with the highest probability is taken as the state of the current frame.

[0018] According to another aspect of the present invention, a method for recognizing the state of a video target is provided, comprising:

[0019] A recognition model for the state of a video target is obtained, wherein the recognition model is obtained by training the recognition model for the state of a video target according to any one of the above-mentioned methods;

[0020] The image frame is input into the recognition model, and the state recognition result is output after passing through the feature extraction module and the time inference module of the recognition model.

[0021] Preferably, before inputting the image frame into the recognition model, the image frame is further preprocessed, and the preprocessing process is as follows:

[0022] Save the bitmap data entity of the bmp format file as a 16-bit unsigned integer x;

[0023] Using the first conversion formula The 16-bit unsigned integer x is normalized to obtain a normalized integer x', where max(x) represents the maximum value of the 16-bit unsigned integer x;

[0024] The normalized integer x' is converted into an 8-bit unsigned integer y, and the second conversion formula is used. Perform a nonlinear transformation on the 8-bit unsigned integer y, and save the transformation result z as a PNG file.

[0025] In summary, this invention fully utilizes the different spatial features of the target under different states to construct a spatial reasoning module. This module effectively expands the target region and enhances the network's learning effect on the target region, enabling faster convergence of network parameters representing the image background. Simultaneously, ConvLSTM is used to memorize the temporal and spatial information of historical image frames, storing and remembering video frame features over a period of time to construct a temporal reasoning module. The combination of temporal and spatial reasoning achieves high state recognition accuracy and realizes automated feature event detection of video image sequences. Attached Figure Description

[0026] Figure 1 This is a system network architecture diagram during model training in one embodiment of this application;

[0027] Figure 2 This is a flowchart of the steps for model training in one embodiment of this application;

[0028] Figure 3 This is a diagram illustrating the processing of feature maps by the spatial reasoning module in one embodiment of this application;

[0029] Figure 4 This is a diagram illustrating the processing of the feature map by the time inference module in one embodiment of this application;

[0030] Figure 5 This is a flowchart illustrating a method for recognizing the state of a video target in one embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0032] like Figure 1 This is a system network architecture diagram for model training in one embodiment of this application. The system network architecture diagram mainly includes a feature extraction module, a spatial inference module, and a temporal inference module. For example... Figure 2 As shown, the model training method of this application is based on this model and performs the following training:

[0033] Step S100: Input consecutive image frames from the training set into the feature extraction module to obtain high-level features.

[0034] The feature extraction module can focus on high-level representations of image features. Since background features have similar representations and the background occupies a large proportion of the image, the module used for high-level semantic representation easily reaches a stable state. Once stable, the high-level representation module only makes minor weight updates depending on the target state, thus avoiding competition between the high-level semantic representation module and the inference module to adjust network weights, ensuring the model's ability to concentrate on state classification.

[0035] When preparing the training set, since two training sessions are required—one for image saliency extraction and the other for state recognition—the images in the training set must have both saliency labels and state labels.

[0036] In one embodiment, when the original image is in BMP format, preprocessing is required before inputting it to the feature extraction module to convert the image format. Specifically, this preprocessing includes:

[0037] Step S1: Save the bitmap data entity of the bmp format file as a 16-bit unsigned integer x.

[0038] Specifically, analyzing the BMP file structure, the file header occupies 14 bits, the bitmap information header occupies 40 bits, and ignoring the palette information, the bitmap data entity starts from the 55th bit of the BMP data. The BMP file is unpacked to obtain the necessary information, and then the specific image data of the bitmap is read based on its width, height, and other information. After successfully reading the image data entity 'x' from the BMP file, the 14-bit data is saved as a 16-bit unsigned integer closest to its bit length.

[0039] Step S2: Using the first conversion formula Normalize the 16-bit unsigned integer x to obtain the normalized integer x', where max(x) represents the maximum value of the 16-bit unsigned integer x.

[0040] The 16-bit unsigned integer is normalized to the range (0, 1) through normalization.

[0041] Step S3: Convert the normalized integer x' to an 8-bit unsigned integer y, and use the second conversion formula. Perform a nonlinear transformation on an 8-bit unsigned integer y, and save the transformation result z as a PNG file.

[0042] Each sampled pixel value in an 8-bit image is between [0, 255], containing 256 levels of grayscale. The 8-bit data undergoes a non-linear transformation to enhance the visual effect of the target.

[0043] In this embodiment, the 16-bit bitmap data entity of the BMP format file is normalized and non-linearly transformed, and saved as an 8-bit PNG format file. The normalization and non-linear transformation process can effectively improve the image balance and avoid image display problems. The content of the 16-bit BMP image can be fully displayed using conventional equipment for human visual observation.

[0044] In one embodiment, a base model trained on the ImageNet dataset using VGG-16 is further trained. The resolution of the input image needs to be reduced to 384 pixels high and 480 pixels wide, resulting in a height-to-width ratio of 0.8.

[0045] In one embodiment, the feature extraction module includes five convolutional layers of a VGG-16 network. The input image frame passes through the five convolutional layers of the VGG-16 network, the pooling layer of the fifth layer is removed, and the fifth layer outputs two high-level features.

[0046] After extracting high-level features, the process also includes:

[0047] Step S200: Input the high-level features into the spatial inference module and the temporal inference module respectively for training.

[0048] The training processes for the spatial reasoning module and the temporal reasoning module can be carried out simultaneously.

[0049] The training process for spatial reasoning is as follows:

[0050] Step S210: Upsample the high-level features through the spatial reasoning module and train them by comparing them with the saliency labels. Calculate the first loss function and train it to converge the first loss function to a preset level.

[0051] The spatial reasoning module upsamples the feature maps extracted by the feature extraction module. Utilizing the spatial reasoning capabilities of convolution, it trains by comparing the feature maps with salient feature maps, using this as an image mask to enlarge the target region and enhance the learning effect.

[0052] In one embodiment, the spatial reasoning module includes four deconvolutional layers and one convolutional layer. For example... Figure 3 As shown, the spatial reasoning training process specifically includes:

[0053] Step S211: Upsample the high-level features using four isomorphic deconvolution layers.

[0054] Specifically, the kernel function used for the decoupling operation has a space size of 4×4.

[0055] Step S212: Use a 1×1 convolution to obtain a salient pixel map with the same spatial dimensions as the input image.

[0056] Step S213: Obtain the output image by performing Sigmoid activation on the salient pixel image.

[0057] After obtaining the output image, calculate the first loss function between the output image and the saliency label. Through multiple training iterations, the first loss function is converged to a preset level.

[0058] Overall, the feature extraction module and the spatial inference network form an encoder and decoder pair. Supervised training of the spatial inference network enhances the feature-extracting encoder network's ability to analyze salient regions of the image. Through spatial inference training, the output map is supervised by the density maps of the labeled target and environment regions during training. This constrains the temporal inference module to primarily expand according to the target and target environment regions, focusing the model's attention on object regions rather than the entire background area. This helps the temporal inference module better infer the current state of the video frame.

[0059] The training process for time-based reasoning is as follows:

[0060] Step S220: Input the high-level features sequentially into the ConvLSTM network, fully connected layer and Softmax layer of the temporal inference module to obtain the predicted state of the current target and compare it with the state label for training. Calculate the second loss function and train it to converge the second loss function to a preset level to obtain the video target state recognition model.

[0061] The ConvLSTM (Long Short-Term Memory) network learns by handling the long-range dependencies and parameter sharing of recurrent neural networks, forming a kind of learning memory. It uses the Tanh hyperbolic tangent activation function, and its input is the output of the feature extraction module, with 512 feature channels. The two fully connected layers are mainly used to flatten the ConvLSTM output to obtain a vector with the same dimension as the final classification value, corresponding to the key frame state to be identified. The softmax layer is used to obtain the probability of the current target's motion state being in each state.

[0062] In one specific embodiment, such as Figure 4 As shown, the time-based reasoning training process specifically includes:

[0063] Step S221: Obtain high-level features with 512 feature channels, and output them after passing through the memory gate, forget gate, and output gate in the ConvLSTM module.

[0064] The memory gate in the ConvLSTM network is used to remember spatial features relevant to spatiotemporal reasoning, while the forget gate is used to forget spatial features less relevant to spatiotemporal reasoning, based on their importance. After the input features pass through the memory gate, forget gate, and output gate in the ConvLSTM network, the number of channels in the output features remains unchanged.

[0065] Step S222: Flatten the features output by ConvLSTM to obtain a one-dimensional vector with 92×160 elements.

[0066] Step S223: After passing through the first fully connected layer, the vector becomes a vector with 32 elements.

[0067] Step S224: After passing through the second fully connected layer, a 6-dimensional vector is obtained to correspond to the 6 key frame states that need to be identified.

[0068] Step S224: After passing through a Softmax layer, obtain the probabilities of various states.

[0069] The state with the highest probability is used as the predicted state of the current target. The second loss function between the predicted state and the state label is calculated. The training is completed by training the second loss function to converge to a preset level, and the recognition model of the video target state is obtained.

[0070] This application also designs a method for recognizing the state of a video target, such as... Figure 5 As shown, it includes:

[0071] A recognition model for the state of a video target is obtained, which is obtained by training any of the above-mentioned video target state recognition model training methods;

[0072] The image frame is input into the recognition model, and the state recognition result is output after passing through the feature extraction module and the time inference module of the recognition model.

[0073] In one specific embodiment, before inputting the image frame into the recognition model, the image frame is further preprocessed. The preprocessing process is as follows:

[0074] Save the bitmap data entity of the bmp format file as a 16-bit unsigned integer x;

[0075] Using the first conversion formula The 16-bit unsigned integer x is normalized to obtain the normalized integer x', where max(x) represents the maximum value of the 16-bit unsigned integer x;

[0076] Convert the normalized integer x' to an 8-bit unsigned integer y, and then use the second conversion formula. Perform a nonlinear transformation on an 8-bit unsigned integer y, and save the transformation result z as a PNG file.

[0077] It should be noted that although the spatial reasoning module participates in the training of the model, it does not participate in the recognition process when using the trained model to recognize the state.

[0078] In one specific embodiment, the identification process includes:

[0079] Step S310: Based on the location of the test folder, read several consecutive frames of images and perform appropriate preprocessing.

[0080] Step S320: Load the trained video target state recognition model.

[0081] Step S330: Input several consecutive image frames into the model for prediction, and output the final result of keyframe state recognition. Specifically, each image returns a vector representing the probability of each state, and the state with the highest probability is taken as the state of the current frame.

[0082] This invention fully utilizes the different spatial characteristics of a target under different states to construct a spatial reasoning module. This module effectively expands the target region and enhances the network's learning effect on the target region, enabling faster convergence of network parameters representing the image background. Simultaneously, ConvLSTM is used to memorize the temporal and spatial information of historical image frames, storing and remembering video frame features over a period of time to construct a temporal reasoning module. The combination of temporal and spatial reasoning achieves high state recognition accuracy, realizing automated feature event detection of video image sequences. The state recognition method for image keyframes designed in this invention has been successfully applied, achieving good practical results and solving the problems of small targets, large background regions, high noise, and difficulties in analysis and mining in video images.

[0083] Those skilled in the art will readily understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a video target state recognition model, characterized in that, include: Input consecutive image frames from the training set into the feature extraction module to obtain high-level features; The high-level features are upsampled by the spatial reasoning module and compared with the saliency labels for training. The first loss function is calculated and trained to converge the first loss function to a preset degree. The high-level features are input into the temporal inference module, which includes a ConvLSTM network, a fully connected layer, and a Softmax layer. After the high-level features are sequentially input into the ConvLSTM network, the fully connected layer, and the Softmax layer, the predicted state of the current target is obtained and compared with the state label for training. The second loss function is calculated and trained to converge the second loss function to a preset degree to obtain the video target state recognition model. The image frames input to the feature extraction module are preprocessed image frames, and the preprocessing process is as follows: Save the bitmap data entity of the bmp format file as a 16-bit unsigned integer x; Using the first conversion formula The 16-bit unsigned integer x is normalized to obtain a normalized integer x', where max(x) represents the maximum value of the 16-bit unsigned integer x; The normalized integer x' is converted into an 8-bit unsigned integer y, and the second conversion formula is used. Perform a nonlinear transformation on the 8-bit unsigned integer y, and save the transformation result z as a PNG file.

2. The video target state recognition model training method as described in claim 1, characterized in that, The feature extraction module includes five convolutional layers of the VGG-16 network. After the input image frame passes through the five convolutional layers of the VGG-16 network, the fifth layer of the VGG-16 outputs two high-level features.

3. The video target state recognition model training method as described in claim 1, characterized in that, The spatial reasoning module includes four deconvolutional layers and one convolutional layer. The high-level features input to the spatial reasoning module are first upsampled through four deconvolutional layers, and then passed through one convolutional layer to obtain a salient pixel image before being output. The output image is then compared with the salient labels for training.

4. The video target state recognition model training method as described in claim 3, characterized in that, The spatial reasoning module also includes a Sigmoid activation layer. After obtaining a salient pixel map through one convolutional layer, the module further includes Sigmoid activation of the salient pixel map before outputting it.

5. The video target state recognition model training method as described in claim 1, characterized in that, The ConvLSTM network includes memory gates, forget gates, and output gates.

6. The video target state recognition model training method as described in claim 1, characterized in that, After passing through the Softmax layer, the probabilities of each state are returned, and the state with the highest probability is taken as the state of the current frame.

7. A method for recognizing the state of a video target, characterized in that, include: A recognition model for the state of a video target is obtained, wherein the recognition model is obtained by the video target state recognition model training method according to any one of claims 1 to 6; The image frame is input into the recognition model, and the state recognition result is output after passing through the feature extraction module and the time inference module of the recognition model. Before inputting the image frame into the recognition model, the image frame is preprocessed. The preprocessing process is as follows: Save the bitmap data entity of the bmp format file as a 16-bit unsigned integer x; Using the first conversion formula The 16-bit unsigned integer x is normalized to obtain a normalized integer x', where max(x) represents the maximum value of the 16-bit unsigned integer x; The normalized integer x' is converted into an 8-bit unsigned integer y, and the second conversion formula is used. Perform a nonlinear transformation on the 8-bit unsigned integer y, and save the transformation result z as a PNG file.

Images