A target recognition method based on multi-dimensional sequence features

Abstract

The present invention provides an object recognition method based on multi-dimensional sequence features. The method first obtains continuous video frame images from a video source; then uses a deep learning object detection algorithm to detect the specific position of a preselected object in the current frame. A new detected object directly creates a tracker N. Secondly, during the tracking process, a normalized region of interest (ROI) is cropped, and the CLIP network is used to extract the image under the ROI, as well as extract the text features of the description texts of positive and negative samples pre-prepared manually, and update the feature sequence of the image text in the object tracker N. After processing, a dynamic change sequence feature map of the image is generated, and the dynamic and static feature sequences are cached in the tracker N. Finally, the recognition result is output through a comprehensive judgment method of voting by raising hands. The present invention realizes the recognition of objects, solves problems such as a large number of false detections and missed detections due to problems such as few samples and difficult samples during the recognition process, and improves and optimizes the accuracy of the object.

Description

Technical Field

[0001] The present invention relates to the field of video signal processing, and particularly to an object recognition method based on multi-dimensional sequence features. Background Art

[0002] In today's society, the related technologies of artificial intelligence are applied in various industries. To apply some advanced algorithms and experiences to their own related businesses, a large amount of high-quality application scenario data is the key basis for training the accuracy of algorithms. With the development of the Internet and the Internet of Things, the amount of data generated and stored by them has increased sharply, providing a data basis for training models of deep learning algorithms. With the development of digitalization and intelligentization, more and more scenarios require customized AI applications to meet the intelligent video surveillance systems in different scenarios. For example, in various scenarios such as gas stations, kitchens, construction sites, warehouses, parks, shopping malls, retail stores, etc., there are those for monitoring human faces and figures, safety helmets, chef hats, smoking, leaving posts, fireworks, and so on. In particular, the object detection algorithms implemented using convolutional neural networks have been widely applied in various computer vision applications.

[0003] In the industry, common object recognition methods include image classification networks such as Resnet, MobileNet, and EfficientNet, which are specifically used to identify which class a certain image belongs to. However, identifying objects from single-network static images, especially when there are few samples and complex scenarios, there will be a large number of misdetection problems. Augmenting data can alleviate the problems, but it cannot meet the requirements for implementation, because when changing to a new scenario, problems such as complex light and a large amount of complex background will still occur. Summary of the Invention

[0004] In order to make up for the deficiencies of the prior art, the present invention provides an object recognition method based on multi-dimensional sequence features.

[0005] The present invention is implemented through the following technical solutions: An object recognition method based on multi-dimensional sequence features, characterized in that: the method includes the following steps:

[0006] Step 1, obtain video frames after decoding from a video source, and then use the single-stage Yolov5 object detection algorithm to detect in the current frame image, and obtain the specific positions of preselected objects, and obtain the labels and coordinate frames of the objects;

[0007] Step 2, directly create a tracker N using the newly detected objects obtained in Step 1. If it is an object that has been tracked, update the tracker N, and continuously track the object until 3 seconds after the object disappears from the frame image, and then end the tracking;

[0008] Step 3, during the tracking process, extract the normalized region of interest (ROI) from the preselected target position to ensure that the ROI sizes of the front and rear frames are consistent;

[0009] Step 4, use the CLIP network to extract the image features under the ROI obtained in Step 3, and extract the text features of the description texts for positive and negative samples pre-prepared manually, and update the feature sequence of the image text in the target tracker N;

[0010] Step 5, obtain the ROI region of the current frame, cache the image of the current ROI into the tracker, and at the same time compare it with the image cached in the previous frame in the tracker N to calculate the difference change between frames and generate a feature map of the image dynamic change sequence;

[0011] Step 6, cache the dynamic change metric value and the static recognition result into the tracker N. Once the target tracking is lost for more than 3 s, it is considered that the target has left. Obtain the dynamic and static sequences from the tracker N and send them to a comprehensive judgment device. Through the integrated comprehensive judgment method, output the recognition result; at the same time, delete the data of the tracker N.

[0012] Further, in Step 2, use the SORT algorithm to track the target. According to the coordinate information of the target obtained in Step 1, calculate the IoU (intersection over union) with the position coordinates in the already created tracker. If the intersection over union is greater than 0.3, it is considered that the newly detected target is the same as the already tracked target, and update it to the tracker N with the existing detected target information.

[0013] Further, in Step 3, the specific steps for extracting the normalized region of interest (ROI) are as follows: take the maximum side of the detection target position box and the center position of the target to re-obtain the ROI. The center point remains unchanged, and the side length is 2 times the maximum side of the target box. Expand it outward to obtain a square ROI region, and then resize it to 224x224 to unify the size.

[0014] Further, if it exceeds the frame boundary during the outward expansion, the exceeded part is filled with white.

[0015] Further, in Step 4, the CLIP network consists of a text feature extraction network and an image feature extraction network. The former is used to represent the features of the target label text description, and the latter represents the features of the target image.

[0016] Further, the CLIP network is used to extract image features from the normalized target. Meanwhile, according to the actual scenario, for the text descriptions of positive samples, few-shot, and zero-shot negative samples, the CLIP network is used to extract the text features of these text description classes. Then, the similarity between the image features and each class is calculated, and the similarity matrix is normalized using softmax to obtain the probability distribution. The maximum value is taken as the classification label, and the recognition result is used to update the picture text recognition sequence in the target tracker N.

[0017] Further, in step 5, the steps to calculate the change in the difference between frames are as follows: First, calculate the grayscale of the ROI region, then use Sobel to calculate the gradient. Subtract the gradient map of the current frame directly from the gradient map of the previous frame. Through the minimum threshold method, the similar points below the threshold are directly set to 0, and then the mean value of the difference map is calculated to obtain the change metric value between the current frame and the previous frame.

[0018] Further, in step 6, the method for calculating the recognition result is as follows:

[0019] S1: Calculate the standard deviation of the dynamic change amount of the sequence to generally describe the dynamic change M of the sequence.

[0020] S2: For the recognition result sequence, through a show of hands voting method, the one with the most votes is the target recognition result: R.

[0021] S3: According to the actual dynamic features of the recognized target, obtain the threshold X of the dynamic features. If M>X, output the recognition result R externally.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. The multi-modal feature target recognition method proposed by the present invention integrates the spatial, color, texture and other features of static pictures, the temporal features, image dynamic change features in multi-frame sequence pictures, the text features of label descriptions, and the association features between people and target objects to achieve target recognition, and solves the problems of a large number of false detections and missed detections due to few samples, difficult samples, etc. in the recognition process.

[0024] 2. The present invention uses the inter-frame difference of consecutive frames to obtain the dynamic temporal change features of the target, improving and optimizing the accuracy of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings.

[0026] Figure 1 It is a flowchart of the target recognition method based on multi-dimensional sequence features of the present invention;

[0027] Figure 2 Structural schematic diagram of the normalized region of interest extracted by the present invention. Detailed implementation manners

[0028] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0029] It should be noted that in the drawings or the description of the specification, similar or identical parts are all denoted by the same reference numerals. The implementation manners illustrated or described in the drawings are in the forms known to those of ordinary skill in the art to which the present invention pertains. It should be pointed out that the described embodiments are merely for illustrative purposes and are not intended to limit the present invention.

[0030] Figure 1 is a flowchart of the object recognition method based on multi-dimensional sequence features according to the present invention. As Figure 1 shown, the object recognition method based on multi-dimensional sequence features includes the following steps:

[0031] Step 1: Decode the video frames from a video file or a video source such as a camera, and then use the single-stage Yolov5 object detection algorithm to detect in the current frame image, and obtain the specific positions of the preselected objects, so as to obtain the labels and coordinate frames of the objects.

[0032] Step 2: Use the newly detected object obtained in Step 1 to directly create a tracker N. If it is an object that has been tracked, update the tracker N, and continuously track the object until 3 seconds after the object disappears from the frame image, and then end the tracking.

[0033] Specifically, use the SORT algorithm to track the object. According to the coordinate information of the object obtained in Step 1, calculate the IoU (intersection over union) with the position coordinates in the existing tracker. If the intersection over union is greater than 0.3, it is considered that the newly detected object and the tracked object are the same object, and update the existing detection object information to the tracker N.

[0034] Step 3: During the tracking process, extract the normalized region of interest (ROI) from the preselected object position to ensure that the ROI sizes of the front and rear frames are consistent.

[0035] As Figure 2 shown, the specific method is as follows: Take the maximum side of the detection object position box and the center position of the object to re-obtain the ROI. The center point remains unchanged, and the side length is 2 times the maximum side of the object box, and expand it outward to obtain a square ROI region, and then resize it to 224x224 to unify the size. If it exceeds the frame image boundary during the outward expansion, the exceeded part is filled with white.

[0036] Step 4: Use the CLIP network to extract the image features under the ROI obtained in Step 3, and extract the text features of the descriptive texts for positive and negative samples pre-prepared manually, and update the feature sequence of the image text in the target tracker N.

[0037] In this implementation case, the CLIP network consists of a text feature extraction network and an image feature extraction network. The former is used to represent the features of the target label text description, and the latter represents the features of the target image. Use the CLIP network to extract image features from the normalized target. At the same time, according to the actual scenario, for the text descriptions of positive samples, few-shot, and zero-shot negative samples, use the CLIP network to obtain the text features of these descriptive texts, then calculate the similarity between the image features and each class, and use softmax to normalize the similarity matrix to obtain a probability distribution. Take the maximum value as the classification label, and update the image text recognition sequence in the target tracker N with the recognition result.

[0038] When using CLIP to extract image features, the text features are extracted from the descriptive texts for positive and negative samples pre-prepared manually through the CLIP network.

[0039] Step 5: Obtain the ROI region of the current frame, cache the image of the current ROI into the tracker, and at the same time compare it with the image cached in the previous frame in the tracker N to calculate the frame-to-frame difference change and generate a feature map of the image dynamic change sequence.

[0040] The steps to calculate the frame-to-frame difference change are as follows: First, calculate the grayscale of the ROI region, then use Sobel to calculate the gradient, directly subtract the gradient map from the gradient map of the previous frame, and use the minimum threshold method to directly set the similar points below the threshold to 0, and then calculate the mean of the difference map to obtain the change metric value between the current frame and the previous frame.

[0041] Step 6: Cache the dynamic change metric value and the static recognition result into the tracker N. Once the target tracking is lost for more than 3s, it is considered that the target has left. Obtain the dynamic and static sequences from the tracker N and send them to a comprehensive judgment device, and output the recognition result through the integrated comprehensive judgment method; at the same time, delete the data in the tracker N.

[0042] The method for calculating the recognition result is as follows:

[0043] S1: Calculate the standard deviation of the dynamic change amount of the sequence to generally describe the dynamic change M of the sequence

[0044] S2: For the recognition result sequence, through the method of taking a vote, the one with the most votes is the target recognition result: R

[0045] S3: Obtain the threshold X of the dynamic feature based on the actually recognized target dynamic feature. If M>X, output the recognition result R externally.

[0046] After the present invention uses the detection network to obtain the ROI region, it then uses the CLIP network to extract the image features of this region, as well as extract the text features of different types of text descriptions for the positive and negative samples in this scenario, calculate the similarity matrix, and achieve target recognition. Among them, different types of text descriptions will be described from different dimensions according to the actual scenario, and during the recognition process, manual intervention is carried out to dynamically update the description text. Use the inter-frame difference of consecutive frames to obtain the dynamic temporal change features of the target and improve the accuracy of the optimized target.

[0047] The specific embodiments described above have further elaborated on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A target recognition method based on multi-dimensional sequence features, characterized in that: The method includes the following steps: Step 1: Obtain video frames after decoding from a video source, then use the single-stage Yolov5 object detection algorithm to detect in the current frame image, and obtain the specific positions of preselected objects, obtaining the labels and coordinate boxes of the objects; Step 2: Use the newly detected objects obtained in Step 1 to directly create a tracker N. If it is an already tracked object, update the tracker N, and continuously track this object until 3 seconds after the object disappears from the frame image, then end the tracking; Step 3: During the tracking process, extract the normalized region of interest ROI from the preselected object positions, ensuring that the ROI sizes of consecutive frames are consistent; Step 4: Use the CLIP network to extract the image features under the ROI, as well as extract the text features of the descriptive text for positive and negative samples pre-prepared manually, and update the feature sequence of the image text in the target tracker N; among them, the CLIP network consists of a text feature extraction network and an image feature extraction network. The former is used to represent the features of the target label text description, and the latter represents the features of the target image; Use the CLIP network to extract image features from the normalized target. At the same time, according to the actual scenario, for the text descriptions of positive samples, few-shot, and zero-shot negative samples, use the CLIP network to extract the text features of these text descriptions, then calculate the similarity between the image features and each class, and use softmax to normalize the similarity matrix to obtain a probability distribution, take the maximum value as the classification label, and update the image text recognition sequence in the target tracker N with the recognized result; Step 5: Obtain the ROI region of the current frame, and cache the image of the current ROI into the tracker. At the same time, compare it with the image cached in the previous frame in the tracker N, calculate the difference change between frames, and generate a feature map of the image dynamic change sequence; The steps to calculate the difference change between frames are as follows: First, calculate the grayscale of the ROI region, then use Sobel to calculate the gradient, directly subtract the gradient map from the previous frame's gradient map, and through the minimum threshold method, set the similar points below the threshold directly to 0, and then calculate the mean of the difference map to obtain the change metric value between the current frame and the previous frame; Step 6: Cache the dynamic change metric value and the static recognition result into the tracker N. Once the target tracking is lost for more than 3 seconds, it is considered that the target has left. Obtain the dynamic and static sequences from this tracker N and send them to a comprehensive judgment device, and output the recognition result through the integrated comprehensive judgment method; at the same time, delete the data of this tracker N.

2. The object recognition method based on multi-dimensional sequence features according to claim 1, characterized in that: In Step 2, the SORT algorithm is used to track the target. According to the coordinate information of the target obtained in Step 1, calculate the IoU with the position coordinates in the already created tracker. If the intersection over union is greater than 0.3, it is considered that the newly detected target and the already tracked target are the same target, and update the existing detection target information into the tracker N.

3. The object recognition method based on multi-dimensional sequence features according to claim 1, characterized in that: In step 3, the normalized region of interest (ROI) is cropped, and the specific steps are as follows: Take the longest side of the detection target position box and the center position of the target to re-obtain the ROI. The center point remains unchanged, and the side length is twice the longest side of the target box. Expand outward to obtain a square ROI region, and then resize it to 224x224 to unify the size.

4. The object recognition method based on multi-dimensional sequence features according to claim 3, wherein: If it exceeds the frame boundary during the outward expansion, the exceeded part is filled with white.

5. The object recognition method based on multi-dimensional sequence features according to claim 1, characterized in that: In step 6, the method for calculating the recognition result is as follows: S1: Calculate the standard deviation of the dynamic change amount of the sequence to generally describe the dynamic change M of the sequence; S2: For the recognition result sequence, through a show of hands, the one with the most votes is the target recognition result: R; S3: According to the actual dynamic characteristics of the recognized target, obtain the threshold X of the dynamic characteristics. If M>X, output the recognition result R externally.

Images