Target feature calculation method based on color and structure

Abstract

The present invention discloses a method for calculating target features based on color and structure. The method for calculating target image features includes: obtaining a target region of interest in an original image, and performing histogram calculation, structural similarity calculation, image quality calculation, depth feature calculation, and position calculation in a target world coordinate system based on the target region of interest image. The beneficial effect of the target feature calculation method provided by this patent is that it can more comprehensively describe the target while ensuring speed and practicality, improve the distinction and judgment of different and identical targets and detection when using functions such as target similarity comparison in engineering, and improve the robustness of feature calculation in the face of different scenes and illumination.

Description

Technical Field

[0001] The invention relates to the technical field of automobile image processing, and in particular to a method for calculating target features based on color and structure. Background Art

[0002] With the development of vehicle autonomous driving technology, the improvement of GPU computing power and the development of deep learning, people are no longer satisfied with the use of traditional image technology, but are more focused on integrating deep learning technology into products to improve product experience. In the field of autonomous driving, the most common function is target detection and tracking. Therefore, the accuracy of target detection needs to be developed to increase customer experience and further improve the competitiveness of autonomous driving of automotive products.

[0003] In the prior art of detecting and tracking vehicle targets, such as CN111784836A, application number

[0004] The invention patent application document of 202011425364.6 for an online vehicle multi-target tracking method based on a neural network discloses that the vehicle tracking data set is obtained by taking images from an on-board camera, and image enhancement processing is performed on the vehicle tracking data set; a pre-trained model is obtained and fine-tuned; the vehicle tracking data set is iteratively trained multiple times using the pre-trained model to obtain a trained vehicle target detection model; the vehicle target detection model is post-frame processed, and an ID is assigned to the target in the vehicle target detection model and the threshold parameter is readjusted; the vehicle target detection model after the threshold parameter is readjusted is used in the image taken by the on-board camera to obtain the tracking result; the method of the present invention realizes the robust tracking of multiple targets in complex road scenes, and the lightweight model structure can achieve real-time online tracking effect while having accuracy, and can be used in fields such as advanced driver assistance systems and autonomous driving. However, the calculation method for multi-target tracking in this application is relatively complex, the hardware cost is also high, and the anti-interference ability (robustness) needs to be further improved.

[0005] Therefore, it is necessary to study a method that meets the requirements of speed and performance under the premise of low hardware cost to determine the uniqueness and difference of the target. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a target feature calculation method based on color and structure, which describes the target features by calculating image quality, structural features and depth features in combination with multiple methods, so that the accuracy and versatility of the target feature description can be significantly improved even when the scales are different.

[0007] The technical solution of the present invention is achieved in this way:

[0008] The method for calculating target features based on color and structure is characterized by comprising the following steps:

[0009] Step 1) Data collection:

[0010] Randomly select targets with different data distributions to label data records and clean the data;

[0011] Step 2) Data preprocessing:

[0012] Cut the video stream into images based on the 25fps frame rate and remove obstructed or blurred images;

[0013] Arrange and remove occluded or blurred images, write scripts, make part of the image dataset into training video files, and make another part of the image dataset into test video files;

[0014] Combine the target frame information with the image data, write a script, extract the target area images, put the images of the same target into the same folder, divide them into multiple folder directories and put them into the neural network for training;

[0015] Step 3) Use the deepsort tracking network to learn target features and train the deepsort model based on the training data;

[0016] Step 4) Test data:

[0017] Input the same target image into the test script and test the output feature matrix; use the cosine distance to calculate the cosine distance value of the same target and obtain the model file;

[0018] Step 5) Use opencv to load the video, and use the opencv library to calculate the histogram features according to the target box obtained by the target detection network, calculate the psrn value of the target image, and set the threshold. When the two values ​​are the same, they are considered to be the same target, otherwise proceed to step 6;

[0019] Step 6) Calculate the values ​​of the three channels of the target frame SSIM feature, calculate the brightness, contrast and structural similarity, if the SSIM feature values ​​of the two targets in the picture are within the set threshold range, they are considered to be the same target, otherwise proceed to step 7;

[0020] Step 7) Input the target box area image into the deepsort network model for feature calculation; then use the cosine distance to calculate the similarity of the two targets. If they are similar, they are considered to be the same target, otherwise they are considered to be two targets.

[0021] In this way, the present invention obtains the target region of interest in the original image, and performs histogram calculation, structural similarity calculation, image quality calculation, depth feature calculation, and position calculation in the target world coordinate system based on the target region of interest image. While ensuring speed and practicality, the present invention can more comprehensively describe the target, improve the distinction and judgment of different and identical targets and detection when using functions such as target similarity comparison in engineering, and improve the robustness of feature calculation in the face of different scenes and lighting.

[0022] Furthermore: the ratio of the number of training video files to the number of test video files is between (9:1) and (8:2).

[0023] Further: in the step 7), the target frame area image is input into the deepsort network model for feature calculation to obtain the deep features, including the following steps:

[0024] Step 1) Data collection:

[0025] Data collection or cleaning: Collect data in proportion to scenes, angles, distances, lighting conditions, and different human targets;

[0026] Step 2) Data preprocessing:

[0027] The collected video images are saved as single-frame images using the opencv library function at a frame rate of 25fps, and the images with occlusion or blur are sorted and removed. A python script is written; the acquired image data is divided into two parts: training images and test images, with a ratio of (8-12):1; the image data is input for detection using the human body detection network to obtain the target position information, and then the target image is cut out from the original image and saved; the images that have been cut out after detection are sorted, and the targets of the same human body are placed in one folder, and the targets of different human bodies are stored in separate folders;

[0028] Step 3) Download the deepsort network model, download the model code, and modify the number of categories in model.py according to the number of categories of the samples;

[0029] Step 4) Train the model:

[0030] Modify the data path in train.py according to the training and test data storage paths, and execute train.py training;

[0031] Step 5) Test data:

[0032] The test data is divided into single-frame test and video test. In single-frame test, the test sample is input into the network and a 512-dimensional feature data is output; in video test, the target feature data of continuous frames is output;

[0033] Step 6) Model deployment:

[0034] By inputting the target image, calling the test function, obtaining the feature value, calculating the similarity using the cosine distance, and returning the judgment value of whether the two target images are the same target.

[0035] Further: The ratio of training images to test images is 10:1.

[0036] Furthermore: the data collected in step 1) is cleaned by corresponding data cleaning personnel every day.

[0037] Further: the image of the region of interest and the ID number of the initially acquired target frame are recorded and the target frame is output.

[0038] Further: if the target frame with similar target features as the previous frame is found through comparison, the position of the current frame is updated and the target ID of the previous frame is kept, otherwise a new target frame and target ID are added to the record information.

[0039] In summary, the present invention has the following beneficial effects:

[0040] 1. This patent describes the target features by calculating the image quality, structural features and depth features in combination with various methods, so that the accuracy and versatility of the target feature description can be significantly improved when the scales are different, thereby laying a solid foundation for the subsequent uniqueness and difference judgment of the target.

[0041] 2. While ensuring speed and practicality, the method of the present invention can more comprehensively describe the target, improve the distinction, judgment and detection of different and identical targets when using functions such as target similarity comparison in engineering, and improve the robustness of feature calculation in the face of different scenes and lighting; the method of the present invention has a major breakthrough in speed, robustness and operability, and there is no need to collect data multiple times for robust training for multiple scenes.

[0042] 3. The method of the present invention combines multiple calculation methods and is more suitable for use in target detection or similarity comparison related products, because it has been deeply tried and tested in terms of speed and scene robustness.

[0043] 4. The present invention obtains the target region of interest in the original image, and performs histogram calculation, structural similarity calculation, image quality calculation, depth feature calculation, and position calculation in the target world coordinate system based on the target region of interest image. While ensuring speed and practicality, the present invention can more comprehensively describe the target, improve the distinction, judgment, and detection of different and identical targets when using functions such as target similarity comparison in engineering, and improve the robustness of feature calculation in different scenes and lighting. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall flow chart of the present invention;

[0045] Figure 2 It is a training flow chart of the present invention;

[0046] Figure 3 Book Figure 1 Key location map;

[0047] Figures 4-1 and 4-2 are schematic diagrams of embodiments of the method of the present invention. DETAILED DESCRIPTION

[0048] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0049] See also Figure 1 — Figure 3 The present invention provides a method for calculating target features based on color and structure, and its specific implementation steps are as follows:

[0050] Step 1) Data collection:

[0051] Randomly select targets with different data distributions to label data records and clean the data;

[0052] Step 2) Data preprocessing:

[0053] Cut the video stream into images based on the 25fps frame rate and remove obstructed or blurred images;

[0054] Arrange and remove occluded or blurred images, write scripts, make part of the image data set into training video files, and make another part of the image data set into test video files; the ratio of images made into training video files to images made into test video files ranges from (9:1) to (8:2);

[0055] Combine the target frame information with the image data, write a script, extract the target area images, put the images of the same target into the same folder, divide them into multiple folder directories and put them into the neural network model for training;

[0056] Step 3) Use the deepsort tracking network to learn target features. Based on the training data, train the deepsort model. The deepsort network model mentioned in this article is the same model, which is used for target feature learning and is part of the tracking algorithm.

[0057] Step 4) Test data:

[0058] Input the same target image into the test script and test the output feature matrix; use the cosine distance to calculate the cosine distance value of the same target, print and display it, and get the model file;

[0059] Step 5) Use opencv to load the video file, and use the opencv library to calculate the histogram features according to the target box obtained by the target detection network (public detection network models yolov3 and yolov5), calculate the psrn value of the target image, set the threshold, and when the two values ​​are the same, they are considered to be the same target, otherwise proceed to step 6); the psrn value is an indicator for evaluating image quality, the peak signal-to-noise ratio, and the larger the value, the smaller the degree of image distortion;

[0060] Step 6) Calculate the values ​​of the three channels of the target frame SSIM feature, calculate the brightness, contrast and structural similarity, if the SSIM feature values ​​of the two targets in the picture are within the set threshold range, they are considered to be the same target, otherwise proceed to step 7;

[0061] Step 7) Input the target box area image into the deepsort network model for feature calculation; then use the cosine distance to calculate the similarity of the two targets. If they are similar, they are considered to be the same target, otherwise they are considered to be two targets.

[0062] The feature calculation method of the present invention is described in the application of human target tracking:

[0063] Step 1) Obtain the target detection frame through deep learning (not the focus of this article, so it will not be elaborated on), then record the region of interest image and ID number of the initially acquired target frame and output the target frame.

[0064] Step 2) For the next frame, the target detection frame is obtained through deep learning, the target region of interest image detected in the current frame is obtained and saved, and the target region image obtained in the current frame is compared with the region of interest image of the previous frame by feature calculation (that is, Figure 1 A series of feature calculation steps) to determine whether it is the same target.

[0065] Step 3) If the target frame with similar target features as the previous frame is found through comparison, the position of the current frame is updated and the target ID of the previous frame is kept. Otherwise, a new target frame and target ID are added to the record information. The result of the comparison between the current frame target and the previous frame is output.

[0066] See also Figure 4-1 , 4-2 The present invention is based on a model training method of a deepsort network, thereby obtaining deep features, comprising the following steps:

[0067] Step 1) Data Collection

[0068] Data collection and cleaning: Data is collected according to the scene, angle, distance, lighting conditions, and proportions of different human targets. The corresponding data cleaning personnel will perform data cleaning at the same time every day;

[0069] Step 2) Data preprocessing:

[0070] According to the frame rate of 25fps, the collected video images are saved as single-frame images using the opencv library function, and the images with occlusion or blur are sorted and removed, and a python script is written; the acquired image data is divided into two parts: training images and test images, with a ratio of (8-12):1, such as 10:1, and it is ensured that the training image set is large enough (more than 10,000 images); the image data is input for detection using the human body detection network (yolov3, yolov5 network), the target position information is obtained, and then the target image is cut out from the original picture and saved; the images that have been cut out after detection are sorted, and the targets of the same human body are placed in one folder, and different human targets are stored in separate folders.

[0071] Step 3) Download the deepsort network model (the paper comes from "simple online and realtime tracking with a deep association metric"), download the model code, and modify the number of categories in model.py according to the number of categories of the samples.

[0072] Step 4) Train the model:

[0073] According to the training and test data storage paths, modify the data path in train.py (this refers to the script in the downloaded public deepsort network model project) and execute train.py training.

[0074] Step 5) Test data:

[0075] When testing data, we divide it into single-frame testing and video testing. When testing single-frame, we input the test sample into the network and output a 512-dimensional feature data. When testing video, we output the target feature data of continuous frames.

[0076] Step 6) Model deployment:

[0077] By inputting the target image, calling the test function, obtaining the feature value, calculating the similarity using the cosine distance, and then Figure 3 The judgment process is combined and calculated to return the judgment value of whether the two target images are one target.

[0078] The processing method of the present invention has a major breakthrough in speed, robustness and operability, and does not need to collect data multiple times for robust training for multiple scenes.

[0079] Finally, it should be noted that the above examples of the present invention are merely examples for illustrating the present invention, and are not intended to limit the embodiments of the present invention. Although the applicant has described the present invention in detail with reference to the preferred embodiments, for those of ordinary skill in the art, other different forms of changes and modifications can be made based on the above description. It is impossible to list all the embodiments here. Any obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for calculating target features based on color and structure, characterized in that The steps include: Step 1) Data collection: Randomly select targets with different data distributions to label data records and clean the data; Step 2) Data preprocessing: Cut the video stream into images based on the 25fps frame rate and remove obstructed or blurred images; Arrange and remove occluded or blurred images, write scripts, make part of the image dataset into training video files, and make another part of the image dataset into test video files; Combine the target frame information with the image data, write a script, extract the target area images, put the images of the same target into the same folder, divide them into multiple folder directories and put them into the neural network for training; Step 3) Use the deepsort tracking network to learn target features and train the deepsort model based on the training data; Step 4) Test data: Input the same target image into the test script and test the output feature matrix; use the cosine distance to calculate the cosine distance value of the same target and obtain the model file; Step 5) Use opencv to load the video, and use the opencv library to calculate the histogram eigenvalues ​​and the peak signal-to-noise ratio of the target frame image based on the target frame obtained by the target detection network; Set a threshold. When these two values ​​are the same, they are considered to be the same target. Otherwise, proceed to step 6. Step 6) Calculate the values ​​of the three channels of the target frame SSIM feature, calculate the brightness, contrast and structural similarity. If the SSIM feature values ​​of the two targets in the picture are within the set threshold range, they are considered to be the same target, otherwise proceed to step 7); Step 7) Input the target box area image into the deepsort network model for feature calculation; then use the cosine distance to calculate the similarity of the two targets. If they are similar, they are considered to be the same target, otherwise they are considered to be two targets.

2. The method for calculating target features based on color and structure according to claim 1, characterized in that: The ratio of the number of training video files to the number of test video files is between (9:1) and (8:2).

3. The method for calculating target features based on color and structure according to claim 1 or 2, characterized in that: In the step 7), the target frame area image is input into the deepsort network model for feature calculation, thereby obtaining deep features, including the following steps: Step 1) Data collection: Data collection: Collect data in proportion to scenes, angles, distances, lighting conditions, and different human targets; Step 2) Data preprocessing: The collected video images are saved as single-frame images using the opencv library function at a frame rate of 25fps, and the images with occlusion or blur are sorted and removed. A python script is written; the acquired image data is divided into two parts: training images and test images, with a ratio of (8-12):1; the image data is input for detection using the human body detection network to obtain the target position information, and then the target image is cut out from the original image and saved; the images that have been cut out after detection are sorted, and the targets of the same human body are placed in one folder, and the targets of different human bodies are stored in separate folders; Step 3) Download the deepsort network model, download the model code, and modify the number of categories in model.py according to the number of categories of the samples; Step 4) Train the model: Modify the data path in train.py according to the training and test data storage paths, and execute train.py training; Step 5) Test data: The test data is divided into single-frame test and video test. In single-frame test, the test sample is input into the network and a 512-dimensional feature data is output; in video test, the target feature data of continuous frames is output; Step 6) Model deployment: By inputting the target image, calling the test function, obtaining the feature value, calculating the similarity using the cosine distance, and returning the judgment value of whether the two target images are the same target.

4. The method for calculating target features based on color and structure according to claim 3, characterized in that: The ratio of training images to test images is 10:

1.

5. The method for calculating target features based on color and structure according to claim 3, characterized in that: The data collected in step 1) is cleaned by corresponding data cleaning personnel every day.

6. The method for calculating target features based on color and structure according to claim 1 or 2, characterized in that: The region of interest image and ID number of the initially acquired target frame are recorded and the target frame is output.

7. The method for calculating target features based on color and structure according to claim 1 or 2, characterized in that: If the comparison result is a target frame with similar target features to the previous frame, the position of the current frame is updated and the target ID of the previous frame is kept. Otherwise, a new target frame and target ID are added to the record information.

Images