Video-based small target recognition method and system, electronic device and storage medium

Abstract

The application discloses a small target identification method and system based on video, electronic equipment and storage medium, comprising: setting a structural similarity threshold, and dividing video frames into multiple groups according to the structural similarity; performing target detection on each group of video frames, calculating the intersection-over-union ratio of the target positions detected from two adjacent pictures in the same group in sequence, regarding the maximum intersection-over-union ratio as the same target, and obtaining a position box by taking the union set of the position information of the same target on all frames, so that all position information is within the position box; intercepting each frame of picture in the group by using the position box to obtain multiple low-resolution pictures; obtaining a high-resolution picture from the multiple low-resolution pictures of each target by using a super-resolution algorithm; and identifying the category of the target by using a target identification algorithm on the generated high-resolution picture. The application can greatly improve the accuracy of fuzzy video target detection and identification without much improvement in floating-point operation amount.

Description

Technical Field

[0001] This invention relates to the field of small target detection and recognition technology, specifically to a video-based small target recognition method, system, electronic device, and storage medium. Background Technology

[0002] With the continuous development of deep learning, traffic sign detection and recognition has become one of the current trends in the field of autonomous driving. However, due to limited computing power on the vehicle and the insufficient clarity of the cameras used, inaccurate traffic sign recognition often occurs. To solve the problem of unclear video images, super-resolution algorithms are commonly used to improve image quality. Super-resolution improves the resolution of the original image through hardware or software methods. The core idea of ​​super-resolution is to exchange temporal bandwidth (acquiring multiple frames of the same scene) for spatial resolution, realizing the conversion from temporal resolution to spatial resolution.

[0003] In the industry, there are two main methods for traffic sign detection and recognition: The first is an end-to-end detection and recognition method, which outputs both the target's location and category information. While faster, this method is less accurate and less commonly used. The second method employs a two-stage algorithm. The first stage uses a target detection algorithm to locate the target in the image, and the second stage uses a target recognition algorithm to identify the local image area defined by the detection algorithm, thus obtaining the target category information. This method is slightly slower than the first, but offers improved accuracy and is more widely used. Because direct target detection and recognition on blurry images is ineffective, the industry typically performs super-resolution testing on blurry images before target detection and recognition. However, super-resolution computation is computationally expensive, and the resulting image is very large, significantly increasing the computational load for subsequent target detection and recognition. Therefore, the industry urgently needs an algorithm that can improve the accuracy of blurry target detection and recognition without significantly increasing computational cost.

[0004] For example, patent document CN201510672286.2 (A method for detecting small targets in UAV videos based on super-resolution reconstruction). This patent uses one frame from the video as a reference frame, estimates pixel displacement in the subsequent three frames, and then uses these four frames to generate a high-resolution target image. The target image is then segmented into multiple target region blocks, and each target region block is detected and identified. Because it uses a fixed number of frames to generate the image, if there are large differences between adjacent frames, the super-resolution will be inaccurate. Furthermore, this method requires super-resolution of the entire image, resulting in significant computational overhead.

[0005] Therefore, it is necessary to develop a new video-based method, system, electronic device, and storage medium for small target recognition. Summary of the Invention

[0006] The purpose of this invention is to provide a video-based small target recognition method, system, electronic device, and storage medium that can significantly improve the accuracy of target detection and recognition in blurry videos without increasing the amount of floating-point operations.

[0007] The present invention discloses a video-based small target recognition method, comprising the following steps:

[0008] Set a structural similarity threshold to divide video frames into multiple groups based on structural similarity;

[0009] For each group of video frames, target detection is performed. Within the same group, the cross-union ratio (CUI) of the detected target positions in two adjacent images is calculated sequentially. The target with the highest CUI is considered to be the same target. The position information of the same target in all frames is combined to obtain a bounding box, so that all position information is within the bounding box. The bounding box is used to crop each frame in the group to obtain multiple low-resolution images.

[0010] For each target, multiple low-resolution images are processed using a super-resolution algorithm to obtain a single high-resolution image;

[0011] The generated high-resolution images are used to identify the category of the target using a target recognition algorithm.

[0012] Optionally, the video frames are divided into multiple groups based on structural similarity, specifically:

[0013] S11. Select the first frame as the previous frame and the second frame as the next frame;

[0014] S12. Calculate the structural similarity between the next frame and the previous frame. If the structural similarity between the previous frame and the next frame is greater than the structural similarity threshold, then move the next frame backward and repeat step S12 until the structural similarity between the previous frame and the next frame is less than or equal to the structural similarity threshold. At this time, all frames between the previous frame and the next frame are grouped together.

[0015] S13. Set the previous frame to the position of the next frame, move the position of the next frame one frame to the right, and continue to repeat steps S12 and S13 until the next frame is the last frame of the video. At this point, the video grouping is completed.

[0016] Optionally, a location bounding box can be obtained by taking the union of the location information of the same target across all frames, specifically as follows:

[0017] Store the locations of the same target in the same queue;

[0018] Based on the generated target location queue, traverse the entire queue to generate a location bounding box that makes all target locations in the queue a subset of it.

[0019] Secondly, the present invention provides a video-based small target recognition system, comprising a similarity detection module, a target detection module, a super-resolution module, and a target recognition module;

[0020] The similarity detection module divides the video frames into multiple groups based on structural similarity, and after grouping, each group is transmitted to the target detection module.

[0021] The target detection module performs target detection on each group of video frames. In the same group, it calculates the intersection-union ratio (IUR) of the detected target positions for two adjacent images in turn, and regards the target with the largest IUR as the same target. It takes the union of the position information of the same target on all frames to obtain a position box, so that all position information is within the position box. It uses the position box to crop each frame image in the group to obtain multiple low-resolution images. The target recognition module is connected to the similarity detection module.

[0022] The super-resolution module uses a super-resolution algorithm to obtain a high-resolution image from multiple low-resolution images of each target. This super-resolution module is connected to the target detection module.

[0023] The target recognition module identifies the category of the target in the generated high-resolution image using a target recognition algorithm. This target recognition module is connected to the super-resolution module.

[0024] Thirdly, the electronic device of the present invention includes:

[0025] One or more processors;

[0026] Storage device for storing one or more programs;

[0027] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the video-based small target recognition method as described in this invention.

[0028] Fourthly, the present invention provides a storage medium storing a computer program thereon, which, when invoked, can implement the steps of the video-based small target recognition method described in the present invention.

[0029] This invention has the following advantages: It utilizes image similarity (such as structural similarity) to group frames, achieving automatic frame grouping. Then, a target detection algorithm determines the location of the target to be super-resolution, and by analyzing the target location, the bounding box to be super-resolution is obtained. This method greatly reduces the computational load of subsequent super-resolution algorithms. By super-resolution of multiple low-resolution target images, a high-resolution target image is obtained, thereby improving the target's clarity. Finally, the high-resolution target image is input into the target detection algorithm to obtain the final target classification. In summary, this method can significantly improve the accuracy of target detection and recognition in blurry videos with only a slight increase in floating-point computation, effectively solving the problem of low accuracy in traditional small target detection and recognition methods in this scenario. Attached Figure Description

[0030] Figure 1 This is a flowchart of small target recognition based on video.

[0031] Figure 2 This is a flowchart of a video grouping method based on structural similarity.

[0032] Figure 3 This is a diagram illustrating the grouping process based on structural similarity.

[0033] Figure 4 This is a diagram illustrating the target queue generation process;

[0034] Figure 5 This is a diagram illustrating the process of determining the bounding box and generating a low-resolution image of the target.

[0035] Figure 6 This is a diagram of the super-resolution process;

[0036] Figure 7 This is a system block diagram. Detailed Implementation

[0037] The present invention will now be described in detail with reference to the accompanying drawings.

[0038] like Figure 1 As shown in this embodiment, a small target recognition method based on video includes the following steps:

[0039] First, set a structural similarity threshold, then... Figure 3 As shown, video frames are divided into multiple groups based on structural similarity. The specific steps for grouping are as follows: Figure 2As shown. For each group of video frames, target detection is performed. Within the same group, the intersection-union ratio (IUR) of the detected target locations in adjacent images is calculated, and the target with the highest IUR is considered the same target. The location information of the same target across all frames is combined to obtain a bounding box, ensuring all location information is within this bounding box. This bounding box is then used to crop each frame in the group, resulting in multiple low-resolution images. Specifically, as shown... Figure 4 and Figure 5 As shown, a high-resolution image is obtained from multiple low-resolution images of each target using a super-resolution algorithm, specifically as follows: Figure 6 As shown in the image. Finally, the target category is identified using a target recognition algorithm on the generated high-resolution image.

[0040] In this embodiment, frames are grouped using image similarity, achieving automatic frame grouping. Then, a target detection algorithm determines the location of the target to be super-resolution, and the bounding box to be super-resolution is obtained by analyzing the target location. This method significantly reduces the computational load of subsequent super-resolution algorithms. By super-resolution of multiple low-resolution target images, a high-resolution target image is obtained, thereby improving the target's clarity. Finally, the high-resolution target image is input into the target detection algorithm to obtain the final target classification. In summary, this method can significantly improve the accuracy of target detection and recognition in blurry videos with only a slight increase in floating-point computation, effectively solving the problem of low accuracy in traditional small target detection and recognition methods in this scenario.

[0041] like Figure 2 As shown, in this embodiment, video frames are divided into multiple groups based on structural similarity, specifically as follows:

[0042] S11. First, obtain the manually set structural similarity threshold. Then, set two variables to represent the previous frame and the next frame respectively, and select the first frame as the previous frame and the second frame as the next frame.

[0043] S12. Calculate the structural similarity between the next frame and the previous frame. If the structural similarity between the previous frame and the next frame is greater than the structural similarity threshold, then move the next frame backward and repeat step S12 until the structural similarity between the previous frame and the next frame is less than or equal to the structural similarity threshold. At this time, all frames between the previous frame and the next frame are grouped together.

[0044] S13. Set the previous frame to the position of the next frame, move the position of the next frame one frame forward, and continue to repeat steps S12 and S13 until the next frame is the last frame of the video. At this point, the video frame grouping is complete, and the grouped diagram is shown below. Figure 3 As shown.

[0045] like Figure 4The diagram shows the target queue generation process. Taking group n as an example, the specific steps are as follows: First, target detection is performed on group n. At this time, multiple targets will be detected in each image. Since the images in the group are similar, the positions of the same target are also similar. By calculating the intersection-union ratio (IUR) of all targets in two adjacent frames, the two targets with the largest IUR are regarded as the same target. The position information of the same target is stored in the same queue. Thus, the target position queue is generated.

[0046] like Figure 5 The diagram shows the process of determining the bounding box and generating the target low-resolution image. The specific steps are as follows: First, based on the previously generated target position queue, the entire queue is traversed to generate a bounding box that makes all target positions in the queue a subset of the bounding box. The image is then cropped using this bounding box to generate multiple low-resolution images for super-resolution.

[0047] like Figure 7 As shown in this embodiment, a video-based small target recognition system includes a similarity detection module, a target detection module, a super-resolution module, and a target recognition module.

[0048] The similarity detection module divides video frames into multiple groups based on structural similarity. Initially, the first frame is selected as the preceding frame, and the second frame as the following frame. The structural similarity between the following and preceding frames is calculated. If the structural similarity between the following and preceding frames is greater than a structural similarity threshold, the following frame is moved forward. This process is repeated until the structural similarity between the following and preceding frames is less than or equal to the structural similarity threshold. At this point, all frames between the preceding and following frames are grouped together. The preceding frame is then set to the position of the following frame, and the position of the following frame is moved forward one frame. This process is repeated until the following frame is the last frame of the video. The video grouping is now complete, and each group is passed to the target detection module. This similarity detection module implements adaptive frame grouping. Compared to grouping with a fixed number of frames, this grouping method solves the problem that fixed-frame-count grouping cannot guarantee that all frames within a group are similar frames.

[0049] The target detection module performs target detection on each group of video frames. Multiple targets are detected in each image. Since each image is similar, the positions of the detected targets within the same group are also similar. By calculating the intersection-union ratio (IUR) of targets detected in the previous frame and the targets detected in the next frame, the two targets with the largest IUR (not equal to 0) are considered the same target. The position information of the same target across all frames is then combined to obtain a bounding box, ensuring that all position information is within this bounding box. This bounding box is used to crop each frame in the group and input into the super-resolution module. This operation reduces the size of the input image, significantly reducing the computational load of the super-resolution algorithm. This target recognition module is connected to the similarity detection module.

[0050] The super-resolution module processes multiple low-resolution images of each target using a super-resolution algorithm to obtain a single high-resolution image, which is then fed into the target recognition module. This super-resolution module is connected to the target detection module. This operation makes the low-resolution images clearer, thereby improving the accuracy of the target recognition module.

[0051] The target recognition module identifies the category of the target in the generated high-resolution image using a target recognition algorithm, thus completing the detection and recognition of small targets. This target recognition module is connected to the super-resolution module.

[0052] In this embodiment, an electronic device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the video-based small target recognition method as described in this embodiment.

[0053] In this embodiment, a storage medium stores a computer program that, when invoked, can implement the steps of the video-based small target recognition method described in this embodiment.

[0054] The above embodiments are preferred implementations of the method of the present invention, but the implementation of the method of the present invention is not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A video-based method for small target recognition, characterized in that, Includes the following steps: Set a structural similarity threshold to divide video frames into multiple groups based on structural similarity; For each group of video frames, target detection is performed. Within the same group, the cross-union ratio (CUI) of the detected target positions in two adjacent images is calculated sequentially. The target with the highest CUI is considered to be the same target. The position information of the same target in all frames is combined to obtain a bounding box, so that all position information is within the bounding box. The bounding box is used to crop each frame in the group to obtain multiple low-resolution images. For each target, multiple low-resolution images are processed using a super-resolution algorithm to obtain a single high-resolution image; The generated high-resolution images are used to identify the category of the target using a target recognition algorithm; A bounding box is obtained by taking the union of the position information of the same target across all frames. Specifically: Store the locations of the same target in the same queue; Based on the generated target location queue, traverse the entire queue to generate a location bounding box that makes all target locations in the queue a subset of it.

2. The video-based small target recognition method according to claim 1, characterized in that: The video frames are divided into multiple groups based on structural similarity, specifically: S11. Select the first frame as the previous frame and the second frame as the next frame; S12. Calculate the structural similarity between the next frame and the previous frame. If the structural similarity between the previous frame and the next frame is greater than the structural similarity threshold, then move the next frame backward and repeat step S12 until the structural similarity between the previous frame and the next frame is less than or equal to the structural similarity threshold. At this time, all frames between the previous frame and the next frame are grouped together. S13. Set the previous frame to the position of the next frame, move the position of the next frame one frame to the right, and continue to repeat steps S12 and S13 until the next frame is the last frame of the video. At this point, the video grouping is completed.

3. A video-based small target recognition system, characterized in that, It includes a similarity detection module, an object detection module, a super-resolution module, and an object recognition module; The similarity detection module divides the video frames into multiple groups based on structural similarity, and after grouping, each group is transmitted to the target detection module. The target detection module performs target detection on each group of video frames. In the same group, it calculates the intersection-union ratio (IUR) of the detected target positions for two adjacent images in turn, and regards the target with the largest IUR as the same target. It takes the union of the position information of the same target on all frames to obtain a position box, so that all position information is within the position box. It uses the position box to crop each frame image in the group to obtain multiple low-resolution images. The target recognition module is connected to the similarity detection module. The super-resolution module uses a super-resolution algorithm to obtain a high-resolution image from multiple low-resolution images of each target. This super-resolution module is connected to the target detection module. The target recognition module identifies the category of the target in the generated high-resolution image using a target recognition algorithm. This target recognition module is connected to the super-resolution module. Specifically, a location bounding box is obtained by taking the union of the location information of the same target across all frames: Store the locations of the same target in the same queue; based on the generated target location queue, traverse the entire queue to generate a location box that makes all target locations in the queue a subset of it.

4. The video-based small target recognition system according to claim 3, characterized in that: The video frames are divided into multiple groups based on structural similarity, specifically: S11. Select the first frame as the previous frame and the second frame as the next frame; S12. Calculate the structural similarity between the next frame and the previous frame. If the structural similarity between the previous frame and the next frame is greater than the structural similarity threshold, then move the next frame backward and repeat step S12 until the structural similarity between the previous frame and the next frame is less than or equal to the structural similarity threshold. At this time, all frames between the previous frame and the next frame are grouped together. S13. Set the previous frame to the position of the next frame, move the position of the next frame one frame to the right, and continue to repeat steps S12 and S13 until the next frame is the last frame of the video. At this point, the video grouping is completed.

5. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the video-based small target recognition method as described in claim 1 or 2.

6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is invoked, it can implement the steps of the video-based small target recognition method as described in claim 1 or 2.

Images