Multi-frame information fusion target detection method and system
Through the combination of multi-frame information fusion and deep learning network, the Kalman filter and forgetting factor mechanism are used to solve the stability and accuracy of object detection in marine environments, and efficient object detection in shaking scenarios is achieved.
Patent Information
- Application Number
- CN202411091214.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
AI Technical Summary
Existing object detection algorithms are difficult to detect temporarily disappearing targets in a shaking scenario in the marine environment, especially due to changes in view angles and features caused by camera shaking, resulting in poor single-frame detection results.
The object detection method of multi-frame information fusion is adopted, through matching modules and prediction units, Kalman filter sequences and forget factor mechanisms are used, combined with a multi-channel input deep learning network, and time dimension information and front-to-back frame association information in the video stream are extracted. , perform target prediction and re-match to improve detection accuracy and robustness.
The target of temporarily disappearing is achieved stably and accurately detected in shaking scenes such as the sea surface, improving the detection rate and real-timeness, and enhancing the accuracy and robustness of target detection.
Smart Images

Figure CN119068389A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of robot perception technology, and specifically relates to a target detection method and system for multi-frame information fusion. Background Art
[0002] Existing target detection algorithms can be roughly divided into two categories according to their development history: target detection methods based on traditional image processing and machine learning algorithms, and target detection methods based on deep learning.
[0003] The process of target detection based on traditional image processing and machine learning algorithms can be mainly expressed as target feature extraction → target recognition → target positioning. The target features are all artificially designed, such as SIFT features (scale invariant features), HOG features (histogram of oriented gradients features), SURF features (accelerated robust features), etc. The target is identified by these features, and then the target is positioned in combination with the corresponding strategy. Since artificially designed features often cannot reflect the real features of the target well, and it is difficult to artificially design features for some objects, this method often cannot achieve a good effect in terms of accuracy, computational efficiency, stability, etc.
[0004] Object detection based on deep learning has become a mainstream method, which is mainly divided into two categories: two-stage object detection algorithm and one-stage object detection algorithm. Representative algorithms of two-stage object detection algorithms include R-CNN, Fast R-CNN, etc. This algorithm is usually divided into two steps. First, a series of candidate boxes are generated on the image through the algorithm, and then the samples of these candidate boxes are classified through the network. Due to its process, the two-stage algorithm has the characteristics of high accuracy, but because it requires two steps to get the final result, it is often slow and difficult to meet the real-time requirements. Representatives of one-stage object detection algorithms include YOLO, SSD, etc. This algorithm does not need to be divided into two steps. It directly predicts the location of the category and the bounding box in the network at the same time, without generating candidate areas separately. Due to this design, the one-stage detection algorithm greatly improves the detection speed compared with the two-stage detection algorithm, but the recognition accuracy is lower than that of the two-stage detection algorithm.
[0005] Nowadays, the process of the target detection algorithm for video streams is generally to first divide the video stream into single-frame images, and then apply the above-mentioned target detector to the divided single-frame images to achieve target detection in the video stream. Therefore, as long as each frame can accurately identify the object, the purpose of high accuracy of target detection in the video stream can be achieved. However, unlike the stable road environment, due to the particularity of the marine environment, the instability of the water surface and the presence of strong winds at sea will cause the unmanned sailboat to tilt and shake up and down and left and right. When using the camera to detect surrounding objects, the captured picture will also shake, and the shaking will cause the perspective and features of the objects in the picture to change, thus posing a challenge to single-frame detection. Summary of the invention
[0006] In order to solve the shortcomings of the prior art and achieve the purpose of stably and accurately detecting the temporarily disappeared target in a shaking scene such as the sea surface, the present invention adopts the following technical solution: A target detection method based on multi-frame information fusion obtains basic target detection results of a group of video frame images, matches the detection results of each single frame image with the detection results of its previous frame image, and for the detection results that are not matched in the current frame image, obtains the temporal sequence of the unmatched detection results in the group of video frame images, and based on the sequence, performs target prediction in the current frame image, matches the predicted results with the unmatched detection results of the current frame again, and obtains a new target detection result after matching.
[0007] Furthermore, each single-frame image detection result is first matched with the detection result of its previous frame image, and the target prediction and the re-matching are performed for the unmatched detection results. For the detection results that are still unmatched in the current frame image, each single-frame image detection result is matched with the detection results of its previous multiple frames of images according to a preset forgetting factor, and the target prediction and the re-matching are performed for the unmatched detection results to obtain a new target detection result after matching.
[0008] Furthermore, if the value of the forgetting factor t≤1, the matching is terminated and the group of video frame images is updated. If t>1, the detection result of the current frame image n is matched with the detection results of the previous n-2 frames, and the target prediction and the re-matching are performed for the unmatched detection results, and so on from back to front until all the detection results of the current frame image are matched or nt rounds of matching are completed to obtain a new target detection result.
[0009] Furthermore, in view of the characteristics of the processed video stream data, the previous and next frames of the video have a large amount of related information, and the traditional target detector cannot fully mine the information in the previous and next frame pictures. Therefore, the present invention changes the separate input of the original detector, and adopts multi-image multi-channel input for the basic target detection. The relationship between video frames is learned through a convolutional network, and the single-frame target features are enhanced by multiple frames to effectively improve the detection rate. When performing target detection on the current frame image, the first k-1 images are selected to be superimposed with the current frame image to form multi-channel data, which is then input into the deep learning network for reasoning.
[0010] Furthermore, the detection result includes a frame number, a position of a detection frame, a category confidence of the detection frame, a category of the detection frame, and update information of the detection frame; and the confidence and category of the detection frame that successfully matches the current frame are adjusted.
[0011] A multi-frame information fusion target detection system, comprising a matching module and a basic target detector, a multi-frame storage unit, a prediction unit and a matching module respectively connected to the matching module; The basic object detector is used to obtain basic object detection results of video frame images; The basic target detection results of a group of historical video frame images stored in the multi-frame storage unit are organized in a chain manner, and the detection results belonging to the same object before and after will be linked together and recorded; The prediction unit uses a set of Kalman filter sequences for predicting targets in the image; The matching module matches the detection results of each single-frame image with the detection results of the previous frame image. For the detection results that are not matched in the current frame image, the prediction unit performs target prediction on the current frame image for the sequence that is not successfully matched in the multi-frame storage unit image. The sequence is a sequence of the unsuccessful matching detection results in time. The prediction result is matched again with the unmatched detection result of the current frame to obtain a new target detection result after matching.
[0012] Furthermore, the matching module first matches the detection results of each single-frame image with the detection results of its previous frame image, and performs the target prediction and the re-matching for the unmatched detection results. For the detection results that have not yet matched in the current frame image, according to a preset forgetting factor, the detection results of each single-frame image are matched with the detection results of the previous multiple frames of images, and the target prediction and the re-matching are performed for the unmatched detection results to obtain new target detection results after matching.
[0013] Furthermore, if the value of the forgetting factor t≤1, the matching is terminated, and the unmatched detection results in the current frame are regarded as newly appeared objects, added to the multi-frame storage unit, and the information in the multi-frame storage unit is updated. If t>1, the detection results are selected from the n-1 frame image in the multi-frame storage unit from back to front for round matching with the n detection results of the current frame image, and the target prediction and the re-matching are performed for the unmatched detection results until all the detection results of the current frame image are matched or nt rounds of matching are completed to obtain a new target detection result.
[0014] Furthermore, in view of the characteristics of the processed video stream data, the previous and next frames of the video have a large amount of related information, and the traditional target detector cannot fully mine the information in the previous and next frame images. Therefore, the present invention changes the separate input of the original detector and adopts multi-image multi-channel input for the basic target detector. The relationship between video frames is learned through a convolutional network, and the single-frame target features are enhanced by multiple frames to effectively improve the detection rate. When performing target detection on the current frame image, the first k-1 images are selected to be superimposed with the current frame image to form multi-channel data, which is then input into the deep learning network for reasoning.
[0015] Furthermore, the detection result includes a frame number, a position of a detection frame, a category confidence of the detection frame, a category of the detection frame, and update information of the detection frame; and the confidence and category of the detection frame that successfully matches the current frame are adjusted.
[0016] The advantages and beneficial effects of the present invention are: The target detection method and system of the present invention with multi-frame information fusion can fully extract the information of the time dimension in the video stream, use the extracted information of the time dimension to correct the correlation between the previous and next frames of the detection result, improve the detection rate, and achieve good real-time performance; through the ladder matching mechanism of the forgetting factor, it can effectively deal with the situation where the target suddenly disappears and then reappears in the video frame, and detect the target again quickly, stably and accurately; finally, the robustness and accuracy of the target detection are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a flow chart of the target detection post-processing method of multi-frame memory-ladder matching in an embodiment of the present invention.
[0018] Figure 2 Schematic diagram of the structure of a multi-frame storage unit in an embodiment of the present invention.
[0019] Figure 3 Schematic diagram of the detector network structure of multi-frame information fusion input in an embodiment of the present invention. DETAILED DESCRIPTION
[0020] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.
[0021] like Figure 1 As shown, a method for post-processing target detection based on information fusion and multi-frame memory-ladder matching is provided. The method involves a multi-frame storage unit, a prediction unit and a matching mechanism.
[0022] like Figure 2 As shown in the figure, the multi-frame storage unit stores the detection information of historical frames, including the frame number (frame number), the position of the detection frame, the category confidence of the detection frame, the category of the detection frame, and the update information of the detection frame, and arranges the frames from early to late in the order of the arrival time of the frames to maintain the update information of the detection frame. The detection frame information of historical frames stored in the multi-frame storage unit is organized in a chain manner, and the detection frames belonging to the same object before and after will be linked together and recorded.
[0023] The prediction unit consists of a Kalman filter sequence, which is mainly used to predict the position where the detection box of the current frame should appear in the memory unit.
[0024] The matching mechanism of the entire post-processing algorithm is as follows: The matching mechanism is mainly divided into two parts, namely the previous frame matching and the forgetting factor t matching. The previous frame matching is a processing step that every frame image (except the initial frame) will go through. The forgetting factor t matching does not necessarily go through every frame image. When t=1, it means that the algorithm will only use the detection frame of the previous frame to match the detection frame of the current frame. When t=2, it means that the algorithm will not only use the detection frame of the previous frame to match the detection frame of the current frame, but also use the previous frame of the current frame to match the detection frame of the current frame, and so on.
[0025] Previous frame matching process: When a picture in a video sequence (assuming it is the nth frame) is detected by the basic single-frame target detector and the basic results are obtained, it will enter the post-processing module. First, the algorithm will retrieve the information in the multi-frame storage unit, and then associate and match the detection frame information of the current frame with the information in the multi-frame storage unit (the n-1th frame, i.e. the previous frame). The confidence and category of the detection frame that successfully matches the current frame will be reprocessed. If it is too low, it will be re-scored to improve its confidence score. After this round of matching, if there are still unmatched detection frames in the current frame, the prediction unit will be called. The Kalman filter in the prediction unit will predict the unmatched sequence in the multi-frame storage unit to obtain the predicted frame position of the current frame. Then the algorithm will match the remaining unmatched detection frames of the current frame with these predicted frames again. If the match is successful, the same operation as before will be performed. After this round of matching, if there are still unmatched detection frames in the current frame, the value of the preset forgetting factor t will be used to determine whether to enter the forgetting factor t matching. Adding this algorithm after a single-frame basic target detector can fully extract the time dimension information in the video stream, use the extracted time dimension information to correct the correlation between the previous and next frames of the detection results, improve the detection rate, and achieve good real-time performance.
[0026] Forgetting factor t matching: If the detection frame of the current frame has not been completely processed after the previous frame matching process, then the n-2, n-3...nt frames in the multi-frame storage unit will be called in turn for matching, and the matching process is the same each time. If the detection frame of the current frame has been completely processed after a certain matching, the forgetting factor t matching process will be terminated in advance. One of the matching processes is as follows: the algorithm will call the information of the prediction unit and the n-2 (previous frame) of the multi-frame storage unit. The Kalman filter in the prediction unit will use the information in the multi-frame storage unit to predict the information of the nth frame (the position of the nth frame detection frame), and then match the unprocessed detection frame of the current frame with these predicted detection frame positions. If there are still unmatched detection frames of the current frame, enter the next round and repeat until all the detection frames of the current frame are processed or the processing length reaches the length of the forgetting factor t. Since the basic single-frame target detector cannot guarantee that the target can be detected in every frame, it is inevitable that the target detection frame will be discontinuous, especially in the sea scene. Due to the problems of waves and sea breeze, the camera angle will shake, the object features will change, and the detection rate will be low due to the discontinuity of the target detection frame. The ladder matching mechanism with forgetting factor t can quickly and accurately detect the target again when the object suddenly disappears for some reason and then reappears. It has the ability to handle the problem of temporary disappearance.
[0027] In view of the characteristics of the processed video stream data, the front and back frames of the video have a large amount of associated information, and the traditional target detector cannot fully mine the information in the front and back frame pictures. In order to solve the above problems, the present invention provides an improved target detector with multi-frame information fusion. On the basis of the detector, the separate input of the original detector is changed, and multiple pictures are input into the detector in multiple channels. The convolution network learns the relationship between video frames, and enhances the single-frame target features by multiple frames, which can effectively improve the detection rate. When the current frame picture is detected, the algorithm will select the first k-1 pictures and the current frame picture to be superimposed together to form a multi-channel data, and then input it into the deep learning network for reasoning. Based on the rich information redundancy in the video frame, the image information of the first few frames is used to strengthen the single-frame image feature extraction, which can effectively deal with the shortcomings of the single-frame image detector: when the camera shakes and causes image blur, feature changes, etc., the single-frame image alone cannot fully and effectively detect the target, and through the inference of the front and back frame images, the clear image features of the first few frames are superimposed and enhanced with the current frame, then the target can be effectively detected, and the robustness and accuracy of marine target detection can be improved.
[0028] like Figure 3 As shown, in the embodiment of the present invention, the video stream is divided into pictures (with three RGB channels) in sequence. When a picture needs to be sent to the network for target detection, the current frame picture and the previous k-1 pictures are superimposed on the channel dimension to form 3*k channel input data, which is then input into the target detection network to obtain the output of the network, which is the position, category and confidence score of the target detection box of the current frame.
[0029] After the current frame passes through the improved multi-frame fusion target detection network, a basic detection result will be obtained. All detection results of a single frame will undergo two major matches: the previous frame match and the forgetting factor t match.
[0030] The single-frame detection result first enters the previous frame matching process. All detection results of the current frame will be uniquely matched with the information in the multi-frame memory unit. The confidence and category of the detection frame that successfully matches the current frame will be reprocessed. If it is too low, it will be re-scored to improve its confidence score; after this round of matching, if there are still unmatched detection frames in the current frame, the prediction unit will be called. The Kalman filter in the prediction unit will predict the unmatched sequence in the multi-frame storage unit to obtain the predicted frame position of the current frame. The sequence here is the sequence of detection frames belonging to the same object in time. Then the algorithm will match the remaining unmatched detection frames of the current frame with these prediction frames again. The confidence and category of the successfully matched detection frame will be reprocessed. If it is too low, it will be re-scored to improve its confidence score. At this point, the previous frame matching process ends. The algorithm will check whether the forgetting factor t is set. If t≤1 is found, the whole process ends, and the unmatched detection frame in the current frame will be regarded as a newly appeared object, added to the multi-frame storage unit, and the information in the multi-frame storage unit will be updated; if t>1, the algorithm will enter the second large match, and the algorithm will select the detection frame information of the last updated n-2 frame, n-3 frame...nt frame in the multi-frame storage unit in sequence for round matching, and then call the information of the prediction unit for these detection frames respectively, predict their position in the current frame, and obtain a prediction frame of the current frame. Then the algorithm will match the remaining unmatched detection frames of the current frame with these prediction frames again, and the confidence and category of the successfully matched detection frame will be reprocessed. If it is too low, it will be re-scored to improve its confidence score. At the end of each round, it will check whether the detection frame of the current frame has been processed. If there is still an unprocessed detection frame in the current frame, it will continue to the next round until all rounds are matched (nt) or all the detection frames of the current frame have been processed.
[0031] Finally, the algorithm outputs all the detection box results of the current frame that have been processed.
[0032] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A target detection method based on multi-frame information fusion, characterized in that: Obtain basic target detection results for a group of video frame images, match the detection results of each single frame image with the detection results of its previous frame image, and for the detection results that are not matched in the current frame image, obtain the temporal sequence of the unmatched detection results in the group of video frame images, and based on the sequence, perform target prediction in the current frame image, match the prediction results with the unmatched detection results of the current frame again, and obtain new target detection results after matching.
2. The target detection method of multi-frame information fusion according to claim 1, characterized in that: First, each single-frame image detection result is matched with the detection result of the previous frame image, and the target prediction and the re-matching are performed for the unmatched detection results. For the detection results that are still unmatched in the current frame image, each single-frame image detection result is matched with the detection results of the previous multiple frames of images according to a preset forgetting factor, and the target prediction and the re-matching are performed for the unmatched detection results to obtain a new target detection result after matching.
3. The target detection method of multi-frame information fusion according to claim 2 is characterized in that: If the value of the forgetting factor t≤1, the matching is terminated. If t>1, the detection result of the current frame image n is matched with the detection results of the previous n-2 frames of images, and the target prediction and the re-matching are performed for the unmatched detection results, and so on from back to front until all the detection results of the current frame image are matched or nt rounds of matching are completed to obtain a new target detection result.
4. The target detection method of multi-frame information fusion according to claim 1, characterized in that: The basic target detection adopts multi-image multi-channel input, and the relationship between video frames is learned through a convolutional network. The single-frame target features are enhanced through multiple frames. When performing target detection on the current frame image, the first k-1 images are selected to be superimposed with the current frame image to form multi-channel data, which is then input into the deep learning network for reasoning.
5. The target detection method of multi-frame information fusion according to claim 1, characterized in that: The detection result includes the frame number, the position of the detection frame, the category confidence of the detection frame, the category of the detection frame, and the update information of the detection frame; the confidence and category of the detection frame that successfully matches the current frame are adjusted.
6. A target detection system for multi-frame information fusion, comprising a matching module and a basic target detector, a multi-frame storage unit, a prediction unit and a matching module respectively connected to the matching module, characterized in that: The basic object detector is used to obtain basic object detection results of video frame images; The basic target detection results of a group of historical video frame images stored in the multi-frame storage unit are organized in a chain manner, and the detection results belonging to the same object before and after will be linked together and recorded; The prediction unit uses a set of Kalman filter sequences for predicting targets in the image; The matching module matches the detection results of each single-frame image with the detection results of the previous frame image. For the detection results that are not matched in the current frame image, the prediction unit performs target prediction on the current frame image for the sequence that is not successfully matched in the multi-frame storage unit image. The sequence is a sequence of the unsuccessful matching detection results in time. The prediction result is matched again with the unmatched detection result of the current frame to obtain a new target detection result after matching.
7. The target detection system of multi-frame information fusion according to claim 6, characterized in that: The matching module first matches each single-frame image detection result with the detection result of the previous frame image, and performs the target prediction and the re-matching for the unmatched detection results. For the detection results that are still unmatched in the current frame image, according to a preset forgetting factor, each single-frame image detection result is matched with the detection results of the previous multiple frames of images, and the target prediction and the re-matching are performed for the unmatched detection results to obtain a new target detection result after matching.
8. The target detection system of multi-frame information fusion according to claim 7, characterized in that: If the value of the forgetting factor t≤1, the matching is terminated, and the unmatched detection results in the current frame are regarded as newly appeared objects, added to the multi-frame storage unit, and the multi-frame storage unit is updated. If t>1, the detection results are selected from the n-1 frame image in the multi-frame storage unit from the back to the front for round matching with the n detection results of the current frame image, and the target prediction and the re-matching are performed for the unmatched detection results until all the detection results of the current frame image are matched or nt rounds of matching are completed to obtain a new target detection result.
9. The target detection system of multi-frame information fusion according to claim 6, characterized in that: The basic target detector adopts multi-image multi-channel input, learns the relationship between video frames through a convolutional network, enhances single-frame target features through multiple frames, and when performing target detection on the current frame image, selects the first k-1 images to be superimposed with the current frame image to form multi-channel data, which is then input into the deep learning network for reasoning.
10. The target detection system of multi-frame information fusion according to claim 6, characterized in that: The detection result includes the frame number, the position of the detection frame, the category confidence of the detection frame, the category of the detection frame, and the update information of the detection frame; the confidence and category of the detection frame that successfully matches the current frame are adjusted.
Citation Information
Cited By
Unmanned aerial vehicle autonomous inspection method, device and equipment for power distribution network and storage medium
CN120236216A
