A data processing method and system for video overlay based on abnormal frame detection

Abstract

The present invention relates to a data processing method and system for video superposition based on abnormal frame detection, and belongs to the field of video processing technology. The method comprises the following steps: obtaining the position and size of the target in each frame in the video to be superimposed, judging whether there is an abnormal frame according to the change of the position and size of the target in each continuous frame, and if there is an abnormal frame, replacing the abnormal frame with correction data to realize the correction of the abnormal frame; adding the image to be added to the target in each frame of the video after the correction; the abnormal frame is a frame in which the slope change of the position or size of the target in each continuous frame exceeds the corresponding set threshold. The method finds the abnormal frame by using the change of the position and size of the target in each continuous frame, and corrects the abnormal frame, so that the problem of relatively large fluctuation of the target in the video to be superimposed can be solved, thereby obtaining relatively stable video data, avoiding dislocation when the video is superimposed, and improving the superposition effect.

Description

Technical Field

[0001] The invention relates to a data processing method and system for video superposition based on abnormal frame detection, belonging to the technical field of video processing. Background Art

[0002] In the process of existing video overlay, the overlay is completely based on the original data analyzed by the image algorithm. However, the poor video quality and blur in the video will lead to inaccurate image algorithm analysis, making the original data of the target in the detected video inaccurate. For example, when a virtual traffic light needs to be superimposed on the traffic light in the original video, if the position coordinates and width and height of the traffic light fluctuate greatly in the original video, that is, the original data of the traffic light as the target is inaccurate, the superimposed virtual traffic light cannot completely fit the traffic light in the original video, causing the video overlay to be misaligned. Summary of the invention

[0003] The purpose of the present invention is to provide a data processing method and system for video superposition based on abnormal frame detection, so as to solve the problem of superposition misalignment caused by large fluctuation of the target in the video.

[0004] To achieve the above object, the solution of the present invention includes: A data processing method for video superposition based on abnormal frame detection of the present invention comprises the following steps: Obtain the position and size of the target in each frame of the video to be superimposed, and determine whether there is an abnormal frame according to the change of the position and size of the target in each consecutive frame. If there is an abnormal frame, replace the abnormal frame with the correction data to realize the correction of the abnormal frame; In the corrected video, the image to be added is added to the target in each frame of the video; the abnormal frame is a frame in which the slope change of the position or size of the target in consecutive frames exceeds the corresponding set threshold.

[0005] Furthermore, the position and size of the target in each frame of the video to be superimposed are obtained by performing image segmentation and recognition on the video to be superimposed.

[0006] Furthermore, the abnormal frame judgment process is as follows: Based on the position and size of the target in each frame of the video to be superimposed, the target position change curve and the target size change curve of the continuous frames are obtained. If one of the slope changes of the target position change curve and the slope change of the target size change curve exceeds the corresponding set threshold, it is considered that there is an abnormal frame.

[0007] Furthermore, when the video quality requirement for the video after correction is higher, the corresponding set threshold value is smaller; when the video quality requirement for the video after correction is lower, the corresponding set threshold value is larger.

[0008] Furthermore, the set threshold is related to the video quality of the video to be superimposed. The higher the video quality of the video, the smaller the corresponding set threshold value; the lower the video quality of the video, the larger the corresponding set threshold value.

[0009] Furthermore, the method further comprises the following steps: After correcting the abnormal frame, the position and size of the target in each frame of the video are re-acquired, and whether there is an abnormal frame is determined based on the changes in the position and size of the target in consecutive frames. If there is an abnormal frame, the abnormal frame is replaced by the corrected data and judged again until there is no abnormal frame in the video. The video without abnormal frames is regarded as the corrected video.

[0010] Furthermore, the correction data is obtained as follows: The parameters of the target in the normal frame are extracted, and the fitting curve is obtained by using the change of the parameter in the normal frame. The correction data corresponding to the abnormal frame is obtained by using the fitting curve; when the abnormal frame is caused by the target position, the parameter is the position data of the target; when the abnormal frame is caused by the target size, the parameter is the size data of the target.

[0011] Furthermore, the correction data is obtained as follows: The parameters of the target in the normal frame adjacent to the abnormal frame are extracted, and the parameters are used for linear interpolation to obtain the correction data corresponding to the abnormal frame; when the abnormal frame is caused by the target position, the parameter is the position data of the target; when the abnormal frame is caused by the target size, the parameter is the size data of the target.

[0012] Furthermore, the method also includes the following step: before adding, Gaussian filtering is performed on the data set of the position and size of the target in each frame of the corrected video.

[0013] A data processing system for video superposition based on abnormal frame detection of the present invention comprises a processor, and the processor is used to execute a computer program to implement the steps of the data processing method for video superposition based on abnormal frame detection as described above.

[0014] Beneficial effects of the present invention: The present invention is a pioneering invention, which provides a data processing method for video superposition based on abnormal frame detection. The method uses the changes in the position and size of the target in consecutive frames to find the abnormal frames and correct the abnormal frames. In this way, the problem of relatively large fluctuations in the target in the video to be superimposed can be solved, thereby obtaining relatively stable video data, avoiding dislocation when the video is superimposed, and improving the superposition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of video overlay. DETAILED DESCRIPTION

[0016] To solve the problem in the background technology, the present invention adopts a filtering algorithm to find abnormal frames using the changes in the original data of the target in consecutive frames, and corrects the abnormal frames to obtain accurate original data of the target to avoid dislocation when the video is superimposed.

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0018] An embodiment of a data processing method for video superposition based on abnormal frame detection: A data processing method for video superposition based on abnormal frame detection comprises the following steps: 1. Get the position and size of the target in each frame of the video to be superimposed.

[0019] The video to be superimposed is generally a video, which is composed of continuous multi-frame data (multi-frame images). Therefore, before superimposing the video, it is necessary to determine the position and size of the target in each continuous frame of the video to be superimposed. Among them, the size of the target refers to the width and height of the target.

[0020] For example, assuming that the video to be superimposed is a train moving video, and the target is a signal light in the video, when a virtual signal light is superimposed on the signal light in the train moving video, it is necessary to determine the position and size of the signal light in consecutive frames of the train moving video.

[0021] The position and size of the target in each frame of the video to be superimposed are obtained by using an image target recognition algorithm on the video to be superimposed. Specifically, the video to be superimposed can be segmented and then image recognized to obtain the position and size of the target in each frame of the video to be superimposed. Among them, image segmentation and recognition can be implemented using deep learning models, such as convolutional neural networks, graph convolutional neural networks, and other deep learning models suitable for image processing.

[0022] 2. Detection of abnormal frames.

[0023] It is determined whether there is an abnormal frame based on the changes in the target position and target size in consecutive frames.

[0024] Specifically, the target position change curve of the target in the continuous frames is obtained according to the position of the target in each frame in the video to be superimposed, and the target size change curve of the target in the continuous frames is obtained according to the size of the target in each frame in the video to be superimposed. If one of the slope change of the target position change curve and the slope change of the target size change curve exceeds the corresponding set threshold, it is considered that there is an abnormal frame. The abnormal frame is the frame corresponding to the point on the corresponding change curve where the slope change exceeds the corresponding set threshold, that is, the frame corresponding to the point on the target position change curve where the slope change exceeds the corresponding set threshold is an abnormal frame, and the frame corresponding to the point on the target size change curve where the slope change exceeds the corresponding set threshold is an abnormal frame.

[0025] Among them, abnormal frames appear in the following three situations: ① When only the target position in the continuous frames of the target position change curve has a slope change that exceeds the set threshold; ② When only the target size in the continuous frames of the target size change curve has a slope change that exceeds the set threshold; ③ When the target position and size in the continuous frames of the corresponding change curves both have slope changes that exceed the corresponding set threshold.

[0026] Among them, the frames exceeding the set threshold are abnormal frames, and the frames not exceeding the threshold are normal frames.

[0027] The above-mentioned threshold value can be calculated using the average slope change rate of the corresponding change curve as a whole, and the specific value of the threshold value can be set according to the following method: When the video quality requirement for the video after correction is higher, the corresponding set threshold value is smaller; when the video quality requirement for the video after correction is lower, the corresponding set threshold value is larger.

[0028] As other implementations, the specific value of the threshold value can be dynamically adjusted according to the video quality of the original video to be superimposed. The higher the quality of the video, the higher the credibility of the change curve data. Therefore, the higher the video quality requirement is. The threshold for determining abnormal frames can be appropriately lowered, and the corresponding threshold value is set to be smaller. On the contrary, the lower the quality of the video, the lower the credibility of the change curve data. Therefore, the video quality requirement is relatively lower. The threshold for determining abnormal frames can be appropriately raised, and the corresponding threshold value is set to be larger. The video quality of the original video here depends on the video quality of the video as the video to be superimposed. The video quality of the video can be the video quality of the original video shot, the video quality after transmission, or the stored video quality.

[0029] Of course, the threshold value can also be set based on experience.

[0030] 3. Correction of abnormal frames.

[0031] If an abnormal frame exists, the correction data is used to replace the abnormal frame to achieve correction of the abnormal frame.

[0032] Specifically, the parameters of the target in the normal frame are extracted, and a fitting curve is obtained by using the variation of the parameters in the normal frame. The correction data for replacing the abnormal frame is obtained according to the fitting curve.

[0033] Among them, when the abnormal frame is caused by the target position, that is, the abnormal frame is a frame in which the slope change of the target position in consecutive frames exceeds the set threshold, then the parameter used for fitting is the target position data of the target in the normal frame, that is, the target position data.

[0034] When the abnormal frame is caused by the target size, that is, the abnormal frame is a frame in which the slope change of the target size in consecutive frames exceeds the set threshold, then the parameter used for fitting is the target size data of the target in the normal frame, that is, the target size data.

[0035] As another implementation, the parameters of the target in the normal frame adjacent to the abnormal frame are extracted, and the parameters of the target in the normal frame adjacent to the abnormal frame are used to perform linear interpolation to obtain the correction data for replacing the abnormal frame.

[0036] Among them, when the abnormal frame is the abnormality caused by the target position, that is, the abnormal frame is a frame in which the slope change of the target position in consecutive frames exceeds the set threshold, then the parameter used for linear interpolation is the target position data (target position data) of the target in the normal frame adjacent to the abnormal frame.

[0037] When the abnormal frame is caused by the target size, that is, the abnormal frame is a frame in which the slope change of the target size in consecutive frames exceeds the set threshold, then the parameter used for linear interpolation is the target size data (target size data) of the target in the normal frame adjacent to the abnormal frame.

[0038] For example, when the two detected abnormal frames are caused by the target position and the target size respectively, the two abnormal frames are corrected by the following methods: The abnormal frame is an anomaly caused by the target position. The target position data of the target in the normal frame is extracted, and the fitting curve is obtained by using the change of the target position data in the normal frame. According to the fitting curve, correction data for replacing the abnormal frame is obtained. The abnormal frame is replaced by the correction data to realize the correction of the abnormal frame with abnormal target position.

[0039] Or the abnormal frame is an abnormality caused by the target position, the target position data of the target in the normal frame adjacent to the abnormal frame is extracted, and the target position data of the target in the normal frame adjacent to the abnormal frame is used to perform linear interpolation to obtain correction data for replacing the abnormal frame, and the abnormal frame is replaced by the correction data to achieve correction of the abnormal frame with the abnormal target position.

[0040] The abnormal frame is an anomaly caused by the target size. The target size data of the target in the normal frame is extracted, and a fitting curve is obtained using the change of the target size data in the normal frame. According to the fitting curve, correction data for replacing the abnormal frame is obtained. The abnormal frame is replaced by the correction data to realize the correction of the abnormal frame with abnormal target size.

[0041] Or the abnormal frame is an abnormality caused by the target size, the target size data of the target in the normal frame adjacent to the abnormal frame is extracted, and the target size data of the target in the normal frame adjacent to the abnormal frame is used to perform linear interpolation to obtain correction data for replacing the abnormal frame, and the abnormal frame is replaced by the correction data to realize the correction of the abnormal frame with abnormal target size.

[0042] The two abnormal frames mentioned above can be corrected simultaneously or in sequence, depending on the actual situation.

[0043] Specifically, after correcting the abnormal frame, the position and size of the target in each frame of the video can be re-acquired, and whether there is an abnormal frame can be determined based on the changes in the position and size of the target in consecutive frames. If there is an abnormal frame, the abnormal frame is replaced by the corrected data, and the judgment is performed again until there is no abnormal frame in the video, and the video without abnormal frames is used as the corrected video.

[0044] 4. Video overlay.

[0045] The image to be added is added to the target of each frame in the corrected video.

[0046] Specifically, before adding, the data set of the position and size of the target in each frame of the corrected video can be Gaussian filtered to smooth the data, that is, the video after Gaussian filtering is superimposed, specifically superimposing the virtual target on the target in the video.

[0047] For example, the video to be superimposed consists of 20 consecutive frames of data (20 images). The 20 images are segmented and recognized, and the position coordinates and sizes of the targets in the 20 images are recorded. Taking the target position as an example, the 20 position coordinates obtained are placed in the same coordinate system to obtain the target position change curve of the target in the 20 consecutive frames. Then, the target size change curve of the target in the 20 consecutive frames is obtained based on the 20 sizes obtained. The abnormal frame is found by the change in the slope of the two change curves of the target position and the target size, and the threshold can be set based on experience.

[0048] For example, if it is found that only the target position coordinates of the 5th frame are abnormal, the target position coordinates of the 5th frame are removed, and the target position parameters of the 4th frame and the target position parameters of the 6th frame are summed and averaged as the target position data of the 5th frame to correct the 5th frame; or the data of the 2nd, 3rd, 4th frame and the 6th, 7th frame are summed and averaged as the data of the 5th frame to correct the 5th frame; or the data of the 5th frame is obtained by fitting the remaining normal frames to correct the 5th frame; or the data of the 5th frame is obtained by linear interpolation of the 2nd, 3rd, 4th frame and the 6th, 7th frame to correct the 5th frame.

[0049] For example, if it is found that only the target size of the 5th frame is abnormal, the target size of the 5th frame is removed, and the data obtained by summing and averaging the target size parameters of the 4th frame and the target size parameters of the 6th frame are used as the target size data of the 5th frame to correct the 5th frame; or the data obtained by summing and averaging the data of the 2nd, 3rd, 4th frames and the 6th, 7th frames are used as the data of the 5th frame to correct the 5th frame; or the data of the 5th frame is obtained by fitting the remaining normal frames to correct the 5th frame; or the data of the 5th frame is obtained by linear interpolation of the 2nd, 3rd, 4th frames and the 6th, 7th frames to correct the 5th frame.

[0050] For example, if the 6th frame and the 15th frame are found to be abnormal frames, the two frames are corrected and the abnormal frame detection is performed again. If there are still abnormal frames, the correction of the abnormal frames is continued. If there are no abnormal frames, the corrected video is obtained.

[0051] Among them, when the slope change of the corresponding change curve does not exceed the corresponding set threshold, it is considered that there is no abnormal frame. That is, when the slope change of the target position change curve does not exceed the set threshold, and the slope change of the target size change curve does not exceed the set threshold, it is considered that there is no abnormal frame in the video to be superimposed.

[0052] like Figure 1 As shown, the original video is analyzed to obtain the original data, the original data is filtered to obtain the processed data, and finally the target in the original video is located using the three-dimensional model, and the found target is superimposed with the target in each frame of the original video.

[0053] The filtering algorithm includes obtaining the position and size of the target in each frame of the video to be superimposed, detecting abnormal frames, and correcting abnormal frames.

[0054] The slope change refers to the slope change speed, and the slope change speed includes the magnitude and direction of the slope change, that is, the slope change rate and direction.

[0055] The present invention corrects abnormal frames to avoid the situation where misalignment occurs during video superposition due to inaccurate image analysis caused by low quality of the video to be superimposed.

[0056] An embodiment of a data processing system for video overlay based on abnormal frame detection: A data processing system for video superposition based on abnormal frame detection includes a processor, and the processor is used to execute a computer program to implement the steps of a data processing method for video superposition based on abnormal frame detection. Among them, a data processing method for video superposition based on abnormal frame detection has been described in detail in an embodiment of a data processing method for video superposition based on abnormal frame detection, and will not be repeated here.

Claims

1. A data processing method for video superposition based on abnormal frame detection, characterized in that: The steps include: Obtain the position and size of the target in each frame of the video to be superimposed, and determine whether there is an abnormal frame according to the change of the position and size of the target in each consecutive frame. If there is an abnormal frame, replace the abnormal frame with the correction data to realize the correction of the abnormal frame; In the corrected video, the image to be added is added to the target in each frame of the video; the abnormal frame is a frame in which the slope change of the position or size of the target in consecutive frames exceeds the corresponding set threshold.

2. The data processing method for video superposition based on abnormal frame detection according to claim 1 is characterized in that: The position and size of the target in each frame of the video to be superimposed are obtained by performing image segmentation and recognition on the video to be superimposed.

3. The data processing method for video superposition based on abnormal frame detection according to claim 1, characterized in that: The abnormal frame judgment process is as follows: Based on the position and size of the target in each frame of the video to be superimposed, the target position change curve and the target size change curve of the continuous frames are obtained. If one of the slope changes of the target position change curve and the slope change of the target size change curve exceeds the corresponding set threshold, it is considered that there is an abnormal frame.

4. The data processing method for video superposition based on abnormal frame detection according to claim 1 or 3, characterized in that: When the video quality requirement for the video after correction is higher, the corresponding set threshold value is smaller; when the video quality requirement for the video after correction is lower, the corresponding set threshold value is larger.

5. The data processing method for video superposition based on abnormal frame detection according to claim 1 or 3, characterized in that: The set threshold is related to the video quality of the video to be superimposed. The higher the video quality of the video, the smaller the corresponding set threshold value; the lower the video quality of the video, the larger the corresponding set threshold value.

6. The data processing method for video superposition based on abnormal frame detection according to claim 1, characterized in that: The method further comprises the steps of: After correcting the abnormal frame, the position and size of the target in each frame of the video are re-acquired, and whether there is an abnormal frame is determined based on the changes in the position and size of the target in consecutive frames. If there is an abnormal frame, the abnormal frame is replaced by the corrected data and judged again until there is no abnormal frame in the video. The video without abnormal frames is regarded as the corrected video.

7. The data processing method for video superposition based on abnormal frame detection according to claim 1 or 6, characterized in that: The correction data is obtained by: Extract the parameters of the target in the normal frame, use the change of the parameters in the normal frame to obtain a fitting curve, and use the fitting curve to obtain the correction data corresponding to the abnormal frame; when the abnormal frame is caused by the target position, the parameter is the position data of the target; When the exception frame is caused by the target size, this parameter is the size data of the target.

8. The data processing method for video superposition based on abnormal frame detection according to claim 1 or 6, characterized in that: The correction data is obtained by: The parameters of the target in the normal frame adjacent to the abnormal frame are extracted, and the parameters are used for linear interpolation to obtain the correction data corresponding to the abnormal frame; when the abnormal frame is caused by the target position, the parameter is the position data of the target; when the abnormal frame is caused by the target size, the parameter is the size data of the target.

9. The data processing method for video superposition based on abnormal frame detection according to claim 1, characterized in that: The method further comprises the following steps: before adding, Gaussian filtering is performed on the data set of the position and size of the target in each frame of the corrected video.

10. A data processing system for video superposition based on abnormal frame detection, comprising a processor, characterized in that: The processor is used to execute a computer program to implement the steps of the data processing method for video superposition based on abnormal frame detection according to any one of claims 1 to 9.

Images