Object tracking-based horizontal screen-to-vertical screen broadcasting method
By identifying and cropping moving objects in the video as the center of the picture, the problem of poor viewing effect when horizontal screen to vertical screen on the phone is solved, and a better vertical screen viewing experience is achieved, especially in competitive competition videos.
Patent Information
- Application Number
- CN202411015406.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-06
AI Technical Summary
When watching TV programs on your mobile phone, the viewing effect is poor when switching to vertical screens, especially competitive competition videos, which leads to poor viewing experience.
By identifying moving objects in the video, such as a sphere as the center of the picture, cropping the video to ensure that the center of the picture is maintained on the moving objects, and using the target recognition model to filter and crop, a better vertical viewing experience is achieved.
Improve the experience of watching competitive game videos in portrait mode, ensuring that the center of the picture is always on moving objects, and enhancing the accuracy and viewing effect of cropping.
Smart Images

Figure CN119094649A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of streaming media processing, and in particular to a method for converting horizontal screen to vertical screen broadcasting based on object tracking. Background Art
[0002] Among the terminal broadcasting devices for TV programs, the proportion of mobile phones as display terminals is increasing. This is because smartphones have become an essential communication device for the public. As smartphones become more and more powerful, network communication speeds and methods become more and more popular, and they are easy to carry and can be moved at any time, many people are watching TV programs directly on their mobile phones.
[0003] Conventional TV programs are usually in 16:9 or 4:3, while the common display screen ratios of mobile phones are 9:16, 9:18, 9:19, etc. Under normal circumstances, if a mobile phone wants to play a TV program, as long as it is displayed in horizontal mode, no matter which ratio is mentioned above, even after scaling, the error can be ignored. However, in reality, most people do not have the habit of watching TV in horizontal mode. This may be due to the inconvenience of carrying in horizontal mode or the difficulty of placing in horizontal mode. At this time, when watching TV programs, the audience would rather watch in vertical mode. To watch the same program, in vertical mode, the player can only open an area in the middle of the screen with the width of the screen as the width and 9 / 16 of the width as the height as the playback area, and the other areas are black backgrounds or display other content. The "masked wide-screen film" effect, this helpless technical processing greatly reduces the viewing effect, especially for some competitive games such as football and basketball, the poor viewing experience is particularly obvious. Summary of the invention
[0004] In view of the above problems existing in the prior art, a method for converting horizontal screen to vertical screen broadcasting based on object tracking is now provided.
[0005] The specific technical solutions are as follows:
[0006] A method for converting horizontal screen to vertical screen broadcasting based on object tracking, comprising:
[0007] Step S1: for a single frame image in the video to be cropped, identify and obtain multiple suspected targets respectively;
[0008] Step S2: for each of the suspected targets, respectively calculating the target distance of the suspected target relative to the center point of the picture, so as to take the suspected target with the smallest target distance as the actual target;
[0009] Step S3: cropping the video to be cropped according to the actual target in the single frame image.
[0010] On the other hand, the step S1 comprises:
[0011] Step S11: extracting the single frame image from the video to be cropped, and using a target recognition model to recognize the single frame image to obtain a recognition frame and a confidence level corresponding to the suspected target output by the target recognition model;
[0012] Step S12: screening the identification boxes according to the confidence and a pre-configured confidence threshold, so as to remove the identification boxes whose confidence is lower than the confidence threshold;
[0013] Step S13: extracting the center coordinates of the identification frame, and outputting the extraction result as the coordinate position of the suspected target.
[0014] On the other hand, the step S2 comprises:
[0015] Step S21: calculating the Euclidean distance of each suspected target relative to the center point of the picture as the target distance;
[0016] Step S22: sorting the target distances to obtain a minimum target distance, and outputting the suspected target corresponding to the minimum target distance as the actual target.
[0017] On the other hand, in the step S22, a target distance sequence is also constructed during the sorting process, and the target distance sequence includes all the target distances;
[0018] In the target distance sequence, the target distances other than the minimum target distance are marked as target distances to be determined;
[0019] The step S2 further comprises:
[0020] Step S23: Subtract each of the target distances to be determined from the minimum target distance to obtain a first subtraction result;
[0021] Step S24: comparing the first subtraction result with a first subtraction threshold to determine whether there is a first subtraction result smaller than the first subtraction threshold;
[0022] If yes, go to step S25;
[0023] If not, outputting the suspected target corresponding to the minimum target distance as the actual target;
[0024] Step S25: The actual target is obtained by screening according to the centerline distance of the suspected target relative to the centerline.
[0025] On the other hand, the step S25 comprises:
[0026] Step S251: respectively calculating the centerline distance of the suspected target relative to the centerline;
[0027] Step S252: sorting the centerline distances to obtain a minimum centerline distance, and outputting the suspected target corresponding to the minimum centerline distance as the actual target.
[0028] On the other hand, in the step S22, a centerline distance sequence is also constructed during the sorting process, and the centerline distance sequence includes all the centerline distances;
[0029] In the centerline distance sequence, the centerline distances other than the minimum centerline distance are marked as centerline distances to be determined;
[0030] The step S2 further comprises:
[0031] Step S26: Subtract each of the to-be-determined centerline distances from the minimum centerline distance to obtain a second subtraction result;
[0032] Step S27: comparing the second subtraction result with a second subtraction threshold to determine whether there is a second subtraction result that is smaller than the second subtraction threshold;
[0033] If yes, go to step S28;
[0034] If not, output the suspected target corresponding to the minimum centerline distance as the actual target;
[0035] Step S28: searching for the nearest suspected target as the actual target according to the actual target in the previous frame.
[0036] On the other hand, when the suspected target is not identified in the step S1, continuous trajectory prediction is performed according to the actual target on other frames to obtain the actual target on the current single-frame image.
[0037] On the other hand, step S3 comprises:
[0038] Step S31: determining the center of the cropped single frame image according to the actual target;
[0039] Step S32: determining the left boundary and the right boundary of the cropped video image according to the center of the picture and the single frame image;
[0040] Step S33: cropping the video image according to the left boundary and the right boundary of the video image to obtain a cropped video.
[0041] The above technical solution has the following advantages or beneficial effects:
[0042] In view of the problem that the video of ball games in the existing technology has poor viewing effect in the vertical screen scene of mobile phones, this solution selects a technical solution to crop the picture around a moving object, such as a sphere. By taking the moving object as the center of the picture and cropping the video, the center of the picture can be kept on the moving object during playback, and a better vertical screen viewing experience is achieved;
[0043] Furthermore, in this process, by identifying the target and eliminating it according to the offset, the moving object can be accurately identified, thereby improving the accuracy of cropping. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The embodiments of the present invention will be described more fully with reference to the attached drawings, which are provided for illustration and description only and are not intended to limit the scope of the present invention.
[0045] Figure 1 is an overall schematic diagram of an embodiment of the present invention;
[0046] Figure 2 This is a schematic diagram of step S1 in an embodiment of the present invention;
[0047] Figure 3 This is a schematic diagram of step S2 in an embodiment of the present invention;
[0048] Figure 4 This is a schematic diagram of step S23 in an embodiment of the present invention;
[0049] Figure 5 This is a schematic diagram of step S25 in an embodiment of the present invention;
[0050] Figure 6 This is a schematic diagram of step S27 in an embodiment of the present invention;
[0051] Figure 7 Schematic diagram of step S3 in an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0054] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.
[0055] The present invention comprises:
[0056] A method for converting horizontal screen to vertical screen broadcasting based on object tracking, such as Figure 1 As shown, including:
[0057] Step S1: for a single frame image in the video to be cropped, identify and obtain multiple suspected targets respectively;
[0058] Step S2: for each suspected target, respectively calculate the target distance of the suspected target relative to the center point of the picture, so as to take the suspected target with the smallest target distance as the actual target;
[0059] Step S3: cropping the video to be cropped according to the actual target in the single frame image.
[0060] Specifically, after analyzing user data, the inventor found that when watching a football game, the audience usually keeps their focus near the ball. Taking football as an example, the photographer will adjust the lens at any time to keep the football in the picture as much as possible, and then determine the position of the football in the picture according to the position of the players and the goal. The audience will most likely focus on the football, and the focus will change with the change of the position of the football.
[0061] Therefore, on this basis, the inventor believes that if one wants to provide viewers with a better viewing experience in portrait mode, one should use the sphere as the center of the screen based on the original video material, and perform secondary composition and cropping around the sphere, and reconstruct the original horizontal playback screen into a vertical screen, so as to achieve a better playback experience on vertical screen platforms, such as some short video platforms.
[0062] To achieve the above functions, in this embodiment, we first select to identify the single-frame image separately to obtain multiple suspected targets. The suspected target is usually a moving object with a small number of pixels, which is similar to a ball in some features. On this basis, the target distance of the suspected target relative to the center point of the picture is also used for discrimination. In the TV broadcast scene, the cameraman of radio and television usually uses the center composition method to focus and shoot the ball. Therefore, the target distance of the suspected target relative to the center point of the picture can be used to determine the object that the cameraman actually wants to shoot in the current frame. On this basis, irrelevant items in the background are filtered out to obtain the actual target, and then the actual target is cropped to obtain a vertical screen video to achieve a better playback experience.
[0063] In one embodiment, Figure 2 As shown, step S1 includes:
[0064] Step S11: extracting a single frame image from the video to be cropped, and using a target recognition model to recognize the single frame image to obtain a recognition frame and a confidence level corresponding to a suspected target output by the target recognition model;
[0065] Step S12: screening the recognition frames according to the confidence and the pre-configured confidence threshold, so as to remove the recognition frames whose confidence is lower than the confidence threshold;
[0066] Step S13: extracting the center coordinates according to the identification frame, and outputting the extraction result as the coordinate position of the suspected target.
[0067] Specifically, in order to achieve a better recognition effect on the target and facilitate subsequent processing, in this embodiment, the corresponding target recognition model is first introduced to recognize a single-frame image. Among them, a single-frame image is an image extracted in sequence from the video to be cut. The target recognition model is an artificial intelligence model that is pre-trained corresponding to the object to be identified. It can identify and predict the position of the target object from a single-frame image, and output the corresponding recognition frame and confidence. Subsequently, the recognition frame is screened according to the confidence and a pre-configured confidence threshold, such as 50%, to remove the recognition frame with a confidence lower than the confidence threshold. Considering that the sphere usually occupies only a few pixels in the actual live broadcast picture and its main body is relatively small, the center coordinates of the recognition frame can be directly used for processing in subsequent processing.
[0068] Center coordinate extraction includes: x 足 =(x 左上 +x 右下 ) / 2,y 足 =(y 左上 +y 右下 ) / 2;
[0069] In the formula, (x 足 ,y 足 ) is the coordinate of the suspected target, (x 左上 ,y 左上 ) is the coordinate of the upper left corner of the recognition box, (x 右下 ,y 右下 ) is the coordinate of the upper right corner of the recognition box.
[0070] In one embodiment, Figure 3 As shown, step S2 includes:
[0071] Step S21: Calculate the Euclidean distance of each suspected target relative to the center point of the image as the target distance;
[0072] Step S22: sorting the target distances to obtain the minimum target distance, and outputting the suspected target corresponding to the minimum target distance as the actual target.
[0073] Specifically, in order to achieve a better calculation effect, in this embodiment, the Euclidean distance of each suspected target relative to the center point of the picture is selected as the target distance, and then the target distances are sorted to obtain the minimum target distance, and the suspected target corresponding to the minimum target distance is output as the actual target. If there is only one suspected target, the suspected target can be directly output as the actual target.
[0074] The target distance calculation method includes: s = sqrt ((x 足 -x 屏 )*(x 足 -x 屏 )+(y 足 -y 屏 )*(y 足 -y 屏 )).
[0075] Where s is the target distance, (x 足 ,y 足 ) is the coordinate of the suspected target, (x 屏 ,y 屏 ) are the coordinates of the center point of the screen.
[0076] In one embodiment, Figure 4 As shown, in step S22, a target distance sequence is also constructed during the sorting process, and the target distance sequence includes all target distances;
[0077] In the target distance sequence, the target distances other than the minimum target distance are marked as target distances to be determined;
[0078] Step S2 also includes:
[0079] Step S23: Subtract each target distance to be determined from the minimum target distance to obtain a first subtraction result;
[0080] Step S24: comparing the first subtraction result with the first subtraction threshold to determine whether there is a first subtraction result that is less than the first subtraction threshold;
[0081] If yes, go to step S25;
[0082] If not, the suspected target corresponding to the minimum target distance is output as the actual target;
[0083] Step S25: Screening the actual target according to the centerline distance of the suspected target relative to the centerline.
[0084] Specifically, in order to improve the accuracy of target recognition, in this embodiment, after calculating the target distance of each suspected target relative to the center point of the picture, sorting to obtain the target distance sequence and screening the suspected target corresponding to the minimum target distance, it is further judged whether there are other similar suspected targets around the center of the picture. Specifically, each target distance to be judged is subtracted from the minimum target distance to obtain a first subtraction result. If there is a first subtraction result less than the first subtraction threshold, it indicates whether there are other similar suspected targets around the center of the picture. Further discrimination is required at this time. For example, when the minimum target distance is 20 pixels, but the second smallest target distance is 30 pixels, the difference may be caused by the photographer not accurately aiming at the target when following the shot, and the target is too close to the interference object. Therefore, on this basis, it is necessary to further screen the suspected targets, for example, according to the center line distance of the suspected target relative to the center line, the actual target is screened.
[0085] In one embodiment, Figure 5 As shown, step S25 includes:
[0086] Step S251: Calculate the centerline distance of the suspected target relative to the centerline respectively;
[0087] Step S252: sort the centerline distances to obtain the minimum centerline distance, and output the suspected target corresponding to the minimum centerline distance as the actual target.
[0088] Specifically, in order to achieve a better screening effect for suspected targets, in this embodiment, the centerline distances of the suspected targets relative to the centerline are calculated respectively, and then the centerline distances are sorted to obtain the minimum centerline distance, and the suspected target corresponding to the minimum centerline distance is output as the actual target.
[0089] The calculation process of the centerline distance includes:
[0090] S2=min(|x 足 -x 屏 |,|y 足 -y 屏 |);
[0091] Where s2 is the center line distance, (x 足 ,y 足 ) is the coordinate of the suspected target, (x 屏 ,y 屏 ) are the coordinates of the center point of the screen.
[0092] In one embodiment, Figure 6 As shown, in step S22, a centerline distance sequence is also constructed during the sorting process, and the centerline distance sequence includes all centerline distances;
[0093] In the centerline distance sequence, the centerline distances other than the minimum centerline distance are marked as centerline distances to be determined;
[0094] Step S2 also includes:
[0095] Step S26: Subtract each centerline distance to be determined from the minimum centerline distance to obtain a second subtraction result;
[0096] Step S27: comparing the second subtraction result with the second subtraction threshold to determine whether there is a second subtraction result that is less than the second subtraction threshold;
[0097] If yes, go to step S28;
[0098] If not, the suspected target corresponding to the minimum centerline distance is output as the actual target;
[0099] Step S28: searching for the nearest suspected target as the actual target according to the actual target of the previous frame.
[0100] Specifically, after calculating the centerline distance, in order to improve the accuracy of target recognition, in this embodiment, after calculating the centerline distance of each suspected target relative to the center point of the picture, sorting to obtain the centerline distance sequence and screening to obtain the suspected target corresponding to the minimum centerline distance, it is further judged whether there are other similar suspected targets around the centerline. Specifically, each centerline distance to be judged is subtracted from the minimum centerline distance to obtain a second subtraction result. If there is a second subtraction result less than the second subtraction threshold, it indicates whether there are other similar suspected targets around the center of the picture. Further judgment is required at this time. For example, when the minimum centerline distance is 20 pixels, but the second minimum centerline distance is 30 pixels, the difference may be caused by the photographer not accurately aiming at the target when following the shot, and the target is too close to the interference object. Therefore, on this basis, it is necessary to further screen the suspected targets, for example, according to the coordinate position of the actual target in the previous frame, find the suspected target closest to the position as the actual target.
[0101] In one embodiment, when no suspected target is identified in step S1, continuous trajectory prediction is performed according to actual targets in other frames to obtain the actual target in the current single-frame image.
[0102] Specifically, in order to achieve a better labeling process, in this embodiment, when a suspected target is not correctly identified in a single-frame image of a certain frame in step S1, the single-frame image can be marked and other single-frame images can be processed. After the actual target is identified in all single-frame images with suspected targets, continuous trajectory prediction is performed according to the actual targets on other frames to obtain the actual target on the current single-frame image, including:i =(x0+(y n -x0)*i / n);y i =(y0+(y n -y0)*i / n), [i:1~n-1].
[0103] In the formula, (x i ,y i ) is the predicted actual target, (x0, y0) is the actual target coordinate in the first frame, (x n ,y n ) is the actual target on the nth frame.
[0104] In one embodiment, Figure 7 As shown, step S3 includes:
[0105] Step S31: determining the center of the cropped single frame image according to the actual target;
[0106] Step S32: determining the left boundary and the right boundary of the cropped video image according to the center of the picture and the single frame image;
[0107] Step S33: cropping the video image according to the left boundary and the right boundary of the video image to obtain a cropped video.
[0108] Specifically, the cutting process includes:
[0109] y mobile phone = y program video; x mobile phone left = x foot - y program video * 9 / 16 / 2
[0110] Specifically, in order to achieve a better cropping effect, the above cropping method is adopted in this embodiment. Specifically, the predicted coordinates of the actual target are first used as the center of the cropped image, and then the size of the cropping frame is determined according to the above formula, that is, y 手机 =y 节目视频 ;x 手机左 =x 足 -y 节目视频 *9 / 16 / 2.
[0111] In the formula, y 手机 The height (H) of the cropping frame after cropping, y 手机 is the height (H) of the original video in pixels, x 手机左 The number of pixels on the left edge of the cropping box after cropping, y 节目视频 is the width (W) of the original video in pixels, x 足 The actual number of pixels on the X axis of the target.
[0112] Based on the above settings, the height of the mobile phone vertical screen can be controlled to the height of the original video of the TV program. After determining the height, it is easy to determine the width of the mobile phone vertical screen as y节目视频 *9 / 16. At the same time, in order to make the sphere fall on the vertical screen center line, the left edge of the mobile phone vertical screen is controlled to be the x coordinate of the football minus half of the mobile phone vertical screen width. If the value is negative, it is assigned to 0. If the value is greater than x 节目视频 -The width of the vertical screen of the mobile phone (y 节目视频 *9 / 16), then force x 手机左 =x 节目视频 -y 节目视频 *9 / 16 / 2. To avoid cropping to the invalid area on the right side of the original video. Regardless of the above, the right border is: x 手机右 =x 手机左 +The width of the vertical screen of the mobile phone (y 节目视频 *9 / 16).
[0113] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for converting horizontal screen to vertical screen broadcasting based on object tracking, characterized in that: include: Step S1: for a single frame image in the video to be cropped, a plurality of suspected targets are identified respectively; Step S2: for each of the suspected targets, respectively calculating the target distance of the suspected target relative to the center point of the picture, so as to take the suspected target with the smallest target distance as the actual target; Step S3: cropping the video to be cropped according to the actual target in the single frame image.
2. The method for converting horizontal screen to vertical screen broadcasting based on object tracking according to claim 1, characterized in that: The step S1 comprises: Step S11: extracting the single frame image from the video to be cropped, and using a target recognition model to recognize the single frame image to obtain a recognition frame and a confidence level corresponding to the suspected target output by the target recognition model; Step S12: screening the identification boxes according to the confidence and a pre-configured confidence threshold, so as to remove the identification boxes whose confidence is lower than the confidence threshold; Step S13: extracting the center coordinates of the identification frame, and outputting the extraction result as the coordinate position of the suspected target.
3. The method for converting horizontal screen to vertical screen broadcasting based on object tracking according to claim 1, characterized in that: The step S2 comprises: Step S21: calculating the Euclidean distance of each suspected target relative to the center point of the picture as the target distance; Step S22: sorting the target distances to obtain a minimum target distance, and outputting the suspected target corresponding to the minimum target distance as the actual target.
4. The method for converting horizontal screen to vertical screen broadcasting based on object tracking according to claim 3, characterized in that: In the step S22, a target distance sequence is also constructed during the sorting process, and the target distance sequence includes all the target distances; In the target distance sequence, the target distances other than the minimum target distance are marked as target distances to be determined; The step S2 further comprises: Step S23: Subtract each of the target distances to be determined from the minimum target distance to obtain a first subtraction result; Step S24: comparing the first subtraction result with a first subtraction threshold to determine whether there is a first subtraction result smaller than the first subtraction threshold; If yes, go to step S25; If not, outputting the suspected target corresponding to the minimum target distance as the actual target; Step S25: The actual target is obtained by screening according to the centerline distance of the suspected target relative to the centerline.
5. The method for converting horizontal screen to vertical screen broadcasting based on object tracking according to claim 4, characterized in that: The step S25 comprises: Step S251: respectively calculating the centerline distance of the suspected target relative to the centerline; Step S252: sorting the centerline distances to obtain a minimum centerline distance, and outputting the suspected target corresponding to the minimum centerline distance as the actual target.
6. The method for converting horizontal screen to vertical screen broadcasting based on object tracking according to claim 5, characterized in that: In the step S22, a centerline distance sequence is also constructed during the sorting process, and the centerline distance sequence includes all the centerline distances; In the centerline distance sequence, the centerline distances other than the minimum centerline distance are marked as centerline distances to be determined; The step S2 further comprises: Step S26: Subtract each of the to-be-determined centerline distances from the minimum centerline distance to obtain a second subtraction result; Step S27: comparing the second subtraction result with a second subtraction threshold to determine whether there is a second subtraction result that is smaller than the second subtraction threshold; If yes, go to step S28; If not, output the suspected target corresponding to the minimum centerline distance as the actual target; Step S28: searching for the nearest suspected target as the actual target according to the actual target in the previous frame.
7. The method for converting horizontal screen to vertical screen broadcasting based on object tracking according to claim 1, characterized in that: When the suspected target is not identified in the step S1, continuous trajectory prediction is performed according to the actual targets on other frames to obtain the actual target on the current single-frame image.
8. The method for converting horizontal screen to vertical screen broadcasting based on object tracking according to claim 1, characterized in that: The step S3 comprises: Step S31: determining the center of the cropped single frame image according to the actual target; Step S32: determining a left boundary and a right boundary of a cropped video image according to the center of the picture and the single frame image; Step S33: cropping the video image according to the left boundary and the right boundary of the video image to obtain a cropped video.
