Video processing method and device and terminal equipment

Abstract

The invention provides a video processing method and device and terminal equipment, and the method comprises the steps: obtaining a first video, and enabling the similarity between first video frames in the first video to be greater than or equal to a first threshold value; determining the number of objects in each first video frame; and for any first video frame, determining a target coordinate based on the number of objects in the first video frame, and performing clipping processing on the first video frame based on a preset size by taking the target coordinate as a clipping center point to obtain a second video frame corresponding to the first video frame.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of video processing technology, and in particular to a video processing method, apparatus, and terminal device. Background Art

[0002] Videos captured by video capture devices are usually in a horizontal ratio of 16:9 or 4:3 (to facilitate video playback on devices such as TVs). With the rise of short videos, more and more users are watching videos on mobile devices. Therefore, users need to crop the captured video to adapt to the screen ratio of the mobile device.

[0003] Currently, users can crop each video frame in a video based on the screen size of a mobile device so that the size of each video frame in the video is adapted to the screen size of the mobile device. For example, users can determine the main playback content in each video frame of a video based on experience, and based on the screen size of the mobile device and the main playback content, crop an image in the video frame that is the same as the screen size of the mobile device, thereby obtaining a cropped video that is adapted to the screen size of the mobile device. However, the labor cost in the above method is high, and the difficulty of cropping based on experience is high, so the efficiency of video cropping is low. Summary of the invention

[0004] The present disclosure provides a video processing method, apparatus and terminal device, which are used to solve one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides a video processing method, the method comprising:

[0006] Acquire a first video, wherein a similarity between first video frames in the first video is greater than or equal to a first threshold;

[0007] determining the number of objects in each first video frame;

[0008] For any first video frame, the target coordinates are determined based on the number of objects in the first video frame, and the first video frame is cropped based on a preset size with the target coordinates as the center point of cropping to obtain a second video frame corresponding to the first video frame.

[0009] In a second aspect, the present disclosure provides a video processing device, the video processing device comprising an acquisition module, a first determination module, a second determination module and a cropping module, wherein:

[0010] The acquisition module is used to acquire a first video, wherein the similarity between first video frames in the first video is greater than or equal to a first threshold;

[0011] The first determination module is used to determine the number of objects in each first video frame;

[0012] The second determination module is used to determine the target coordinates for any first video frame based on the number of objects in the first video frame;

[0013] The cropping module is used to crop the first video frame based on a preset size with the target coordinates as the center point of cropping to obtain a second video frame corresponding to the first video frame.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video processing method as described in the first aspect and various possible aspects of the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the video processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0018] The present disclosure provides a video processing method, apparatus and terminal device, wherein the terminal device can obtain a first video, wherein the similarity between each first video frame in the first video is greater than or equal to a first threshold, determine the number of objects in each first video frame, and for any first video frame, determine the target coordinates based on the number of objects in the first video frame, and use the target coordinates as the center point of cropping, and based on a preset size, crop the first video frame to obtain a second video frame corresponding to the first video frame. In the above method, the terminal device can determine the center point of cropping the first video frame based on the number of objects included in the first video frame in the first video, and then can accurately crop the first video frame in combination with the video content, thereby improving the accuracy of video cropping, reducing the difficulty of video cropping, and improving the efficiency of video cropping. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of a process for determining a second video frame provided by an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a method for determining target coordinates provided in an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of determining a first area provided in an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of another process for determining a first video frame provided by an embodiment of the present disclosure;

[0026] Figure 7 A minimum area schematic diagram provided for an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of another method for determining target coordinates provided by an embodiment of the present disclosure;

[0028] Fig. 9 A schematic diagram of a process for determining target coordinates provided in an embodiment of the present disclosure;

[0029] Fig.10 A schematic diagram of a weighted undirected graph provided in an embodiment of the present disclosure;

[0030] Fig.11 A schematic diagram of a process for determining target coordinates provided in an embodiment of the present disclosure;

[0031] Fig.12 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure; and

[0032] Fig.13 A schematic diagram of the structure of a terminal device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0035] Terminal device: a device with wireless transceiver function. The terminal device can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted. The terminal device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a vehicle-mounted terminal device, a wireless terminal in self-driving, a wireless terminal device in remote medical, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, a wireless terminal device in smart home, a wearable terminal device, etc. The terminal device involved in the embodiments of the present disclosure can also be called a terminal, a user equipment (UE), an access terminal device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote terminal device, a mobile device, a UE terminal device, a wireless communication device, a UE agent or a UE device, etc. Terminal devices can also be fixed or mobile.

[0036] In the related art, in order to adapt to the screen size of video playback devices such as video TVs and computers, the video size usually collected by the video capture device can be a horizontal 16:9 or 4:3 ratio. However, with the rise of short videos, more and more users can watch short videos played vertically in mobile devices. Therefore, it is necessary to crop the size of the video collected by the video capture device to meet the needs of vertical playback of mobile devices. At present, the user can crop each video frame in the video based on the screen size of the mobile device, so that the size of each video frame is adapted to the screen size of the mobile device to obtain a cropped video. For example, the user can determine the main playback content (characters in the video, etc.) in the video based on experience, and crop an image including the main playback content in the video. The size of the image can be the same as the screen size of the mobile device, and then a video adapted to the mobile device can be obtained. However, the labor cost in the above method is high, and the difficulty and accuracy of experience-based cropping are high, which leads to low efficiency of video cropping.

[0037] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides a video processing method, and a terminal device can obtain a first video, wherein the similarity between each first video frame in the first video is greater than or equal to a first threshold, and the number of objects in each first video frame is determined. For any first video frame, if the number of objects in the first video frame is 0, a first area is determined in the first video frame, and based on the first area corresponding to the first video frame, the target coordinates are determined. If the number of objects in the first video frame is greater than 0, the video frame type of the first video frame is determined, wherein the video frame type can be a type of a single object or a type of multiple objects. The terminal device can determine the target coordinates based on the video frame type, and use the target coordinates as the center point of cropping, based on a preset size, crop the first video frame to obtain a cropped video frame corresponding to the first video frame. After all the first video frames in the first video are processed based on the above-mentioned video processing method, a cropped video can be obtained, wherein the size of each video frame in the cropped video is a preset size. In this way, since the terminal device can determine the cropping method of the first video frame based on the number of objects included in the first video frame in the first video, the terminal device can accurately and flexibly crop each first video frame in the first video based on the video content to obtain a video frame of a preset size, thereby improving the accuracy and flexibility of video cropping, reducing the difficulty of video cropping, and improving the efficiency of video cropping.

[0038] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0039] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 , including a terminal device and a mobile device. The terminal device can play a first video, the first video can be a video of horizontal size, and the size of the first video is adapted to the screen size of the terminal device. The terminal device can crop the first video to obtain a cropped video, wherein the cropped video can be a video of vertical size. The terminal device can send the cropped video to the mobile device, and the mobile device can play the cropped video. In this way, the terminal device can crop the first video based on the content of the first video to obtain a cropped video adapted to the screen size of the mobile device, thereby reducing the difficulty of video cropping and improving the efficiency and accuracy of video cropping.

[0040] It should be noted that Figure 1 The application scenarios of the embodiments of the present disclosure are merely illustrative and are not intended to limit the application scenarios of the embodiments of the present disclosure.

[0041] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0042] Figure 2 A flowchart of a video processing method provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0043] S201, obtaining a first video.

[0044] The execution subject of the embodiments of the present disclosure may be a terminal device or a video processing device arranged in the terminal device, wherein the video processing device may be implemented based on software or a combination of software and hardware, which is not limited by the embodiments of the present disclosure.

[0045] The similarity between the first video frames in the first video is greater than or equal to the first threshold. For example, the first video may include multiple first video frames, wherein the image similarity between any two first video frames is greater than or equal to the first threshold, so that the image content included in each first video frame in the first video is similar, which facilitates cropping of the first video frames and improves the accuracy of video cropping.

[0046] It should be noted that the terminal device can determine the similarity between video frames in a video based on any feasible implementation method (such as cosine similarity, Euclidean distance, etc.) to obtain the first video, and the embodiments of the present disclosure are not limited to this.

[0047] Optionally, a long video shot by a video capture device usually includes multiple scenes or pictures, so the terminal device can segment the video shot by the video capture device to obtain multiple first videos. For example, a video shot by a video capture device may include multiple scene transitions (indoor scenes are converted to outdoor scenes, etc.), so before the terminal device performs cropping processing on the video frame, it can divide the video into multiple first videos based on the multiple scenes in the video, so that the scenes in each first video are relatively continuous, which can facilitate video cropping processing and improve the efficiency of video cropping.

[0048] Optionally, the terminal device can obtain the first video based on the following feasible implementation method: for any video frame in the long video, the terminal device can determine the first image feature of the video frame and the second image feature of the previous frame of the video frame, and the terminal device can determine the similarity between the first image feature and the second image feature. If the similarity is greater than or equal to a preset threshold, the terminal device can segment before the video frame, thereby obtaining multiple first videos.

[0049] Optionally, the terminal device may determine the first image feature and the second image feature based on a convolutional neural network. For example, the terminal device may input a video frame into the convolutional neural network to obtain a first image feature corresponding to the video frame, and the terminal device may input a previous frame of the video frame into the convolutional neural network to obtain a second image feature.

[0050] Optionally, the terminal device may obtain a color distribution histogram of the video frame, and then obtain an image feature of the video frame. For example, the terminal device may process the video frame and the previous frame of the video frame based on an algorithm for determining a color distribution histogram, and then obtain a color distribution histogram (first image feature) of the video frame and a color distribution histogram (first image feature) of the previous frame of the video frame.

[0051] It should be noted that the image features in the embodiments of the present disclosure may include color distribution histograms, image hash value features, high-dimensional features extracted by convolutional neural networks, etc., and the embodiments of the present disclosure are not limited to this.

[0052] S202: Determine the number of objects in each first video frame.

[0053] Optionally, the object may be a person, an animal, an object, etc., which is not limited in the embodiments of the present disclosure. For example, if the first video frame includes 1 person, the terminal device may determine that the number of objects in the first video frame is 1; if the first video frame includes a cat and a dog, the terminal device may determine that the number of objects in the first video frame is 2; if the first video frame includes 3 fruits, the terminal device may determine that the number of objects in the first video frame is 3.

[0054] It should be noted that the category of the object may be a pre-set category (eg, the terminal device may pre-set the object in each first video frame as a person), and this embodiment of the present disclosure does not limit this.

[0055] Optionally, the terminal device may determine the number of objects in each first video frame based on the category of the object. For example, if the terminal device determines that the object is a person, the terminal device may process the first video frame based on a human detection algorithm (such as a yolov5 algorithm, etc.) to obtain the number of people in the first video frame; if the terminal device determines that the object is an animal, the terminal device may process the first video frame based on an image recognition model (such as a trained neural network model for identifying animals) to obtain the number of animals in the first video frame.

[0056] It should be noted that the terminal device may also determine the number of objects in each first video frame based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0057] It should be noted that the number of objects in the first video frame may be 0, 1 or more, which is not limited in the embodiment of the present disclosure.

[0058] S203 . For any first video frame, determine target coordinates based on the number of objects in the first video frame.

[0059] The target coordinates may indicate the center point of the cropped first video frame. For example, if the target coordinates are (10, 10), the terminal device uses the coordinates (10, 10) as the center of the crop when cropping the first video frame; if the target coordinates are (3, 3), the terminal device uses the coordinates (3, 3) as the center of the crop when cropping the first video frame.

[0060] It should be noted that the maximum value of the horizontal coordinate of the coordinate in the embodiment of the present disclosure may be the number of pixels of the image in the horizontal direction, and the maximum value of the vertical coordinate may be the number of pixels of the image in the vertical direction, and the embodiment of the present disclosure is not limited to this.

[0061] Optionally, since the number of objects in the first video frame can be 0, 1, or M (an integer greater than 1), the terminal device determines the target coordinates in different ways for different numbers of objects. For example, if the number of objects in the first video frame is 0, it means that there is no obvious main image in the first video frame. Therefore, the terminal device can analyze the content in the first video frame, determine a more prominent area in the first video frame, and then determine the coordinates of the center point of the area as the target coordinates. For example, if the number of objects in the first video frame is 1, the terminal device can determine the area surrounding the object, and then determine the coordinates of the center point of the area as the target coordinates. For example, if the number of objects in the first video frame is greater than 1, the terminal device can determine multiple areas surrounding the object in the first video frame, and then determine the target coordinates based on the positional relationship of the multiple areas.

[0062] S204 , taking the target coordinates as the center point of cropping and based on a preset size, cropping the first video frame to obtain a second video frame corresponding to the first video frame.

[0063] Optionally, the preset size may be the size of the screen of the mobile device. For example, if the first video can be played on a mobile device with a screen ratio of 4:9, the aspect ratio of the preset size may be 4:9; if the first video can be played on a mobile device with a screen ratio of 9:16, the aspect ratio of the preset size may be 9:16.

[0064] It should be noted that the preset size may also be an arbitrarily set size, and the embodiments of the present disclosure are not limited to this.

[0065] The second video frame may be a video frame obtained by cropping the first video frame. Optionally, the size of the second video frame may be the same as the preset size. For example, if the aspect ratio of the preset size may be 4:9, the aspect ratio of the second video frame may be 4:9.

[0066] Optionally, the terminal device may obtain the second video frame corresponding to the first video frame based on the following feasible implementation method: using the target coordinates as the cropping center, cropping an image matching the preset size in the first video frame to obtain the second video frame. For example, if the number of pixels of the first video frame is 1600 (the number of pixels in the horizontal direction)*1200 (the number of pixels in the vertical direction), the preset size is 400*800, and the target coordinates are (300, 300), the terminal device may determine the target coordinates (300, 600) as the cropping center, intercept 150-450 pixels in the horizontal direction of the first video frame, and intercept 200-1000 pixels in the vertical direction of the first video frame, thereby obtaining the second video frame.

[0067] Next, combine Figure 3 , the process of determining the second video frame is described.

[0068] Figure 3 A schematic diagram of a process for determining a second video frame provided by an embodiment of the present disclosure. Figure 3 , including: a first video frame. The first video frame includes point A (a point at a target coordinate position) and a cropping area (an area determined based on a preset size and centered on A). The terminal device ( Figure 3 (not shown) can obtain an image within the cropped area in the first video frame to obtain a second video frame, wherein the size of the second video frame is the same as the size of the cropped area, and the content of the second video frame is the content of the cropped area of ​​the first video frame.

[0069] It should be noted that, based on the above-mentioned video processing method, the terminal device crops each first video frame in the first video to obtain multiple second video frames, and the terminal device splices the multiple second video frames to obtain a cropped video matching a preset size.

[0070] The present disclosure provides a video processing method, in which a terminal device can obtain a first video and determine the number of objects included in each first video frame in the first video. For any first video frame, the target coordinates are determined based on the number of objects in the first video frame, and the target coordinates are used as the center point of cropping. Based on a preset size, the first video frame is cropped to obtain a second video frame corresponding to the first video frame, and multiple second video frames are spliced ​​to obtain a video matching the preset size. In this way, since the terminal device can determine the cropping method for the first video frame based on the number of objects included in the first video frame in the first video, the terminal device can accurately and flexibly crop each first video frame in the first video in combination with the video content, thereby improving the accuracy and flexibility of video cropping, reducing the difficulty of video cropping, and improving the efficiency of video cropping.

[0071] exist Figure 2 Based on the embodiment shown in the figure, in the above video processing method, there are two cases for determining the target coordinates based on the number of objects in the first video. Figure 4-Figure 8 , the methods for determining the target in two situations are explained.

[0072] Figure 4 A schematic diagram of a method for determining target coordinates provided by an embodiment of the present disclosure. Figure 4 In the embodiment shown, the number of objects in the first video frame is 0, see Figure 4 The method flow includes:

[0073] S401: Determine a first area in a first video frame.

[0074] Among them, the first area can be a significant area in the first video frame. For example, since the number of objects in the first video frame is 0, the terminal device can determine the significant area in the first video frame, and then determine the target coordinates. For example, the significant area can be a visually significant area of ​​the first video frame. For example, some areas in the image can attract the user's attention more than other areas in the image. This part of the area can be a significant area in the image. The higher the visual significance, the more the part of the area attracts the user's attention. Among them, visual significance is also called visual attention, unpredictability, etc., which is not limited in the embodiments of the present disclosure.

[0075] Optionally, the terminal device determines the first area in the first video frame, and there are two cases as follows:

[0076] Case 1: The first video frame is the first frame in the first video.

[0077] If the first video frame is the first frame in the first video, the terminal device may perform convolution processing on the first video frame to obtain a first region corresponding to the first video frame. For example, the terminal device may process the first video frame based on a pre-trained convolutional neural network model to obtain the first region in the first video frame, wherein the convolutional neural network model is trained based on sample images of manually marked salient regions, which can improve the accuracy of the first region.

[0078] Optionally, the terminal device may also predict the first area in the first video frame based on underlying image features such as contrast, color, grayscale, etc. of different parts in the first video frame, which is not limited in the embodiments of the present disclosure.

[0079] Next, combine Figure 5 , in this case, the process of determining the first area is explained.

[0080] Figure 5 A schematic diagram of determining a first area provided in an embodiment of the present disclosure. Figure 5 , including a salient region detection model and a first video frame. The salient region detection model may include multiple downsampling layers, multiple convolutional layers, and multiple upsampling layers. After the salient region detection model obtains the first video frame, it may downsample the first video frame based on multiple downsampling layers, and convolve the downsampling results based on multiple convolutional layers, and then upsample the convolution results to obtain a mask of the first region. After the mask of the first region is superimposed on the first video frame, the first region in the first video frame may be obtained. In the process of upsampling the convolution results, the output results of multiple downsampling layers may be input, thereby improving the accuracy of upsampling and improving the accuracy of the first region.

[0081] Case 2: The first video frame is not the first frame in the first video.

[0082] If the first video frame is not the first frame in the first video, the first area corresponding to the previous frame of the first video frame and the optical flow information between the first video frame and the previous frame are obtained, and the first area corresponding to the first video frame is determined based on the first video frame, the first area corresponding to the previous frame and the optical flow information.

[0083] Among them, since the first video frame is not the first frame in the first video, the first video frame has a previous frame, and the terminal device can obtain the first area corresponding to the previous frame (for example, the terminal device can obtain the first area of ​​the first frame based on the method in situation 1, and obtain the first area of ​​each frame in turn based on the method in situation 2).

[0084] Optionally, the terminal device can process the first video frame and the previous frame of the first video frame based on any feasible implementation method to obtain optical flow information between the first video frame and the previous frame of the first video frame (optical flow is the movement of an object between a continuous frame sequence), which is not limited to the embodiments of the present disclosure.

[0085] Among them, the terminal device determines the first area corresponding to the first video frame based on the first video frame, the first area corresponding to the previous frame and the optical flow information. Specifically, it can be: based on the optical flow information, the first area corresponding to the previous frame is offset to obtain the predicted area corresponding to the first video frame, and the first video frame and the predicted area are convolved to obtain the first area corresponding to the first video frame.

[0086] Optionally, since the optical flow information may include offset information between pixels in the first video frame and pixels in the previous frame, the terminal device may offset the first area corresponding to the previous frame based on the optical flow information, and thereby estimate the first area in the first video frame (i.e., the predicted area).

[0087] It should be noted that the first area corresponding to the previous frame may also be the saliency mask of the previous frame. Therefore, the terminal device may process the saliency mask based on the optical flow information to estimate the predicted area of ​​the first video frame.

[0088] Optionally, the terminal device can perform convolution processing on the first video frame and the predicted area based on a pre-trained neural network model, and thus obtain the first area corresponding to the first video frame. In this way, when the neural network model predicts the first area in the first video frame, not only the image information in the first video frame can be obtained, but also the estimated predicted area of ​​the first video frame can be obtained, thereby improving the prediction accuracy of the first video frame.

[0089] Next, combine Figure 6 In this case, the process of determining the first area corresponding to the first video frame is explained.

[0090] Figure 6 A schematic diagram of another process for determining a first video frame provided by an embodiment of the present disclosure. Figure 6, including: a salient region detection model, a first video frame, a previous frame of the first video frame, and a mask of the previous frame (saliency mask). The salient region detection model may include multiple downsampling layers, multiple convolutional layers, and multiple upsampling layers. The terminal device ( Figure 6 (not shown) optical flow estimation can be performed on the first video frame and the previous frame to obtain optical flow information between the first video frame and the previous frame, and based on the optical flow information and the mask of the previous frame, a predicted area of ​​the first video frame is predicted.

[0091] See also Figure 6 , the terminal device can input the first video frame into the salient region detection model, the multiple downsampling layers can downsample the first video frame, the convolution layer can convolve the output result of the downsampling layer, and input the convolution result to the upsampling layer. The upsampling layer 1 can upsample the convolution result and input the upsampled result to the upsampling layer 2. The upsampling layer 2 can determine the output result of the upsampling layer 2 based on the output result of the upsampling layer 1, the predicted region after two downsamplings (in order to align the predicted region with the size of the first video frame in the convolution process), and the output result of the downsampling layer 2.

[0092] See also Figure 6 , the upsampling layer 2 can input the output result of the upsampling layer 2 to the upsampling layer 3, and the upsampling layer 3 can output the mask of the first area based on the output result of the upsampling layer 2, the predicted area of ​​1 downsampling, and the output result of the downsampling layer 1. In this way, when determining the first area in the non-first frame, the terminal device can not only be based on the image information of the first video frame, but also can be combined with the image information of the previous frame, thereby improving the accuracy of the first area.

[0093] S402: Determine target coordinates based on a first area corresponding to a first video frame.

[0094] The terminal device determines the target coordinates based on the first area corresponding to the first video frame, and there are two situations as follows:

[0095] Case 1: The number of the first region is 1.

[0096] If the number of first regions is 1, the coordinates of the center of the first region are determined as the target coordinates. For example, after the terminal device acquires the first region in the first video frame, if there is only one first region in the first video frame, it means that there is one significant region in the first video frame, so the terminal device can determine the coordinates of the center of the first region as the target coordinates.

[0097] It should be noted that the terminal device may also determine the coordinates of any point in the first area as the target coordinates, which is not limited in the embodiment of the present disclosure.

[0098] Case 2: The number of first regions is greater than one.

[0099] If the number of first areas is greater than 1, the size of the smallest area surrounding the multiple first areas is obtained; if the size of the smallest area is less than or equal to a preset size, the target coordinates are determined based on the coordinates of the centers of the multiple first areas.

[0100] The minimum area may be the smallest area among the multiple areas surrounding the multiple first areas. For example, if the first video frame includes the first area 1, the first area 2 and the first area 3, the minimum area may be the smallest area surrounding the first area 1, the first area 2 and the first area 3.

[0101] Next, combine Figure 7 , describing the smallest area.

[0102] Figure 7 A minimum area schematic diagram provided for an embodiment of the present disclosure. Figure 7 , including a first video frame. The first video frame includes three first regions A, B, and C. The three first regions have different positions and areas in the first video frame. The terminal device ( Figure 7 (Not shown) The minimum area surrounding area A, area B, and area C may be determined in the first video frame based on the positions of area A, area B, and area C in the first video frame.

[0103] Optionally, if the size of the smallest area is less than or equal to the preset size, the terminal device may determine the target coordinates based on the coordinates of the centers of the multiple first areas. For example, if the size of the smallest area is less than or equal to the preset size, it means that after the terminal device crops the first video frame based on the preset size, the obtained second video frame may include all the first areas, and therefore, the terminal device may crop the first video frame.

[0104] For example, the terminal device can calculate the average value of the horizontal coordinate and the average value of the vertical coordinate based on the coordinates of multiple first area centers, and then determine the average value of the horizontal coordinate as the horizontal coordinate of the target coordinate, and determine the average value of the vertical coordinate as the vertical coordinate of the target coordinate. It should be noted that in this case, the terminal device can also determine the coordinates of the center of the smallest area as the target coordinate, and the embodiments of the present disclosure are not limited to this.

[0105] It should be noted that the terminal device may also determine the target coordinates based on the coordinates of any point in each first area, and the embodiments of the present disclosure are not limited to this.

[0106] Optionally, if the size of the minimum area is larger than a preset size, it means that after the terminal device crops the first video frame, the second video frame obtained cannot include all of the first area (significant area). Therefore, in this case, the terminal device can stop cropping the first video frame, and the terminal device can scale the first video frame so that the mobile device can display all the contents of the first video frame. However, since the first video frame after scaling does not match the size of the screen of the mobile device (e.g., blank areas will appear at the upper and lower ends of the screen of the mobile device), the terminal device can extract the main color in the first video frame (based on a preset color extraction algorithm or a pre-trained model, which is not limited in the embodiment of the present disclosure), and then fill the scaled first video frame with the extracted main color to a video frame of a preset size (i.e., splicing the image of the main color at the upper and lower ends of the scaled first video frame to obtain a new video frame of a preset size). In order to improve the visual effect of the video frame, the terminal device can perform Gaussian blur processing on the filled main color, thereby improving the user experience.

[0107] It should be noted that before the terminal device determines the size of the minimum area, the terminal device may filter the obtained multiple first areas, thereby reducing the number of first areas. For example, after the terminal device obtains multiple first areas, if the area of ​​some first areas is less than or equal to the preset area, the terminal device may remove the first area (i.e., filter the small connected domain), which can facilitate the determination of the target coordinates and improve the efficiency of video cropping.

[0108] The disclosed embodiment provides a method for determining target coordinates, wherein a first region is determined in a first video frame, and if the number of first regions is 1, the coordinates of the center of the first region are determined as the target coordinates, and if the number of first regions is greater than 1, the size of the smallest region surrounding multiple first regions is obtained, and if the size of the smallest region is less than or equal to a preset size, the target coordinates are determined based on the coordinates of the centers of multiple first regions. In this way, when the first video frame does not include an object, the terminal device can crop the first video frame based on the salient region (i.e., the first region) in the first video frame to improve the accuracy of video cropping.

[0109] Figure 8 A schematic diagram of another method for determining target coordinates provided by an embodiment of the present disclosure. Figure 8 In the embodiment shown, the number of objects in the first video frame is greater than 0, see Figure 8 The method flow includes:

[0110] S801: Determine a video frame type of a first video frame.

[0111] The video frame type is a type of a single object or a type of multiple objects. For example, if the first video frame includes one object, the video frame type of the first video frame is a type of a single object; if the first video frame includes multiple objects, the video frame type of the first video frame is a type of multiple objects.

[0112] S802: Determine target coordinates based on the video frame type and the first video frame.

[0113] Optionally, the object in the embodiment of the present disclosure may be a person, and the terminal device determines the target coordinates based on the video frame type and the first video frame. There are two cases as follows:

[0114] Case 1: The video frame type is a single object type.

[0115] If the video frame type is a single object type, then in the first video frame, the second area surrounding the person is determined, and the coordinates of the center of the second area are determined as the target coordinates. For example, if the video frame type of the first video frame is a single object type, it means that the first video frame includes a person, and the person can be the main content of the first video frame. Therefore, the terminal device can obtain the second area surrounding the person, and determine the coordinates of the center of the second area as the target coordinates. In this way, if the first video frame includes a task, the terminal device can quickly determine the target coordinates, thereby improving the efficiency of video cropping.

[0116] Optionally, the terminal device may determine the second area surrounding the person based on a human body detection algorithm, or the terminal device may determine the second area surrounding the face of the person based on a face detection algorithm, which is not limited in the embodiments of the present disclosure.

[0117] Optionally, the terminal device may also determine the coordinates of any point in the second area as the target coordinates, which is not limited in the embodiment of the present disclosure.

[0118] It should be noted that the second area in the first video frame may be the smallest area among all areas surrounding the person, or may be any area among all areas surrounding the person, which is not limited in the embodiment of the present disclosure.

[0119] Case 2: The video frame type is the type of multiple objects.

[0120] If the video frame type is a type of multiple objects, the audio of the first video is obtained, and based on the audio, speaker detection is performed on each character in the first video frame to obtain the voice parameters corresponding to each character, and based on the voice parameters of each character, the target coordinates are determined in the first video frame.

[0121] The terminal device may process the first video and the audio of the first video based on the current speaker detection algorithm, and then obtain the voice parameters corresponding to the characters in each first video frame in the first video. Optionally, the terminal device may also determine the voice parameters corresponding to each character in the first video frame based on other methods, which is not limited in the embodiments of the present disclosure.

[0122] Optionally, the voice parameter is proportional to the volume of the character's voice. For example, if the voice parameter corresponding to the character in the first video frame is positive, it means that at the position of the first video frame in the first video, the character is speaking; if the voice parameter corresponding to the character in the first video frame is negative, it means that at the position of the first video frame in the first video, the character is not speaking. For example, if the voice parameters of character 1 and character 2 in the first video frame are both positive, and the voice parameter corresponding to character 1 is greater than the voice parameter corresponding to character 2, it means that at the position of the first video frame in the first video, the volume of character 1's speech is higher than the volume of character 2's speech.

[0123] Optionally, the terminal device determines the target coordinates in the first video frame based on the voice parameters of each character, which can be specifically: if the voice parameters of one or more characters in the first video frame are greater than the second threshold, the character with the highest voice parameter is determined among the one or more characters, and the coordinates of the center of the third area surrounding the character with the highest voice parameter are determined as the target coordinates. For example, if the voice parameter of only one character in the first video frame is positive, the terminal device can determine the area surrounding the character (which can be the minimum area surrounding the character, or any area surrounding the character, which is not limited in the embodiment of the present disclosure) as the third area, and determine the coordinates of the center of the third area as the target coordinates. For example, if the voice parameters of multiple characters in the first video frame are positive, the terminal device can obtain the character with the largest voice parameter, and determine the third area in the first video frame based on the character with the largest voice parameter, and then determine the target coordinates based on the coordinates in the third area. In this way, the terminal device can accurately and quickly determine the main content in the first video frame, thereby improving the accuracy of video cropping.

[0124] It should be noted that the third area may also be an area surrounding the face of the person, which is not limited in the embodiment of the present disclosure.

[0125] It should be noted that the target coordinates may also be the coordinates of any point in the third area, which is not limited in the embodiment of the present disclosure.

[0126] Next, combine Fig. 9 , in this case, the process of determining the target coordinates is explained.

[0127] Fig. 9A schematic diagram of a process for determining target coordinates provided by an embodiment of the present disclosure. Fig. 9 , including: a first video frame. The first video frame includes person 1 and person 2. The terminal device ( Fig. 9 (not shown) The video in which the first video frame is located and the audio of the video can be input into the speaker detection algorithm to obtain the voice parameters of person 1 in the first video frame and the voice parameters of person 2 in the first video frame.

[0128] See also Fig. 9 Since the voice parameters of character 1 are greater than those of character 2, the terminal device can determine that the third area in the first video frame is the minimum area surrounding character 1. The terminal device can determine the center point A in the third area and determine the coordinates of point A in the first video frame as the target coordinates. In this way, after the terminal device crops the first video frame, the second video frame obtained can include character 1 who is speaking, thereby improving the user experience and the accuracy of video cropping.

[0129] Among them, if the voice parameter of each character in the first video frame is less than or equal to the second threshold, then in the first video frame, multiple fourth areas surrounding the character are determined, and the target coordinates are determined based on the multiple fourth areas. For example, if the voice parameter of each character in the first video frame is less than the second threshold, it means that there is no character speaking at the position of the first video frame of the first video. Therefore, the terminal device can determine the fourth area corresponding to each character (which can be the smallest area surrounding the character), and determine the target coordinates based on the multiple fourth areas.

[0130] Optionally, the terminal device determines the target coordinates based on multiple fourth areas, specifically by constructing a weighted undirected graph corresponding to the multiple fourth areas based on the distances between the multiple fourth areas, and determining the target coordinates based on the weighted undirected graph and the multiple fourth areas.

[0131] Among them, the weighted undirected graph can be a data structure in graph theory. The weighted undirected graph is composed of a group of vertices and a group of edges. Each edge connects 2 vertices, and there is a weight associated with the edge. In the embodiment of the present disclosure, a group of nodes of the weighted undirected graph are related to multiple fourth area objects, and the weight associated with the edge between any two nodes is related to the distance between the two fourth areas corresponding to the two nodes.

[0132] Next, combine Fig.10 , the weighted undirected graph is explained.

[0133] Fig.10 A schematic diagram of a weighted undirected graph provided by an embodiment of the present disclosure. Fig.10, including: a first video frame. The first video frame includes four fourth areas A, B, C, and D. The terminal device ( Fig.10 (not shown) A weighted undirected graph corresponding to the four fourth regions of the first video frame can be constructed based on the distances between the four fourth regions. Among them, the weighted undirected graph includes node 1 corresponding to the fourth region A, node 2 corresponding to the fourth region B, node 3 corresponding to the fourth region C, and node 4 corresponding to the fourth region D. Among them, each node includes 3 edges, and the edge weight can be the distance between the fourth regions corresponding to the nodes. For example, the edge weights of node 1 and node 4 can be determined based on the distance between the fourth region A and the fourth region D (can be proportional or inversely proportional, which is not limited in the embodiments of the present disclosure).

[0134] Among them, the terminal device determines the target coordinates based on the weighted undirected graph and multiple fourth areas, which can be specifically: determining the minimum spanning tree corresponding to the weighted undirected graph, clustering multiple nodes of the minimum spanning tree based on the edge weights between each node in the minimum spanning tree and L clustering centers to obtain L subgraphs, and determining the target coordinates based on the L subgraphs.

[0135] The minimum spanning tree of a weighted undirected graph refers to a subtree that includes all vertices in the weighted undirected graph, and the sum of the edge weights of the subtree is the smallest. Fig.11 In the embodiment shown, the terminal device can remove the edge between node 1 and node 2, remove the edge between node 1 and node 4, and remove the edge area between node 2 and node 4, thereby obtaining a subtree, which includes node 1, node 2, node 3, node 4, and the edge between node 1 and node 3, the edge between node 2 and node 3, and the edge between node 3 and node 4. The terminal device can obtain all subtrees corresponding to the weighted undirected graph, and among all the subtrees, determine the subtree with the smallest edge weight as the minimum spanning tree of the weighted undirected graph.

[0136] Wherein, L may be an integer greater than 1. For example, if L is 2, the terminal device may classify multiple nodes in the minimum spanning tree into 2 categories based on the weight of each edge in the minimum spanning tree; if L is 3, the terminal device may classify multiple nodes in the minimum spanning tree into 3 categories based on the weight of each edge in the minimum spanning tree.

[0137] The subgraph may be a graph obtained after the minimum spanning tree is clustered. For example, the minimum spanning tree includes nodes 1, 2, 3, and 4. If the minimum spanning tree is clustered into two categories, two subgraphs may be obtained, one of which may include nodes 1 and 2, and the other may include nodes 3 and 4.

[0138] It should be noted that, in actual application, L can be determined based on the size of the first video frame and the preset size. For example, if the first video frame can be a 16:9 size and the preset size can be a 5:11 size, the first video frame can be cut into three longitudinal video frames along the vertical direction, and therefore, L can be determined to be 3.

[0139] Optionally, the terminal device determines the target coordinates based on the L subgraphs, which may be specifically: determining a subgraph with the largest number of nodes among the L subgraphs, and determining the target coordinates based on a fourth area corresponding to each node in the subgraph with the largest number of nodes.

[0140] For example, after the terminal device performs clustering on the nodes in the minimum spanning tree, three subgraphs are obtained, among which Figure 1 There are 2 nodes in it, Figure 2 It includes 1 node, Figure 3 The terminal device can be based on the sub Figure 3 The three fourth regions corresponding to the three nodes in the 3D image are used to determine the target coordinates.

[0141] It should be noted that if there are multiple subgraphs with the largest number of nodes among the L subgraphs, the terminal device can determine the target subgraph based on the edge weights in the multiple subgraphs with the largest number of nodes, and determine the target coordinates based on the fourth area corresponding to the nodes in the target subgraph. For example, after the terminal device clusters the nodes in the minimum spanning tree, three subgraphs are obtained, among which Figure 1 There are 2 nodes in it, Figure 2 There are 3 nodes in it, Figure 3 There are 3 nodes in it, so Figure 2 Kazuko Figure 3 The number of nodes is the same as that of Figure 2 Kazuko Figure 3 are the subgraphs with the largest number of nodes. The terminal device can determine the subgraphs Figure 2 The sum of the edge weights in Figure 3 The sum of the edge weights in Figure 2 The sum of the edge weights in is less than Figure 3 The sum of the edge weights in the Figure 2 Determine as the target subgraph, where Figure 2 The sum of the edge weights in Figure 3 If the sum of the edge weights in is the same, the terminal device can determine any subgraph as the target subgraph, which is not limited in the embodiment of the present disclosure.

[0142] Optionally, the terminal device may determine the target coordinates based on the fourth area corresponding to the node in the subgraph with the largest number of nodes. For example, if the subgraph with the largest number of nodes includes node 1 and node 2, the terminal device may determine the target coordinates based on coordinate A of the center of the fourth area corresponding to node 1 and coordinate B of the center of the fourth area corresponding to node 2 (it may also be the coordinates of any point in the fourth area corresponding to node 1 and the coordinates of any point in the fourth area corresponding to node 2, which is not limited in the embodiments of the present disclosure), wherein the terminal device may calculate the mean of coordinate A and coordinate B to obtain the center coordinates, and the terminal device may also determine the target coordinates based on any other feasible implementation method, which is not limited in the embodiments of the present disclosure.

[0143] Next, combine Fig.11 , the process of determining the target coordinates is explained.

[0144] Fig.11 A schematic diagram of a process for determining target coordinates provided by an embodiment of the present disclosure. Fig.11 , including: a first video frame. The first video frame may include a fourth region A (region A), a fourth region B (region B), a fourth region C (region C) and a fourth region D (region D). The terminal device ( Fig.11 (Not shown) A directed unweighted graph may be constructed based on the distances between the fourth region A, the fourth region B, the fourth region C, and the fourth region D.

[0145] See also Fig.11 , the weighted undirected graph includes node 1 corresponding to the fourth region A, node 2 corresponding to the fourth region B, node 3 corresponding to the fourth region C, and node 4 corresponding to the fourth region D. Each node includes three edges, and the edge weight may be the distance between the fourth regions corresponding to the nodes. The terminal device may determine the minimum spanning tree corresponding to the weighted undirected graph.

[0146] See also Fig.11 , the minimum spanning tree may include node 1, node 2, node 3 and node 4, wherein node 3 is connected to node 1, node 2 and node 3 respectively. The terminal device may cluster the four nodes in the minimum spanning tree based on the two cluster centers and the edge weights in the minimum spanning tree to obtain sub-nodes. Figure 1 Kazuko Figure 2 .

[0147] See also Fig.11 ,son Figure 1 It can include node 1, node 2 and node 3. Figure 2 The terminal device can determine the sub Figure 1Therefore, the terminal device can determine the target coordinates based on the fourth area A, the fourth area B and the fourth area C in the first video frame.

[0148] See also Fig.11 , the terminal device can calculate the mean of the coordinates of the center of the fourth area A, the coordinates of the center of the fourth area B, and the coordinates of the center of the fourth area C, and obtain point E in the first video frame. The terminal device can determine the coordinates of point E in the first video frame as the target coordinates. In this way, when the first video frame includes multiple silent characters, the terminal device can determine the area (salient area) where the characters are dense in the first video frame based on the clustering processing method, thereby improving the accuracy of video cropping.

[0149] It should be noted that, in any one of the embodiments shown in the present disclosure, before the terminal device crops the first video frame, since the first video frame may include text (e.g., identifying the text and the position of the text in the first video frame through OCR), the terminal device can erase the text in the first video frame (e.g., erasing the text based on the inpainting image repair algorithm) and regenerate a text area that matches the preset size. In this way, after the terminal device crops the first video frame, the matching text can be displayed in the second video frame, thereby improving the accuracy of video cropping.

[0150] It should be noted that, in any of the embodiments shown in the present disclosure, before the terminal device crops the first video frame, the terminal device may perform jitter correction on the first video frames of the continuous multiple frames. For example, if the difference between the target coordinates of any two consecutive first video frames is greater than the preset difference, the terminal device may smooth the target coordinates between the two first video frames based on the index of the exponential average, thereby improving the display effect after the video is cropped. For example, if there are first video frames with different video frame types in multiple continuous first video frames, the terminal device may perform category correction on the first video frames with different video frame categories, so that the video frame categories of the multiple continuous first video frames are the same. For example, if the video frame type of the fifth first video frame among 10 continuous first video frames is the type of a single object, and the video frame types of the other first video frames are the types of multiple objects, the terminal device may modify the video frame type of the fifth first video frame to the type of multiple objects, so that the jitter problem after video cropping can be avoided.

[0151] The disclosed embodiment provides another method for determining target coordinates, which determines the video frame type of the first video frame. If the video frame type is a type of a single object, then in the first video frame, a second area surrounding a person is determined, and the coordinates of the center of the second area are determined as the target coordinates. If the video frame type is a type of multiple objects, then the audio of the first video is obtained, and based on the audio, speaker detection processing is performed on each person in the first video frame to obtain the voice parameters corresponding to each person, and based on the voice parameters of each person, the target coordinates are determined in the first video frame. In this way, the terminal device can determine the target coordinates based on different methods for different numbers of objects, thereby improving the flexibility of determining the target coordinates and improving the accuracy of the determined target coordinates.

[0152] Fig.12 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure. Fig.12 The video processing device 120 includes an acquisition module 121, a first determination module 122, a second determination module 123 and a cropping module 124, wherein:

[0153] The acquisition module 121 is used to acquire a first video, wherein the similarity between first video frames in the first video is greater than or equal to a first threshold;

[0154] The first determination module 122 is used to determine the number of objects in each first video frame;

[0155] The second determination module 123 is used to determine the target coordinates for any first video frame based on the number of objects in the first video frame;

[0156] The cropping module 124 is configured to crop the first video frame based on a preset size with the target coordinates as the center point of cropping, to obtain a second video frame corresponding to the first video frame.

[0157] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0158] If the number of objects in the first video frame is 0, determining a first area in the first video frame, and determining the target coordinates based on the first area corresponding to the first video frame;

[0159] If the number of objects in the first video frame is greater than 0, a video frame type of the first video frame is determined, and based on the video frame type and the first video frame, the target coordinates are determined, the video frame type being a type of a single object or a type of multiple objects.

[0160] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0161] If the first video frame is the first frame in the first video, performing convolution processing on the first video frame to obtain a first region corresponding to the first video frame;

[0162] If the first video frame is not the first frame in the first video, the first area corresponding to the previous frame of the first video frame and the optical flow information between the first video frame and the previous frame are obtained, and based on the first video frame, the first area corresponding to the previous frame and the optical flow information, the first area corresponding to the first video frame is determined.

[0163] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0164] Based on the optical flow information, performing an offset process on the first area corresponding to the previous frame to obtain a predicted area corresponding to the first video frame;

[0165] Convolution processing is performed on the first video frame and the predicted area to obtain a first area corresponding to the first video frame.

[0166] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0167] If the number of the first regions is 1, the coordinates of the center of the first region are determined as the target coordinates;

[0168] If the number of the first areas is greater than 1, the size of the smallest area surrounding the multiple first areas is obtained; if the size of the smallest area is less than or equal to the preset size, the target coordinates are determined based on the coordinates of the centers of the multiple first areas.

[0169] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0170] If the video frame type is a type of a single object, then in the first video frame, a second area surrounding the person is determined, and the coordinates of the center of the second area are determined as the target coordinates;

[0171] If the video frame type is a type of multiple objects, the audio of the first video is obtained, and based on the audio, speaker detection processing is performed on each character in the first video frame to obtain the voice parameters corresponding to each character, and based on the voice parameters of each character, the target coordinates are determined in the first video frame.

[0172] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0173] If the voice parameters of one or more characters in the first video frame are greater than the second threshold, the character with the highest voice parameter is determined among the one or more characters, and the coordinates of the center of the third area surrounding the character with the highest voice parameter are determined as the target coordinates.

[0174] If the voice parameter of each character in the first video frame is less than or equal to the second threshold, a plurality of fourth regions surrounding the characters are determined in the first video frame, and the target coordinates are determined based on the plurality of fourth regions.

[0175] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0176] Based on the distances between the plurality of fourth regions, constructing a weighted undirected graph corresponding to the plurality of fourth regions, wherein the nodes in the weighted undirected graph correspond to the fourth regions one-to-one, and the edges of the nodes in the weighted undirected graph are associated with the distances between the fourth regions corresponding to the nodes;

[0177] The target coordinates are determined based on the weighted undirected graph and the plurality of fourth regions.

[0178] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0179] Determine a minimum spanning tree corresponding to the weighted undirected graph;

[0180] Based on the edge weights between the nodes in the minimum spanning tree and L cluster centers, clustering the multiple nodes in the minimum spanning tree to obtain L subgraphs, wherein the subgraph includes at least one node, and L is an integer greater than 1;

[0181] Based on the L sub-images, the target coordinates are determined.

[0182] According to one or more embodiments of the present disclosure, the second determining module 123 is specifically configured to:

[0183] Among the L subgraphs, determine the subgraph with the largest number of nodes;

[0184] The target coordinates are determined based on a fourth region corresponding to the nodes in the subgraph having the largest number of nodes.

[0185] The video processing device provided in the embodiment of the present disclosure may be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0186] Fig.13 This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present disclosure. Fig.13 , which shows a schematic diagram of the structure of a terminal device 1300 suitable for implementing the embodiments of the present disclosure. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0187] like Fig.13 As shown, the terminal device 1300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1308 to a random access memory (RAM) 1303. Various programs and data required for the operation of the terminal device 1300 are also stored in the RAM 1303. The processing device 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0188] Typically, the following devices may be connected to the I / O interface 1305: input devices 1306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1309. The communication devices 1309 may allow the terminal device 1300 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.13 The terminal device 1300 is shown as having various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0189] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1309, or installed from a storage device 1308, or installed from a ROM 1302. When the computer program is executed by the processing device 1301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0190] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0191] The computer-readable medium may be included in the terminal device, or may exist independently without being installed in the terminal device.

[0192] The computer-readable medium carries one or more programs. When the one or more programs are executed by the terminal device, the terminal device executes the method shown in the above embodiment.

[0193] The embodiments of the present disclosure provide a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video processing methods that may be involved in the above embodiments are implemented.

[0194] The embodiments of the present disclosure provide a computer program product, including a computer program. When the computer program is executed by a processor, the video processing method that may be involved in the above embodiments is implemented.

[0195] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0196] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0197] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0198] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0199] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0200] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0201] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0202] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0203] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as a terminal device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure based on the prompt message. As an optional but non-limiting implementation method, in response to receiving an active request from a user, the method of sending a prompt message to the user can be, for example, a pop-up window, and the prompt message can be presented in text form in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the terminal device.

[0204] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0205] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws and regulations. The data may include information, parameters and messages, such as flow switching indication information.

[0206] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0207] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0208] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that: include: Acquire a first video, wherein a similarity between first video frames in the first video is greater than or equal to a first threshold; determining the number of objects in each first video frame; For any first video frame, the target coordinates are determined based on the number of objects in the first video frame, and the first video frame is cropped based on a preset size with the target coordinates as the center point of cropping to obtain a second video frame corresponding to the first video frame.

2. The method according to claim 1, characterized in that The determining the target coordinates based on the number of objects in the first video frame includes: If the number of objects in the first video frame is 0, determining a first area in the first video frame, and determining the target coordinates based on the first area corresponding to the first video frame; If the number of objects in the first video frame is greater than 0, a video frame type of the first video frame is determined, and based on the video frame type and the first video frame, the target coordinates are determined, the video frame type being a type of a single object or a type of multiple objects.

3. The method according to claim 2, characterized in that Determining a first area in the first video frame includes: If the first video frame is the first frame in the first video, performing convolution processing on the first video frame to obtain a first region corresponding to the first video frame; If the first video frame is not the first frame in the first video, the first area corresponding to the previous frame of the first video frame and the optical flow information between the first video frame and the previous frame are obtained, and based on the first video frame, the first area corresponding to the previous frame and the optical flow information, the first area corresponding to the first video frame is determined.

4. The method according to claim 3, characterized in that The determining, based on the first video frame, the first area corresponding to the previous frame, and the optical flow information, the first area corresponding to the first video frame includes: Based on the optical flow information, performing an offset process on the first area corresponding to the previous frame to obtain a predicted area corresponding to the first video frame; Convolution processing is performed on the first video frame and the predicted area to obtain a first area corresponding to the first video frame.

5. The method according to any one of claims 2 to 4, characterized in that: The determining the target coordinates based on the first area corresponding to the first video frame includes: If the number of the first regions is 1, the coordinates of the center of the first region are determined as the target coordinates; If the number of the first areas is greater than 1, the size of the smallest area surrounding the multiple first areas is obtained; if the size of the smallest area is less than or equal to the preset size, the target coordinates are determined based on the coordinates of the centers of the multiple first areas.

6. The method according to any one of claims 2 to 4, characterized in that: The object is a person; and determining the target coordinates based on the video frame type and the first video frame includes: If the video frame type is a type of a single object, then in the first video frame, a second area surrounding the person is determined, and the coordinates of the center of the second area are determined as the target coordinates; If the video frame type is a type of multiple objects, the audio of the first video is obtained, and based on the audio, speaker detection is performed on each character in the first video frame to obtain the voice parameters corresponding to each character, and based on the voice parameters of each character, the target coordinates are determined in the first video frame.

7. The method according to claim 6, characterized in that The determining the target coordinates in the first video frame based on the voice parameters of each character includes: If the voice parameter of one or more characters in the first video frame is greater than a second threshold, determining a character with the highest voice parameter among the one or more characters, and determining the coordinates of the center of a third area surrounding the character with the highest voice parameter as the target coordinates; If the voice parameter of each character in the first video frame is less than or equal to the second threshold, a plurality of fourth regions surrounding the characters are determined in the first video frame, and the target coordinates are determined based on the plurality of fourth regions.

8. The method according to claim 7, characterized in that The determining the target coordinates based on the plurality of fourth areas comprises: Based on the distances between the plurality of fourth regions, constructing a weighted undirected graph corresponding to the plurality of fourth regions, wherein the nodes in the weighted undirected graph correspond to the fourth regions one-to-one, and the edges of the nodes in the weighted undirected graph are associated with the distances between the fourth regions corresponding to the nodes; The target coordinates are determined based on the weighted undirected graph and the plurality of fourth regions.

9. The method according to claim 8, characterized in that The step of determining the target coordinates based on the weighted undirected graph and the plurality of fourth regions includes: Determine a minimum spanning tree corresponding to the weighted undirected graph; Based on the edge weights between the nodes in the minimum spanning tree and L cluster centers, clustering the multiple nodes in the minimum spanning tree to obtain L subgraphs, wherein the subgraph includes at least one node, and L is an integer greater than 1; Based on the L sub-images, the target coordinates are determined.

10. The method according to claim 9, characterized in that The determining the target coordinates based on the L sub-graphs includes: Among the L subgraphs, determine the subgraph with the largest number of nodes; The target coordinates are determined based on a fourth region corresponding to the nodes in the subgraph having the largest number of nodes.

11. A video processing device, characterized in that: It includes an acquisition module, a first determination module, a second determination module and a clipping module, wherein: The acquisition module is used to acquire a first video, wherein the similarity between first video frames in the first video is greater than or equal to a first threshold; The first determination module is used to determine the number of objects in each first video frame; The second determination module is used to determine the target coordinates for any first video frame based on the number of objects in the first video frame; The cropping module is used to crop the first video frame based on a preset size with the target coordinates as the center point of cropping to obtain a second video frame corresponding to the first video frame.

12. A terminal device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the video processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the video processing method according to any one of claims 1 to 10 is implemented.

Images