A multimedia data processing method, communication device and readable storage medium

Abstract

The present invention provides a multimedia data processing method, a communication device and a readable storage medium, wherein the multimedia data processing method comprises: determining a target object according to reference parameter information corresponding to at least two objects in a picture; intercepting the picture according to the target object; obtaining target multimedia data according to the intercepted picture; wherein the reference parameter information comprises: at least one of speech parameter information, identity parameter information and motion parameter information. This solution can realize the maximum degree of automatic cutting and greatly reduce time consumption, and well solves the problem of long time consumption of multimedia data processing solutions for video processing in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimedia data processing method, communication equipment and a readable storage medium. Background Art

[0002] Existing video processing methods usually require the use of video editing tools or manual processing, but this method is too time-consuming and not conducive to the conversion of large amounts of videos.

[0003] As can be seen from the above, existing multimedia data processing solutions for video processing have problems such as long time consumption. Summary of the invention

[0004] The object of the present invention is to provide a multimedia data processing method, a communication device and a readable storage medium to solve the problem that the multimedia data processing solution for video processing in the prior art is time-consuming.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a multimedia data processing method, including:

[0006] Determine the target object according to the reference parameter information respectively corresponding to at least two objects in the picture;

[0007] According to the target object, intercepting the picture;

[0008] Obtain target multimedia data according to the captured images;

[0009] The reference parameter information includes at least one of speech parameter information, identity parameter information and motion parameter information.

[0010] Optionally, the value of the reference parameter information is equal to the sum of the values ​​of the speech parameter information, the identity parameter information and the motion parameter information;

[0011] The step of determining the target object according to the reference parameter information respectively corresponding to at least two objects in the picture includes:

[0012] sorting the at least two objects according to values ​​of the reference parameter information respectively corresponding to the at least two objects;

[0013] Select the object in the preset position as the target object.

[0014] Optionally, before determining the target object according to the reference parameter information respectively corresponding to at least two objects in the picture, the method further includes:

[0015] Matching at least two objects in the picture with objects in a preset object library respectively;

[0016] According to the matching result, identity parameter information corresponding to at least two objects in the picture is obtained.

[0017] Optionally, obtaining identity parameter information corresponding to at least two objects in the picture respectively according to the matching result includes:

[0018] When the first object in the picture successfully matches the second object in the preset object library, obtaining identity parameter information corresponding to the first object according to ranking information of the second object in the preset object library;

[0019] In the case where the first object in the picture fails to match each object in the preset object library, using the preset identity parameter information as the identity parameter information corresponding to the first object;

[0020] The first object is any object in the picture.

[0021] Optionally, obtaining identity parameter information corresponding to the first object according to ranking information of the second object in the preset object library includes:

[0022] Obtaining difference information between the total ranking information of the preset object library and the ranking information;

[0023] The identity parameter information corresponding to the first object is obtained according to the difference information and the total ranking information.

[0024] Optionally, the capturing the picture according to the target object includes:

[0025] When the number of the target objects is at least two, obtaining at least one candidate interception method corresponding to the target objects;

[0026] Pre-capturing the picture by using the at least one candidate capture method to obtain at least one pre-capturing result;

[0027] According to the at least one pre-interception result, obtaining a target interception method from the at least one candidate interception method;

[0028] The picture is intercepted according to the target interception method.

[0029] Optionally, acquiring a target interception method from the at least one candidate interception method according to the at least one pre-interception result includes:

[0030] Performing an object integrity check on the at least one pre-cut result;

[0031] According to the detection result, a target interception method is obtained from the at least one candidate interception method.

[0032] Optionally, acquiring a target interception method from the at least one candidate interception method according to the detection result includes:

[0033] The candidate interception method corresponding to the detection result with the highest value is used as the target interception method.

[0034] Optionally, the capturing the picture according to the target object further includes:

[0035] When the highest value in the detection result is less than the preset threshold, the longitudinal middle points of the target objects at the left and right ends of the picture are taken as the left and right center points of the picture;

[0036] The picture is captured according to the left and right center points of the picture.

[0037] Optionally, the capturing the picture according to the target object includes:

[0038] Determine a cropping range according to the target object;

[0039] The picture is intercepted according to the cropping range.

[0040] Optionally, the intercepting the picture according to the cropping range includes:

[0041] When there is an incomplete object picture within the cropping range, narrowing the cropping range to obtain a target cropping range;

[0042] According to the target cropping range, the picture is intercepted;

[0043] Wherein, there is no incomplete object picture within the target cropping range;

[0044] The incomplete object picture refers to an object in the picture lacking some object features corresponding to the object type to which it belongs.

[0045] Optionally, obtaining target multimedia data according to the captured picture includes:

[0046] Stretch the size of the image within the target cropping range to the target size;

[0047] Before stretching the size of the image within the target cropping range to the target size, it also includes:

[0048] A preset definition enhancement method is used to process the image captured according to the target cropping range.

[0049] The embodiment of the present invention further provides a multimedia data processing device, comprising:

[0050] A first determination module, used to determine a target object according to reference parameter information respectively corresponding to at least two objects in the picture;

[0051] A first processing module, used for intercepting the picture according to the target object;

[0052] A second processing module, used for obtaining target multimedia data according to the captured picture;

[0053] The reference parameter information includes at least one of speech parameter information, identity parameter information and motion parameter information.

[0054] Optionally, the value of the reference parameter information is equal to the sum of the values ​​of the speech parameter information, the identity parameter information and the motion parameter information;

[0055] The step of determining the target object according to the reference parameter information respectively corresponding to at least two objects in the picture includes:

[0056] sorting the at least two objects according to values ​​of the reference parameter information respectively corresponding to the at least two objects;

[0057] Select the object in the preset position as the target object.

[0058] Optionally, also include:

[0059] A first matching module, used for matching the at least two objects in the picture with objects in a preset object library respectively before determining the target object according to the reference parameter information respectively corresponding to the at least two objects in the picture;

[0060] The third processing module is used to obtain identity parameter information corresponding to at least two objects in the picture according to the matching result.

[0061] Optionally, obtaining identity parameter information corresponding to at least two objects in the picture respectively according to the matching result includes:

[0062] When the first object in the picture successfully matches the second object in the preset object library, obtaining identity parameter information corresponding to the first object according to ranking information of the second object in the preset object library;

[0063] In the case where the first object in the picture fails to match each object in the preset object library, using the preset identity parameter information as the identity parameter information corresponding to the first object;

[0064] The first object is any object in the picture.

[0065] Optionally, obtaining identity parameter information corresponding to the first object according to ranking information of the second object in the preset object library includes:

[0066] Obtaining difference information between the total ranking information of the preset object library and the ranking information;

[0067] The identity parameter information corresponding to the first object is obtained according to the difference information and the total ranking information.

[0068] Optionally, the capturing the picture according to the target object includes:

[0069] When the number of the target objects is at least two, obtaining at least one candidate interception method corresponding to the target objects;

[0070] Pre-capturing the picture by using the at least one candidate capture method to obtain at least one pre-capturing result;

[0071] According to the at least one pre-interception result, obtaining a target interception method from the at least one candidate interception method;

[0072] The picture is intercepted according to the target interception method.

[0073] Optionally, acquiring a target interception method from the at least one candidate interception method according to the at least one pre-interception result includes:

[0074] Performing an object integrity check on the at least one pre-cut result;

[0075] According to the detection result, a target interception method is obtained from the at least one candidate interception method.

[0076] Optionally, acquiring a target interception method from the at least one candidate interception method according to the detection result includes:

[0077] The candidate interception method corresponding to the detection result with the highest value is used as the target interception method.

[0078] Optionally, the capturing the picture according to the target object further includes:

[0079] When the highest value in the detection result is less than the preset threshold, the longitudinal middle points of the target objects at the left and right ends of the picture are taken as the left and right center points of the picture;

[0080] The picture is captured according to the left and right center points of the picture.

[0081] Optionally, the capturing the picture according to the target object includes:

[0082] Determine a cropping range according to the target object;

[0083] The picture is intercepted according to the cropping range.

[0084] Optionally, the intercepting the picture according to the cropping range includes:

[0085] When there is an incomplete object picture within the cropping range, narrowing the cropping range to obtain a target cropping range;

[0086] According to the target cropping range, the picture is intercepted;

[0087] Wherein, there is no incomplete object picture within the target cropping range;

[0088] The incomplete object picture refers to an object in the picture lacking some object features corresponding to the object type to which it belongs.

[0089] Optionally, obtaining target multimedia data according to the captured picture includes:

[0090] Stretch the size of the image within the target cropping range to the target size;

[0091] The device also includes:

[0092] The fourth processing module is used to process the picture captured according to the target cropping range by using a preset definition enhancement method before stretching the size of the picture within the target cropping range to the target size.

[0093] An embodiment of the present invention further provides a communication device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, the above-mentioned multimedia data processing method is implemented.

[0094] The embodiment of the present invention further provides a readable storage medium on which a program is stored. When the program is executed by a processor, the steps in the above-mentioned multimedia data processing method are implemented.

[0095] The beneficial effects of the above technical solution of the present invention are as follows:

[0096] In the above scheme, the multimedia data processing method determines the target object according to the reference parameter information corresponding to at least two objects in the picture; intercepts the picture according to the target object; obtains the target multimedia data according to the intercepted picture; wherein the reference parameter information includes: at least one of speech parameter information, identity parameter information and motion parameter information; it can achieve the maximum degree of automatic cropping and greatly reduce the time consumption, which well solves the problem of long time consumption of multimedia data processing schemes for video processing in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 A schematic diagram of a multimedia data processing method according to an embodiment of the present invention;

[0098] Figure 2 A schematic diagram of the cutting ratio of an embodiment of the present invention;

[0099] Figure 3 A schematic diagram of a cut-off area according to an embodiment of the present invention;

[0100] Figure 4 The motion amplitude scoring process of the embodiment of the present invention is shown as follows Figure 1 ;

[0101] Figure 5 The motion amplitude scoring process of the embodiment of the present invention is shown as follows Figure 2 ;

[0102] Figure 6 The motion amplitude scoring process of the embodiment of the present invention is shown as follows Figure 3 ;

[0103] Figure 7 The motion amplitude scoring process of the embodiment of the present invention is shown as follows Figure 4 ;

[0104] Figure 8 This is a schematic diagram of pre-cropping of multiple people in the same frame according to an embodiment of the present invention;

[0105] Fig. 9 A schematic diagram of human skeleton points according to an embodiment of the present invention;

[0106] Fig.10 A schematic diagram of facial coordinates of a person according to an embodiment of the present invention;

[0107] Fig.11 A schematic diagram of the horizontal coordinates of a person's face according to an embodiment of the present invention;

[0108] Fig.12 Schematic diagram of a pre-cropped image according to an embodiment of the present invention Figure 1 ;

[0109] Fig.13Schematic diagram of a pre-cropped image according to an embodiment of the present invention Figure 2 ;

[0110] Fig.14 A schematic diagram of a candidate graph according to an embodiment of the present invention;

[0111] Fig.15 A schematic diagram of image cropping according to an embodiment of the present invention;

[0112] Fig.16 Schematic diagram of the structure of a multimedia data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0113] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0114] The present invention aims to solve the problem that the multimedia data processing scheme for video processing in the prior art takes a long time, and provides a multimedia data processing method, such as Figure 1 As shown, including:

[0115] Step 11: determining a target object according to reference parameter information corresponding to at least two objects in the picture;

[0116] Step 12: Capturing the image according to the target object;

[0117] Step 13: Obtain target multimedia data based on the captured image; wherein the reference parameter information includes at least one of speech parameter information, identity parameter information and motion parameter information.

[0118] The picture may be an image picture or a video picture, which is not limited here.

[0119] The multimedia data processing method provided by the embodiment of the present invention determines the target object according to the reference parameter information corresponding to at least two objects in the picture; captures the picture according to the target object; obtains the target multimedia data according to the captured picture; wherein the reference parameter information includes: at least one of speech parameter information, identity parameter information and motion parameter information; it can realize the maximum degree of automatic cropping and greatly reduce the time consumption, which well solves the problem of long time consumption of multimedia data processing solutions for video processing in the prior art.

[0120] Among them, the value of the reference parameter information is equal to the sum of the values ​​of the speech parameter information, the identity parameter information and the motion parameter information; determining the target object according to the reference parameter information corresponding to at least two objects in the picture includes: sorting the at least two objects according to the values ​​of the reference parameter information corresponding to the at least two objects; selecting the object in a preset position as the target object.

[0121] The preset position may be the first position, the last position, the middle position, etc., and is not limited here.

[0122] Furthermore, before determining the target object based on the reference parameter information corresponding to at least two objects in the picture, it also includes: matching the at least two objects in the picture with objects in a preset object library; and obtaining the identity parameter information corresponding to the at least two objects in the picture based on the matching results.

[0123] Among them, the identity parameter information corresponding to at least two objects in the picture is obtained according to the matching result, including: when the first object in the picture successfully matches the second object in the preset object library, the identity parameter information corresponding to the first object is obtained according to the ranking information of the second object in the preset object library; when the first object in the picture fails to match each object in the preset object library, the preset identity parameter information is used as the identity parameter information corresponding to the first object; wherein the first object is any object in the picture.

[0124] In an embodiment of the present invention, obtaining identity parameter information corresponding to the first object based on ranking information of the second object in the preset object library includes: obtaining difference information between the total ranking information of the preset object library and the ranking information; and obtaining identity parameter information corresponding to the first object based on the difference information and the total ranking information.

[0125] The ranking information may be updated according to a preset time period. More specifically, the ranking information may be real-time ranking information. Among them, "obtaining the identity parameter information corresponding to the first object according to the difference information and the total ranking information" may specifically be obtaining the identity parameter information according to the difference information divided by the total ranking information (i.e., the obtained ratio is used as the identity parameter information), but is not limited to this.

[0126] In the embodiment of the present invention, the capturing of the screen according to the target object includes: when the number of the target objects is at least two, obtaining at least one candidate capturing method corresponding to the target object; pre-capturing the screen by using the at least one candidate capturing method to obtain at least one pre-capturing result; obtaining a target capturing method from the at least one candidate capturing method according to the at least one pre-capturing result; and capturing the screen according to the target capturing method.

[0127] Wherein, obtaining the target interception method from the at least one candidate interception method according to the at least one pre-interception result includes: performing an object integrity check on the at least one pre-interception result; and obtaining the target interception method from the at least one candidate interception method according to the check result.

[0128] Specifically, acquiring the target interception method from the at least one candidate interception method according to the detection result includes: taking the candidate interception method corresponding to the detection result with the highest value as the target interception method.

[0129] Furthermore, the capturing of the picture according to the target object also includes: when the highest value in the detection result is less than a preset threshold, taking the longitudinal middle points of the target objects at the left and right ends of the picture as the left and right center points of the picture; and capturing the picture according to the left and right center points of the picture.

[0130] The longitudinal middle point may be the longitudinal middle point of the face, but is not limited thereto; the intercepting of the picture according to the left and right center points of the picture may include: determining a cropping range according to the left and right center points of the picture (the left and right center points of the picture are determined according to the target object); intercepting the picture according to the cropping range. "Determining a cropping range according to the left and right center points of the picture" may include determining a picture cropping center point according to the left and right center points of the picture; determining a cropping range according to the picture cropping center point.

[0131] In the embodiment of the present invention, the capturing of the picture according to the target object includes: determining a cropping range according to the target object; and capturing the picture according to the cropping range.

[0132] In combination with the above, "determine the cropping range according to the target object" can specifically be: determine the left and right center points of the picture according to the target object; determine the picture cropping center point according to the left and right center points of the picture; determine the cropping range according to the picture cropping center point; but it is not limited to this.

[0133] Among them, the intercepting of the picture according to the cropping range includes: when there is an incomplete object picture within the cropping range, narrowing the cropping range to obtain a target cropping range; intercepting the picture according to the target cropping range; wherein there is no incomplete object picture within the target cropping range; the incomplete object picture means that the object in the picture lacks some object features corresponding to the object type to which it belongs.

[0134] In an embodiment of the present invention, obtaining target multimedia data based on the captured picture includes: stretching the size of the picture within the target cropping range to the target size; before stretching the size of the picture within the target cropping range to the target size, it also includes: using a preset clarity enhancement method to process the picture captured according to the target cropping range.

[0135] This can reduce the blurriness caused by video stretching to a certain extent.

[0136] The multimedia data processing method provided in the embodiment of the present invention is described below by taking an example of video processing, taking the scenario of converting a horizontal screen to a vertical screen as an example.

[0137] In view of the above technical problems, an embodiment of the present invention provides a multimedia data processing method, which can be specifically implemented as a method for converting a horizontal screen video into a vertical screen video, mainly involving: shot segmentation, cropping ratio setting, cropping method (which can also be understood as a cropping method, corresponding to the above target interception method) determination and video output; specifically, it can be implemented as follows:

[0138] 1. Shot segmentation;

[0139] The video is analyzed frame by frame by using the image similarity comparison method. When the confidence of the previous and next frames is lower than a certain threshold, it is considered that the scene in the video has been switched and the video is segmented.

[0140] This segmentation facilitates object tracking, which is to segment the video and then determine the cropping method for each frame in each segment.

[0141] 2. Setting the cropping ratio;

[0142] Before cropping the video, you can set the ratio of the video playback area, for example, the ratio can be: 1:1, 4:3 or 9:16; Figure 2 As shown, a means cropping the original video at a ratio of 9:16; b means cropping the original video at a ratio of 1:1; c means cropping the original video at a ratio of 4:3; further, no matter how the cropping ratio is set, the final output video can be output in a vertical screen (usually 9:16), and the insufficient part is filled with black, but it is not limited to this.

[0143] 3. Determine the cropping method (which is executed for each frame);

[0144] The basic rules for determining the interception method can be as follows:

[0145] 1. Capture target (corresponding to the above target object): People are given priority. If there is no person, the target (object) in motion is given priority; if there is no person, the center can be captured by default;

[0146] 2. Capture range: cannot exceed the original video area;

[0147] 3. Target audience confirmation:

[0148] (1) Characters:

[0149] For the captured person, try to ensure the integrity of their face, that is, when there is only one person, the face is centered on the left or right side of the screen; when there are two people, ensure that the faces are displayed completely, and the center point of the distance between the two faces is centered on the left or right side of the screen;

[0150] If the area of ​​the captured person is less than 1 / 9 of the screen (this value can be adjusted according to the actual situation), the captured area can be reduced (the captured height ≥ 1 / 2 of the original video height). Figure 3 As shown in the dotted box in the middle; on this basis, the proportion of the character in the picture can be increased as much as possible.

[0151] When there are multiple (at least 3) characters in the picture, the main character (i.e. the target object) can be determined according to the wonderfulness score of the dynamic object (corresponding to determining the above reference parameter information, see the following "wonderfulness evaluation") for details; when intercepting, the object with the highest wonderfulness (i.e. the highest score) is used as the main interception point (i.e. the main character), corresponding to the above preset position as the first;

[0152] The closer the excitement between multiple objects (characters), the greater the probability that they are in the same frame (that is, the closer the excitement, the more likely they are to be captured and retained at the same time). For details, please refer to the subsequent "How to deal with multiple people in the same frame"; if the excitement differs greatly, the one with the higher excitement will occupy the center of the screen;

[0153] (2) Object:

[0154] When capturing an object, its size may not be considered; when there is an object moving quickly in the picture, the object is used as the capture center; when there are at least two objects moving quickly in the picture, the excitement rule is used (the excitement processing method when there are multiple characters in the same picture).

[0155] (3) Evaluation of the degree of excitement:

[0156] 1) The wonderfulness of the person speaking (corresponding to the above-mentioned speech parameter information): lip movement recognition technology can be used to identify whether the person in the picture is speaking. The wonderfulness of the person speaking +1;

[0157] 2) Increased excitement (corresponding to the above-mentioned identity parameter information): Establish a celebrity, internet celebrity, and well-known person recognition library (a specific implementation of the above-mentioned preset object library). If the person in the picture matches the person library, the excitement of the person is increased;

[0158] The increased value can be determined based on the popularity ranking of the character (corresponding to the above-mentioned obtaining the identity parameter information corresponding to the first object based on the ranking information of the second object in the preset object library), and the popularity ranking can be determined based on the number of topic views within a certain period of time.

[0159] Increased excitement = (total number of rankings - ranking of the character) / total number of rankings (corresponding to the identity parameter information corresponding to the first object obtained based on the difference information and the total number of rankings information).

[0160] 3) Scoring the motion amplitude of the object (or person) in the picture (corresponding to the above motion parameter information);

[0161] a. Oriented Rapid Rotation (ORB) or Scale Invariant Feature Transform (SIFT) algorithm can be used to extract feature points of the two key frames (I frames) before and after;

[0162] b. Match these feature points and use random sampling consistent RANSAC to filter out abnormal feature points;

[0163] c. Make the feature points overlap, keep the intersecting parts of the image, and cut off the non-intersecting parts from the edge; specifically, Figure 4 and Figure 5 As shown in FIG. 1 , the first and last two frames of the video are shown in FIG. 1 . In the picture, the background is offset and the character's movements are changed.

[0164] This solution can extract the feature points separately and match them, overlap the feature points of the two, and then calculate the intersection of the sizes of the two pictures, such as Figure 6 As shown, after the feature points are matched, the intersecting parts of the image area are retained and the non-intersecting parts are removed ( Figure 6 (filled with slashes in the middle).

[0165] d. Calculate the areas of the different parts in the two pictures.

[0166] Specifically, Figure 7 As shown in the figure, the diagonal filled parts on the left and right sides are the removed parts. For the convenience of viewing, they are retained and represented by diagonal filled parts.

[0167] Figure 7 The dotted part where the two characters overlap is the difference between the two pictures.

[0168] In this solution, the structural similarity index (SSIM) or image difference comparison and other methods can be used to calculate the difference area between two images.

[0169] The motion amplitude score between two frames = the difference part of the intersection area of ​​the two pictures / the intersection area of ​​the two pictures.

[0170] If it is a video, the average value of the motion amplitude score in the time period (the average value of the motion amplitude score between every two frames) can be obtained.

[0171] In an embodiment of the present invention, the final excitement (corresponding to the above-mentioned reference parameter information) = the excitement of the speaking character + the increased excitement + the movement amplitude score, corresponding to the reference parameter information including: speech parameter information, identity parameter information and movement parameter information. Specifically, the value of the reference parameter information is equal to the sum of the values ​​of the speech parameter information, the identity parameter information and the movement parameter information.

[0172] (4) Regarding the processing method of multiple people (at least two people) in the same frame (corresponding to the number of the above-mentioned target objects being at least two):

[0173] 1) At least two people have similar weights (i.e., the above-mentioned final excitement), but they are not fully displayed in the same frame (cropping the left and right sides of the picture), such as Figure 8 As shown, if the weights of the three people are similar (all three are stars, all are talking, or none of them are talking); the integrity test can be used to determine the target object later. The integrity test can be understood as: the object in the pre-cropped image is compared with the object in the original image to obtain the completeness, and the completeness is compared; for details, please refer to the following processing method (corresponding to the above-mentioned object integrity test for the at least one pre-cut result; according to the test result, the target cut method is obtained from the at least one candidate cut method):

[0174] A. If it is not possible to ensure that at least two people are displayed completely, reduce the display range of the characters to ensure that the displayed characters are complete as much as possible;

[0175] a. Character integrity detection (when there are complete characters in the picture and incomplete characters, the incomplete characters can be quickly filtered out):

[0176] a1) The skeleton points of the characters in the picture can be detected through the skeleton point recognition algorithm;

[0177] a2) When the skeleton points are complete, but the limb part to which the skeleton belongs has a horizontal or vertical contour fault (which can be performed in conjunction with human body contour detection), the skeleton point closest to the fault is considered missing (when the human body part is outside the picture, there will definitely be a horizontal or vertical contour fault. The length of the fault can be defined by yourself, for example, the length of the fault is 2% of the width or height of the corresponding contour);

[0178] a3) If the human body outline does not show obvious horizontal or vertical faults, the figure is considered complete but just blocked, and its skeleton points are counted as 100%;

[0179] a4) Calculate the integrity of the character according to the number of bone points. Integrity = number of identified bone points / number of complete bone points;

[0180] For example Fig. 9 As shown in the figure, the human body has 22 bone points. The left figure shows the complete outline of the human body and the complete bone points. The palm of the right figure has a fault, so the bone points at the wrist are invalid. Therefore, the integrity of the figures in the left and right figures are 22 / 22 and 21 / 22 respectively.

[0181] b. Character integrity detection (when the screen is full of complete characters, or full of incomplete characters):

[0182] b1) Use face detection to obtain the coordinates of the person's face in the picture (taking the center of the forehead as an example), see Fig.10 The dots on the faces of the characters shown;

[0183] b2) Get the horizontal coordinates of each character's face, see Fig.11 A vertical line through a dot on the face of the person shown;

[0184] b3) respectively from left to right, taking a single person as the center, find out the pre-cropped pictures as candidate cropped pictures (included in the above-mentioned pre-cropping of the pictures by the at least one candidate cropping method to obtain at least one pre-cropping result), such as Fig.12 As shown, d, e, f and g represent different pre-cropped images (see the dotted boxes in the figure);

[0185] b4) The center of the horizontal coordinate of the face of the two characters is used as the center of the pre-cropped image, such as Fig.13 As shown (of course, the center of the facial horizontal coordinates of the three characters can also be used as the center of the pre-cropped picture, or the center of the facial horizontal coordinates of more characters can be used as the center of the pre-cropped picture to obtain the corresponding pre-cropped picture; these operations are included in the above-mentioned pre-cropping of the picture by at least one candidate cropping method to obtain at least one pre-cropping result), h, i and j respectively represent different pre-cropped pictures (see the solid line frame and the dotted line frame in the figure).

[0186] In the embodiment of the present invention, the candidate cropping images may satisfy the condition that the cropping ratio of the characters is ≤ n%. Fig.13 As shown in the figure, after the dotted mask is cropped, very little of the person is cropped, so it meets the conditions. However, in the solid mask part, more of the person is cropped, so it does not meet the conditions.

[0187] Specifically, the proportion of the character being cropped = (pixel area of ​​the character in the original picture - pixel area of ​​the character in the picture after cropping) / pixel area of ​​the character in the original picture.

[0188] b5) From the candidate cropping pictures, according to the completeness principle (i.e., sorting according to the display completeness of the characters in the picture, each candidate cropping picture corresponds to a completeness), select the candidate cropping pictures corresponding to the first two or three completenesses in the completeness sorting (if the total number of candidate cropping pictures obtained is less than this number, then select all the candidate cropping pictures); in the embodiment of the present invention, of course, it is also possible to directly execute "taking the candidate cropping method corresponding to the detection result (completeness) with the highest value as the target cropping method".

[0189] Specifically, assuming that the order is based on the principle of completeness (without considering the degree of excitement), it can be as follows Fig.14 As shown; the reason why there are multiple candidate pictures in the picture is mainly because the characters in them are different. In the future, you can consider referring to the excitement calculation rules for further selection.

[0190] B. Based on the above integrity test, further:

[0191] (1) When the complete person is displayed, priority is given to ensuring that no other person’s body parts appear in the image (which can be understood as the above-mentioned incomplete object images such as Figure 8 In the figure on the right, the effect of the lower figure is better than that of the upper figure);

[0192] That is, the characters in the picture are sorted according to their completeness. Under the same completeness conditions, characters with higher completeness (pictures without parts of other people's bodies) are given priority.

[0193] (2) Centering method for multiple people: For example, the vertical middle point of the faces of two characters (not the straight-line distance) is used as the left and right center points of the picture (if there are three people, the vertical middle point of the faces of the two characters on the left and right are selected as the left and right center points of the picture (two endpoints); further, the center point of these two endpoints can be selected as the picture cropping center point).

[0194] In the embodiment of the present invention, if the weights differ greatly, the target object is locked according to the weight calculation method (corresponding to the above-mentioned if the differences in the wonderfulness are large, the one with higher wonderfulness occupies the center of the screen). Fig.14 The correct cropping range is shown in the figure on the right.

[0195] Specifically, the above multi-person centering method is given as an example:

[0196] Assuming the weights are similar, but the images are not fully displayed after being put on the same screen (pre-cropped) (cropped on all four sides), for example, Fig.15 As shown in the two figures on the left, the weights of the three people are similar, but at most only two people can be framed, and none of them can be framed completely. The following processing methods can be used to deal with this situation:

[0197] Method for centering multiple people: Use the vertical middle point of the two characters' faces (not the straight-line distance) as the left and right center points of the picture;

[0198] The weights are similar, and multiple roles can be selected according to completeness, such as: Fig.15 The effect of the figure in the lower left corner is better than that in the upper left corner.

[0199] In the embodiment of the present invention, when the weights are uneven, for example, when only one role has a higher weight, Fig.15 The 3 images in the middle column are cropped based on the principle of centering the face.

[0200] In the embodiment of the present invention, when cropping, it is possible to detect whether there are other objects within the cropping range (which can be achieved by human body detection). If so, the cropping frame (i.e., the cropping range) can be reduced to ensure that no part of other objects is displayed within it, such as Fig.15 As shown in the two figures on the right;

[0201] Specifically, the rules for reducing the cropping frame may include:

[0202] (1) The proportion does not change;

[0203] (2) From the subject's head to the bottom, ensure integrity, with the top of the head left blank for about 7% to 10% of the image;

[0204] (3) If the subject’s head is below the body (e.g. a person stretching his hands toward the sky), you can leave a reserved space (blank space) of about 20% of the picture;

[0205] In the embodiment of the present invention, the cropped area is reduced and can be stretched to the same resolution as other images in the video when it is finally output;

[0206] 4. Video output;

[0207] After the video output ratio (ie the above cropping ratio) is determined, the cropping range of each shot (each frame) is marked by the above method, and the transcoding can be performed according to a unified encoding format.

[0208] If the sizes of the images of the cropped videos are different, the sizes of at least a portion of the images (for the reduced cropped areas) are stretched to a target size (for example, consistent with other images).

[0209] Furthermore, in the embodiment of the present invention, super-resolution, generative adversarial network GAN and other methods (corresponding to the above-mentioned preset definition improvement method) can also be used for those video clips (pictures) whose sizes are inconsistent with the target, such as Fig.15 The two figures on the right of the figure are shown in Figure 1. The method of providing increased definition is used to reduce the blur caused by video stretching. Specifically, it may include:

[0210] a. Determine the ratio between the current cropped image and the original image. For example, the resolution of the current original image is 4000×2000, and the current cropped image is 200×100. Then, the ratio of the two images is 20:1;

[0211] b. Use ffmpeg (transcoding) to deframe the cropped image, extract its key frames, and extract its audio.

[0212] c. The deframed image is enlarged by a super-resolution algorithm or a super-resolution model based on a convolutional neural network (CNN). The magnification ratio is 20 times (i.e., the previous 20:1).

[0213] d. After the image is enlarged, it is synthesized into a video using ffmpeg and the original audio track (the audio above) is added.

[0214] The solution provided by the embodiment of the present invention involves: according to the degree of excitement = the degree of excitement of the speaking person + the increased degree of excitement + the amplitude of movement, the cropping range is determined in three dimensions, so as to realize a method of cropping a horizontal video into a vertical video; based on this, the solution can achieve the maximum degree of automated cropping, greatly reduce time consumption, and has good versatility, and can be used for almost all types of videos.

[0215] The embodiment of the present invention also provides a multimedia data processing device, such as Fig.16 As shown, including:

[0216] A first determination module 161 is used to determine a target object according to reference parameter information corresponding to at least two objects in the picture;

[0217] A first processing module 162, configured to capture the image according to the target object;

[0218] The second processing module 163 is used to obtain target multimedia data according to the captured picture;

[0219] The reference parameter information includes at least one of speech parameter information, identity parameter information and motion parameter information.

[0220] The multimedia data processing device provided by the embodiment of the present invention determines the target object according to the reference parameter information corresponding to at least two objects in the picture; captures the picture according to the target object; obtains the target multimedia data according to the captured picture; wherein the reference parameter information includes: at least one of speech parameter information, identity parameter information and motion parameter information; it can realize the maximum degree of automatic cropping and greatly reduce the time consumption, which well solves the problem of long time consumption of multimedia data processing solutions for video processing in the prior art.

[0221] Among them, the value of the reference parameter information is equal to the sum of the values ​​of the speech parameter information, the identity parameter information and the motion parameter information; determining the target object according to the reference parameter information corresponding to at least two objects in the picture includes: sorting the at least two objects according to the values ​​of the reference parameter information corresponding to the at least two objects; selecting the object in a preset position as the target object.

[0222] Furthermore, the multimedia data processing device also includes: a first matching module, used to match the at least two objects in the picture with the objects in the preset object library before determining the target object based on the reference parameter information corresponding to the at least two objects in the picture; and a third processing module, used to obtain the identity parameter information corresponding to the at least two objects in the picture based on the matching results.

[0223] Among them, the identity parameter information corresponding to at least two objects in the picture is obtained according to the matching result, including: when the first object in the picture successfully matches the second object in the preset object library, the identity parameter information corresponding to the first object is obtained according to the ranking information of the second object in the preset object library; when the first object in the picture fails to match each object in the preset object library, the preset identity parameter information is used as the identity parameter information corresponding to the first object; wherein the first object is any object in the picture.

[0224] Specifically, obtaining the identity parameter information corresponding to the first object based on the ranking information of the second object in the preset object library includes: obtaining the difference information between the total ranking information of the preset object library and the ranking information; and obtaining the identity parameter information corresponding to the first object based on the difference information and the total ranking information.

[0225] In the embodiment of the present invention, the capturing of the screen according to the target object includes: when the number of the target objects is at least two, obtaining at least one candidate capturing method corresponding to the target object; pre-capturing the screen by using the at least one candidate capturing method to obtain at least one pre-capturing result; obtaining a target capturing method from the at least one candidate capturing method according to the at least one pre-capturing result; and capturing the screen according to the target capturing method.

[0226] Wherein, obtaining the target interception method from the at least one candidate interception method according to the at least one pre-interception result includes: performing an object integrity check on the at least one pre-interception result; and obtaining the target interception method from the at least one candidate interception method according to the check result.

[0227] Specifically, acquiring the target interception method from the at least one candidate interception method according to the detection result includes: taking the candidate interception method corresponding to the detection result with the highest value as the target interception method.

[0228] Furthermore, the capturing of the picture according to the target object also includes: when the highest value in the detection result is less than a preset threshold, taking the longitudinal middle points of the target objects at the left and right ends of the picture as the left and right center points of the picture; and capturing the picture according to the left and right center points of the picture.

[0229] In the embodiment of the present invention, the capturing of the picture according to the target object includes: determining a cropping range according to the target object; and capturing the picture according to the cropping range.

[0230] Among them, the intercepting of the picture according to the cropping range includes: when there is an incomplete object picture within the cropping range, narrowing the cropping range to obtain a target cropping range; intercepting the picture according to the target cropping range; wherein there is no incomplete object picture within the target cropping range; the incomplete object picture means that the object in the picture lacks some object features corresponding to the object type to which it belongs.

[0231] In an embodiment of the present invention, obtaining target multimedia data based on the captured picture includes: stretching the size of the picture within the target cropping range to the target size; the device also includes: a fourth processing module, which is used to process the picture captured according to the target cropping range using a preset clarity enhancement method before stretching the size of the picture within the target cropping range to the target size.

[0232] Among them, the implementation embodiments of the above-mentioned multimedia data processing method are all applicable to the embodiments of the multimedia data processing device and can achieve the same technical effects.

[0233] An embodiment of the present invention further provides a communication device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, the above-mentioned multimedia data processing method is implemented.

[0234] Among them, the implementation embodiments of the above-mentioned multimedia data processing method are all applicable to the embodiments of the communication device and can achieve the same technical effects.

[0235] The embodiment of the present invention further provides a readable storage medium on which a program is stored. When the program is executed by a processor, the steps in the above-mentioned multimedia data processing method are implemented.

[0236] Among them, the implementation embodiments of the above-mentioned multimedia data processing method are all applicable to the embodiments of the readable storage medium and can achieve the same technical effects.

[0237] It should be noted that many functional components described in this specification are referred to as modules in order to more particularly emphasize the independence of their implementation methods.

[0238] In the embodiment of the present invention, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0239] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0240] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0241] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimedia data processing method, characterized in that: include: Determine the target object according to the reference parameter information respectively corresponding to at least two objects in the picture; According to the target object, intercepting the picture; Obtain target multimedia data according to the captured images; Wherein, the reference parameter information includes: at least one of speech parameter information, identity parameter information and motion parameter information; The step of intercepting the picture according to the target object includes: When the highest value in the detection result is less than a preset threshold, the longitudinal middle points of the target objects at the left and right ends of the picture are used as the left and right center points of the picture; the picture is intercepted according to the left and right center points of the picture; wherein the detection result is obtained by performing object integrity detection on the pre-interception result, and the pre-interception result is obtained by pre-interception of the picture according to the candidate interception method of the target object; Alternatively, the capturing the picture according to the target object comprises: determining a cropping range according to the target object; capturing the picture according to the cropping range; Determining the cropping range according to the target object includes: determining left and right center points of the picture according to the target object; determining a picture cropping center point according to the left and right center points of the picture; and determining a cropping range according to the picture cropping center point; And / or, obtaining target multimedia data according to the captured image includes: A preset definition enhancement method is adopted to process the picture captured according to the target cropping range; and the size of the picture within the target cropping range is stretched to the target size to obtain the target multimedia data.

2. The multimedia data processing method according to claim 1, characterized in that: The value of the reference parameter information is equal to the sum of the values ​​of the speech parameter information, the identity parameter information and the motion parameter information; The step of determining the target object according to the reference parameter information respectively corresponding to at least two objects in the picture includes: sorting the at least two objects according to values ​​of the reference parameter information respectively corresponding to the at least two objects; Select the object in the preset position as the target object.

3. The multimedia data processing method according to claim 1, characterized in that: Before determining the target object according to the reference parameter information respectively corresponding to at least two objects in the picture, the method further includes: Matching at least two objects in the picture with objects in a preset object library respectively; According to the matching result, identity parameter information corresponding to at least two objects in the picture is obtained.

4. The multimedia data processing method according to claim 3, characterized in that: The obtaining, according to the matching result, identity parameter information corresponding to at least two objects in the picture, respectively, includes: When the first object in the picture successfully matches the second object in the preset object library, obtaining identity parameter information corresponding to the first object according to ranking information of the second object in the preset object library; In the case where the first object in the picture fails to match each object in the preset object library, using the preset identity parameter information as the identity parameter information corresponding to the first object; The first object is any object in the picture.

5. The multimedia data processing method according to claim 1, characterized in that: The step of intercepting the picture according to the target object includes: When the number of the target objects is at least two, obtaining at least one candidate interception method corresponding to the target objects; Pre-capturing the picture by using the at least one candidate capture method to obtain at least one pre-capturing result; According to the at least one pre-interception result, obtaining a target interception method from the at least one candidate interception method; The picture is intercepted according to the target interception method.

6. The multimedia data processing method according to claim 5, characterized in that: The acquiring a target interception method from the at least one candidate interception method according to the at least one pre-interception result includes: Performing an object integrity check on the at least one pre-cut result; According to the detection result, a target interception method is obtained from the at least one candidate interception method.

7. The multimedia data processing method according to claim 1, characterized in that: The step of intercepting the picture according to the cropping range includes: When there is an incomplete object picture within the cropping range, narrowing the cropping range to obtain a target cropping range; According to the target cropping range, the picture is intercepted; Wherein, there is no incomplete object picture within the target cropping range; The incomplete object picture refers to an object in the picture lacking some object features corresponding to the object type to which it belongs.

8. A communication device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the multimedia data processing method according to any one of claims 1 to 7 is implemented.

9. A readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the multimedia data processing method according to any one of claims 1 to 7 are implemented.

Images