Multimedia resource processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a multimedia resource processing method and device, electronic equipment and storage medium. The method comprises: obtaining a plurality of resource segment pairs based on a plurality of resource segments corresponding to each multimedia resource in a plurality of multimedia resources; obtaining a segment coefficient of each resource segment, and obtaining a first similarity between two resource segments in each resource segment pair according to the segment coefficient; determining at least two candidate resource segment pairs from the plurality of resource segment pairs according to the first similarity; obtaining a second similarity between the two resource segments in the candidate resource segment pair according to a second segment feature set of each of the two resource segments in the candidate resource segment pair; determining a target resource segment pair from the at least two candidate resource segment pairs according to the second similarity; and obtaining target resource information based on the target resource segment pair. The method can quickly determine the target resource information and provide the function of skipping the target resource information to the user in time.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, and particularly relates to a multimedia resource processing method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] With the development of Internet technology, uploading multimedia resources on a multimedia platform for users to listen to or view is a common way for anchors, and anchors will implant target resource information such as advertisements in the multimedia resources they upload, which brings trouble to users. Therefore, how to detect target resource information and provide users with an entry that can skip target resource information is very important to improve the user experience.

[0003] At present, the way to detect target resource information is mainly through pairwise resource comparison. However, the number of multimedia resources uploaded on the multimedia platform every day is huge, and this pairwise resource comparison detection method is low in efficiency and difficult to meet the timeliness requirement. SUMMARY

[0004] The present disclosure provides a multimedia resource processing method, device, electronic equipment, storage medium and program product to at least solve the problem of low efficiency of detection methods in related technologies and difficulty in meeting the timeliness requirement. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a multimedia resource processing method is provided, comprising:

[0006] Based on a plurality of resource segments corresponding to each of the plurality of multimedia resources, a plurality of resource segment pairs are obtained, each resource segment pair including two resource segments belonging to different multimedia resources;

[0007] Obtain a segment coefficient of each resource segment, and obtain a first similarity between the two resource segments in each resource segment pair according to the segment coefficient; determine at least two candidate resource segment pairs from the plurality of resource segment pairs according to the first similarity; the segment coefficient represents the statistical characteristics of the first segment feature set of the resource segment;

[0008] Obtain a second similarity between the two resource segments in the candidate resource segment pair according to the second segment feature set of each of the two resource segments in the candidate resource segment pair; determine a target resource segment pair from the at least two candidate resource segment pairs according to the second similarity; the dimension of the first segment feature set of each resource segment is less than the dimension of its second segment feature set;

[0009] Obtain target resource information based on the target resource segment pair.

[0010] In an example embodiment, the obtaining of the segment coefficients of each of the resource segments comprises:

[0011] The first segment feature set of each of the resource segments is obtained.

[0012] For each of the resource segments, the frame features of each dimension in the first segment feature set of the resource segment are averaged to obtain the segment coefficient of the resource segment.

[0013] In an example embodiment, the obtaining of the first segment feature set of each of the resource segments comprises:

[0014] The second segment feature set of each of the resource segments is obtained.

[0015] The second segment feature set is reduced in dimension to obtain the first segment feature set of each of the resource segments.

[0016] In an example embodiment, the determining of the at least two candidate resource segment pairs from the plurality of resource segment pairs according to the first similarity comprises:

[0017] For each resource segment in each of the multimedia resources, a first similarity between the resource segment and each resource segment in another multimedia resource is obtained.

[0018] Based on the first similarity, a target resource segment is determined from each resource segment in the other multimedia resource; the first similarity corresponding to the target resource segment is greater than the first similarity corresponding to the remaining resource segments in the other multimedia resource, the remaining resource segments being the resource segments in the other multimedia resource except the target resource segment.

[0019] Based on each resource segment in each of the multimedia resources and the target resource segment in the other multimedia resource, the candidate resource segment pair is obtained.

[0020] In an example embodiment, the obtaining of the second similarity of the two resource segments in the candidate resource segment pair according to the second segment feature set of each of the two resource segments in the candidate resource segment pair comprises:

[0021] The second segment feature set of the two resource segments in the candidate resource segment pair is compared by a dynamic time warping algorithm to obtain the second similarity of the two resource segments in the candidate resource segment pair.

[0022] In an example embodiment, the determining the target resource segment pair from the at least two candidate resource segment pairs according to the second similarity comprises:

[0023] For any one resource segment, the candidate resource segment pair with the maximum second similarity in the candidate resource segment pairs corresponding to the resource segment is determined as the target resource segment pair corresponding to the resource segment.

[0024] In an example embodiment, the obtaining the target resource information based on the target resource segment pair comprises:

[0025] determining a continuous resource segment pair from the target resource segment pair; the resource segments in the continuous resource segment pair are continuous in the two multimedia resources to which the resource segments belong;

[0026] performing a merging process on the continuous resource segment pair to obtain a merged resource segment pair;

[0027] when the segment length of a repeated resource segment in the merged resource segment pair is greater than a length threshold, determining the repeated resource segment as the target resource information.

[0028] In an example embodiment, after the obtaining the target resource information based on the target resource segment pair, the method further comprises:

[0029] traversing a resource information template library of an account corresponding to the target resource information;

[0030] when a first resource information template in the resource information template library already exists and matches the target resource information, obtaining information lengths of the target resource information and the first resource information template respectively; a similarity between the first resource information template and the target resource information is greater than a first similarity threshold;

[0031] if the information length of the target resource information is greater than the information length of the first resource information template, updating the first resource information template in the resource information template library according to the target resource information.

[0032] In an example embodiment, the method further comprises:

[0033] obtaining a new multimedia resource and traversing a resource information template library of an account corresponding to the new multimedia resource;

[0034] when a second resource information template matching the new multimedia resource already exists in the resource information template library, obtaining a resource identifier of the new multimedia resource and starting resource segment information in the new multimedia resource matching the second resource information template, and adding the resource identifier and the starting resource segment information as multimedia resource information corresponding to the second resource information template in the resource information template library;

[0035] wherein a similarity between the second resource information template and the new multimedia resource is greater than a second similarity threshold.

[0036] In an exemplary embodiment, after the traversing of the resource information template library of the account corresponding to the new multimedia resource, the method further comprises:

[0037] when the second resource information template matching the new multimedia resource does not exist in the resource information template library, storing the new multimedia resource in an unprocessed resource library of the account corresponding to the new multimedia resource;

[0038] when a number of unprocessed multimedia resources in the unprocessed resource library exceeds a number threshold, returning the step of obtaining a plurality of resource segment pairs based on a plurality of resource segments corresponding to each of the plurality of multimedia resources.

[0039] According to a second aspect of the embodiments of the present disclosure, a multimedia resource processing apparatus is provided, comprising:

[0040] a segmentation unit configured to perform obtaining a plurality of resource segment pairs based on a plurality of resource segments corresponding to each of a plurality of multimedia resources, each resource segment pair including two resource segments respectively belonging to different multimedia resources;

[0041] a first determination unit configured to perform obtaining a segment coefficient of each resource segment, obtaining a first similarity between two resource segments in each resource segment pair according to the segment coefficient, and determining at least two candidate resource segment pairs from the plurality of resource segment pairs according to the first similarity; the segment coefficient representing a statistical feature of a first segment feature set of the resource segment;

[0042] a second determination unit configured to perform obtaining a second similarity between two resource segments in each candidate resource segment pair according to a second segment feature set of each of the two resource segments in the candidate resource segment pair, and determining a target resource segment pair from the at least two candidate resource segment pairs according to the second similarity; a dimension of the first segment feature set of each resource segment being less than a dimension of the second segment feature set thereof.

[0043] The target acquisition unit is configured to obtain target resource information based on the target resource pair.

[0044] In an example embodiment, the first determining unit is configured to obtain the first segment feature set of each resource segment; the first segment feature set includes frame features of multiple dimensions; for each resource segment, the frame features of each dimension in the first segment feature set of the resource segment are processed by mean value to obtain a segment coefficient of the resource segment.

[0045] In an example embodiment, the first determining unit is further configured to obtain the second segment feature set of each resource segment; the second segment feature set is processed by dimension reduction to obtain the first segment feature set of each resource segment.

[0046] In an example embodiment, the first determining unit is further configured to obtain, for each resource segment in each multimedia resource, a first similarity between the resource segment and each resource segment in another multimedia resource; based on the first similarity, a target resource segment is determined from each resource segment in the another multimedia resource; the first similarity corresponding to the target resource segment is greater than the first similarity corresponding to the remaining resource segments in the another multimedia resource, and the remaining resource segments are the resource segments in the another multimedia resource except the target resource segment; based on each resource segment in each multimedia resource and the target resource segment in the another multimedia resource, the candidate resource pair is obtained.

[0047] In an example embodiment, the second determining unit is further configured to perform, by a dynamic time warping algorithm, comparison processing on the second segment feature sets of the two resource segments in the candidate resource pair to obtain the second similarity of the two resource segments in the candidate resource pair.

[0048] In an example embodiment, the second determining unit is further configured to determine, for any resource segment, the candidate resource pair with the maximum second similarity in the candidate resource pair corresponding to the resource segment as the target resource pair corresponding to the resource segment.

[0049] In an example embodiment, the target determining unit is further configured to determine a continuous resource segment pair from the target resource segment pairs; the resource segments in the continuous resource segment pair are continuous in the two multimedia resources to which the resource segments belong; and perform a merging process on the continuous resource segment pair to obtain a merged resource segment pair; and determine a repeated resource segment in the merged resource segment pair as the target resource information when a segment length of the repeated resource segment is greater than a length threshold.

[0050] In an example embodiment, the apparatus further includes a first updating unit configured to traverse a resource information template library of an account corresponding to the target resource information; obtain information lengths of the target resource information and a first resource information template in the resource information template library when the first resource information template in the resource information template library matches the target resource information; the similarity between the first resource information template and the target resource information is greater than a first similarity threshold; and update the first resource information template in the resource information template library according to the target resource information when the information length of the target resource information is greater than the information length of the first resource information template.

[0051] In an example embodiment, the apparatus further includes a second updating unit configured to obtain a new multimedia resource and traverse a resource information template library of an account corresponding to the new multimedia resource; obtain a resource identifier of the new multimedia resource and starting resource segment information of the new multimedia resource that matches a second resource information template in the resource information template library when the second resource information template in the resource information template library matches the new multimedia resource; and add the resource identifier and the starting resource segment information as multimedia resource information corresponding to the second resource information template in the resource information template library; wherein the similarity between the second resource information template and the new multimedia resource is greater than a second similarity threshold.

[0052] In an example embodiment, the second updating unit is further configured to store the new multimedia resource in an unprocessed resource library of an account corresponding to the new multimedia resource when there is no second resource information template in the resource information template library that matches the new multimedia resource; and return to the step of obtaining a plurality of resource segment pairs based on a plurality of resource segments corresponding to the multimedia resources in the plurality of multimedia resources when the number of unprocessed multimedia resources in the unprocessed resource library exceeds a number threshold.

[0053] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0054] a processor;

[0055] a memory for storing the processor-executable instructions;

[0056] The processor is configured to execute the instructions to implement the method according to any one of the preceding embodiments.

[0057] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of the preceding embodiments.

[0058] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, the computer program product includes instructions, when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of the preceding embodiments.

[0059] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0060] The method first obtains a segment coefficient through a first segment feature set with a smaller dimension than a second segment feature set, and obtains a first similarity between two resource segments in each resource segment pair according to the segment coefficient, so as to realize the determination of the similarity between the two resource segments at a coarse scale, and then determines a second similarity through the second segment feature set with more dimensions, so as to realize the detection of the target resource information from multiple scales, which is convenient for quickly determining the target resource information when a user views a new multimedia resource same as the multimedia resource corresponding to the target resource information, and timely providing the function of skipping the target resource information to the user.

[0061] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0063] Figure 1 is a flowchart of a multimedia resource processing method according to an exemplary embodiment.

[0064] Figure 2 is a flowchart of a multi-scale advertisement template discovery algorithm according to an exemplary embodiment.

[0065] Figure 3 is a flowchart of an offline-online advertisement template detection and updating method according to an exemplary embodiment.

[0066] Figure 4 is a structural block diagram of a multimedia resource processing device according to an example embodiment.

[0067] Figure 5 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0068] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0069] It should be noted that the implementations described in the following example embodiments do not represent all the implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0070] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0071] In an example embodiment, as shown in Figure 1 A multimedia resource processing method is provided, and the present embodiment is exemplified by the method applied to a terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be realized through the interaction of the terminal and the server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server can be realized by an independent server or a server cluster composed of multiple servers. In the present embodiment, the method includes the following steps:

[0072] In step S110, based on a plurality of resource segments corresponding to each multimedia resource in a plurality of multimedia resources, a plurality of resource segment pairs are obtained, each resource segment pair including two resource segments belonging to different multimedia resources.

[0073] The multimedia resource can be audio and video, etc.

[0074] In a specific implementation, a plurality of multimedia resources can be acquired in advance, each multimedia resource is divided by a sliding window method to obtain a plurality of resource segments corresponding to the multimedia resource, for each resource segment of each multimedia resource, each resource segment of another multimedia resource can form a resource segment pair, and thus a plurality of resource segment pairs are obtained.

[0075] More specifically, the resources in a time period of each multimedia resource can be divided according to a preset step size and window width to obtain a plurality of resource segments of the multimedia resource. For example, the first 60 seconds of the audio are divided according to a 3-second window width and a 0.5-second step size to obtain a plurality of audio segments.

[0076] In step S120, a segment coefficient of each resource segment is acquired, a first similarity between two resource segments in each resource segment pair is acquired according to the segment coefficient, at least two candidate resource segment pairs are determined from the plurality of resource segment pairs according to the first similarity, and the segment coefficient represents a statistical feature of a first segment feature set of the resource segment.

[0077] The first similarity can be obtained by calculating the distance between the segment coefficients of the two resource segments in each resource segment pair. The smaller the distance, the higher the first similarity between the two resource segments.

[0078] The first segment feature set includes a plurality of dimensional frame features, each dimension corresponds to a frame feature of a certain feature frame number, and each dimension can be represented by a matrix. D represents the feature dimension of the multimedia resource, and N represents the feature frame number. The segment coefficient can be obtained by performing mean value processing on the frame features of each dimension.

[0079] In a specific implementation, after obtaining the multimedia resources, feature extraction can be performed on each multimedia resource to obtain resource features corresponding to each multimedia resource, and then the resource features of each multimedia resource are divided by a sliding window method to obtain a plurality of resource segments of each multimedia resource, each resource segment has a second segment feature set, the second segment feature set is processed by dimension reduction to obtain a first segment feature set of each resource segment, and the frame features of each dimension in the first segment feature set are further processed by mean value processing to obtain the segment coefficient of each resource segment. The distance between the segment coefficients of the two resource segments in each resource segment pair is calculated to obtain the first similarity between the two resource segments in each resource segment pair.

[0080] Further, for each resource segment of each multimedia resource, the first n resource segment pairs with the highest first similarity from the resource segment pairs corresponding to the resource segment are determined as the candidate resource segment pairs corresponding to the resource segment.

[0081] In step S130, a second similarity between the two resource segments in the candidate resource segment pair is obtained according to the second segment feature sets of the two resource segments in the candidate resource segment pair; and the target resource segment pair is determined from the at least two candidate resource segment pairs according to the second similarity; the dimension of the first segment feature set of each resource segment is less than the dimension of the second segment feature set of the resource segment.

[0082] The second similarity can be obtained by calculating the distance between the second segment feature sets of the two resource segments in each resource segment pair. The smaller the distance, the higher the second similarity between the two resource segments.

[0083] The first segment feature set is obtained by dimension reduction processing of the second segment feature set.

[0084] In a specific implementation, the second similarity between the two resource segments in the candidate resource segment pair can be obtained by comparing the second segment feature sets of the two resource segments in the candidate resource segment pair using a dynamic time warping (DTW) algorithm. For each resource segment, the candidate resource segment pair with the highest second similarity can be determined from the multiple candidate resource segment pairs corresponding to the resource segment as the target resource segment pair. The candidate resource segment pairs corresponding to each resource segment can include resource segment pairs composed of the resource segment and different resource segments in the same multimedia resource, or resource segment pairs composed of the resource segment and resource segments in different multimedia resources. Therefore, only one target resource segment pair can be determined from the candidate resource segment pairs corresponding to each resource segment, or multiple target resource segment pairs can be determined according to different multimedia resources, i.e., for a resource segment, a resource segment can be determined from each multimedia resource other than the multimedia resource to which the resource segment belongs to form a target resource segment pair with the resource segment.

[0085] In step S130, the target resource information is obtained based on the target resource segment pair.

[0086] In one application scenario, the target resource information can be understood as an advertisement template, and the advertisement template is detected by the embodiment.

[0087] In a specific implementation, after the target resource segment pair is obtained, the repeated resource segments in the target resource segment pair can be determined as the target resource information. More specifically, to avoid the length of the obtained target resource information being too short, the segment length of the repeated resource segments in the target resource segment pair can also be obtained. If the segment length is greater than a length threshold, the repeated resource segment is taken as the target resource information.

[0088] Further, since the target resource segment pair is composed of resource segments obtained by cutting the multimedia resources, the resource information can be incomplete, i.e., there can be continuous target resource segment pairs, and therefore, after obtaining the target resource segment pair, a continuity detection process of the target resource segment pair can be performed, if the resource segments in two target resource segment pairs are continuous in the two multimedia resources to which they belong, it is determined that the two target resource segments are continuous resource segment pairs, and then a merging process of the continuous resource segment pairs can be further performed, and the repeated resource segments in the merged resource segment pair are determined as the target resource information.

[0089] In the above multimedia resource processing method, first, based on the plurality of resource segments corresponding to each multimedia resource in the plurality of multimedia resources, a plurality of resource segment pairs is obtained, further, first, the segment coefficients of each resource segment are obtained, according to the segment coefficients, the first similarity of the two resource segments in each resource segment pair is calculated at a coarse scale, and according to the first similarity, at least two candidate resource segment pairs are determined from the plurality of resource segment pairs; then, according to the second segment feature sets of the two resource segments in the candidate resource segment pairs, the second similarity of the two resource segments in the candidate resource segment pairs is obtained at a finer granularity, and according to the second similarity, the target resource segment pair is determined from the at least two candidate resource segment pairs, and finally, based on the target resource segment pair, the target resource information is obtained. This method first obtains the segment coefficients through the first segment feature set with a smaller dimension than the second segment feature set, obtains the first similarity of the two resource segments in each resource segment pair according to the segment coefficients, realizes the determination of the similarity of the two resource segments at a coarse scale, and then determines the second similarity through the second segment feature set with more dimensions, realizes the detection of the target resource information from multiple scales, and facilitates the subsequent provision of the function of skipping the target resource information to the user based on the target resource information when the user views a new multimedia resource identical to the multimedia resource corresponding to the target resource information.

[0090] In an exemplary embodiment, in step S120, the segment coefficients of each resource segment can be obtained by the following steps:

[0091] In step S121, the first segment feature set of each resource segment is obtained; the first segment feature set includes a plurality of dimensional frame features.

[0092] In step S122, for each resource segment, the frame features of each dimension in the first segment feature set of the resource segment are averaged to obtain the segment coefficients of the resource segment.

[0093] In a specific implementation, the first segment feature set can be a matrix of feature dimensions x feature frame numbers, and an element in the matrix can represent a feature value of a certain feature frame in a certain dimension. The frame features of each dimension in the first segment feature set are subjected to mean value processing, i.e., the frame features of each dimension are added and then subjected to mean value processing, to obtain a matrix of feature dimensions x 1, which is taken as a segment coefficient.

[0094] In an example embodiment, in step S121, the first segment feature set of each resource segment is obtained, which can be achieved by the following steps: obtaining the second segment feature set of each resource segment; and performing dimension reduction processing on the second segment feature set to obtain the first segment feature set of each resource segment.

[0095] In a specific implementation, the multimedia resources can be subjected to feature extraction by using the first window width and the first step length to obtain resource features of each multimedia resource, and then the resource features of each multimedia resource are segmented by using the second window width and the second step length to obtain the second segment feature set of the multiple resource segments. Meanwhile, after obtaining the resource features of each multimedia resource, the resource features are analyzed by using PCA (principal component analysis) to obtain multiple principal component vectors. The second segment feature set of each resource segment is subjected to dimension reduction processing by using the multiple principal component vectors to obtain the first segment feature set of each resource segment, and the first segment feature set of each resource segment can be further subjected to mean value processing to obtain a segment coefficient of each resource segment.

[0096] For example, taking audio as the multimedia resource, it is assumed that the MFCC (Mel Frequency Cepstral Coefficient) feature is extracted from the first 60 seconds of multiple audios as audio features of the audios by using 0.0116 seconds as the first step length and 0.0464 seconds as the first window width. It is assumed that the extracted audio features of the audio i are segmented by using 3 seconds as the second window width and 0.5 seconds as the second step length to obtain 60 / 0.5=120 audio segments j, denoted as Each audio segment is a matrix of , i.e. The matrix of can represent the second segment feature set, where 258=3 seconds / 0.0116 seconds. The second segment feature set is subjected to dimension reduction processing to obtain the first segment feature set of each resource segment, denoted as where D * is less than D, and the feature values of each dimension in the first segment feature set of each resource segment are further subjected to mean value processing to obtain a segment coefficient which is a matrix of D * x 1.

[0097] In the above embodiment, the second segment feature set of each resource segment is subjected to dimension reduction processing, the segment coefficient of each resource segment is determined by the first segment feature set obtained through the dimension reduction processing, the data amount for calculating the segment coefficient is reduced, and thus the efficiency for determining the first similarity can be improved.

[0098] In an example embodiment, in step S120, at least two candidate resource segment pairs are determined from the plurality of resource segment pairs according to the first similarity, which can be achieved by the following steps:

[0099] In step S123, for each resource segment in each multimedia resource, the first similarity between the resource segment and each resource segment in another multimedia resource is obtained respectively;

[0100] In step S124, based on the first similarity, a target resource segment is determined from each resource segment in another multimedia resource; the first similarity of the target resource segment is greater than the first similarity of the remaining resource segments in another multimedia resource, and the remaining resource segments are the resource segments in another multimedia resource except the target resource segment;

[0101] In step S125, based on each resource segment in each multimedia resource and the target resource segment in another multimedia resource, a candidate resource segment pair is obtained.

[0102] In a specific implementation, each resource segment in each multimedia resource can form a resource segment pair with each resource segment in another multimedia resource. For ease of description, the following describes this embodiment by taking the determination of a candidate resource segment pair from a first multimedia resource and a second multimedia resource as an example.

[0103] Suppose the first multimedia resource is i and the second multimedia resource is k. For a resource segment j in the first multimedia resource i, the first similarity between the resource segment j and each resource segment l1, l2, … in the second multimedia resource is obtained respectively, based on the first similarity, a target resource segment is determined from each resource segment in the second multimedia resource, the first similarity corresponding to the target resource segment is greater than the first similarity corresponding to the remaining resource segments in the second multimedia resource, and the resource segment j and the target resource segment form a candidate resource segment pair. That is, the resource segment pairs (j, l1), (j, l2), … formed by the resource segment j and each resource segment in the second multimedia resource are sorted according to the value of the first similarity in order, and the first n resource segment pairs with the highest first similarity from the sorted resource segment pairs are determined as the candidate resource segment pairs corresponding to the resource segment j.

[0104] In the embodiment, for any resource segment, the target resource segment is determined from each resource segment of another multimedia resource based on the first similarity, and the first similarity corresponding to the target resource segment is greater than the first similarity corresponding to the remaining resource segments in the another multimedia resource, so that the candidate resource segment pair based on the target resource segment is the most similar resource segment pair to the resource segment.

[0105] In an example embodiment, in step S130, the second similarity of the two resource segments in the candidate resource segment pair is obtained according to the second segment feature set of each of the two resource segments in the candidate resource segment pair. The specific implementation can be: the second similarity of the two resource segments in the candidate resource segment pair is obtained by comparing the second segment feature set of the two resource segments in the candidate resource segment pair through a dynamic time warping algorithm.

[0106] In the implementation, the distance between the second segment feature set of the two resource segments in the candidate resource pair can be calculated through the dynamic time warping algorithm, and the distance is taken as the second similarity of the two resource segments in each candidate resource segment pair. The greater the distance value is, the smaller the second similarity represented is, and the smaller the distance value is, the greater the second similarity represented is.

[0107] In the embodiment, the dynamic time warping algorithm is a similarity algorithm for processing sequence features. The second similarity of the two resource segments in the candidate resource segment pair is calculated through the dynamic time warping algorithm on the second segment feature set of the two resource segments in the candidate resource segment pair, and a more accurate similarity between the two resource segments can be obtained, which has better performance than the traditional Euclidean distance algorithm.

[0108] In an example embodiment, in step S130, the target resource segment pair is determined from at least two candidate resource segment pairs according to the second similarity. The specific implementation can be: for any resource segment, the candidate resource segment pair with the greatest second similarity corresponding to the resource segment is determined as the target resource segment pair corresponding to the resource segment.

[0109] For example, there is a resource segment j, and the candidate resource segment pairs corresponding to the resource segment j are (j, l1), (j, l2), (j, l3) and (j, l4). The candidate resource segment pairs are sorted in descending order of the second similarity as (j, l3)→(j, l4)→(j, l2)→(j, l1). The candidate resource segment pair with the greatest second similarity (j, l3) is determined as the target resource segment pair corresponding to the resource segment j.

[0110] In this embodiment, the candidate resource segment pair with the second largest similarity in the candidate resource segment pairs corresponding to the resource segment is determined as the target resource segment pair corresponding to the resource segment, so as to subsequently determine the accurate target resource information from the target resource segment pair.

[0111] In an exemplary embodiment, in step S140, target resource information is obtained based on the target resource segment pair, which can be achieved by the following steps:

[0112] In step S141, a continuous resource segment pair is determined from the target resource segment pair; the resource segments in the continuous resource segment pair are continuous in the two multimedia resources to which they belong;

[0113] In step S142, the continuous resource segment pair is merged to obtain a merged resource segment pair;

[0114] In step S143, when the segment length of the repeated resource segment in the merged resource segment pair is greater than a length threshold, the repeated resource segment is determined as the target resource information.

[0115] Specifically, assuming that there are resource segments {j1, j2, j3, j4} in the first multimedia resource i and resource segments {l1, l2, l3, l4} in the second multimedia resource k, the two multimedia resources form two target resource segment pairs (j2, l3) and (j3, l4), the resource segments j2 and j3 in the two target resource segment pairs are continuous in the first multimedia resource, and the resource segments l3 and l4 are continuous in the second multimedia resource, therefore, the two target resource segment pairs can be merged to obtain a merged resource segment pair (j2-l3, j3-l4). 2- When the segment length meets the preset condition, i.e., is greater than the length threshold, the resource segment composed of j2 and j3 is determined as the target resource information.

[0116] In this embodiment, by performing continuity detection processing on the target resource segment pair, and performing merging processing on the continuous target resource segment pair when the continuous target resource segment pair is detected, it can be ensured that the obtained target resource information is complete resource information. At the same time, by performing screening processing on the target resource segment pair through the length threshold, it can avoid obtaining target resource information with too short length, and ensure the quality of the obtained target resource information.

[0117] In an exemplary embodiment, in step S140, after obtaining the target resource information based on the target resource segment pair, the following steps are further included:

[0118] In step S150, the resource information template library of the account corresponding to the target resource information is traversed.

[0119] In step S160, when the first resource information template matching the target resource information already exists in the resource information template library, the information lengths of the target resource information and the first resource information template are obtained; the similarity between the first resource information template and the target resource information is greater than the first similarity threshold.

[0120] In step S170, if the information length of the target resource information is greater than the information length of the first resource information template, the first resource information template in the resource information template library is updated according to the target resource information.

[0121] The information length can be the playing time length of the target resource information.

[0122] The template identifier is an identifier representing the uniqueness of the resource information template.

[0123] The resource information template library stores a plurality of resource information templates, and each resource information template includes a plurality of multimedia resource information. The multimedia resource information includes a multimedia resource identifier and starting resource segment information of the multimedia resource matching the resource information template. The multimedia resource information can be expressed in the form of a key-valve pair, such as {pid[i,i]} and {pid[k,k]}. In the formula, pid represents the identifier of the multimedia resource i, i represents the starting segment of the multimedia resource i, and k represents the ending segment of the multimedia resource i. The resource information template library can be stored in the form of key-key-valve, such as {tid:{pid[i,i]},…} In the formula, tid represents the template identifier of the resource information template. i h t k h t i h t i h t n i h t n

[0124] ​​​​​​​​​​​​​​​​​In a specific implementation, the resource information template library can be set according to different accounts of the published multimedia resources. After obtaining the target resource information, the resource information template library corresponding to the account of the target resource information can be traversed to check whether a first resource information template matching the target resource information exists in the resource information template library. According to the matching result, the resource information template library can be updated correspondingly.

[0125] More specifically, the similarity between the target resource information and the resource information template in the resource information template library can be represented by an interaction over union (IOU). When a resource information template with a similarity greater than a first similarity threshold is found from the resource information template library, the resource information template is determined to be a first resource information template matching the target resource information. Then, the information lengths of the target resource information and the first resource information template can be obtained. When the information length of the first resource information template is less than that of the target resource information, the first resource information template can be updated to the target resource information with a greater information length. When no first resource information template is found from the resource information template library, it is considered that a new resource information template appears. A new resource information template identifier tid is created, and the resource identifiers of the two multimedia resources to which the target resource segment corresponding to the target resource information belongs and the starting resource segment information of the target resource information in each multimedia resource are obtained. The new resource information template identifier tid, the target resource information, the resource identifiers of the multimedia resources, and the starting resource segment information are stored in the resource information template library in a key-key-value structure.

[0126] Further, the first resource information template in the resource information template library can be updated according to the target resource information. The first resource information template in the resource information template library can be replaced by the target resource information, or the target resource information and the first resource information template can be merged, and the merged resource information can replace the first resource information template.

[0127] In the embodiment, after obtaining the target resource information, the resource information template library corresponding to the account of the target resource information is traversed, and when a first resource information template matching the target resource information is found, the resource information with a greater information length between the target resource information and the first resource information template is stored as a resource information template in the resource information template library, thereby realizing offline updating of the resource information template library.

[0128] In an example embodiment, the method further comprises: obtaining a new multimedia resource, and traversing a resource information template library of an account corresponding to the new multimedia resource; when a second resource information template that matches the new multimedia resource already exists in the resource information template library, obtaining a resource identifier of the new multimedia resource and starting resource segment information in the new multimedia resource that matches the second resource information template, and adding the resource identifier and the starting resource segment information as multimedia resource information corresponding to the second resource information template in the resource information template library; wherein a similarity between the second resource information template and the new multimedia resource is greater than a second similarity threshold.

[0129] The resource identifier is an identifier that represents uniqueness of the multimedia resource.

[0130] In a specific implementation, for a new multimedia resource of an account, resource features of the new multimedia resource are first extracted, and then a resource information template library of an account corresponding to the new multimedia resource is traversed, and resource features of each resource information template in the resource information template library are slidingly matched on the resource features of the new multimedia resource to calculate a similarity between the new multimedia resource and each resource information template, wherein the similarity can be represented by a correlation coefficient, and a resource information template with a similarity greater than a second similarity threshold is determined as a second resource information template that matches the new multimedia resource, and a resource identifier of the new multimedia resource and starting resource segment information in the new multimedia resource that matches the second resource information template are obtained, and the resource identifier and the starting resource segment information are added as multimedia resource information corresponding to the second resource information template in the resource information template library.

[0131] In the embodiment, when the new multimedia resource is obtained, the new multimedia resource is matched with each resource information template in the resource information template library, and when the second resource information template is matched, the multimedia resource information corresponding to the second resource information template is updated according to the resource identifier and the starting resource segment information of the new multimedia resource, so as to realize online updating of the resource information template library.

[0132] In an example embodiment, the method further comprises: when the second resource information template that matches the new multimedia resource does not exist in the resource information template library, storing the new multimedia resource in an unprocessed resource library of an account corresponding to the new multimedia resource; and when a number of unprocessed multimedia resources in the unprocessed resource library exceeds a number threshold, returning to the step of obtaining the plurality of resource segment pairs based on the plurality of resource segments corresponding to each multimedia resource in the plurality of multimedia resources.

[0133] In a specific implementation, when there is no second resource information template in the resource information template library that matches the newly added multimedia resource, i.e., the similarity between each resource information template in the resource information template library and the newly added multimedia resource is less than the second similarity threshold, it is determined that the existing resource information template does not appear in the newly added multimedia resource. Therefore, the newly added multimedia resource can be temporarily stored in the unprocessed resource library of the account corresponding to the newly added multimedia resource. In this way, when the number of unprocessed multimedia resources in the unprocessed resource library exceeds the number threshold, the step S110 of obtaining a plurality of resource segment pairs based on a plurality of resource segments corresponding to each multimedia resource in the plurality of multimedia resources is performed to determine the target resource information.

[0134] In this embodiment, the newly added multimedia resource that does not exist in the resource information template library is stored, and when the number of stored unprocessed multimedia resources reaches a preset number threshold, the resource information template discovery processing of each unprocessed multimedia resource is performed, and the resource information template library is updated according to the discovered target resource information, thereby realizing offline updating of the resource information template library.

[0135] In an exemplary embodiment, in order to facilitate those skilled in the art to understand the embodiments of the present disclosure, the following will take the multimedia resource as an audio, take detecting the advertisement in the audio as an example, and combine the specific examples of the drawings to describe the multimedia resource processing method proposed by the present disclosure.

[0136] It can be understood that the basic logic of the podcast platform is that different hosts upload audio content respectively for users to listen to, and the hosts will implant advertisements at the beginning of part of the audio they upload. These advertisement contents often have the following properties:

[0137] o Repetition: After receiving the advertisement demand, the host will implant the same advertisement content in the beginning of multiple audios (the volume size may differ, but after audio feature extraction and energy dimension removal, the same features can be obtained), to achieve more extensive dissemination.

[0138] o Transmission: Assuming that a host uploads three audios p1, p2, and p3, where p1[1.5, 11.5] (time interval, unit: seconds) and p2[2, 12] are the same implanted advertisements, and p2[2, 12] and p3[0, 10] are the same implanted advertisements, then p1[1.5, 11.5] and p3[0, 10] are the same implanted advertisements. Such identical implanted advertisements can be referred to as an advertisement template.

[0139] According to the basic properties of the advertisement template described above, the present disclosure proposes a new multi-scale advertisement template discovery algorithm, wherein all algorithms in the present disclosure will be used on the audio data of each host. Referring to Figure 2Fig. 1 is a schematic diagram of a flow of a multi-scale advertisement template discovery algorithm, and the specific steps are as follows:

[0140] (1) For multiple audios of a host, first, the Mel-frequency cepstral coefficient (MFCC) features of the first 60 seconds of each audio are extracted with a step size of 0.0116 seconds and a window width of 0.0464 seconds. Thus, the first 60 seconds of content of each audio is converted into a matrix of D x N, where D is the feature dimension of the audio features, and N is the number of feature frames. The MFCC features are features after removing the first dimension, i.e., the direct current component, so that the obtained audio features are not affected by the original volume size.

[0141] (2) After obtaining the audio features of each audio of the current host, the audio features are processed using principal component analysis (PCA) to obtain multiple principal component components.

[0142] The audio features of each audio are divided into multiple audio segments j according to a 3-second window width and a 0.5-second step size, and there are a total of 60 / 0.5 = 120 audio segments, which can be denoted as The second segment feature set of each audio segment is wherein 258 = 3 seconds / 0.0116 seconds. The second segment feature set of each audio segment is processed by dimension reduction according to the multiple principal component components obtained by principal component analysis to obtain the first segment feature set of each audio segment. The frame features of each feature dimension in the first segment feature set are processed by mean value to obtain the segment coefficient of each audio segment

[0143] (3) Coarse-grained similarity calculation: the similarity between two audios is calculated by the segment coefficient. For audio i and audio k of the current host, the distance between the segment coefficients of the 120 audio segments of the two audios is calculated to obtain a distance matrix and wherein represents the distance between the jth segment (3-second length segment described in step (2)) of audio i and the lth segment of audio k. For each element in the matrix D, it is determined whether the corresponding distance value is in the top 3 minimum values of the current row and the current column. If yes, the element is set to 1, otherwise, the element is set to 0. Thus, the first similarity matrix with element values of 0 / 1 is obtained The 1 in the matrix indicates the position where the features of the two audios are close, i.e., the position where the advertisement template may appear.

[0144] (4) Fine-grained similarity calculation: through the coarse-grained similarity calculation of step (3), for each segment j in audio i, at most 3 audio segments in audio k that are closest to it can be obtained, i.e., ​There are at most 3 non-zero elements. For the 3 candidate audio segment pairs consisting of segment j and the three segments closest to segment j in coarse granularity, the distance between each candidate audio segment pair is calculated by dynamic time warping algorithm (DTW), and the elements corresponding to the target audio segment pair with the minimum distance value are set to 1, and the rest are set to 0, thereby further obtaining a second similarity matrix in fine granularity wherein, There is at most one non-zero element. In the foregoing calculation, O(N 2 ) times of low-complexity vector distance calculation in coarse granularity and O(N) times of high-complexity DTW calculation in fine granularity, through this multi-scale distance calculation mode, efficiency and accuracy can be considered, and the two-by-two comparison of the audio files can be effectively performed.

[0145] (5) Continuous sequence detection: the second similarity matrix is subjected to continuity detection, and the steps of the continuity detection on the two-dimensional 0 / 1 matrix include:

[0146] (a) finding all continuous row numbers with non-zero elements:

[0147] (b) finding all continuous column numbers with non-zero elements:

[0148] (c) for the non-zero elements (j, k) in the second similarity matrix , if j belongs to the continuous row numbers in (a), k belongs to the continuous column numbers in (b), and then (j, k) and (j+1, k+1) belong to two points in the continuous sequence, and steps (a)-(c) are repeated to find all non-zero sequences in the second similarity matrix, and each non-zero sequence is recorded as a candidate advertisement template.

[0149] (6) Length screening of the detected continuous sequences: sequences with a length less than 5 seconds are removed, and repeated audio segments in sequences with a length greater than or equal to 5 seconds are taken as advertisement templates.

[0150] (7) The advertisement template is expressed in the form of key-value pair: {pid i : [i h , i t ]} and {pid k : [k h , k t ]}. pid i represents the podcast id of audio i, i h and i t represent that the current advertisement template starts from the i h th segment of audio it The segment ends. At the same time, an advertisement template library is saved, which is stored in the form of key-key-value, such as {tid n : {pid i : [i h , i t ]},…}. For the newly obtained templates {pid i : [i h , i t ]} and {pid k : [k h , k t ]}, the advertisement template library can be traversed, if the same pid as the template exists in the advertisement template library and the interval IOU (interaction over union) is higher than 0.7, it is considered that the advertisement template already exists, at this time, the interval of the corresponding template is updated to a template with a longer length; otherwise, it is confirmed that a new template appears, at this time, a new tid is created in the advertisement template library, and [pid i : [i h , i t ]} and {pid k : [k h , k t ]} are stored in the corresponding key-key-value structure.

[0151] Referring to Figure 3 , it is a flowchart of an online advertisement template detection and updating method in an application instance, the steps are as follows:

[0152] (1) For the newly added audio of a host, firstly, the features of the first 60 seconds of the content are extracted, then the advertisement template library of the host is traversed, the feature matrix of each advertisement template is used to perform sliding matching on the current audio feature matrix and calculate the correlation coefficient. (Each column of the feature matrix is normalized, and the final correlation coefficient is between 0 and 1.) When the correlation coefficient at a certain position exceeds 0.8, it is determined that the task template matches successfully, the pid of the current audio and the corresponding position are updated to the matched template kkv structure.

[0153] (2) If the correlation coefficients of all template matches are less than 0.8, it is determined that the existing template does not appear in the current audio. The current audio is temporarily stored in the “unprocessed audio library” of the host, when the number of unprocessed files exceeds 100, the template discovery algorithm is started Figure 2 , the multi-scale advertisement template discovery and the updating of the template library are performed.

[0154] The disclosure proposes an algorithm for multi-scale advertisement template discovery, which quickly locates the positions of advertisements at a coarse scale and accurately corrects the positions of advertisements at a fine scale, while meeting the requirements of accuracy and speed. On the other hand, using the combination of online template detection and offline template discovery can not only ensure that existing advertisements are accurately located, but also timely discover new advertisements.

[0155] It should be understood that, although each step in the flowchart involved in each of the above-described embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above-described embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0156] It should be understood that the same / similar parts between each embodiment of the above-described method in this specification can be mutually referred to, and each embodiment focuses on the differences from other embodiments, and the related parts can be referred to the description of other method embodiments.

[0157] Based on the same inventive concept, the embodiments of the disclosure also provide a multimedia resource processing apparatus for implementing the above-mentioned multimedia resource processing method.

[0158] Figure 4 is a structural block diagram of a multimedia resource processing apparatus according to an exemplary embodiment. Referring to Figure 4 The apparatus includes a segmentation unit 410, a first determination unit 420, a second determination unit 430, and a target acquisition unit 440, wherein

[0159] The segmentation unit 410 is configured to perform, based on a plurality of resource segments corresponding to each of a plurality of multimedia resources, obtain a plurality of resource segment pairs, each resource segment pair including two resource segments respectively belonging to different multimedia resources;

[0160] The first determination unit 420 is configured to perform obtaining a segment coefficient of each of the resource segments, obtaining a first similarity of the two resource segments in each of the resource segment pairs according to the segment coefficient; and determining at least two candidate resource segment pairs from the plurality of resource segment pairs according to the first similarity; the segment coefficient represents a statistical feature of a first segment feature set of the resource segment;

[0161] The second determining unit 430 is configured to perform second segment feature set of each of the two resource segments in the candidate resource segment pair, obtain the second similarity of the two resource segments in the candidate resource segment pair; and determine the target resource segment pair from the at least two candidate resource segment pairs according to the second similarity; the dimension of the first segment feature set of each resource segment is less than the dimension of the second segment feature set of the resource segment.

[0162] The target obtaining unit 440 is configured to perform target resource information based on the target resource segment pair.

[0163] In an example embodiment, the first determining unit is configured to perform the first segment feature set of each resource segment; the first segment feature set includes frame features of multiple dimensions; and for each resource segment, the frame features of each dimension in the first segment feature set of the resource segment are subjected to mean value processing to obtain a segment coefficient of the resource segment.

[0164] In an example embodiment, the first determining unit is further configured to perform the second segment feature set of each resource segment; and the second segment feature set is subjected to dimension reduction processing to obtain the first segment feature set of each resource segment.

[0165] In an example embodiment, the first determining unit is further configured to perform, for each resource segment in each multimedia resource, respectively, first similarity between the resource segment and each resource segment in another multimedia resource; and based on the first similarity, determine a target resource segment from each resource segment in the another multimedia resource; the first similarity corresponding to the target resource segment is greater than the first similarity corresponding to the remaining resource segments in the another multimedia resource, and the remaining resource segments are resource segments in the another multimedia resource except the target resource segment; and based on each resource segment in each multimedia resource and the target resource segment in the another multimedia resource, obtain the candidate resource segment pair.

[0166] In an example embodiment, the second determining unit is further configured to perform comparison processing on the second segment feature set of the two resource segments in the candidate resource segment pair by dynamic time warping algorithm to obtain the second similarity of the two resource segments in the candidate resource segment pair.

[0167] In an example embodiment, the second determining unit is further configured to determine, for any one resource segment, a candidate resource segment pair with the maximum second similarity among candidate resource segment pairs corresponding to the resource segment as the target resource segment pair corresponding to the resource segment.

[0168] In an example embodiment, the target determining unit is further configured to determine a continuous resource segment pair from the target resource segment pair; the resource segments in the continuous resource segment pair are continuous in the two multimedia resources to which the resource segments belong; and perform merging processing on the continuous resource segment pair to obtain a merged resource segment pair; and determine a repeated resource segment in the merged resource segment pair as the target resource information when a segment length of the repeated resource segment is greater than a length threshold.

[0169] In an example embodiment, the apparatus further includes a first updating unit configured to traverse a resource information template library of an account corresponding to the target resource information; obtain information lengths of the target resource information and a first resource information template in the resource information template library when the first resource information template in the resource information template library matches the target resource information; the similarity between the first resource information template and the target resource information is greater than a first similarity threshold; and update the first resource information template in the resource information template library according to the target resource information if the information length of the target resource information is greater than the information length of the first resource information template.

[0170] In an example embodiment, the apparatus further includes a second updating unit configured to obtain a new multimedia resource and traverse a resource information template library of an account corresponding to the new multimedia resource; obtain a resource identifier of the new multimedia resource and starting resource segment information of the new multimedia resource that matches a second resource information template in the resource information template library when the second resource information template in the resource information template library matches the new multimedia resource; and add the resource identifier and the starting resource segment information as multimedia resource information corresponding to the second resource information template in the resource information template library; wherein the similarity between the second resource information template and the new multimedia resource is greater than a second similarity threshold.

[0171] In an example embodiment, the second updating unit is further configured to, when there is no second resource information template in the resource information template library that matches the added multimedia resource, store the added multimedia resource in an unprocessed resource library of an account corresponding to the added multimedia resource; and when the number of unprocessed multimedia resources in the unprocessed resource library exceeds a number threshold, return to the step of obtaining a plurality of resource segment pairs based on a plurality of resource segments corresponding to each of the plurality of multimedia resources.

[0172] With regard to the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0173] Figure 5 FIG. 5 is a block diagram of an electronic device 500 for implementing a multimedia resource processing method according to an example embodiment. The electronic device 500 can be, for example, a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0174] Referring to Figure 5 The electronic device 500 can include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0175] The processing component 502 generally controls the overall operations of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 can include one or more processors 520 to execute instructions to complete all or part of the steps of the methods described above. In addition, the processing component 502 can include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0176] The memory 504 is configured to store various types of data to support the operations of the electronic device 500. Examples of these data include instructions for any application programs or methods operating on the electronic device 500, contact data, phonebook data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or nonvolatile memory, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disc, optical disc, or graphene memory.

[0177] The power supply component 506 supplies power for various components of the electronic device 500. The power supply component 506 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 500.

[0178] The multimedia component 508 includes a screen providing an output interface between the electronic device 500 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the electronic device 500 is in an operating mode, such as a photographing mode or a video mode. Each of the front and rear cameras can be a fixed optical lens system or have a focal length and optical zoom capability.

[0179] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) configured to receive external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0180] The I / O interface 512 provides an interface between the processing component 502 and peripheral interface modules, which can be a keyboard, a click wheel, a button, etc. These buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0181] The sensor component 514 includes one or more sensors for providing status assessments for various aspects of the electronic device 500. For example, the sensor component 514 can detect an open / closed position of the electronic device 500, relative positioning of components, such as a display and a keypad of the electronic device 500, a change in position of the electronic device 500 or a component of the electronic device 500, presence or absence of user contact with the electronic device 500, orientation or acceleration / deceleration / g-force and temperature of the electronic device 500. The sensor component 514 can include an optical sensor for detecting ambient light, a proximity sensor configured to detect proximity of an object, a motion sensor, a temperature sensor, a magnetic sensor, an acceleration sensor, a gyroscope sensor, a pressure sensor, or the like.

[0182] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a corresponding communication standard, such as WiFi, a cellular network (e.g., 2G, 3G, 4G, or 5G), or a combination thereof. In an example embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0183] In an example embodiment, the electronic device 500 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements to perform the above-described methods.

[0184] In an example embodiment, a computer-readable storage medium, such as the memory 504 including instructions, is also provided, which can be executed by the processor 520 of the electronic device 500 to implement the above-described methods. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0185] In an exemplary embodiment, a computer program product is also provided, which comprises instructions executable by the processor 520 of the electronic device 500 to perform the above method.

[0186] It should be noted that the above apparatus, electronic device, computer readable storage medium, computer program product and the like according to the description of the method embodiments can also include other implementation manners, and the specific implementation manners can be referred to the description of the related method embodiments, which will not be repeated here.

[0187] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the concepts disclosed herein. The disclosure is intended to cover any variations, uses or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice in the art to which the disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the disclosure are indicated by the appended claims.

[0188] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A multimedia resource processing method, characterized in that, include: Based on multiple resource segments corresponding to each of the multiple multimedia resources, multiple resource segment pairs are obtained, each resource segment pair including two resource segments belonging to different multimedia resources; the multimedia resources are audio. Mel-Cepstral Coefficients features of each resource fragment are extracted to form a second fragment feature set for each resource fragment; wherein the energy dimension of the Mel-Cepstral Coefficients features has been removed. The second fragment feature set is subjected to dimensionality reduction processing to obtain the first fragment feature set of each resource fragment; the first fragment feature set is a matrix of feature dimension and feature frame number, and the elements in the matrix represent the feature value of a certain feature frame in a certain dimension; The frame features of each dimension in the first fragment feature set of the resource fragment are averaged to obtain the fragment coefficients of each resource fragment; Based on the fragment coefficient, a first similarity is obtained between two resource fragments in each resource fragment pair; based on the first similarity, at least two candidate resource fragment pairs are determined from the plurality of resource fragment pairs; the fragment coefficient characterizes the statistical features of the first fragment feature set of the resource fragment; Based on the second segment feature sets of each of the two resource segments in the candidate resource segment pair, a second similarity is obtained between the two resource segments in the candidate resource segment pair; based on the second similarity, a target resource segment pair is determined from the at least two candidate resource segment pairs. Based on the target resource fragment pairs, target resource information is obtained.

2. The method according to claim 1, characterized in that, The step of determining at least two candidate resource fragment pairs from the plurality of resource fragment pairs based on the first similarity includes: For each resource segment in each of the multimedia resources, a first similarity is obtained between the resource segment and each resource segment in another multimedia resource; Based on the first similarity, a target resource segment is determined from each resource segment in the other multimedia resource; the first similarity corresponding to the target resource segment is greater than the first similarity corresponding to the other resource segments in the other multimedia resource, and the other resource segments are resource segments other than the target resource segment from each resource segment in the other multimedia resource. The candidate resource fragment pair is obtained based on each resource fragment in each multimedia resource and the target resource fragment in the other multimedia resource.

3. The method according to claim 1, characterized in that, The step of obtaining the second similarity between two resource fragments in the candidate resource fragment pair based on their respective second fragment feature sets includes: The second segment feature set of the two resource segments in the candidate resource segment pair is compared by a dynamic time-normalization algorithm to obtain the second similarity between the two resource segments in the candidate resource segment pair.

4. The method according to claim 1, characterized in that, The step of determining the target resource fragment pair from the at least two candidate resource fragment pairs based on the second similarity includes: For any given resource fragment, the candidate resource fragment pair with the highest second similarity among the candidate resource fragment pairs corresponding to the resource fragment is determined as the target resource fragment pair corresponding to the resource fragment.

5. The method according to claim 4, characterized in that, The process of obtaining target resource information based on the target resource fragment pair includes: From the target resource fragment pairs, consecutive resource fragment pairs are determined; the resource fragments in the consecutive resource fragment pairs are consecutive in the two multimedia resources to which they belong; The consecutive resource fragment pairs are merged to obtain merged resource fragment pairs; When the length of a duplicate resource fragment in the merged resource fragment pair is greater than a length threshold, the duplicate resource fragment is identified as the target resource information.

6. The method according to claim 1, characterized in that, After obtaining the target resource information based on the target resource fragment pair, the process further includes: Traverse the resource information template library of the account corresponding to the target resource information; When a first resource information template matching the target resource information already exists in the resource information template library, the information lengths of the target resource information and the first resource information template are obtained; the similarity between the first resource information template and the target resource information is greater than a first similarity threshold. If the length of the target resource information is greater than the length of the first resource information template, then the first resource information template in the resource information template library is updated according to the target resource information.

7. The method according to claim 1, characterized in that, The method further includes: Retrieve newly added multimedia resources and iterate through the resource information template library of the account corresponding to the newly added multimedia resources; When a second resource information template that matches the newly added multimedia resource already exists in the resource information template library, the resource identifier of the newly added multimedia resource and the starting resource fragment information that matches the second resource information template in the newly added multimedia resource are obtained, and the resource identifier and the starting resource fragment information are added as the multimedia resource information corresponding to the second resource information template in the resource information template library. Wherein, the similarity between the second resource information template and the newly added multimedia resource is greater than the second similarity threshold.

8. The method according to claim 7, characterized in that, After traversing the resource information template library of the account corresponding to the newly added multimedia resource, the method further includes: When there is no second resource information template matching the newly added multimedia resource in the resource information template library, the newly added multimedia resource is stored in the unprocessed resource library of the account corresponding to the newly added multimedia resource; When the number of unprocessed multimedia resources in the unprocessed resource library exceeds the number threshold, the step of obtaining multiple resource fragment pairs based on multiple resource fragments corresponding to each of the multiple multimedia resources is returned.

9. A multimedia resource processing device, characterized in that, include: The segmentation unit is configured to execute multiple resource segments corresponding to each of the multiple multimedia resources to obtain multiple resource segment pairs, each resource segment pair including two resource segments belonging to different multimedia resources; the multimedia resources are audio. A first determining unit is configured to extract Mel-Cepstral Coefficient (MCC) features from each of the resource fragments to form a second fragment feature set for each resource fragment; wherein the MCC features have had their energy dimension removed; perform dimensionality reduction processing on the second fragment feature set to obtain a first fragment feature set for each of the resource fragments; the first fragment feature set is a matrix of feature dimensions and feature frame number, where each element represents the feature value of a feature frame in a certain dimension; perform mean averaging on the frame features in each dimension of the first fragment feature set of the resource fragments to obtain fragment coefficients for each resource fragment; obtain a first similarity between two resource fragments in each resource fragment pair based on the fragment coefficients; determine at least two candidate resource fragment pairs from the plurality of resource fragment pairs based on the first similarity; the fragment coefficients characterize the statistical features of the first fragment feature set of the resource fragments; The second determining unit is configured to perform the following: obtaining a second similarity between two resource fragments in the candidate resource fragment pair based on their respective second fragment feature sets; and determining a target resource fragment pair from the at least two candidate resource fragment pairs based on the second similarity. The target acquisition unit is configured to perform operations based on the target resource fragment pairs to obtain target resource information.

10. The apparatus according to claim 9, characterized in that, The first determining unit is further configured to perform the following operations: for each resource segment in each of the multimedia resources, obtain a first similarity between the resource segment and each resource segment in another multimedia resource; based on the first similarity, determine a target resource segment from each resource segment in the other multimedia resource; the first similarity corresponding to the target resource segment is greater than the first similarity corresponding to the remaining resource segments in the other multimedia resource, the remaining resource segments being resource segments from each resource segment in the other multimedia resource excluding the target resource segment; and obtain the candidate resource segment pair based on each resource segment in each multimedia resource and the target resource segment in the other multimedia resource.

11. The apparatus according to claim 9, characterized in that, The second determining unit is further configured to perform a comparison of the second segment feature sets of two resource segments in the candidate resource segment pair using a dynamic time-normalization algorithm to obtain the second similarity between the two resource segments in the candidate resource segment pair.

12. The apparatus according to claim 9, characterized in that, The second determining unit is further configured to perform the following operation for any resource fragment: determining the candidate resource fragment pair with the highest second similarity among the candidate resource fragment pairs corresponding to the resource fragment as the target resource fragment pair corresponding to the resource fragment.

13. The apparatus according to claim 12, characterized in that, The target determination unit is further configured to perform the following: determine consecutive resource fragment pairs from the target resource fragment pairs; the resource fragments in the consecutive resource fragment pairs are consecutive in their respective multimedia resources; merge the consecutive resource fragment pairs to obtain merged resource fragment pairs; when the fragment length of the duplicate resource fragments in the merged resource fragment pairs is greater than a length threshold, determine the duplicate resource fragments as the target resource information.

14. The apparatus according to claim 9, characterized in that, The device further includes a first update unit configured to traverse a resource information template library of the account corresponding to the target resource information; when a first resource information template matching the target resource information already exists in the resource information template library, the unit obtains the information length of the target resource information and the first resource information template respectively; the similarity between the first resource information template and the target resource information is greater than a first similarity threshold; if the information length of the target resource information is greater than the information length of the first resource information template, the unit updates the first resource information template in the resource information template library according to the target resource information.

15. The apparatus according to claim 9, characterized in that, The device further includes a second updating unit configured to perform the following operations: acquiring newly added multimedia resources and traversing the resource information template library of the account corresponding to the newly added multimedia resources; when a second resource information template matching the newly added multimedia resources already exists in the resource information template library, acquiring the resource identifier of the newly added multimedia resources and the starting resource fragment information in the newly added multimedia resources that matches the second resource information template, and adding the resource identifier and the starting resource fragment information as multimedia resource information corresponding to the second resource information template in the resource information template library; wherein the similarity between the second resource information template and the newly added multimedia resources is greater than a second similarity threshold.

16. The apparatus according to claim 15, characterized in that, The second update unit is further configured to store the new multimedia resource in the unprocessed resource library of the account corresponding to the new multimedia resource when there is no second resource information template matching the new multimedia resource in the resource information template library. When the number of unprocessed multimedia resources in the unprocessed resource library exceeds the number threshold, the step of obtaining multiple resource fragment pairs based on multiple resource fragments corresponding to each of the multiple multimedia resources is returned.

17. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the multimedia resource processing method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the multimedia resource processing method as described in any one of claims 1 to 8.

19. A computer program product, the computer program product comprising instructions, characterized in that, When the instruction is executed by the processor of the electronic device, the electronic device is able to perform the multimedia resource processing method as described in any one of claims 1 to 8.

Images