A multimedia data processing method and apparatus

Abstract

The embodiment of the application provides a kind of multimedia data processing method and device, it is related to data processing technical field, the above-mentioned method includes: based on multimedia data obtains the multiple object features of target object to which the multimedia data is directed;For each object feature, obtain the first representation information of the feature offset degree of representation that object feature is relative to each object feature, and obtain the second representation information of representation that the matching degree between this object feature and preset reference feature;Based on the first representation information and the second representation information corresponding to each object feature, obtain the quality description information of each object feature;According to the quality description information obtained, determine target feature from each object feature, based on the determined target feature, downstream data processing is carried out to the multimedia data.The application of the embodiment of the application can improve the processing effect of downstream data processing to multimedia data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multimedia data processing method and apparatus. Background Technology

[0002] Multimedia data is becoming increasingly common in daily work and life, containing a wealth of information. In many scenarios, electronic devices can process multimedia data to obtain the required information, and then perform corresponding downstream data processing based on the obtained information.

[0003] For example, in access control scenarios, access control devices can acquire video footage of visitors captured by cameras, extract facial and body features from the video data, and determine whether the visitor is a pre-registered authorized person based on the extracted features, thereby deciding whether to open the access control. Similarly, in smart home appliance control scenarios, after a home appliance picks up a user's voice command, it can extract the user's voice features from the voice data, determine whether the user issuing the voice command has the necessary permissions based on these features, and thus decide whether to execute the command.

[0004] It is evident that in various scenarios, the data processing of multimedia data by electronic devices involves the extraction of object features. However, the quality of the extracted object features often varies, which can lead to poor processing results when downstream data processing is performed based on these features. Summary of the Invention

[0005] The purpose of this application is to provide a multimedia data processing method and apparatus to improve the processing effect of downstream data processing. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a multimedia data processing method, including:

[0007] Based on the multimedia data, obtain multiple object features of the target object for which the multimedia data is targeted;

[0008] For each object feature, first representation information is obtained that characterizes the degree of feature offset of the object feature relative to other object features, and second representation information is obtained that characterizes the degree of matching between the object feature and a preset benchmark feature.

[0009] Based on the first and second characterization information corresponding to each object feature, the quality description information of each object feature is obtained.

[0010] Based on the obtained quality description information, target features are determined from the features of each object, and downstream data processing is performed on the multimedia data based on the determined target features.

[0011] Secondly, embodiments of this application provide a multimedia data processing apparatus, including:

[0012] An object feature acquisition module is used to obtain multiple object features of the target object for which the multimedia data is targeted, based on the multimedia data.

[0013] The representation information acquisition module is used to obtain, for each object feature, first representation information representing the degree of feature offset of the object feature relative to other object features, and second representation information representing the degree of matching between the object feature and a preset benchmark feature.

[0014] The quality description information acquisition module is used to obtain the quality description information of each object feature based on the first and second characterization information corresponding to each object feature.

[0015] The data processing module is used to determine target features from the features of each object based on the obtained quality description information, and to perform downstream data processing on the multimedia data based on the determined target features.

[0016] Thirdly, embodiments of this application provide an electronic device, including:

[0017] Memory, used to store computer programs;

[0018] A processor, when executing a program stored in memory, implements the method described in the first aspect.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0020] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect above.

[0021] As can be seen from the above, when applying the solution provided in the embodiments of this disclosure to perform multimedia data processing, firstly, multiple object features of the target object for the multimedia data are obtained. Then, for each object feature, first characterization information and second characterization information corresponding to the object feature are obtained, and quality description information of each object feature is obtained based on the first characterization information and the second characterization information. In this way, the target feature can be determined from each object feature based on the obtained quality description information, and then the multimedia data can be processed downstream based on the determined target feature.

[0022] For each feature of the target object, the first representation information characterizes the degree of feature deviation of the object feature relative to other object features, and the second representation information characterizes the degree of matching between the object feature and a preset benchmark feature. The feature deviation reflects the uniformity of the object feature and can measure its feature quality. Based on this feature deviation, extreme features with large deviations relative to other object features can be filtered out in the subsequent feature selection stage. The matching degree reflects the similarity between the object feature and the preset benchmark feature and can also measure its feature quality. Based on this matching degree, features with high similarity to the preset benchmark feature can be identified in the subsequent feature selection stage. In summary, both the feature deviation and matching degree reflect the feature quality of the object feature. Therefore, the quality representation information obtained based on the first and second representation information can effectively reflect the feature quality of the object feature. Furthermore, based on the quality representation information of the object feature, target features with superior feature quality can be identified from the object features. This, in turn, improves the reliability and accuracy of downstream data processing, thus enhancing the processing effect of downstream data processing.

[0023] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0025] Figure 1 A flowchart illustrating the first multimedia data processing method provided in this application embodiment;

[0026] Figure 2 A flowchart illustrating the second multimedia data processing method provided in this application embodiment;

[0027] Figure 3 A schematic diagram of the first characterization information acquisition process provided in the embodiments of this application;

[0028] Figure 4 A flowchart illustrating the third multimedia data processing method provided in this application embodiment;

[0029] Figure 5 A schematic diagram illustrating the second characterization information acquisition process provided in this application embodiment;

[0030] Figure 6This is a schematic diagram of the structure of a multimedia data processing device provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0033] First, the implementing entity of the solution provided in the embodiments of this application will be described.

[0034] The implementation subject of the solution provided in this application embodiment is any electronic device with data processing and data storage functions. Specifically, it can be an electronic device that needs to process multimedia data in a real-world application scenario.

[0035] For example, in access control scenarios, the aforementioned electronic devices can be access control devices that process video or images of visitors; in smart home appliance control scenarios, the aforementioned electronic devices can be smart home appliance devices that process user voice.

[0036] The multimedia data processing scheme provided in the embodiments of this application will be described in detail below.

[0037] See Figure 1 This is a flowchart illustrating the first multimedia data processing method provided in this application embodiment. The method includes the following steps S101-S104.

[0038] Step S101: Obtain multiple object features of the target object targeted by the multimedia data based on the multimedia data.

[0039] First, let's explain the multimedia data mentioned above.

[0040] In terms of data type, the aforementioned multimedia data can be video, audio, images, etc.

[0041] In terms of data processing level, the aforementioned multimedia data can be raw data collected by multimedia data acquisition devices such as recording devices, photography devices, and video cameras, or it can be pre-processed multimedia data with a defined target object.

[0042] In this application, the target object is not limited to the above-mentioned target object; the target object can be any object that needs attention in the actual scene. For example, the target object can be a person, other animals, buildings, documents, etc.

[0043] The following explains how to obtain object characteristics when the multimedia data has already been preprocessed.

[0044] In one scenario, if the multimedia data is an image with the target object already marked, then the object features of the target object can be extracted based on an object feature extraction algorithm.

[0045] For example, the aforementioned object feature extraction algorithms can be convolutional neural network algorithms, LBP (Local Binary Patterns) algorithms, HOG (Histogram of Oriented Gradient) algorithms, etc.

[0046] In another scenario, if the multimedia data is a video containing video frames with labeled target objects, then video frames containing target objects can be extracted from the video, and then the object features of the target objects can be extracted based on the extracted video frames using the object feature extraction algorithm described above.

[0047] In another scenario, if the multimedia data is audio targeting a specific object, audio features can be extracted using an audio feature extraction algorithm, and the extracted features can be used as the object features of the target object.

[0048] For example, the aforementioned audio feature extraction algorithms can be MFCC (Mel Frequency Cepstral Coefficients), PLP (Perceptual Linear Predictive), DWT (Discrete Wavelet Transform), LPC (Linear Prediction Coefficient), etc.

[0049] Depending on the audio feature extraction algorithm used, the extracted audio features can be audio amplitude, frequency, spectrum, timbre, spectral flux, and other features.

[0050] Next, we will explain how to obtain object characteristics when the multimedia data is unprocessed raw data.

[0051] In this case, the target object of the multimedia data can be identified first, and then the object characteristics of the target object can be obtained in the manner described above. The method for identifying the target object is explained below.

[0052] In one scenario, if the multimedia data is the original image, preliminary object detection can be performed on the image, and the objects that meet the preset criteria among the detected objects can be identified as target objects.

[0053] Preliminary object detection can be performed using simple methods such as feature point extraction, or it can be performed using object detection algorithms such as edge extraction operators and texture feature extraction algorithms. This application does not limit the comparison.

[0054] In another scenario, if the multimedia data is the original video, preliminary object detection can be performed on the target frames in the video to identify the detected objects as target objects, and then the video frames containing the target objects can be identified from each video frame.

[0055] The target frame mentioned above can be the first frame of the video, or a key frame within the video frame, etc.

[0056] The following describes how to determine the video frame containing the target object from each video frame.

[0057] The first approach is to use a multi-frame target association algorithm based on Kalman filtering to predict video frames that may contain the target object and are located after the target frame in time, and use these as the determined video frames.

[0058] The second method can identify objects in each video frame, then determine the center position of the objects in each video frame, the area where the objects are located, etc., and then, based on the center position of the objects in each video frame and / or the area where the objects are located, determine the video frames containing the target objects from each video frame.

[0059] For example, video frames whose distance between their center position and the center position of the target object is less than a first preset threshold are determined as video frames containing the target object, and video frames whose overlap between the area where the object is located and the area where the target object is located in the target frame is greater than a second preset threshold are determined as video frames containing the target object.

[0060] In another case, if the multimedia data is audio, the target of the audio can be determined based on the human voice separation algorithm, and audio frames for different objects can be segmented from the audio. Then, any one of these objects can be identified as the target object, and audio frames for the target object can be determined from the aforementioned audio frames.

[0061] It should be noted that in this case, each object can be treated as a target object, and the multimedia data processing method provided in the embodiments of this application can be executed multiple times to obtain the target features of each object, and downstream data processing can be performed based on the target features.

[0062] Step S102: For each object feature, obtain first representation information that characterizes the degree of feature offset of the object feature relative to other object features, and obtain second representation information that characterizes the degree of matching between the object feature and the preset benchmark feature.

[0063] First, we will explain how the first representation information is obtained.

[0064] The first representation information represents the degree of feature offset of the object feature relative to the features of each object, and the aforementioned feature offset reflects the feature difference between the object feature and each feature.

[0065] In one implementation, the distance between the object feature and each other object feature can be calculated separately, and then the first representation information can be obtained based on the obtained distance.

[0066] The aforementioned distances can be Euclidean distance, Manhattan distance, etc.

[0067] Specifically, representative values ​​of the calculated distances can be obtained, and first characterization information can be derived based on these representative values. For example, representative values ​​such as the average and median of the aforementioned distances can be obtained, and then the reciprocal of these representative values ​​can be used as the first characterization information.

[0068] In this case, the larger the first representation information, the greater the distance between the object feature and the features of other objects, that is, the greater the feature offset relative to the features of other objects; conversely, the smaller the first representation information, the smaller the distance between the object feature and the features of other objects, that is, the smaller the feature offset relative to the features of other objects. In other words, the magnitude of the first representation information can reflect the degree of feature offset of the object feature relative to the features of other objects.

[0069] In another implementation, the central features of each object feature can be obtained first, then the distance between the object feature and the central features can be calculated, and the first representation information can be obtained based on the obtained distance. Detailed implementation methods will be provided later. Figure 2 Steps S202-S204 in the illustrated embodiment will not be described in detail here.

[0070] Next, we will explain how the second representation information is obtained.

[0071] First, the similarity between the object's features and preset benchmark features can be calculated. Then, second representation information is obtained based on the obtained similarity. The preset benchmark features can be pre-stored in a feature library, so that when obtaining the second representation information, the preset benchmark features can be retrieved from the direct feature library.

[0072] For detailed implementation methods, please refer to the subsequent sections. Figure 4 Steps S403-S405 in the illustrated embodiment will not be described in detail here.

[0073] It is evident that the second representation information is obtained based on the similarity between the object features and the preset benchmark features. Therefore, the second representation information can reflect the degree of matching between the object features and the preset benchmark features.

[0074] Step S103: Based on the first and second characterization information corresponding to each object feature, obtain the quality description information of each object feature.

[0075] Specifically, the quality description information of each object's characteristics can be obtained in the following ways.

[0076] In one implementation, the first and second characterization information corresponding to each object feature can be weighted and calculated according to preset weights corresponding to the first and second characterization information, and quality description information can be obtained based on the calculation results.

[0077] The aforementioned preset weights can be set by staff according to the actual situation.

[0078] In another implementation, the sum of the first and second characterization information corresponding to each object feature can be obtained, and the resulting sum can be determined as the quality description information of each object feature.

[0079] In another embodiment, for each object feature, a first weight of the first representation information corresponding to the object feature and a second weight of the second representation information corresponding to the object feature can be determined based on a first quantity and a second quantity. Based on the first weight and the second weight, the first representation information and the second representation information corresponding to the object feature are weighted and calculated to obtain the quality description information of the object feature.

[0080] The relative size relationship between the first weight and the second weight is consistent with the relative size relationship between the first quantity and the second quantity. The first quantity is the number of object features obtained, and the second quantity is the number of preset benchmark features used to obtain the second representation information of the object features.

[0081] The aforementioned consistency in relative size means that the relative size trend between the first weight and the second weight is consistent with the relative size trend between the first quantity and the second quantity. Specifically, when the first quantity is greater than the second quantity, the first weight is also greater than the second weight; when the first quantity is less than the second quantity, the first weight is also less than the second weight; and when the first quantity is equal to the second quantity, the first weight is also equal to the second weight.

[0082] The embodiments of this application do not limit the specific way of determining the first weight and the second weight based on the first quantity and the second quantity. It is only necessary to ensure that the relative size relationship between the first weight and the second weight is consistent with the relative size relationship between the first quantity and the second quantity. Several possible methods are described below.

[0083] The first method can determine the size of the first quantity and the second quantity, and determine the first weight and the second weight based on the judgment result and the preset fixed weight.

[0084] For example, if the judgment result is that the first weight is larger, then the first weight is determined as the first fixed weight, and the second weight is determined as the second fixed weight; if the judgment result is that the second weight is larger, then the first weight is determined as the second fixed weight, and the second weight is determined as the first fixed weight; if the judgment result is that the first quantity and the second quantity are the same, then both the first weight and the second weight are determined as the third fixed weight.

[0085] The aforementioned fixed weights can be set by staff based on experience, such as a first fixed weight of 0.7, a second fixed weight of 0.3, and a third fixed weight of 0.5, etc.

[0086] The second method is to calculate the sum of the first quantity and the second quantity, and then calculate the ratio of the first quantity and the second quantity to the obtained sum, and use the obtained ratio as the first weight and the second weight mentioned above.

[0087] For example, if the first quantity is 80 and the second quantity is 120, then the sum is 80 + 120 = 200. Therefore, the first weight is 80 / 200 = 0.4 and the second weight is 120 / 200 = 0.6.

[0088] The third method is to determine the first weight and the second weight corresponding to the first quantity and the second quantity based on the preset correspondence between the quantity and the weight.

[0089] The above correspondence can be a step function, where the independent variable represents the quantity, and the constant value corresponding to the independent variable represents the weight corresponding to the quantity.

[0090] For example, the step function described above can be expressed as follows:

[0091]

[0092] Where y represents the weight, x represents the quantity, n is a positive integer, AN is a constant, and A < B < C < ... < N.

[0093] By substituting the first and second quantities as values ​​of the independent variable x into the above step function, we can obtain the corresponding values ​​of y, which are the first and second weights.

[0094] As can be seen, the first quantity is the number of object features obtained, that is, the number of object features used to obtain the first representation information of the object features. The larger the first quantity, the more comprehensive and accurate the first representation information obtained can be considered, that is, the higher the importance of the first representation information in measuring feature quality. Similarly, the second quantity is the number of preset benchmark features used to obtain the second representation information of the object features. The larger the second quantity, the more comprehensive and accurate the second representation information obtained, and the higher the importance of the second representation information in measuring feature quality.

[0095] Therefore, this makes the relative size relationship between the first weight and the second weight consistent with the relative size relationship between the first quantity and the second quantity. In other words, it makes the relative size of the first weight and the second weight consistent with the relative importance of the first and second characterization information in measuring feature quality. Thus, when the first and second characterization information are weighted according to the above weights to obtain quality characterization information, the accuracy of the obtained quality characterization information can be improved.

[0096] Step S104: Based on the obtained quality description information, determine the target features from the features of each object, and perform downstream data processing on the multimedia data based on the determined target features.

[0097] First, we will explain how to determine the target features from the features of each object.

[0098] In one implementation, object features whose quality description information values ​​are greater than a third preset threshold can be identified as target features.

[0099] In another implementation, the object features corresponding to the top preset number of quality description information with the largest values ​​in the obtained quality description information can be determined as target features.

[0100] It should be noted that after determining the target features from the features of each object, the embodiments of this application do not limit the specific downstream data processing of the multimedia data based on the determined target features. Examples are given below.

[0101] Specifically, based on the identified target characteristics, at least one of the following downstream data processing methods can be performed on the multimedia data:

[0102] 1. Perform feature fusion on the determined target features to obtain the fused features of the target object targeted by the multimedia data.

[0103] Among them, various feature fusion algorithms such as feature concatenation algorithm, weighted linear fusion algorithm, and cross fusion algorithm can be used to perform feature fusion on target features, which will not be elaborated here.

[0104] In this way, feature fusion can fully leverage the complementarity between target features, smooth out the differences between target features, and obtain fused features with stronger representational capabilities, and then be used as a basis for further analysis.

[0105] 2. Perform object identification on the target objects of multimedia data.

[0106] The preset object corresponding to the target object can be determined based on the similarity between the target feature and the preset object features of each preset object. See the examples shown in steps A-E and F-J in the following embodiments for details.

[0107] 3. Classify the target objects of the multimedia data.

[0108] The object category to which the target object belongs can be determined based on the similarity between the target features and the preset category features of each preset object category.

[0109] For example, the similarity between the target feature and the preset category feature can be calculated. If there is a target similarity greater than the first similarity threshold among the obtained similarities, the target object can be classified into the object category corresponding to the largest target similarity.

[0110] 4. Obtain associated data for multimedia data.

[0111] It is possible to obtain associated data of multimedia data based on the similarity between the target features and the preset multimedia features of existing multimedia data in the multimedia database.

[0112] For example, the similarity between the target feature and the preset multimedia features can be calculated, and existing multimedia data in the multimedia database with a similarity greater than the second similarity threshold can be identified as associated data of the multimedia data.

[0113] As can be seen, after determining the target features using the scheme provided in this application, various downstream data processing can be performed on multimedia data, such as target recognition, object classification, and related data acquisition. Since the target features are selected from the object features of the target object based on quality description information, their feature quality is superior. Therefore, the processing effect of the above-mentioned various downstream data processing can be improved based on the target features.

[0114] In one embodiment of this application, before performing the various downstream data processing steps described above on the multimedia data, feature fusion can be performed on the determined target features to obtain fused features, and then the multimedia data can be processed downstream based on the fused features.

[0115] This fusion feature, with its stronger representational capabilities, can further improve the processing efficiency of downstream data processing.

[0116] As can be seen from the above, when applying the solution provided in the embodiments of this disclosure to perform multimedia data processing, firstly, multiple object features of the target object for the multimedia data are obtained. Then, for each object feature, first characterization information and second characterization information corresponding to the object feature are obtained, and quality description information of each object feature is obtained based on the first characterization information and the second characterization information. In this way, the target feature can be determined from each object feature based on the obtained quality description information, and then the multimedia data can be processed downstream based on the determined target feature.

[0117] For each feature of the target object, the first representation information characterizes the degree of feature deviation of the object feature relative to other object features, and the second representation information characterizes the degree of matching between the object feature and a preset benchmark feature. The feature deviation reflects the uniformity of the object feature and can measure its feature quality. Based on this feature deviation, extreme features with large deviations relative to other object features can be filtered out in the subsequent feature selection stage. The matching degree reflects the similarity between the object feature and the preset benchmark feature and can also measure its feature quality. Based on this matching degree, features with high similarity to the preset benchmark feature can be identified in the subsequent feature selection stage. In summary, both the feature deviation and matching degree reflect the feature quality of the object feature. Therefore, the quality representation information obtained based on the first and second representation information can effectively reflect the feature quality of the object feature. Furthermore, based on the quality representation information of the object feature, target features with superior feature quality can be identified from the object features. This, in turn, improves the reliability and accuracy of downstream data processing, thus enhancing the processing effect of downstream data processing.

[0118] exist Figure 1 Based on the illustrated embodiment, when obtaining first characterization information representing the degree of feature offset of a specific object feature relative to each other object feature, the distance between the object feature and the central feature of each other object feature can be determined first, and the first characterization information can be obtained based on the obtained distance. In view of the above, embodiments of this application provide a second multimedia data processing method.

[0119] See Figure 2This is a flowchart illustrating a second multimedia data processing method provided in an embodiment of this application. The method includes the following steps S201-S207.

[0120] Step S201: Obtain multiple object features of the target object targeted by the multimedia data based on the multimedia data.

[0121] The above step S201 is the same as the aforementioned Figure 1 Step S101 is the same in the illustrated embodiment, and will not be repeated here.

[0122] Step S202: Obtain the central features of each object.

[0123] Specifically, the above central features can be obtained in the following ways.

[0124] In one implementation, the object features are in the form of feature vectors. In this case, the average vector of each feature vector can be calculated, and the calculation result can be used as the central feature of the object features.

[0125] Specifically, the central feature f can be calculated according to the following expression. center :

[0126]

[0127] Among them, the above f i Let m represent the i-th object feature, and m represent the number of object features.

[0128] By summing up the object features and calculating the average value, the central feature of each object can be obtained.

[0129] In another implementation, the object features can be clustered based on a clustering algorithm, and the cluster centers can be determined based on the clusters obtained. The obtained cluster centers can be used as the central features of the object features.

[0130] The clustering algorithms mentioned above can be BIRCH (Balanced Iterative Reducing and Clustering Using Hierarchies), CURE (Clustering Using Representative), K-Means, etc.

[0131] After obtaining the central feature, for each object feature, the first representation information corresponding to that object feature can be obtained based on the following steps S203-S204.

[0132] Step S203: Determine the first distance between the object feature and the central feature.

[0133] Specifically, the Euclidean distance, Manhattan distance, or other feature distances between the object's features and the central features can be calculated as the aforementioned first distance.

[0134] Step S204: Based on the first distance, obtain first representation information that characterizes the degree of feature offset of the object feature relative to the features of each object.

[0135] In one case, the reciprocal of the first distance can be directly used as the first representation information.

[0136] Thus, the magnitude of the first distance is inversely proportional to the magnitude of the first representation information. Since the magnitude of the first distance is inversely proportional to the degree of feature offset, this allows the magnitude of the first representation information to characterize the degree of feature offset of the object feature relative to the features of other objects.

[0137] In another implementation, the first characterization information can be obtained based on the determined first distance and second distance.

[0138] The second distance is the distance between the other object features (excluding the object feature itself) and the central feature.

[0139] Specifically, the following will introduce two methods as examples.

[0140] The first method allows us to obtain the first representation information L using the following expression. i :

[0141]

[0142] Where m represents the number of object features obtained, l i Indicates the first distance, l j This represents the second distance corresponding to the j-th object feature.

[0143] It can be seen that the above expression means: the product of the reciprocal of the sum of the second distances corresponding to each object feature and the reciprocal of the first distance corresponding to that object feature.

[0144] For example, if object features 1-3 are obtained, where the first distance corresponding to object feature 1 is 1, and the second distances corresponding to object features 2 and 3 are 2 and 2 respectively, then when obtaining the first representation information corresponding to object feature 1, we can first obtain the reciprocal of the sum of the second distances corresponding to each object feature as 1 / (1+2+2) = 0.2, and the reciprocal of the first distance corresponding to object feature 1 as 1. Then the first representation information corresponding to object feature 1 is the product of the reciprocal of the sum of the second distances corresponding to each object feature and the reciprocal of the first distance corresponding to that object feature: 0.2 × 1 = 0.2.

[0145] Based on the above expression, the first representation information can be obtained quickly and conveniently. Furthermore, when obtaining the first representation information based on the above expression, the first representation information is the product of the reciprocal of the sum of the second distances of each object's features and the reciprocal of the first distance of that object's feature. In this way, the first representation information is not only related to the first and second distances, but also reflects the difference between the sum of the first and second distances, making the obtained first representation information more comprehensive and able to reflect more information.

[0146] The second method is to obtain the first representation information L according to the following expression. i :

[0147]

[0148] The meanings of the parameters in the above expression are the same as those in the previous method, and will not be repeated here.

[0149] It can be seen that, in addition to the first distance between the object feature and the central feature, the first representation information can also be obtained by combining the second distances between other object features and the central feature. This makes the first representation information not only related to the first distance but also to the second distances of each object feature. Since both the first and second distances can reflect the degree of feature shift of the object feature relative to the central feature, this allows the first representation information obtained based on the first and second distances to more comprehensively represent the degree of feature shift of the object feature relative to other object features.

[0150] Step S205: Obtain second representation information that characterizes the degree of matching between the object's features and the preset benchmark features.

[0151] Step S206: Based on the first and second characterization information corresponding to each object feature, obtain the quality description information of each object feature.

[0152] Step S207: Based on the obtained quality description information, determine the target features from the features of each object, and perform downstream data processing on the multimedia data based on the determined target features.

[0153] Steps S205-S207 have been described above. Figure 1 The steps S102-S104 in the illustrated embodiment are explained and will not be repeated here.

[0154] As can be seen from the above, when obtaining the first representation information corresponding to the object features using the scheme provided in the embodiments of this disclosure, the central feature of each object feature is first obtained. The central feature reflects the information of the average representation of each object feature and can serve as a representative of each object feature. Based on this, the first distance between the object feature and the central feature represents the degree of feature offset of the object feature relative to the central feature, which also represents the overall feature offset of the object feature relative to each other object features. Therefore, based on the first distance, the first representation information representing the degree of feature offset of the object feature relative to each other object features can be obtained relatively conveniently and accurately.

[0155] The following is combined with Figure 3 The overall process of obtaining the first characterization information shown in steps S202-S204 above will be explained.

[0156] See Figure 3 This is a schematic diagram of the first characterization information acquisition process provided in the embodiments of this application. The process includes the following steps S301-S304.

[0157] Step S301: Obtain multiple object features.

[0158] The aforementioned object characteristics are the same as the object characteristics of the multimedia data mentioned above.

[0159] Step S302: Obtain the central features of each object.

[0160] Step S303: Obtain the distance between each object feature and the central feature.

[0161] The distances obtained include the distances between all object features and the central feature.

[0162] Specifically, for a certain object feature, the obtained distance includes a first distance between the object feature and the central feature, and a second distance between other object features and the central feature.

[0163] Step S304: Obtain first representation information based on the obtained first distance and second distance.

[0164] For details on how to obtain the first representation information based on the first distance and the second distance, please refer to the foregoing. Figure 2 Step S204 in the illustrated embodiment will not be repeated here.

[0165] Since both the first distance and the second distance can reflect the degree of feature shift of the object feature relative to the central feature, the first representation information that characterizes the degree of feature shift of the object feature relative to the central feature can be obtained based on the first distance and the second distance.

[0166] exist Figure 1 Based on the illustrated embodiment, when obtaining second characterization information representing the degree of matching between a specific object feature and preset benchmark features, the similarity between the object feature and each preset benchmark feature can be obtained first, and then the second characterization information can be obtained based on the obtained similarity. In view of the above, embodiments of this application provide a third multimedia data processing method.

[0167] See Figure 4 This is a flowchart illustrating a third multimedia data processing method provided in an embodiment of this application. The method includes the following steps S401-S407.

[0168] Step S401: Obtain multiple object features of the target object targeted by the multimedia data based on the multimedia data.

[0169] The above step S401 is the same as the aforementioned Figure 1 Step S101 is the same in the illustrated embodiment, and will not be repeated here.

[0170] After obtaining multiple object features, for each object feature, the first representation information and the second representation information corresponding to that object feature can be obtained based on the following steps S402-S405.

[0171] Step S402: Obtain first representation information that characterizes the degree of feature offset of the object feature relative to the features of each object.

[0172] The above step S402 is the same as the aforementioned Figure 1 Step S102 is the same in the illustrated embodiment, and will not be repeated here.

[0173] Step S403: Obtain the similarity between the object's features and each preset benchmark feature.

[0174] The aforementioned similarity can be cosine similarity, Pearson correlation coefficient, etc., and the embodiments of this application do not limit it.

[0175] Step S404: Determine the maximum similarity from the obtained similarities.

[0176] Step S405: Based on the maximum similarity, obtain second representation information that characterizes the degree of matching between the object's features and the preset benchmark features.

[0177] In one implementation, the maximum similarity can be directly determined as the second representation information.

[0178] The magnitude of the maximum similarity is proportional to the magnitude of the second representation information. Since the magnitude of the maximum similarity is proportional to the positive match, this allows the second representation information to characterize the degree of matching between the object's features and the preset benchmark features.

[0179] In another implementation, the maximum number of target similarities can be determined from the obtained similarities, and then the second characterization information can be obtained based on the maximum similarity and the target similarity.

[0180] Specifically, the following will introduce two methods as examples.

[0181] The first method allows obtaining the second representation information q using the following expression. i :

[0182]

[0183] Among them, sim(top) i, ) represents the maximum similarity, sim(top) i,j ) represents the target similarity sorted by j, and the sorting is as follows: the obtained similarity is sorted from largest to smallest, and k represents the preset number.

[0184] As can be seen, the above expression means: the proportion of the maximum similarity in the sum of the top K similarities.

[0185] For example, for object feature i, first obtain the similarity between the object feature and each preset benchmark feature from 1 to 10. The top 3 target similarities with the largest similarity are 0.8, 0.05, and 0.15. Obviously, the largest similarity is 0.8. Then, the sum of the top 3 largest similarities is 0.8 + 0.05 + 0.15 = 1. The second representation information is the proportion of the largest similarity in the sum of the top K largest similarities: 0.8 / 1 = 0.8.

[0186] The second representation information can be obtained quickly and conveniently based on the above expression. Furthermore, when obtaining the second representation information based on the above expression, the second representation information is: the proportion of the maximum similarity in the sum of the top K similarities. The second representation information obtained in this way is correlated with all of the top K similarities and can also reflect the leading margin of the maximum similarity relative to the subsequent K-1 similarities, making the obtained second representation information more comprehensive and able to reflect more information.

[0187] The second method is to obtain q based on the first method mentioned above. i Then, the second representation information Q can be obtained according to the following expression. i :

[0188]

[0189] As can be seen, based on the above expression, the q of the object's characteristics can be... i q of each object feature jThe proportion in the sum is used as the second representation information. This is equivalent to normalizing the second representation information, making its value more stable and able to reflect more information.

[0190] The third method is to obtain the second representation information q according to the following expression. i :

[0191]

[0192] The meanings of the parameters in the above expression are the same as those in the previous method, and will not be repeated here.

[0193] It can be seen that, in addition to the maximum similarity between the object feature and the preset benchmark feature, a second representation information can also be obtained by combining the maximum number of similarities between the object feature and the preset benchmark feature. This makes the second representation information related not only to the maximum similarity but also to the maximum number of similarities. Since the maximum number of similarities can all reflect the degree of matching between the object feature and the preset benchmark feature, this allows the second representation information to more comprehensively represent the degree of matching between the object feature and the preset benchmark feature.

[0194] Step S406: Based on the first and second characterization information corresponding to each object feature, obtain the quality description information of each object feature.

[0195] Step S407: Based on the obtained quality description information, determine the target features from the features of each object, and perform downstream data processing on the multimedia data based on the determined target features.

[0196] Steps S406-S407 above are the same as those mentioned above. Figure 1 In the illustrated embodiment, steps S103-S104 are the same and will not be repeated here.

[0197] As can be seen from the above, when obtaining the second characterization information corresponding to the object feature by applying the scheme provided in the embodiments of this disclosure, the similarity between the object feature and each preset benchmark feature is first obtained. The similarity can reflect the degree of matching between the object feature and each preset benchmark feature. Then, based on the similarity, the second characterization information representing the degree of matching between the object feature and the preset benchmark feature can be obtained more conveniently and accurately.

[0198] In one embodiment of this application, when obtaining the second characterization information, a target reference feature can be selected from preset reference features according to the feature type of the object feature, and then the second characterization information representing the degree of matching between the object feature and the target reference feature can be obtained.

[0199] The aforementioned feature types can be data types of object features, such as image features, audio features, etc.; or they can be content types of object features, such as face features, body features, pitch features, spectrum features, etc.

[0200] Specifically, when storing preset benchmark features, the correspondence between each benchmark feature and feature type can be recorded in advance. In this way, based on the above object relationship, the target benchmark feature corresponding to the feature type of the object feature can be selected from the preset benchmark features.

[0201] After obtaining the target baseline features, the method for obtaining second representation information that characterizes the degree of matching between the object features and the target baseline features can be described above. Figure 4 The embodiments shown are based on steps S402-S405, with the only difference being that the preset reference feature is replaced with the target reference feature, which will not be described in detail here.

[0202] It is evident that when determining the benchmark features used for comparison with the object features, the target benchmark features are selected from the preset benchmark features according to the feature type of the object features. That is, the preset benchmark features participating in the comparison are further selected. In this way, when obtaining the second representation information based on the object features and the target benchmark features, the comparison granularity between the object features and the benchmark features is refined, the accuracy of the comparison is improved, and thus, based on the object features and the target benchmark features, it is beneficial to obtain more accurate and reasonable second representation information.

[0203] The following is combined with Figure 5 Regarding the above Figure 4 The overall process of obtaining the second characterization information shown in steps S403-S405 of the illustrated embodiment will be described.

[0204] See Figure 5 This is a schematic diagram of the second characterization information acquisition process provided in the embodiments of this application. The process includes the following steps S501-S505.

[0205] Step S501: Obtain multiple object features.

[0206] Step S502: Obtain the similarity between the features of each object and the target baseline features.

[0207] Step S503: Sort the obtained similarities.

[0208] The obtained similarity scores can be sorted in descending order of similarity.

[0209] Step S504: Select the first preset number of target similarities in the sorting.

[0210] Of course, the first preset number of target similarities selected from the sorting in this step include the maximum similarity.

[0211] Step S505: Obtain second representation information based on the selected similarity.

[0212] As can be seen, the first preset number of target similarities in the above sorting are the largest preset number of similarities among the obtained similarities. Since the target similarities can all reflect the degree of matching between the object features and the preset benchmark features, the second representation information that represents the degree of matching between the object features and the preset benchmark features is obtained based on the target similarities.

[0213] The multimedia data processing method provided in this application embodiment will be explained more intuitively below in conjunction with specific application scenarios.

[0214] First, the application of the solution provided in the embodiments of this application in the access control scenario will be introduced through the following steps A-E.

[0215] Step A: Extract the visitor's characteristics from the video captured by the camera equipment.

[0216] Specifically, after detecting visitors from the target frames of the video, multiple video frames containing visitors can be identified based on the aforementioned multi-frame target association algorithm, and personnel features can be extracted from the obtained video frames.

[0217] Step B: For each person feature, obtain first representation information that represents the degree of feature offset of the person feature relative to other person features, and obtain second representation information that represents the degree of matching between the person feature and the preset benchmark feature.

[0218] Step C: Based on the first and second characterization information corresponding to each person's characteristics, obtain the quality description information of each person's characteristics.

[0219] Step D: Based on the obtained quality description information, determine the target characteristics from the characteristics of each person.

[0220] The specific implementation methods for steps B-D above have been described in the foregoing. Figure 1 , Figure 2 , Figure 4 As illustrated in the embodiments, the only difference is that the object features are replaced with personnel features, which will not be elaborated further here.

[0221] Step E: Based on the determined target characteristics and the personnel characteristics of pre-registered authorized personnel, identify the visitor. If the visitor is confirmed to be a pre-registered authorized person, control the access control to open.

[0222] Specifically, the similarity between the target feature and the personnel features of the allowed personnel can be calculated. If there is a target similarity greater than the third similarity threshold among the obtained similarities, the visitor can be identified as the allowed personnel corresponding to the largest target similarity, and the access control can be opened. If there is no target similarity greater than the third similarity threshold, it can be determined that the target personnel are not allowed personnel, so the access control can be left unopened.

[0223] Of course, we can also first perform feature fusion on the determined target features, and then identify visitors based on the fused features and the personnel features of pre-registered authorized personnel.

[0224] Since the target feature is a quality description of personnel features selected from personnel features, its feature quality is relatively high and its identification ability of personnel is strong. Therefore, by calculating the similarity between the target feature and the personnel features of the allowed personnel, it is possible to accurately identify whether the visitor is an allowed person, and then to accurately control the access control switch based on the identification results.

[0225] The following steps, F-J, will further introduce the application of the solution provided in the embodiments of this application in the smart home appliance control scenario.

[0226] Step F: Extract the user's voice features from the picked-up user command speech.

[0227] Step G: For each speech feature, obtain first representation information that represents the degree of feature offset of the speech feature relative to other speech features, and obtain second representation information that represents the degree of matching between the speech feature and the preset benchmark feature.

[0228] Step H: Based on the first and second representation information corresponding to each speech feature, obtain the quality description information of each speech feature.

[0229] Step 1: Based on the obtained quality description information, determine the target features from each speech feature.

[0230] The specific implementation methods of steps G-I above have been described in the foregoing. Figure 1 , Figure 2 , Figure 4 As illustrated in the embodiments, the only difference is that the object features are replaced with speech features, which will not be elaborated further here.

[0231] Step J: Based on the determined target features and the voice features of pre-registered users with command permissions, perform user identification. If the user is determined to be a user with command permissions, perform speech recognition and semantic analysis on the user's command speech, and execute the corresponding action based on the analysis results.

[0232] Among them, speech recognition and semantic analysis can be performed based on algorithms such as Hidden Markov Models and Dynamic Time Warping.

[0233] Specifically, the similarity between the target feature and the voice feature of a user with command authority can be calculated. If there is a target similarity greater than the fourth similarity threshold, the user can be identified as a user with command authority, and the voice can be recognized and analyzed. Based on the analysis results, corresponding actions can be executed. If there is no target similarity greater than the fourth similarity threshold, the user can be determined not to have command authority, and therefore no action can be executed.

[0234] Of course, feature fusion can also be performed on the identified target features first, and user identification can be performed based on the obtained fused features and the voice features of pre-registered users with command permissions.

[0235] Since the target feature is a feature selected from speech features based on the quality description information of speech features, its feature quality is relatively high and its ability to identify user speech is strong. Therefore, by calculating the similarity between the target feature and the speech features of users with command permissions, it is possible to more accurately determine whether a user has command permissions. Then, based on the determination result, it is possible to determine whether to execute an action, which improves the accuracy of determining whether a user has command permissions, reduces the occurrence of errors in user identity confirmation, and improves the precision of controlling smart home appliances.

[0236] Corresponding to the above-described multimedia data processing method, this application also provides a multimedia data processing apparatus.

[0237] See Figure 6 This is a schematic diagram of the structure of a first multimedia data processing device provided in the embodiments of this application. The device includes the following modules 601-604.

[0238] The object feature acquisition module 601 is used to obtain multiple object features of the target object targeted by the multimedia data based on the multimedia data.

[0239] The representation information acquisition module 602 is used to obtain, for each object feature, first representation information representing the degree of feature offset of the object feature relative to other object features, and second representation information representing the degree of matching between the object feature and a preset benchmark feature.

[0240] The quality description information acquisition module 603 is used to obtain the quality description information of each object feature based on the first characterization information and the second characterization information corresponding to each object feature;

[0241] The data processing module 604 is used to determine target features from the features of each object based on the obtained quality description information, and to perform downstream data processing on the multimedia data based on the determined target features.

[0242] As can be seen from the above, when applying the solution provided in the embodiments of this disclosure to perform multimedia data processing, firstly, multiple object features of the target object for the multimedia data are obtained. Then, for each object feature, first characterization information and second characterization information corresponding to the object feature are obtained, and quality description information of each object feature is obtained based on the first characterization information and the second characterization information. In this way, the target feature can be determined from each object feature based on the obtained quality description information, and then the multimedia data can be processed downstream based on the determined target feature.

[0243] For each feature of the target object, the first representation information characterizes the degree of feature deviation of the object feature relative to other object features, and the second representation information characterizes the degree of matching between the object feature and a preset benchmark feature. The feature deviation reflects the uniformity of the object feature and can measure its feature quality. Based on this feature deviation, extreme features with large deviations relative to other object features can be filtered out in the subsequent feature selection stage. The matching degree reflects the similarity between the object feature and the preset benchmark feature and can also measure its feature quality. Based on this matching degree, features with high similarity to the preset benchmark feature can be identified in the subsequent feature selection stage. In summary, both the feature deviation and matching degree reflect the feature quality of the object feature. Therefore, the quality representation information obtained based on the first and second representation information can effectively reflect the feature quality of the object feature. Furthermore, based on the quality representation information of the object feature, target features with superior feature quality can be identified from the object features. This, in turn, improves the reliability and accuracy of downstream data processing, thus enhancing the processing effect of downstream data processing.

[0244] In one embodiment of this application, after the object feature acquisition module 601, the following is further included:

[0245] The central feature acquisition module is used to obtain the central features of each object.

[0246] The characterization information acquisition module 602 includes:

[0247] The first distance determination submodule is used to determine the first distance between the object feature and the central feature;

[0248] The first representation information acquisition submodule is used to obtain first representation information based on the first distance, representing the degree of feature offset of the object feature relative to the features of each object.

[0249] As can be seen from the above, when obtaining the first representation information corresponding to the object features using the scheme provided in the embodiments of this disclosure, the central feature of each object feature is first obtained. The central feature reflects the information of the average representation of each object feature and can serve as a representative of each object feature. Based on this, the first distance between the object feature and the central feature represents the degree of feature offset of the object feature relative to the central feature, which also represents the overall feature offset of the object feature relative to each other object features. Therefore, based on the first distance, the first representation information representing the degree of feature offset of the object feature relative to each other object features can be obtained relatively conveniently and accurately.

[0250] In one embodiment of this application, the first characterization information acquisition submodule includes:

[0251] The first characterization information acquisition unit is used to obtain first characterization information representing the degree of feature offset of the object feature relative to each object feature based on the determined first distance and second distance, wherein the second distance is the distance between the obtained object features other than the object feature and the central feature.

[0252] It can be seen that, in addition to the first distance between the object feature and the central feature, the first representation information can also be obtained by combining the second distances between other object features and the central feature. This makes the first representation information not only related to the first distance but also to the second distances of each object feature. Since both the first and second distances can reflect the degree of feature shift of the object feature relative to the central feature, this allows the first representation information obtained based on the first and second distances to more comprehensively represent the degree of feature shift of the object feature relative to other object features.

[0253] In one embodiment of this application, the first characterization information obtaining unit is specifically used to obtain first characterization information L, which characterizes the degree of feature offset of the object feature relative to each object feature, according to the following expression. i :

[0254]

[0255] Where m represents the number of object features obtained, l i Let l represent the first distance. j This represents the second distance corresponding to the j-th object feature.

[0256] Based on the above expression, the first representation information can be obtained quickly and conveniently. Furthermore, when obtaining the first representation information based on the above expression, the first representation information is the product of the reciprocal of the sum of the second distances of each object's features and the reciprocal of the first distance of that object's feature. In this way, the first representation information is not only related to the first and second distances, but also reflects the difference between the sum of the first and second distances, making the obtained first representation information more comprehensive and able to reflect more information.

[0257] In one embodiment of this application, the characterization information acquisition module 602 includes:

[0258] The similarity acquisition submodule is used to obtain the similarity between the object's features and each preset benchmark feature;

[0259] The maximum similarity determination submodule is used to determine the maximum similarity from the obtained similarities;

[0260] The second representation information acquisition submodule is used to obtain second representation information that represents the degree of matching between the object's features and the preset benchmark features based on the maximum similarity.

[0261] As can be seen from the above, when obtaining the second characterization information corresponding to the object feature by applying the scheme provided in the embodiments of this disclosure, the similarity between the object feature and each preset benchmark feature is first obtained. The similarity can reflect the degree of matching between the object feature and each preset benchmark feature. Then, based on the similarity, the second characterization information representing the degree of matching between the object feature and the preset benchmark feature can be obtained more conveniently and accurately.

[0262] In one embodiment of this application, the second characterization information obtaining submodule includes:

[0263] The target similarity determination unit is used to determine the top preset number of target similarities from the obtained similarities;

[0264] The second characterization information acquisition unit is used to obtain second characterization information that characterizes the degree of matching between the object's features and the preset benchmark features based on the maximum similarity and the target similarity.

[0265] It can be seen that, in addition to the maximum similarity between the object feature and the preset benchmark feature, a second representation information can also be obtained by combining the maximum number of similarities between the object feature and the preset benchmark feature. This makes the second representation information related not only to the maximum similarity but also to the maximum number of similarities. Since the maximum number of similarities can all reflect the degree of matching between the object feature and the preset benchmark feature, this allows the second representation information to more comprehensively represent the degree of matching between the object feature and the preset benchmark feature.

[0266] In one embodiment of this application, the second characterization information obtaining unit is specifically used to obtain second characterization information q, which characterizes the degree of matching between the object features and the preset benchmark features, according to the following expression. i :

[0267]

[0268] Among them, sim(top) i, ) represents the maximum similarity, sim(top) i,j ) represents the target similarity sorted by j, where the sorting is: the obtained similarity sorted from largest to smallest, and k represents the preset number.

[0269] The second representation information can be obtained quickly and conveniently based on the above expression. Furthermore, when obtaining the second representation information based on the above expression, the second representation information is: the proportion of the maximum similarity in the sum of the top K similarities. The second representation information obtained in this way is correlated with all of the top K similarities and can also reflect the leading margin of the maximum similarity relative to the subsequent K-1 similarities, making the obtained second representation information more comprehensive and able to reflect more information.

[0270] In one embodiment of this application, the characterization information acquisition module 602 is specifically used to: obtain first characterization information for each object feature, characterizing the degree of feature offset of the object feature relative to other object features; select a target reference feature from preset reference features according to the feature type of the object feature; and obtain second characterization information characterizing the degree of matching between the object feature and the target reference feature.

[0271] It is evident that when determining the benchmark features used for comparison with the object features, the target benchmark features are selected from the preset benchmark features according to the feature type of the object features. That is, the preset benchmark features participating in the comparison are further selected. In this way, when obtaining the second representation information based on the object features and the target benchmark features, the comparison granularity between the object features and the benchmark features is refined, the accuracy of the comparison is improved, and thus, based on the object features and the target benchmark features, it is beneficial to obtain more accurate and reasonable second representation information.

[0272] In one embodiment of this application, the quality description information acquisition module 603 is specifically used to determine, for each object feature, a first weight of the first representation information corresponding to the object feature and a second weight of the second representation information corresponding to the object feature based on a first quantity and a second quantity; and to perform a weighted calculation on the first representation information and the second representation information corresponding to the object feature based on the first weight and the second weight to obtain the quality description information of the object feature. The relative size relationship between the first weight and the second weight is consistent with the relative size relationship between the first quantity and the second quantity. The first quantity is the number of object features obtained, and the second quantity is the number of preset benchmark features used to obtain the second representation information of the object feature.

[0273] As can be seen, the first quantity is the number of object features obtained, that is, the number of object features used to obtain the first representation information of the object features. The larger the first quantity, the more comprehensive and accurate the first representation information obtained can be considered, that is, the higher the importance of the first representation information in measuring feature quality. Similarly, the second quantity is the number of preset benchmark features used to obtain the second representation information of the object features. The larger the second quantity, the more comprehensive and accurate the second representation information obtained, and the higher the importance of the second representation information in measuring feature quality.

[0274] Therefore, this makes the relative size relationship between the first weight and the second weight consistent with the relative size relationship between the first quantity and the second quantity. In other words, it makes the relative size of the first weight and the second weight consistent with the relative importance of the first and second characterization information in measuring feature quality. Thus, when the first and second characterization information are weighted according to the above weights to obtain quality characterization information, the accuracy of the obtained quality characterization information can be improved.

[0275] In one embodiment of this application, the data processing module 604 is specifically used to perform at least one of the following downstream data processing on the multimedia data based on the determined target features: performing feature fusion on the determined target features to obtain the fusion features of the target object targeted by the multimedia data; performing object recognition on the target object targeted by the multimedia data; performing object classification on the target object targeted by the multimedia data; and obtaining the associated data of the multimedia data.

[0276] As can be seen, after determining the target features using the scheme provided in this application, various downstream data processing can be performed on multimedia data, such as target recognition, object classification, and related data acquisition. Since the target features are selected from the object features of the target object based on quality description information, their feature quality is superior. Therefore, the processing effect of the above-mentioned various downstream data processing can be improved based on the target features.

[0277] In the technical solution of this application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0278] It should be noted that the two-dimensional face images in this embodiment are from a publicly available dataset.

[0279] This application also provides an electronic device, such as... Figure 7 As shown, it includes:

[0280] Memory 701 is used to store computer programs;

[0281] The processor 702 is used to execute the program stored in the memory 701 to implement the aforementioned multimedia data processing method.

[0282] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 702, the communication interface, and the memory 701 communicating with each other via the communication bus.

[0283] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0284] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0285] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0286] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0287] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the aforementioned multimedia data processing method.

[0288] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the multimedia data processing methods described above.

[0289] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0290] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0291] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0292] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A multimedia data processing method, characterized in that, include: Based on multimedia data, multiple object features of the target object for which the multimedia data is targeted are obtained. The multimedia data is video or audio, and the multiple object features are multiple image features extracted from the video or multiple audio features extracted from the audio. For each object feature, first representation information is obtained that characterizes the degree of feature offset of the object feature relative to other object features, and second representation information is obtained that characterizes the degree of matching between the object feature and a preset benchmark feature. Based on the first and second characterization information corresponding to each object feature, the quality description information of each object feature is obtained. Based on the obtained quality description information, target features are determined from the features of each object, and downstream data processing is performed on the multimedia data based on the determined target features.

2. The method according to claim 1, characterized in that, After obtaining multiple object features of the target object based on the multimedia data, the method further includes: obtaining the central feature of each object feature; obtaining the first characterization information representing the degree of feature offset of the object feature relative to each object feature includes: determining the first distance between the object feature and the central feature; and obtaining the first characterization information representing the degree of feature offset of the object feature relative to each object feature based on the first distance. or The step of obtaining the second characterization information representing the degree of matching between the object feature and the preset benchmark feature includes: obtaining the similarity between the object feature and each preset benchmark feature; determining the maximum similarity from the obtained similarity; and obtaining the second characterization information representing the degree of matching between the object feature and the preset benchmark feature based on the maximum similarity.

3. The method according to claim 2, characterized in that, The step of obtaining first characterization information based on the first distance, which characterizes the degree of feature offset of the object feature relative to each object feature, includes: obtaining first characterization information based on the determined first distance and second distance, which characterizes the degree of feature offset of the object feature relative to each object feature, wherein the second distance is: the distance between the other object features (excluding the object feature) and the central feature among the obtained object features; or The step of obtaining second characterization information based on the maximum similarity to represent the degree of matching between the object feature and the preset benchmark feature includes: determining the largest number of target similarities from the obtained similarities; and obtaining second characterization information based on the maximum similarity and the target similarities to represent the degree of matching between the object feature and the preset benchmark feature.

4. The method according to claim 3, characterized in that, The first representation information, based on the determined first and second distances, characterizing the degree of feature offset of the object's features relative to the features of other objects, includes: The first representation information, which characterizes the feature offset of the object's feature relative to the features of other objects, is obtained using the following expression. : ; in, This indicates the number of object features obtained. Represents the first distance, except Other This represents the second distance corresponding to the j-th object feature.

5. The method according to claim 3, characterized in that, The second representation information, obtained based on the maximum similarity and target similarity, characterizing the degree of matching between the object's features and preset benchmark features, includes: The second representation information, which characterizes the degree of matching between the object's features and the preset benchmark features, is obtained using the following expression. : ; in, This represents the maximum similarity. Let j represent the target similarity, where the similarity is sorted from largest to smallest, and k represents the preset number.

6. The method according to any one of claims 1-4, characterized in that, The step of obtaining the second representation information characterizing the degree of matching between the object feature and the preset benchmark feature includes: selecting a target benchmark feature from the preset benchmark features according to the feature type of the object feature; and obtaining the second representation information characterizing the degree of matching between the object feature and the target benchmark feature. or The step of obtaining quality description information for each object feature based on the first and second representation information corresponding to each object feature includes: for each object feature, determining a first weight of the first representation information corresponding to the object feature and a second weight of the second representation information corresponding to the object feature based on a first quantity and a second quantity; performing a weighted calculation on the first and second representation information corresponding to the object feature based on the first and second weights to obtain the quality description information of the object feature, wherein the relative size relationship between the first and second weights is consistent with the relative size relationship between the first and second quantities, the first quantity is: the number of object features obtained, and the second quantity is: the number of preset benchmark features used to obtain the second representation information of the object feature; or The downstream data processing of the multimedia data based on the determined target features includes: performing at least one of the following downstream data processing steps on the multimedia data based on the determined target features: performing feature fusion on the determined target features to obtain the fusion features of the target object targeted by the multimedia data; performing object recognition on the target object targeted by the multimedia data; performing object classification on the target object targeted by the multimedia data; and obtaining the associated data of the multimedia data.

7. A multimedia data processing device, characterized in that, include: An object feature acquisition module is used to obtain multiple object features of a target object for which the multimedia data is applied, based on the multimedia data, wherein the multimedia data is video or audio, and the multiple object features are multiple image features extracted from the video or multiple audio features extracted from the audio. The representation information acquisition module is used to obtain, for each object feature, first representation information representing the degree of feature offset of the object feature relative to other object features, and second representation information representing the degree of matching between the object feature and a preset benchmark feature. The quality description information acquisition module is used to obtain the quality description information of each object feature based on the first and second characterization information corresponding to each object feature. The data processing module is used to determine target features from the features of each object based on the obtained quality description information, and to perform downstream data processing on the multimedia data based on the determined target features.

8. The apparatus according to claim 7, characterized in that, Following the object feature acquisition module, the system further includes: a center feature acquisition module, used to acquire the center feature of each object feature; the representation information acquisition module includes: a first distance determination submodule, used to determine a first distance between the object feature and the center feature; and a first representation information acquisition submodule, used to acquire first representation information representing the degree of feature offset of the object feature relative to each object feature based on the first distance. or The first characterization information acquisition submodule includes: a first characterization information acquisition unit, used to obtain first characterization information representing the degree of feature offset of the object feature relative to each object feature based on a determined first distance and a second distance, wherein the second distance is: the distance between the obtained object features other than the object feature and the central feature; or The first representation information obtaining unit is specifically used to obtain first representation information representing the degree of feature offset of the object feature relative to each object feature according to the following expression. : ,in, This indicates the number of object features obtained. Represents the first distance, except Other This represents the second distance corresponding to the j-th object feature; or The representation information acquisition module includes: a similarity acquisition submodule, used to acquire the similarity between the object feature and each preset benchmark feature; a maximum similarity determination submodule, used to determine the maximum similarity from the acquired similarities; and a second representation information acquisition submodule, used to acquire second representation information representing the degree of matching between the object feature and the preset benchmark features based on the maximum similarity. or The second characterization information acquisition submodule includes: a target similarity determination unit, used to determine the largest number of target similarities from the obtained similarities; and a second characterization information acquisition unit, used to obtain second characterization information representing the degree of matching between the object's features and preset benchmark features based on the largest similarity and the target similarity. or The second representation information obtaining unit is specifically used to obtain second representation information representing the degree of matching between the object features and the preset benchmark features according to the following expression. : ,in, This represents the maximum similarity. Let j represent the target similarity, where the similarity is sorted from largest to smallest, and k represents the preset number. or The representation information acquisition module is specifically used to obtain, for each object feature, first representation information representing the degree of feature offset of the object feature relative to other object features; select a target reference feature from preset reference features according to the feature type of the object feature; and obtain second representation information representing the degree of matching between the object feature and the target reference feature. or The quality description information acquisition module is specifically used to, for each object feature, determine a first weight of the first representation information corresponding to the object feature and a second weight of the second representation information corresponding to the object feature based on a first quantity and a second quantity; and perform a weighted calculation on the first representation information and the second representation information corresponding to the object feature based on the first weight and the second weight to obtain the quality description information of the object feature. The relative size relationship between the first weight and the second weight is consistent with the relative size relationship between the first quantity and the second quantity. The first quantity is the number of object features obtained, and the second quantity is the number of preset benchmark features used to obtain the second representation information of the object feature. or The data processing module is specifically used to perform at least one of the following downstream data processing operations on the multimedia data based on the determined target features: performing feature fusion on the determined target features to obtain the fused features of the target object targeted by the multimedia data; performing object recognition on the target object targeted by the multimedia data; performing object classification on the target object targeted by the multimedia data; and obtaining the associated data of the multimedia data.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images