Multimedia resource labeling monitoring method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a multimedia resource labeling monitoring method and device, electronic equipment and storage medium. The multimedia resource labeling monitoring method comprises: acquiring a multimedia resource to be monitored, and object information corresponding to a labeling object performing a labeling task on the multimedia resource; determining a content feature of the multimedia resource, and determining an object feature corresponding to the labeling object according to the object information; fusing the content feature and the object feature to obtain a fused feature; and determining a monitoring result based on the fused feature, wherein the monitoring result is used to represent a prediction probability of the multimedia resource being labeled incorrectly. The present disclosure fuses the content feature of the multimedia resource and the object feature of the labeling object to obtain the fused feature, so that the fused feature links the multimedia resource and the labeling object, realizes the prediction of the labeling accuracy of the multimedia resource, and realizes the end-to-end reminding of the possible labeling error of the labeling object.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a multimedia resource labeling monitoring method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the progress of computer technology, the artificial intelligence (AI) technology develops rapidly and is widely applied in the related field of image processing. For example, a video image is classified by an image recognition model, and the classification result is provided to a user for reference, so that the user can conveniently select a category.

[0003] In the related art, in addition to the classification of the video image, a labeling personnel labels the video image, such as labeling what the content of the video image contains, whether the video image is abnormal content, etc., for example, whether there is a dog in the video image.

[0004] However, when the labeling personnel labels the video image, there is a problem of labeling error due to different understanding of the labeling rule content, labeling work duration of the labeling personnel, time period of the labeling work, etc. In addition, the labeling personnel cannot be reminded in time to avoid possible labeling errors, which often causes the problem of more labeling errors of the labeling personnel on certain or some categories of video images. SUMMARY

[0005] The present disclosure provides a multimedia resource labeling monitoring method and device, electronic equipment and storage medium to at least solve the problem that the labeling accuracy of the multimedia resource cannot be predicted in the related art.

[0006] According to an aspect of an embodiment of the present disclosure, a multimedia resource labeling monitoring method is provided, comprising:

[0007] obtaining a multimedia resource to be monitored, and object information corresponding to a labeling object performing a labeling task on the multimedia resource;

[0008] determining a content feature of the multimedia resource, and determining an object feature corresponding to the labeling object according to the object information;

[0009] fusing the content feature and the object feature to obtain a fusion feature;

[0010] determining a monitoring result based on the fusion feature, wherein the monitoring result is used to represent a prediction probability of the multimedia resource being labeled incorrectly.

[0011] In a possible implementation manner, the determination of the content feature of the multimedia resource comprises:

[0012] processing the multimedia resource by using a neural network model, wherein the neural network model is configured to obtain an image feature of a target subject in the multimedia resource, and obtain the content feature according to the image feature.

[0013] In a possible implementation, the neural network model comprises an image classification network configured to obtain the image feature, and a cross-modal feature interaction network configured to obtain the content feature according to the image feature.

[0014] In a possible implementation, the processing the multimedia resource by using the neural network model comprises:

[0015] preprocessing the multimedia resource to obtain an effective multimedia resource containing the target subject;

[0016] inputting the effective multimedia resource into the neural network model.

[0017] In a possible implementation, the determining the object feature corresponding to the annotation object according to the object information comprises:

[0018] processing the object information by using a neural network model, wherein the neural network model is configured to encode the object information, and determine the object feature based on an encoding vector obtained by encoding.

[0019] In a possible implementation, the encoding vector is a one-hot vector obtained by one-hot encoding the object information.

[0020] the neural network model comprises an embedding dictionary matrix configured to obtain an embedding vector of the object information according to the one-hot vector.

[0021] the object feature is determined according to the embedding vector.

[0022] In a possible implementation, the fusing the content feature and the object feature to obtain a fusion feature comprises:

[0023] fusing the content feature and the object feature by using a neural network model to obtain the fusion feature.

[0024] In a possible implementation, the neural network model is configured to fuse the content feature and the object feature by concatenating the content feature and the object feature.

[0025] In a possible implementation, the determining a monitoring result based on the fusion feature comprises:

[0026] The fusion features are processed using a neural network model, wherein the neural network model is used to obtain the predicted probability of the labeled object's labeling error for the multimedia resource, and to determine the predicted probability of the labeling error that is not lower than a preset probability threshold as the output monitoring result;

[0027] Obtain the monitoring results output by the neural network model.

[0028] In one possible implementation, the neural network model includes a label prediction compression classification network for obtaining a predicted probability of labeling errors of the labeled object for the multimedia resource.

[0029] In one possible implementation, the labeled prediction compressed classification network includes:

[0030] A labeled feature compression network is used to obtain labeled compressed features based on the fused features;

[0031] A labeling classification network is used to obtain the predicted probability of labeling errors based on the label compression features.

[0032] In one possible implementation, the method further includes:

[0033] Based on the content characteristics, the content classification result of the multimedia resources is determined.

[0034] In one possible implementation, the content classification result is obtained by processing the content features using a neural network model, wherein the neural network model includes a content classification network, which is used to output the content classification result based on the content features.

[0035] In one possible implementation, the content classification result is obtained by processing the content features using a neural network model;

[0036] The monitoring results are obtained by using the neural network model to determine the fused features.

[0037] In one possible implementation, the method further includes:

[0038] The sample multimedia resources and sample object information used to characterize the labeled objects are input into the neural network model;

[0039] obtaining a sample monitoring result and a sample content classification result output by the neural network model, wherein the sample monitoring result is determined according to a sample fusion feature, the sample fusion feature is obtained by fusing a sample content feature and a sample object feature, the sample content feature is obtained according to the sample multimedia resource, the sample object feature is determined according to the sample object information, and the sample content classification result is obtained according to the sample content feature;

[0040] determining a loss parameter of the neural network model based on the sample monitoring result and the sample content classification result;

[0041] adjusting a model parameter of the neural network model according to the loss parameter until the loss parameter is lower than a preset loss threshold.

[0042] In a possible implementation, the determining the loss parameter of the neural network model based on the sample monitoring result and the sample content classification result includes:

[0043] comparing the sample monitoring result with a preset reference sample monitoring result to obtain a prediction loss;

[0044] comparing the sample content classification result with a preset reference classification result to obtain a classification loss;

[0045] determining the loss parameter according to the prediction loss and the classification loss.

[0046] In a possible implementation, the determining the loss parameter according to the prediction loss and the classification loss includes:

[0047] determining a first weight component obtained by multiplying the prediction loss by a first weight coefficient;

[0048] determining a second weight component obtained by multiplying the classification loss by a second weight coefficient;

[0049] determining the loss parameter as a sum of the first weight component and the second weight component.

[0050] In a possible implementation, the loss parameter is a cross-entropy loss.

[0051] According to another aspect of the embodiments of the present disclosure, a multimedia resource annotation monitoring device is provided, including:

[0052] a data obtaining unit configured to perform obtaining a multimedia resource to be monitored, and obtaining object information corresponding to an annotation object performing an annotation task on the multimedia resource;

[0053] The feature determination unit is configured to determine a content feature of the multimedia resource, and determine an object feature corresponding to the labeling object according to the object information.

[0054] The fusion unit is configured to fuse the content feature and the object feature to obtain a fusion feature.

[0055] The monitoring result acquisition unit is configured to determine a monitoring result based on the fusion feature, where the monitoring result is used to represent a prediction probability of the multimedia resource being labeled incorrectly.

[0056] In a possible implementation, the feature determination unit is further configured to:

[0057] process the multimedia resource by using a neural network model, where the neural network model is used to obtain an image feature of a target subject in the multimedia resource, and obtain the content feature according to the image feature.

[0058] In a possible implementation, the neural network model includes an image classification network used to obtain the image feature, and a cross-modal feature interaction network used to obtain the content feature according to the image feature.

[0059] In a possible implementation, the feature determination unit includes:

[0060] The preprocessing unit is configured to pre-process the multimedia resource to obtain an effective multimedia resource containing the target subject.

[0061] The input unit is configured to input the effective multimedia resource into the neural network model.

[0062] In a possible implementation, the feature determination unit is further configured to:

[0063] process the object information by using a neural network model, where the neural network model is used to encode the object information, and determine the object feature based on an encoded vector obtained by encoding.

[0064] In a possible implementation, the encoded vector is a one-hot vector obtained by one-hot encoding the object information.

[0065] The neural network model includes an embedding dictionary matrix used to obtain an embedding vector of the object information according to the one-hot vector.

[0066] The object feature is determined according to the embedding vector.

[0067] In a possible implementation, the fusing unit is further configured to perform:

[0068] fusing the content feature and the object feature by using a neural network model to obtain the fused feature.

[0069] In a possible implementation, the neural network model is used to fuse the content feature and the object feature by concatenating the content feature and the object feature.

[0070] In a possible implementation, the monitoring result obtaining unit is further configured to perform:

[0071] processing the fused feature by using a neural network model, wherein the neural network model is used to obtain a predicted probability of annotation error of the annotation object on the multimedia resource, and a predicted probability of annotation error that is not lower than a preset probability threshold is determined as the monitoring result output;

[0072] obtaining the monitoring result output by the neural network model.

[0073] In a possible implementation, the neural network model includes an annotation prediction compression classification network, and the annotation prediction compression classification network is used to obtain a predicted probability of annotation error of the annotation object on the multimedia resource.

[0074] In a possible implementation, the annotation prediction compression classification network includes:

[0075] an annotation feature compression network, used to obtain an annotation compression feature according to the fused feature;

[0076] an annotation classification network, used to obtain the predicted probability of annotation error according to the annotation compression feature.

[0077] In a possible implementation, the apparatus further includes:

[0078] a classification result obtaining unit configured to perform determining a content classification result of the multimedia resource based on the content feature.

[0079] In a possible implementation, the content classification result is obtained by processing the content feature by using a neural network model, wherein the neural network model includes a content classification network, and the content classification network is used to output the content classification result according to the content feature.

[0080] In a possible implementation, the content classification result is obtained by processing the content feature by using a neural network model;

[0081] The monitoring result is determined by using the neural network model on the fusion feature.

[0082] In a possible implementation, the apparatus further includes:

[0083] a sample input module configured to perform inputting a sample multimedia resource and sample object information for representing a labeled object into the neural network model;

[0084] a sample result obtaining module configured to perform obtaining a sample monitoring result and a sample content classification result output by the neural network model, wherein the sample monitoring result is determined according to a sample fusion feature, the sample fusion feature is obtained by fusing a sample content feature and a sample object feature, the sample content feature is obtained according to the sample multimedia resource, the sample object feature is determined according to the sample object information, and the sample content classification result is obtained according to the sample content feature;

[0085] a loss parameter obtaining module configured to perform determining a loss parameter of the neural network model based on the sample monitoring result and the sample content classification result;

[0086] a model adjusting module configured to perform adjusting a model parameter of the neural network model according to the loss parameter until the loss parameter is lower than a preset loss threshold.

[0087] In a possible implementation, the loss parameter obtaining module is further configured to perform:

[0088] comparing the sample monitoring result with a preset reference sample monitoring result to obtain a prediction loss;

[0089] comparing the sample content classification result with a preset reference classification result to obtain a classification loss;

[0090] determining the loss parameter according to the prediction loss and the classification loss.

[0091] In a possible implementation, the loss parameter obtaining module is further configured to perform:

[0092] determining a first weight component obtained by multiplying the prediction loss by a first weight coefficient;

[0093] determining a second weight component obtained by multiplying the classification loss by a second weight coefficient;

[0094] determining a sum of the first weight component and the second weight component as the loss parameter.

[0095] In a possible implementation, the loss parameter is a cross-entropy loss.

[0096] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0097] a processor;

[0098] a memory for storing executable instructions of the processor;

[0099] The processor is configured to execute the executable instructions to implement the multimedia resource labeling monitoring method according to any of the above embodiments.

[0100] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, when at least one instruction in the computer readable storage medium is executed by a processor of an electronic device, the electronic device can implement the multimedia resource labeling monitoring method according to any of the above embodiments.

[0101] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the multimedia resource labeling monitoring method according to any of the above embodiments.

[0102] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0103] The multimedia resource labeling monitoring method, device, electronic device and storage medium provided by the embodiments of the present disclosure fuse the content features of the multimedia resource to be monitored and the object features of the labeling object to obtain the fusion features. Since the fusion features simultaneously contain the related information of the multimedia resource and the labeling object, the fusion features associate the multimedia resource with the labeling object, and the monitoring result obtained based on the fusion features is simultaneously associated with the multimedia resource and the labeling object. The content features of the multimedia resource are obtained in the classification process of the multimedia resource, so that the prediction of the labeling accuracy of the multimedia resource is realized, the end-to-end reminding of the possible labeling errors of the labeling object is realized, and the labeling quality of the labeling object is improved, and the labeling errors are found in time.

[0104] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0105] The accompanying drawings incorporated in the specification and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0106] Figure 1is a schematic diagram of an implementation environment of a multimedia resource labeling monitoring method according to an exemplary embodiment;

[0107] Figure 2 is a flowchart of a multimedia resource labeling monitoring method according to an exemplary embodiment;

[0108] Figure 3 is a flowchart of a multimedia resource labeling monitoring method according to an exemplary embodiment;

[0109] Figure 4 is a logic structure block diagram of a multimedia resource labeling monitoring device according to an exemplary embodiment;

[0110] Figure 5 is a logic structure block diagram of a multimedia resource labeling monitoring neural network device according to an exemplary embodiment;

[0111] Figure 6 is an application scenario structure diagram of a multimedia resource labeling monitoring neural network device according to an exemplary embodiment;

[0112] Figure 7 is a structural schematic diagram of a server provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0113] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0114] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0115] AI is a theory, method, technology and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0116] At present, AI technology has been widely used in video image recognition, classification and other aspects. The classification of video images has basically been completed by AI technology. However, for in-depth analysis of video images, AI technology alone cannot be relied on to complete the task, and human participation is needed to complete the task, especially for video image annotation analysis to mine the inherent meaning of video content, which requires subjective judgment and analysis of people.

[0117] However, due to the influence of the experience, knowledge, age, and physical condition of the annotator, the annotation of the annotator may be wrong, which requires a way to remind or notify the annotator that the current video image may be annotated incorrectly, thereby assisting the annotator to pay attention to the current video content and avoiding annotation errors.

[0118] Currently, there is no targeted end-to-end technical solution to monitor whether the annotator has annotated the video image incorrectly.

[0119] After analysis, it is found that there is a correlation between the video category and the annotation accuracy of the annotator for the video image. The recognition accuracy of video classification will affect the annotation accuracy of the annotator for the video. Based on this, the relationship between video classification and annotators can be established, and the AI process of video classification can be used as an auxiliary task to optimize the current annotation accuracy of the annotator for the current video image to obtain the probability of annotation error, and then timely remind the current annotator and the reviewer that the annotator and the reviewer may make annotation errors, so as to assist the annotator to improve the annotation quality and assist the reviewer to timely find the annotation error problem.

[0120] Figure 1 It is an implementation environment schematic diagram of a multimedia resource annotation monitoring method according to an example embodiment, see Figure 1 In this implementation environment, at least one terminal 101 and a server 102 can be included, which will be described in detail below.

[0121] The at least one terminal 101 is used for browsing video images, obtaining annotation reminders, annotating video images, etc. An application program can be installed on each terminal in the at least one terminal 101, which can be any client capable of providing video browsing, annotation, and reminder functions. A user can start the application program to browse video images, annotate video images, and obtain annotation reminders. The at least one terminal 101 can also be used for browsing multimedia resources. An application program can be installed on each terminal in the at least one terminal 101, which can be any client capable of providing multimedia resource browsing services. A user can start the application program to browse multimedia resources. The application program can be at least one of a short video application, an audio / video application, a shopping application, a takeout application, a travel application, a game application, or a social application. The multimedia resources can include at least one of video resources, audio resources, picture resources, text resources, or web page resources.

[0122] The at least one terminal 101 can be directly or indirectly connected to the server 102 through wired or wireless communication. The present disclosure does not limit this.

[0123] The server 102 is a computer device for classifying video images, establishing a connection between video classification and annotation personnel, optimizing the AI process of video image classification as an auxiliary task to obtain the probability of annotation errors of the current video image by the current annotation personnel, generating and sending annotation reminders to the terminal 101 according to the probability of annotation errors. The server 102 can also be used to provide multimedia resource services to the at least one terminal 101. The server 102 can include at least one of a server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, the server 102 can undertake primary computing work, and the at least one terminal 101 can undertake secondary computing work. Alternatively, the server 102 can undertake secondary computing work, and the at least one terminal 101 can undertake primary computing work. Alternatively, the server 102 and the at least one terminal 101 can adopt a distributed computing architecture for collaborative computing.

[0124] It should be noted that the device type of any of the at least one terminal 101 can include at least one of a smart phone, a tablet computer, an electronic book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. For example, the any terminal can be a smart phone or other handheld portable electronic device.

[0125] Figure 2 is a flowchart of a multimedia resource labeling monitoring method according to an illustrative embodiment. Referring to FIG. 2, Figure 2 The multimedia resource labeling monitoring method is applied to a computer device, and the following description takes a computer device as a server as an example.

[0126] In step 201, the server acquires a multimedia resource to be monitored and object information corresponding to a labeling object performing a labeling task on the multimedia resource.

[0127] In step 202, the server determines a content feature of the multimedia resource and determines an object feature corresponding to the labeling object according to the object information.

[0128] In step 203, the server fuses the content feature and the object feature to obtain a fused feature.

[0129] In step 204, the server determines a monitoring result based on the fused feature, wherein the monitoring result is used to represent a prediction probability of the multimedia resource being labeled incorrectly.

[0130] The multimedia resource labeling monitoring method provided by the embodiments of the present disclosure fuses the content feature of the multimedia resource to be monitored and the object feature of the labeling object to obtain a fused feature. Since the fused feature simultaneously contains information about the multimedia resource and the labeling object, the fused feature links the multimedia resource and the labeling object, and the monitoring result obtained based on the fused feature is simultaneously associated with the multimedia resource and the labeling object. The content feature of the multimedia resource is acquired in the classification process of the multimedia resource, thereby realizing prediction of the labeling accuracy of the multimedia resource, realizing end-to-end reminding of possible labeling errors of the labeling object, thereby assisting the labeling object to improve labeling quality and assisting the reviewer to timely find problems of labeling errors.

[0131] In some examples, the multimedia resource includes a video image, and the labeling object is a labeling person labeling the video image.

[0132] As AI technology has been widely used in information recognition and the like, the accuracy of recognition can be improved, and in this case, in some examples, the determining of the content feature of the multimedia resource in step 202 includes:

[0133] processing the multimedia resource by using a neural network model, wherein the neural network model is used to obtain an image feature of a target subject in the multimedia resource and obtain the content feature according to the image feature.

[0134] In the above manner, accurate content features can be obtained, and the accuracy of the obtained monitoring result is ensured.

[0135] In some examples, the neural network model includes an image classification network used to obtain the image feature and a cross-modal feature interaction network used to obtain the content feature according to the image feature.

[0136] In some examples, the image classification network includes at least one of an EfficientNet-B3 network and a resnet-50D network.

[0137] In some examples, the cross-modal feature interaction network is a transformer network.

[0138] In some examples, the multimedia resource contains content irrelevant to the target subject, for example, if the target subject is a person, there will be other content such as scenery, buildings, etc. in the video image of the multimedia resource in addition to the person, which is irrelevant to the target subject, so the part of the video image containing only the person needs to be extracted, and in this case, the processing of the multimedia resource by using the neural network model in step 202 includes:

[0139] preprocessing the multimedia resource to obtain an effective multimedia resource containing the target subject;

[0140] inputting the effective multimedia resource into the neural network model.

[0141] In the above manner, more accurate image features and content features can be obtained.

[0142] For multimedia resources of video images, the effective multimedia resource containing the target subject exists in the form of video frames. In this case, in some embodiments, the multimedia resource is a video image, and the effective multimedia resource is a plurality of local images containing the target subject. In this case, the preprocessing of the multimedia resource to obtain the effective multimedia resource containing the target subject includes:

[0143] sampling a plurality of global images from the video image;

[0144] The target subject is detected from the multiple global images to obtain multiple local images containing the target subject;

[0145] The cover frame of the video image is supplemented to the multiple local images when the number of frames of the multiple local images does not reach the threshold of the number of frames.

[0146] The cover frame is supplemented to the multiple local images to ensure that the multiple local images meet the requirements of the image classification network for input.

[0147] In some examples, the above-mentioned obtaining of the image feature includes:

[0148] The multiple local images are input into the image classification network to obtain the image feature of the video image.

[0149] In the above manner, the image classification network can obtain accurate image features, providing reliable image features for subsequent accurate monitoring results.

[0150] In some examples, the above-mentioned obtaining of the content feature according to the image feature includes:

[0151] The image feature is input into the transformer network, the transformer network learns the interaction relationship between the multiple local images, and the maximum value is obtained to obtain the maximum feature value in each dimension, and the maximum feature value in all dimensions is determined as the content feature.

[0152] The transformer network is used to learn the interaction relationship between the multiple local images, which can obtain accurate content features, providing reliable image features for subsequent accurate monitoring results.

[0153] In some examples, the step 202 includes determining the object feature corresponding to the labeled object according to the object information, including:

[0154] The object information is processed by using a neural network model, wherein the neural network model is used to encode the object information and determine the object feature based on the obtained encoding vector.

[0155] In the above manner, the neural network model can obtain the object feature accurately reflecting the labeled object, providing reliable image features for subsequent accurate monitoring results.

[0156] In some examples, the encoding vector is a one-hot vector obtained by one-hot encoding the object information.

[0157] The neural network model includes an embedding dictionary matrix, the embedding dictionary matrix is used to obtain an embedding vector related to the object information according to the one-hot vector, and the object feature is determined according to the embedding vector.

[0158] In some examples, the object feature is expressed by an embedding vector.

[0159] In the above manner, the embedding vector representing the object feature obtained by using the one-hot encoding and the embedding dictionary matrix can accurately reflect the object feature of the labeled object, thereby providing reliable image features for obtaining accurate monitoring results.

[0160] In some examples, the step 203 of fusing the content feature and the object feature to obtain a fused feature includes:

[0161] The fusion of the content feature and the object feature is performed by using a neural network model to obtain the fused feature.

[0162] In some examples, the neural network model is configured to fuse the content feature and the object feature by concatenating the content feature and the object feature.

[0163] In some examples, the content feature and the object feature have equal dimensions.

[0164] In the above manner, the dimensions of the content feature and the object feature are set to be equal, which ensures the complete concatenation of the content feature and the object feature, ensures the accuracy of the fused feature, and provides reliable image features for obtaining accurate monitoring results.

[0165] In some examples, the step 204 of determining the monitoring result based on the fused feature includes:

[0166] The fused feature is processed by using a neural network model, wherein the neural network model is configured to obtain a predicted probability of a labeling error of the labeled object with respect to the multimedia resource and determine a predicted probability of a labeling error that is not lower than a preset probability threshold as the output monitoring result.

[0167] The monitoring result output by the neural network model is obtained.

[0168] In the above manner, the neural network model is used to obtain a predicted probability of a labeling error associated with the fused feature. The obtained predicted probability of a labeling error is also associated with the multimedia resource and the labeled object, that is, the predicted probability of a labeling error reflects the probability of a possible labeling error of the labeled object with respect to the multimedia resource. In the case where a probability threshold is preset, the predicted probability of a labeling error that is not lower than the preset probability threshold is determined as the output monitoring result, which can timely remind the labeled object of the labeling work with respect to the multimedia resource, so as to improve the labeling concentration of the labeled object with respect to the multimedia resource, thereby reducing the possibility of a labeling error.

[0169] In some examples, the neural network model comprises a label prediction compression classification network, which is configured to obtain the predicted probability of the labeling error of the labeling object on the multimedia resource.

[0170] In some examples, the label prediction compression classification network belongs to a classification network, which can be composed of fully connected layers. In this case, in some examples, the label prediction compression classification network comprises:

[0171] a label feature compression network configured to obtain label compression features from the fused features;

[0172] a label classification network configured to obtain the predicted probability of the labeling error from the label compression features.

[0173] In some examples, the label feature compression network comprises at least one fully connected layer.

[0174] In some examples, the label classification network comprises at least one fully connected layer.

[0175] In some examples, the labeling object tends to have a higher labeling error rate on some content categories of multimedia resources. The multimedia resource labeling monitoring method of the present disclosure can be combined with the process of content classification of multimedia resources to obtain better monitoring results. In this case, the multimedia resource labeling monitoring method of the present disclosure further comprises:

[0176] determining a content classification result of the multimedia resource based on the content features.

[0177] In some examples, the content classification result is obtained by processing the content features by using a neural network model, wherein the neural network model comprises a content classification network, and the content classification network is configured to output the content classification result based on the content features.

[0178] In some examples, the content classification network belongs to a classification network, which can be composed of fully connected layers. In this case, the content classification network comprises:

[0179] a content compression sub-network configured to obtain content compression features from the content features;

[0180] a content classification sub-network configured to obtain the content classification result based on the content compression features.

[0181] In some examples, the content compression sub-network comprises at least one fully connected layer.

[0182] In some examples, the content classification sub-network comprises at least one fully connected layer.

[0183] The predicted probability of the multimedia resource being incorrectly labeled reflects on the individual of the labeling object, for example, the labeling personnel has understanding problems of the multimedia resource of a certain content classification due to personal reasons, resulting in a high labeling error rate. In this case, the object information needs to correspond to the individual of the labeling object, and in some examples, the object information includes the information identifier of the labeling object, such as an identity document (ID).

[0184] In some cases, the labeling object is also affected by age, labeling time, and continuous working time, for example, the understanding of multimedia resources by labeling personnel of different age groups will have an era bias, labeling time directly affects the labeling experience of the labeling personnel, and continuous working time relates to the fatigue of the labeling personnel and affects the labeling accuracy, and therefore, these aspects can be considered as labeling information. In this case, in some examples, the object information further includes at least one of the age, labeling time, and continuous working time of the labeling object.

[0185] In the multimedia resource labeling monitoring method of the present disclosure, a neural network model is used to obtain a monitoring result. In some examples, the content classification result is obtained by processing the content features using the neural network model, and the monitoring result is determined by processing the fusion features using the neural network model. It can be seen that the content classification result and the monitoring result are respectively output by the neural network model. In this case, the neural network model can be trained using sample multimedia resources and sample object information, and the parameters of the neural network model are adjusted using the sample monitoring result and the sample content classification result output by the neural network model during the training process, and finally an accurate inference neural network model is obtained.

[0186] In this case, in some examples, the multimedia resource labeling monitoring method of the present disclosure further includes:

[0187] inputting the sample multimedia resource and the sample object information representing the labeling object into the neural network model;

[0188] obtaining the sample monitoring result and the sample content classification result output by the neural network model, wherein the sample monitoring result is determined according to the sample fusion features, the sample fusion features are obtained by fusing the sample content features and the sample object features, the sample content features are obtained according to the sample multimedia resource, the sample object features are determined according to the sample object information, and the sample content classification result is obtained according to the sample content features;

[0189] determining the loss parameter of the neural network model based on the sample monitoring result and the sample content classification result;

[0190] The model parameters of the neural network model are adjusted according to the loss parameter until the loss parameter is lower than a preset loss threshold.

[0191] In the embodiments of the present disclosure, the content classification result and the monitoring result are outputs of the neural network model, and the monitoring result is also affected by the content classification result. Therefore, in the training process, the loss parameter contains both the sample monitoring result and the sample content classification result.

[0192] In this case, in some examples, the determination of the loss parameter of the neural network model based on the sample monitoring result and the sample content classification result includes:

[0193] comparing the sample monitoring result with a preset reference sample monitoring result to obtain a prediction loss;

[0194] comparing the sample content classification result with a preset reference classification result to obtain a classification loss;

[0195] determining the loss parameter according to the prediction loss and the classification loss.

[0196] In the present disclosure, the neural network model is mainly used to determine the monitoring result, so the proportion of the monitoring result in the loss parameter is higher than that of the content classification result during training. In this case, in some examples, the determination of the loss parameter according to the prediction loss and the classification loss includes:

[0197] determining a first weight component obtained by multiplying the prediction loss by a first weight coefficient;

[0198] determining a second weight component obtained by multiplying the classification loss by a second weight coefficient;

[0199] determining the loss parameter as the sum of the first weight component and the second weight component.

[0200] In some examples, the first weight component is 1 and the second weight component is 0.4. In addition, the sizes of the first weight component and the second weight component can be adjusted as needed, and since the neural network model is mainly used to determine the monitoring result, it is necessary to ensure that the first weight component is greater than the second weight component.

[0201] In some examples, the loss parameter is a cross-entropy loss.

[0202] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0203] Figure 3 is a flowchart of a multimedia resource labeling and monitoring method according to an example embodiment, as shown in Figure 3As shown, the multimedia resource labeling monitoring method is applied to a computer device, and the computer device is taken as an example to illustrate the case that the computer device is a server, the multimedia resource is a video image, and the labeling object is a labeling personnel. The embodiment includes the following steps.

[0204] In step 301, the server samples multiple frames of global images from the video image.

[0205] In some embodiments, the server samples N frames of global images from the video image.

[0206] In step 302, the server performs target subject detection on the multiple frames of global images to obtain multiple frames of local images containing target subjects.

[0207] In some embodiments, the server performs target subject detection on the N frames of global images to obtain M frames of local images containing target subjects.

[0208] In some embodiments, the target subject is a person in the video. Based on this, the labeling content of the labeling personnel on the image video can be a label about whether the person in the image video has a violation behavior, for example, whether the person in the image video has a smoking behavior.

[0209] In some embodiments, the M frames of local images are positioned to a specific target subject (such as a person in the video image) and the background other than the target subject is removed.

[0210] In some embodiments, when the number of frames of the multiple frames of local images does not reach a frame threshold, a cover frame of the video image is supplemented to the multiple frames of local images.

[0211] In some embodiments, the total number of frames M of the local images is a set value, and if the number of frames of the obtained local images containing target subjects is less than M frames, a cover frame is supplemented to fill the local images to M frames.

[0212] In step 303, the server inputs the multiple frames of local images into an image classification network to obtain image features of the video image.

[0213] In some embodiments, the image classification network is an EfficientNet-B3 network or a resnet-50D network.

[0214] EfficientNet is a standardized convolutional network expansion scheme that can achieve high accuracy and can fully save computing resources. The EfficientNet-B3 network is composed of 16 mobile flip bottleneck convolution modules, 2 convolution layers, 1 global average pooling layer, and 1 classification layer. The process of extracting image features of the video image by using the EfficientNet-B3 network can be implemented by using the existing technology, and thus will not be described here.

[0215] The resnet-50D network is a residual network. The process of extracting image features of a video image by using the resnet-50D network can be implemented by using the prior art, which will not be described herein.

[0216] In some embodiments, if the feature dimension is D dimensions, the dimension of the image features obtained after the M frame local images pass through the image classification network is M*D dimensions.

[0217] In step 304, the server inputs the image features into the cross-modal feature interaction network to obtain the content features of the video image.

[0218] In some embodiments, the cross-modal feature interaction network is a transformer network.

[0219] The transformer is a pose-based action recognition model, which can accurately capture pose information and recognize subtle changes in human poses.

[0220] In some embodiments, the server inputs the image features into the transformer network, learns the interaction relationship between the M frame local images by the transformer network, and obtains the maximum feature value of each dimension by taking the maximum value. The maximum feature value of all dimensions is determined as the content features.

[0221] In some embodiments, the feature interaction is specifically implemented as follows: the M*D dimensional feature vector (the image features are represented by the feature vector) is input into the transformer network, the interaction relationship between the M frame local images is learned by the transformer network, and finally the reduce_max is used to obtain the maximum feature value corresponding to each dimension (a total of D dimensions), thereby obtaining the content features X.

[0222] Wherein, reduce_max is an operation performed on the data output by the transhformer network. For the transhformer network, the dimension of the input image features is M*D dimensions, and the output is still M*D dimensions. After reduce_max, it becomes D dimensions. The operation of reduce_max is to take the max value (i.e. the maximum feature value of all dimensions) of each row of data in the M row of the M*D dimensional matrix to obtain the D dimensional content features X.

[0223] The process of extracting content features of a video image by using the transformer network can be implemented by using the prior art, which will not be described herein.

[0224] In step 305, the server inputs the content features into the content classification network to obtain the video content classification result of the video image.

[0225] In some embodiments, the content classification network comprises a content compression sub-network and a content classification sub-network. The content compression sub-network is configured to compress the content features to obtain content compressed features, and the content classification sub-network is configured to classify the content compressed features to obtain the video content classification result of the video image.

[0226] In some embodiments, the video content classification comprises at least one of the following categories: life, game, talent, technology, education, etc.

[0227] In some embodiments, the content compression sub-network comprises at least one fully connected layer.

[0228] In some embodiments, the content classification sub-network comprises at least one fully connected layer.

[0229] In some embodiments, the steps 301 to 305 are completed by the server to perform the classification.

[0230] In step 306, the server one-hot encodes the annotator information to obtain a one-hot vector of the annotator information.

[0231] In some embodiments, the annotator is the annotator of the video image classified in the steps 301 to 305, that is, the annotator is associated with the classified video image.

[0232] In some embodiments, the annotator information at least comprises an annotator ID. On this basis, in some embodiments, the annotator information can further comprise at least one of the following: age, annotation time, and continuous working time.

[0233] In some embodiments, the annotator ID is one-hot encoded. Assuming that there are T annotators in total, the one-hot encoding can be used to map the IDs of the T annotators into (0, 1, …, T-1) using a dictionary.

[0234] One-hot encoding, also known as one-bit effective encoding or one-hot encoding, is mainly to use a multi-bit state register to encode multiple states, each state has its own independent register bit, and only one bit is effective at any time.

[0235] In some embodiments, in addition to the annotator ID, other information of the annotator such as age, annotation time, and continuous working time are one-hot encoded. For example, for age, age is divided into age ranges, and the age ranges are one-hot encoded; for annotation time, the time is divided into time ranges, such as 0-1 time range, 1-2 time range, etc., and the time ranges are one-hot encoded; for continuous working time, the working time is divided into working time ranges, such as 0-3 hours working time range, 3-8 hours working time range, 8-12 hours working time range, and 12 hours or more working time range, and the working time ranges are one-hot encoded.

[0236] In step 307, the server obtains an embedding vector of the annotator information according to the one-hot vector and the embedding dictionary matrix.

[0237] The embedding vector is the object feature of the annotator.

[0238] In some embodiments, the server obtains an embedding vector of the one-hot encoded corresponding position of the annotator information in the embedding dictionary matrix (T, D). Here, the embedding dictionary matrix is different from the one-hot encoding dictionary mentioned in step 306.

[0239] In some embodiments, in the case where the annotator information only includes the annotator ID, the embedding vector of the one-hot encoded corresponding position of the annotator ID in the embedding dictionary matrix is obtained.

[0240] In some embodiments, in the case where the annotator information includes the annotator ID and other information of the annotator, the annotator ID and the other information of the annotator are one-hot encoded together, and the embedding vector of the one-hot encoded corresponding position of the annotator ID and the other information of the annotator together in the embedding dictionary matrix is obtained.

[0241] In some embodiments, the dimension of the embedding dictionary matrix (T, D) is D, and the dimension of the embedding dictionary matrix is equal to the dimension of the content feature X, so as to facilitate the interaction between the features of the annotator (embedding vector of the annotator) and the content feature in the subsequent process.

[0242] In some embodiments, the embedding dictionary matrix is randomly initialized initially, and is automatically updated according to the model loss function during training.

[0243] In step 308, the server concatenates the content features and the embedding vectors to obtain fused features.

[0244] The dimension of the embedding dictionary matrix is equal to the dimension of the content features X, both of which are D dimensions. The fused features obtained after concatenation have a dimension of 2D, and the fused features have information features about both the video images and the annotators.

[0245] The content features are generated by step 304, and the embedding vectors are generated by step 308.

[0246] In step 309, the server inputs the fused features into the annotation feature compression network to obtain annotation compressed features.

[0247] In some embodiments, the annotation feature compression network includes at least one fully connected layer. The annotation feature compression network is used to compress the third features.

[0248] In step 310, the server inputs the annotation compressed features into the annotation classification network to obtain a predicted probability of annotation errors.

[0249] In some embodiments, the annotation classification network includes at least one fully connected layer. In some embodiments, the annotation classification network is one fully connected layer.

[0250] In some embodiments, the annotation classification network obtains at least one probability, i.e., the predicted probability of annotation errors.

[0251] In step 311, the server determines whether the predicted probability of annotation errors reaches a preset probability threshold, and determines the predicted probability of annotation errors that is not lower than the preset probability threshold as the output monitoring result.

[0252] In some embodiments, the probability threshold is determined according to the actual application of the business. For example, if high accuracy is required, the probability threshold can be set to a higher value, for example, the probability threshold is set to 0.9. When the predicted probability of annotation errors output by the annotation classification network reaches or exceeds the probability threshold, the monitoring result is issued to inform the annotator that the video image currently annotated by the annotator is easy to be annotated incorrectly, and the annotator should carefully check.

[0253] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described again.

[0254] The multimedia resource labeling monitoring method provided by the embodiments of the present disclosure fuses the content features of the multimedia resource to be monitored and the object features of the labeling object to obtain fusion features. Since the fusion features simultaneously contain the related information of the multimedia resource and the labeling object, the fusion features link the multimedia resource and the labeling object, and the monitoring result obtained based on the fusion features is simultaneously associated with the multimedia resource and the labeling object. The content features of the multimedia resource are obtained in the classification process of the multimedia resource, thereby realizing the prediction of the labeling accuracy of the multimedia resource, realizing the end-to-end reminding of the possible labeling errors of the labeling object, thereby assisting the labeling object to improve the labeling quality and assisting the reviewer to timely find the labeling error problems.

[0255] Figure 4 is a logic structure block diagram of a multimedia resource labeling monitoring device according to an illustrative embodiment, as shown in Figure 4 The device comprises a data acquisition unit 401, a feature determination unit 402, a fusion unit 403 and a monitoring result acquisition unit 404.

[0256] The data acquisition unit 401 is configured to perform the acquisition of the multimedia resource to be monitored, and the acquisition of the object information corresponding to the labeling object performing the labeling task on the multimedia resource.

[0257] The feature determination unit 402 is configured to perform the determination of the content features of the multimedia resource, and the determination of the object features corresponding to the labeling object according to the object information.

[0258] The fusion unit 403 is configured to perform the fusion of the content features and the object features to obtain fusion features.

[0259] The monitoring result acquisition unit 404 is configured to perform the determination of the monitoring result based on the fusion features, wherein the monitoring result is used to represent the prediction probability of the labeling error of the multimedia resource.

[0260] The multimedia resource labeling monitoring device provided by the embodiments of the present disclosure fuses the content features of the multimedia resource to be monitored and the object features of the labeling object to obtain fusion features. Since the fusion features simultaneously contain the related information of the multimedia resource and the labeling object, the fusion features link the multimedia resource and the labeling object, and the monitoring result obtained based on the fusion features is simultaneously associated with the multimedia resource and the labeling object. The content features of the multimedia resource are obtained in the classification process of the multimedia resource, thereby realizing the prediction of the labeling accuracy of the multimedia resource, realizing the end-to-end reminding of the possible labeling errors of the labeling object, thereby assisting the labeling object to improve the labeling quality and assisting the reviewer to timely find the labeling error problems.

[0261] In some examples, the monitoring result is obtained based on Figure 4The feature determination unit 402 is further configured to perform the following features in the device composition:

[0262] The multimedia resource is processed by using the neural network model, wherein the neural network model is used to obtain an image feature of a target subject in the multimedia resource, and obtain a content feature according to the image feature.

[0263] In some examples, the neural network model comprises an image classification network used to obtain the image feature, and a cross-modal feature interaction network used to obtain the content feature according to the image feature.

[0264] In some examples, the image classification network comprises at least one of an EfficientNet-B3 network and a resnet-50D network.

[0265] In some examples, the cross-modal feature interaction network is a transformer network.

[0266] In some examples, the feature determination unit 402 comprises:

[0267] A preprocessing unit configured to perform preprocessing on the multimedia resource to obtain an effective multimedia resource containing the target subject;

[0268] An input unit configured to perform inputting the effective multimedia resource into the neural network model.

[0269] In some examples, the multimedia resource is a video image, and the effective multimedia resource is a plurality of frames of local images containing the target subject.

[0270] The preprocessing unit is further configured to perform the following:

[0271] Sampling a plurality of frames of global images from the video image;

[0272] Performing target subject detection on the plurality of frames of global images to obtain the plurality of frames of local images containing the target subject;

[0273] Wherein, when the number of frames of the plurality of frames of local images does not reach a frame threshold, a cover frame of the video image is supplemented to the plurality of frames of local images.

[0274] In some examples, the image feature is obtained, comprising:

[0275] Inputting the plurality of frames of local images into the image classification network to obtain the image feature of the video image.

[0276] In some examples, the content feature is obtained according to the image feature, comprising:

[0277] The image features are input into a transformer network, the transformer network learns the interaction relationship between multiple frames of local images, and the maximum value is obtained, to obtain the maximum eigenvalue of each dimension, and the maximum eigenvalue of all dimensions is determined as the content feature.

[0278] In some examples, the feature determination unit 402 is further configured to perform:

[0279] The object information is processed by using a neural network model, where the neural network model is used to encode the object information and determine an object feature based on an encoding vector obtained through the encoding.

[0280] In some examples, the encoding vector is a one-hot vector obtained by one-hot encoding the object information.

[0281] The neural network model includes an embedding dictionary matrix, which is used to obtain an embedding vector related to the object information according to the one-hot vector.

[0282] The object feature is determined according to the embedding vector.

[0283] In some examples, the fusion unit 403 is further configured to perform:

[0284] The fusion of the content feature and the object feature is performed by using a neural network model to obtain a fusion feature.

[0285] In some examples, the neural network model is used to fuse the content feature and the object feature by concatenating the content feature and the object feature.

[0286] In some examples, the number of dimensions of the content feature and the object feature is equal.

[0287] In some examples, the monitoring result acquisition unit 404 is further configured to perform:

[0288] The fusion feature is processed by using a neural network model, where the neural network model is used to obtain a prediction probability of a labeling error of a labeling object on the multimedia resource and determine a prediction probability of a labeling error that is not lower than a preset probability threshold as the output monitoring result.

[0289] The monitoring result output by the neural network model is obtained.

[0290] In some examples, the neural network model includes a labeling prediction compression classification network, which is used to obtain a prediction probability of a labeling error of a labeling object on the multimedia resource.

[0291] In some examples, the labeling prediction compression classification network includes:

[0292] The labeling feature compression network is configured to obtain labeled compressed features according to the fused features.

[0293] The labeling classification network is configured to obtain a prediction probability of labeling errors according to the labeled compressed features.

[0294] In some examples, the labeling feature compression network includes at least one fully connected layer.

[0295] In some examples, the labeling classification network includes at least one fully connected layer.

[0296] In some examples, the multimedia resource labeling monitoring apparatus further includes:

[0297] The classification result acquisition unit is configured to determine a content classification result of the multimedia resource based on the content features.

[0298] In some examples, the content classification result is obtained by processing the content features by using a neural network model, where the neural network model includes a content classification network configured to output the content classification result according to the content features.

[0299] In some examples, the content classification network includes:

[0300] The content compression sub-network is configured to obtain content compressed features according to the content features.

[0301] The content classification sub-network is configured to obtain the content classification result according to the content compressed features.

[0302] In some examples, the content compression sub-network includes at least one fully connected layer.

[0303] In some examples, the content classification sub-network includes at least one fully connected layer.

[0304] In some examples, the object information includes an information identifier of the labeling object.

[0305] In some examples, the object information further includes at least one of an age, a labeling time, and a continuous working time of the labeling object.

[0306] In some examples, the content classification result is obtained by processing the content features by using a neural network model.

[0307] The monitoring result is obtained by processing the fused features by using a neural network model.

[0308] In some examples, the multimedia resource labeling monitoring apparatus further includes:

[0309] The sample input module is configured to input a sample multimedia resource and sample object information for representing a labeled object into a neural network model.

[0310] The sample result obtaining module is configured to obtain a sample monitoring result and a sample content classification result output by the neural network model, wherein the sample monitoring result is determined according to sample fusion features, the sample fusion features are obtained by fusing sample content features and sample object features, the sample content features are obtained according to the sample multimedia resource, the sample object features are determined according to the sample object information, and the sample content classification result is obtained according to the sample content features.

[0311] The loss parameter obtaining module is configured to determine a loss parameter of the neural network model based on the sample monitoring result and the sample content classification result.

[0312] The model adjusting module is configured to adjust model parameters of the neural network model according to the loss parameter until the loss parameter is lower than a preset loss threshold.

[0313] In some examples, the loss parameter obtaining module is further configured to:

[0314] compare the sample monitoring result with a preset reference sample monitoring result to obtain a prediction loss;

[0315] compare the sample content classification result with a preset reference classification result to obtain a classification loss;

[0316] determine the loss parameter according to the prediction loss and the classification loss.

[0317] In some examples, the loss parameter obtaining module is further configured to:

[0318] determine a first weight component obtained by multiplying the prediction loss by a first weight coefficient;

[0319] determine a second weight component obtained by multiplying the classification loss by a second weight coefficient;

[0320] determine the loss parameter as a sum of the first weight component and the second weight component.

[0321] In some examples, the loss parameter is a cross-entropy loss.

[0322] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described again.

[0323] It should be noted that the above embodiments are only used as an example for the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0324] It should be noted that the above embodiments are only used as an example for the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0325] Figure 5 is a logical structure block diagram of a multimedia resource annotation monitoring neural network device according to an illustrative embodiment, as shown in Figure 5 The device includes a content feature acquisition module 501, an object feature acquisition module 502, a fusion unit 503, and a result acquisition module 504.

[0326] The content feature acquisition module 501 is configured to perform acquisition of content features of a multimedia resource to be monitored.

[0327] The object feature acquisition module 502 is configured to perform acquisition of object information corresponding to an annotation object performing an annotation task on the multimedia resource, and determine object features corresponding to the annotation object according to the object information.

[0328] The fusion unit 503 is configured to perform fusion of the content features and the object features to obtain fusion features.

[0329] The result acquisition module 504 is configured to perform determination of a monitoring result based on the fusion features, wherein the monitoring result is used to represent a prediction probability of the multimedia resource being annotated incorrectly.

[0330] The multimedia resource annotation monitoring neural network device provided by the embodiments of the present disclosure fuses the content features of the multimedia resource to be monitored and the object features of the annotation object to obtain fusion features. Since the fusion features simultaneously contain information about the multimedia resource and the annotation object, the fusion features associate the multimedia resource with the annotation object, and the monitoring result obtained based on the fusion features is simultaneously associated with the multimedia resource and the annotation object. The content features of the multimedia resource are obtained in the classification process of the multimedia resource, thereby realizing prediction of the annotation accuracy of the multimedia resource, realizing end-to-end reminding of possible annotation errors of the annotation object, thereby assisting the annotation object to improve annotation quality and assisting the reviewer to timely find problems of annotation errors.

[0331] In some examples, based on Figure 5 The content feature acquisition module 501 includes:

[0332] a resource preprocessing unit configured to perform preprocessing on the multimedia resource to obtain an effective multimedia resource containing a target subject;

[0333] an image classification network configured to perform obtaining an image feature of the target subject in the multimedia resource;

[0334] a cross-modal feature interaction network configured to perform obtaining a content feature according to the image feature.

[0335] In some examples, based on Figure 5 The device consists of, the resource preprocessing unit includes:

[0336] a resource sampling unit configured to perform sampling a plurality of global images from the multimedia resource;

[0337] a video detection unit configured to perform target subject detection on the plurality of global images to obtain a plurality of local images containing the target subject;

[0338] wherein when the number of frames of the plurality of local images does not reach a frame threshold, a cover frame of the video image is completed to the plurality of local images.

[0339] In some examples, the image classification network includes at least one of an EfficientNet-B3 network, a resnet-50D network.

[0340] In some examples, the cross-modal feature interaction network is a transformer network, and the transformer network is configured to learn an interaction relationship between the plurality of local images and obtain a maximum feature value in each dimension, and determine the maximum feature value in all dimensions as the content feature.

[0341] In some examples, based on Figure 5 The device consists of, the multimedia resource labeling monitoring neural network device further includes:

[0342] a content classification network configured to output a content classification result according to the content feature.

[0343] In some examples, the content classification network includes:

[0344] a content compression subnetwork for obtaining a content compression feature according to the content feature;

[0345] a content classification subnetwork for obtaining a content classification result according to the content compression feature.

[0346] In some examples, the content compression subnetwork includes at least one fully connected layer.

[0347] In some examples, the content classification sub-network includes at least one fully connected layer.

[0348] In some examples, the object feature obtaining module is configured to obtain the object feature based on Figure 5

[0349] a one-hot encoding unit configured to perform one-hot encoding on the object information to obtain a one-hot vector;

[0350] an embedding vector unit configured to obtain an embedding vector about the object information according to the one-hot vector and an embedding dictionary matrix;

[0351] an object feature determining unit configured to determine the object feature according to the embedding vector.

[0352] In some examples, the fusion unit is configured to perform:

[0353] concatenate the content feature and the object feature to obtain a fusion feature.

[0354] In some examples, the result obtaining module is configured to obtain the prediction probability about the labeling error of the multimedia resource by the labeling object based on Figure 5 a labeling prediction compression classification network configured to obtain the prediction probability about the labeling error of the multimedia resource by the labeling object according to the fusion feature;

[0355] a labeling reminding unit configured to set a probability threshold, and issue a labeling reminder if the prediction probability about the labeling error is not lower than the probability threshold.

[0356] In some examples, the labeling prediction compression classification network is configured to obtain the prediction probability about the labeling error of the multimedia resource by the labeling object based on

[0357] Figure 5 a labeling feature compression network configured to obtain a labeling compression feature according to the fusion feature;

[0358] a labeling classification network configured to obtain the prediction probability about the labeling error according to the labeling compression feature.

[0359] In some examples, the labeling feature compression network includes at least one fully connected layer.

[0360] In some examples, the labeling classification network includes at least one fully connected layer.

[0361] In some examples, the object information includes an information identifier of the labeling object.

[0362] In some examples, the object information further includes at least one of an age, a labeling time, and a continuous working time of the labeling object.

[0363] In some examples, the object information further includes at least one of an age, a labeling time, and a continuous working time of the labeling object.

[0364] ​​In some examples, any combination can be employed to form optional embodiments of the present disclosure, which will not be described one by one here.

[0365] As to the multimedia resource labeling monitoring neural network device in the above embodiments, the specific manner in which each unit performs operations has been described in detail in the embodiments of the multimedia resource labeling monitoring method, and will not be described in detail here.

[0366] It should be noted that: the above embodiments are only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0367] Figure 6 is a schematic diagram of an application scenario using multimedia resource labeling monitoring according to an illustrative embodiment, as shown in Figure 6 The embodiment is applied to a computer device, and the computer device is taken as a server for illustration. The embodiment includes the following steps.

[0368] The scenario mainly includes two task branches. The upper task branch task1 is an end-to-end video classification task, and the lower task branch task2 is to encode part of the information of the labeling personnel such as the labeling personnel ID, labeling time, and continuous working time into an embedding vector and splice the content features obtained in the upper task branch task1, and then determine whether the labeling personnel has labeled the video incorrectly.

[0369] In step 601, the server samples N frames of global images from a video image.

[0370] In step 602, the server performs target subject detection on the N frames of global images to obtain M frames of local images containing target subjects.

[0371] In some embodiments, the target subject is a person, for example Figure 6 two people in which one is smoking.

[0372] In some embodiments, the M frames of local images are positioned to a specific target subject (such as a person in a video image) and the background other than the target subject is removed.

[0373] In some embodiments, if the number of frames of the obtained local images containing target subjects is less than M frames, cover frames are supplemented to supplement the local images to M frames.

[0374] In step 603, the server inputs the M frames of local images into an image classification network to obtain image features about the video image.

[0375] In some embodiments, the image classification network is an EfficientNet-B3 network or a resnet-50D network.

[0376] In some embodiments, if the feature dimension is D dimensions, the dimension of the image features obtained after the M frame local images pass through the image classification network is M*D dimensions in total.

[0377] In step 604, the server inputs the image features into the cross-modal feature interaction network to obtain the content features of the video image.

[0378] In some embodiments, the cross-modal feature interaction network is a transformer network.

[0379] In some embodiments, the server inputs the image features into the transformer network, learns the interaction relationship between the multiple frame local images by the transformer network, and obtains the maximum feature value of each dimension by taking the maximum value, to determine the maximum feature value of all dimensions as the content features.

[0380] In some embodiments, the feature interaction is specifically implemented as follows: the M*D dimensional feature vector (the image features are represented by a feature vector) is input into the transformer network, the interaction relationship between the M frame local images is learned by the transformer network, and finally the reduce_max is used to obtain the maximum feature value corresponding to each dimension (a total of D dimensions), thereby obtaining the content features X.

[0381] wherein reduce_max is an operation performed on the data output by the transhformer network. For the transhformer network, the dimension of the input image features is M*D dimensions, and the output is still M*D dimensions, which becomes D dimensions after reduce_max. The operation of reduce_max is to take the max value (i.e., the maximum feature value of all dimensions) of each row of data in the M row of the M*D dimensional matrix to obtain the D dimensional content features X.

[0382] In step 605, the server inputs the content features into the content feature compression and classification network to obtain the video classification information of the video image.

[0383] In some embodiments, the content feature compression and classification network includes a content feature compression network and a video classification network. The content feature compression network is used to compress the content features to obtain content compression features, and the video classification network is used to classify the content compression features to obtain the video classification information of the video image.

[0384] In some embodiments, the video classification information includes at least one of the categories of life, game, talent, technology, education, etc.

[0385] In some embodiments, the content feature compression network comprises at least one fully connected layer.

[0386] In some embodiments, the video classification network comprises at least one fully connected layer.

[0387] The steps 601 to 605 above are steps of the upper task branch task1.

[0388] In step 606, the server one-hot encodes the annotator information to obtain a one-hot vector about the annotator information.

[0389] In some embodiments, the annotator is the annotator of the classified video image in the annotation steps 301 to 305, that is, the annotator is associated with the classified video image.

[0390] In some embodiments, the annotator information at least comprises an annotator ID. On this basis, in some embodiments, the annotator information can further comprise at least one of age, annotation time, and continuous working time.

[0391] In some embodiments, the annotator ID is one-hot encoded. Assuming that there are T annotators in total, the IDs of the T annotators can be mapped into (0, 1, …, T-1) by using a dictionary for one-hot encoding.

[0392] In some embodiments, other information of the annotator in addition to the annotator ID, such as age, annotation time, and continuous working time, is one-hot encoded. For example, as for age, age stages are divided and one-hot encoding is performed by using the age stages; as for annotation time, time stages are divided, such as 0 to 1 time stage, 1 to 2 time stage, etc., and one-hot encoding is performed by using the time stages; as for continuous working time, working time stages are divided, such as 0 to 3 hours working time stage, 3 to 8 hours working time stage, 8 to 12 hours working time stage, and 12 hours or more working time stage, and one-hot encoding is performed by using the working time stages.

[0393] In step 607, the server obtains an embedding vector about the annotator information according to the one-hot vector and an embedding dictionary matrix.

[0394] In some embodiments, the server utilizes an embedding dictionary matrix (T, D) to obtain an embedding vector at a position corresponding to the one-hot encoding of the annotator information. Here, the embedding dictionary matrix is not the same as the one-hot encoding dictionary mentioned in step 606.

[0395] In some embodiments, when the annotator information only contains the annotator ID, the embedding vector at a position corresponding to the one-hot encoding of the annotator ID is obtained in the embedding dictionary matrix.

[0396] In some embodiments, when the annotator information contains the annotator ID and other information of the annotator, the annotator ID and the other information of the annotator are one-hot encoded together, and the embedding vector at a position corresponding to the one-hot encoding obtained by the annotator ID and the other information of the annotator is obtained in the embedding dictionary matrix.

[0397] In some embodiments, the dimension of the embedding dictionary matrix (T, D) is D, and the dimension of the embedding dictionary matrix is equal to the dimension of the content feature X, so as to facilitate the interaction between the features of the annotator (embedding vector of the annotator) and the content feature.

[0398] In some embodiments, the embedding dictionary matrix is randomly initialized initially, and is automatically updated according to the model loss function during training.

[0399] In step 608, the server concatenates the content feature and the embedding vector to obtain a third feature.

[0400] The dimension of the embedding dictionary matrix is equal to the dimension of the content feature X, both of which are D dimensions, and the dimension of the third feature obtained after concatenation is 2D, and the third feature simultaneously has information features about the video image and the annotator.

[0401] The content feature is generated by step 604 of the upper task branch task1, and the embedding vector is generated by step 608.

[0402] In step 609, the server inputs the third feature into the annotation feature compression and classification network to obtain a prediction probability of annotation error.

[0403] In some embodiments, the annotation feature compression and classification network includes an annotation feature compression network and an annotation classification network.

[0404] In some embodiments, the annotation feature compression network comprises at least one fully connected layer. The annotation feature compression network is configured to compress the third features.

[0405] In some embodiments, the annotation classification network comprises at least one fully connected layer. In some embodiments, the annotation classification network is one fully connected layer.

[0406] In some embodiments, the annotation classification network obtains at least one probability, i.e., a prediction probability of annotation error.

[0407] In step 610, the server determines whether the prediction probability of annotation error reaches a probability threshold, and issues an annotation reminder if the prediction probability of annotation error reaches the probability threshold.

[0408] In some embodiments, the probability threshold is determined according to the actual application of the business. For example, if high accuracy is required, the probability threshold can be set to a higher value, for example, the probability threshold is set to 0.9. When the prediction probability of annotation error output by the annotation classification network reaches or exceeds the probability threshold, an annotation reminder is issued to inform the annotator that the video image currently annotated by the annotator is easy to be annotated incorrectly, and please check carefully.

[0409] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described here.

[0410] In some embodiments, the multimedia resource annotation monitoring neural network device needs to be trained and tested before reminding the annotator.

[0411] In some embodiments, the training process mainly includes two parts of sample collection and model training.

[0412] In some embodiments, sample collection includes three aspects:

[0413] Aspect X1, collect video images: collect video images annotated by the annotator.

[0414] Aspect X2, collect annotator information: obtain relevant personal information of the annotator, such as annotator ID, annotation time, annotation duration, etc.

[0415] Aspect Y, obtain video category label (tag) and whether the video is annotated incorrectly. As training videos, the video category label and whether the video is annotated incorrectly are standard correct data. The video category is the correct category of the video image, i.e., if the annotation of the annotator is incorrect, the video category is the correct category after verification, not the category annotated incorrectly by the annotator.

[0416] In some embodiments, the model training comprises:

[0417] The pair consisting of the video image and the annotator information is input into the above multimedia resource annotation monitoring neural network device to obtain a sample monitoring result, and then the sample monitoring result is calculated with a sample label (including a video category label and a video annotation error label) to obtain a loss. In some embodiments, the loss function adopts a cross-entropy loss function. The training is continuously performed until the loss no longer decreases or the number of network iterations is reached. The main task is the "video annotation error task" (task 2), and the auxiliary task is the "video classification task" (task 1). Therefore, in some embodiments, a is set to be greater than b, that is,

[0418] Loss = a * "video annotation error task" + b * "video classification task"

[0419] In one example, a = 1 and b = 0.4. The loss function is a loss function, and in some embodiments, the loss function adopts a cross-entropy loss function. The values of a and b represent the weights of the "video annotation error task" and the "video classification task" in the loss function. Since the disclosure focuses on determining whether the annotation is correct, a > b, which ensures that the "video annotation error task" has a greater weight in the loss function.

[0420] In some embodiments, the test process mainly comprises:

[0421] 1) Determine the sample video image to be predicted, the annotator assigned to the video image, and the annotator information of the annotator.

[0422] 2) Input the sample to be predicted and the annotator information into the multimedia resource annotation monitoring neural network device to obtain a video category prediction probability P1 and a video annotation error prediction probability P2, respectively. When the prediction probability P2 meets the threshold requirement R, it is determined to be "yes", and the annotator is reminded that the video is easy to be annotated incorrectly and should be carefully checked.

[0423] All the optional technical solutions described above can be combined to form optional embodiments of the disclosure, and will not be described one by one here.

[0424] Figure 7Fig. 7 is a structural schematic diagram of a server provided by an embodiment of the present disclosure. The server 700 can have great differences due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 701 and one or more memories 702. The memory 702 stores at least one program code, which is loaded and executed by the processor 701 to implement the multimedia resource labeling monitoring method provided by the above-mentioned embodiments. Of course, the server 700 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for realizing the functions of the device, and the details are not described herein.

[0425] In an example embodiment, a computer readable storage medium including at least one instruction, such as a memory including at least one instruction, is also provided. The at least one instruction is executable by a processor in a computer device to complete the multimedia resource labeling monitoring method in the above-mentioned embodiments.

[0426] Optionally, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium can include a ROM (Read-Only Memory), a RAM (Random-Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0427] In an example embodiment, a computer program product including one or more instructions is also provided. The one or more instructions are executable by a processor of a computer device to complete the multimedia resource labeling monitoring method provided by the above-mentioned embodiments.

[0428] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the aspects of the present disclosure disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including combinations or permutations of individual features of the present disclosure. The true scope of the present disclosure is indicated by the appended claims, along with the full scope of equivalents to which such claims are entitled. The specification and examples given herein are to be considered exemplary of the application, and are presented for the purpose of illustration and description only.

[0429] It should be understood that the present disclosure is not limited to the precise structures as set forth above and as shown in the attached drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for monitoring multimedia resource labeling, characterized in that, The method comprises: acquiring a multimedia resource to be monitored, and acquiring object information corresponding to a labeling object performing a labeling task on the multimedia resource, wherein the object information comprises at least one of a labeling time of the labeling object and a continuous working time; determining a content feature of the multimedia resource, and determining an object feature corresponding to the labeling object according to the object information; fusing the content feature and the object feature to obtain a fused feature; determining a monitoring result based on the fused feature, wherein the monitoring result is used to represent a prediction probability of the multimedia resource being labeled incorrectly.

2. The multimedia resource tagging monitoring method of claim 1, wherein, The method further comprises: processing the multimedia resource by using a neural network model, wherein the neural network model is used to acquire an image feature of a target subject in the multimedia resource, and acquire the content feature according to the image feature.

3. The method according to claim 2, wherein: the neural network model comprises an image classification network used to acquire the image feature, and a cross-modal feature interaction network used to acquire the content feature according to the image feature.

4. The multimedia resource tagging monitoring method of claim 2, wherein, The method further comprises: preprocessing the multimedia resource to obtain an effective multimedia resource containing the target subject; inputting the effective multimedia resource into the neural network model.

5. The multimedia resource tagging monitoring method of claim 1, wherein, The method further comprises: processing the object information by using a neural network model, wherein the neural network model is used to encode the object information, and determine the object feature based on an encoded vector obtained by encoding.

6. The method according to claim 5, wherein: the encoded vector is a one-hot vector obtained by one-hot encoding the object information; the neural network model comprises an embedding dictionary matrix used to obtain an embedding vector about the object information according to the one-hot vector; the object feature is determined according to the embedding vector.

7. The multimedia resource tagging monitoring method of claim 1, wherein, The method further comprises: fusing the content feature and the object feature by using a neural network model to obtain the fused feature.

8. The method according to claim 7, wherein: the neural network model is used to fuse the content feature and the object feature by splicing the content feature and the object feature.

9. The multimedia resource tagging monitoring method of claim 1, wherein, The method further comprises: processing the fused feature by using a neural network model, wherein the neural network model is used to obtain a prediction probability of the labeling object labeling the multimedia resource incorrectly, and determine a prediction probability of labeling incorrectly that is not lower than a preset probability threshold as the monitoring result output; acquiring the monitoring result output by the neural network model.

10. The method according to claim 9, wherein: The neural network model comprises a label prediction compression classification network, which is configured to obtain a prediction probability of a labeling error of the label object on the multimedia resource.

11. The multimedia resource tagging monitoring method of claim 10, wherein, The label prediction compression classification network comprises: a label feature compression network, configured to obtain label compression features according to the fusion features; a label classification network, configured to obtain the prediction probability of the labeling error according to the label compression features.

12. The multimedia resource tagging and monitoring method of claim 1, wherein, The method further comprises: determining a content classification result of the multimedia resource based on the content features.

13. The multimedia resource labeling monitoring method according to claim 12, wherein: the content classification result is obtained by processing the content features by using a neural network model, wherein the neural network model comprises a content classification network, and the content classification network is configured to output the content classification result according to the content features.

14. The multimedia resource labeling monitoring method according to claim 12, wherein: the content classification result is obtained by processing the content features by using a neural network model; and the monitoring result is determined by using the neural network model on the fusion features.

15. The multimedia resource tagging monitoring method of claim 14, wherein, The method further comprises: inputting a sample multimedia resource and sample object information for representing a label object into the neural network model; obtaining a sample monitoring result and a sample content classification result output by the neural network model, wherein the sample monitoring result is determined according to sample fusion features, the sample fusion features are obtained by fusing sample content features and sample object features, the sample content features are obtained according to the sample multimedia resource, the sample object features are determined according to the sample object information, and the sample content classification result is obtained according to the sample content features; determining a loss parameter of the neural network model based on the sample monitoring result and the sample content classification result; adjusting model parameters of the neural network model according to the loss parameter until the loss parameter is lower than a preset loss threshold.

16. The multimedia resource marking monitoring method of claim 15, wherein, The determination of the loss parameter of the neural network model based on the sample monitoring result and the sample content classification result comprises: comparing the sample monitoring result with a preset reference sample monitoring result to obtain a prediction loss; comparing the sample content classification result with a preset reference classification result to obtain a classification loss; determining the loss parameter according to the prediction loss and the classification loss.

17. The multimedia resource tagging monitoring method of claim 16, wherein, The determination of the loss parameter according to the prediction loss and the classification loss comprises: determining a first weight component obtained by multiplying the prediction loss by a first weight coefficient; determining a second weight component obtained by multiplying the classification loss by a second weight coefficient; determining the loss parameter as a sum of the first weight component and the second weight component.

18. The multimedia resource tagging monitoring method of claim 17, wherein, The loss parameter is a cross-entropy loss.

19. A multimedia resource marking monitoring apparatus, characterized by comprising: comprises: The data acquisition unit is configured to perform acquisition of a multimedia resource to be monitored, and acquisition of object information corresponding to an annotation object performing an annotation task on the multimedia resource, wherein the object information comprises at least one of an annotation time of the annotation object and a continuous working time; The feature determination unit is configured to perform determination of a content feature of the multimedia resource, and determination of an object feature corresponding to the annotation object according to the object information; The fusion unit is configured to perform fusion of the content feature and the object feature to obtain a fusion feature; The monitoring result acquisition unit is configured to perform determination of a monitoring result based on the fusion feature, wherein the monitoring result is used to represent a prediction probability of annotation error of the multimedia resource.

20. The apparatus for monitoring of multimedia resource marking according to claim 19, wherein, The feature determination unit is further configured to perform: processing of the multimedia resource by using a neural network model, wherein the neural network model is used to acquire an image feature of a target subject in the multimedia resource, and acquire the content feature according to the image feature.

21. The multimedia resource annotation monitoring apparatus according to claim 20, wherein: the neural network model comprises an image classification network used to acquire the image feature, and a cross-modal feature interaction network used to acquire the content feature according to the image feature.

22. The apparatus for monitoring of multimedia resource marking according to claim 20, wherein, The feature determination unit comprises: a preprocessing unit configured to perform preprocessing of the multimedia resource to obtain an effective multimedia resource containing the target subject; an input unit configured to perform input of the effective multimedia resource into the neural network model.

23. The apparatus for monitoring of multimedia resource marking according to claim 19, wherein, The feature determination unit is further configured to perform: processing of the object information by using a neural network model, wherein the neural network model is used to encode the object information, and determine the object feature based on an encoded vector obtained by encoding.

24. The multimedia resource annotation monitoring apparatus according to claim 23, wherein: the encoded vector is a one-hot vector obtained by one-hot encoding of the object information; the neural network model comprises an embedding dictionary matrix used to obtain an embedding vector about the object information according to the one-hot vector; the object feature is determined according to the embedding vector.

25. The apparatus for monitoring of multimedia resource marking according to claim 19, wherein, The fusion unit is further configured to perform: fusion of the content feature and the object feature by using a neural network model to obtain the fusion feature.

26. The multimedia resource annotation monitoring apparatus according to claim 25, wherein: the neural network model is used to realize fusion of the content feature and the object feature by splicing the content feature and the object feature.

27. The apparatus for monitoring of multimedia resource marking according to claim 19, wherein, The monitoring result acquisition unit is further configured to perform: processing of the fusion feature by using a neural network model, wherein the neural network model is used to obtain a prediction probability of annotation error of the multimedia resource by the annotation object, and determine a prediction probability of annotation error not lower than a preset probability threshold as the monitoring result output; acquisition of the monitoring result output by the neural network model.

28. The multimedia resource labeling monitoring apparatus of claim 27, wherein the neural network model comprises a labeling prediction compression classification network configured to obtain a predicted probability of a labeling error of the labeling object on the multimedia resource. The neural network model comprises a labeling prediction compression classification network configured to obtain a predicted probability of a labeling error of the labeling object on the multimedia resource.

29. The multimedia resource marking monitoring apparatus of claim 28, wherein, The neural network model comprises a labeling prediction compression classification network configured to obtain a predicted probability of a labeling error of the labeling object on the multimedia resource. The neural network model comprises a labeling prediction compression classification network configured to obtain a predicted probability of a labeling error of the labeling object on the multimedia resource. The apparatus further comprises:

30. The apparatus for marking and monitoring of multimedia resources of claim 19, wherein, a classification result obtaining unit configured to determine a content classification result of the multimedia resource based on the content feature.

31. The multimedia resource labeling monitoring apparatus of claim 30, wherein the content classification result is obtained by processing the content feature using a neural network model, and wherein the neural network model comprises a content classification network configured to output the content classification result based on the content feature.

32. The multimedia resource labeling monitoring apparatus of claim 30, wherein the content classification result is obtained by processing the content feature using a neural network model, and wherein the monitoring result is obtained by processing the fusion feature using the neural network model. The apparatus further comprises: a sample input module configured to input a sample multimedia resource and sample object information representing a labeling object into the neural network model; a sample result obtaining module configured to obtain a sample monitoring result and a sample content classification result output by the neural network model, wherein the sample monitoring result is obtained based on a sample fusion feature obtained by fusing a sample content feature obtained from the sample multimedia resource and a sample object feature obtained from the sample object information, and wherein the sample content classification result is obtained based on the sample content feature; a loss parameter obtaining module configured to determine a loss parameter of the neural network model based on the sample monitoring result and the sample content classification result; 33. The apparatus for marking and monitoring of multimedia resources according to claim 32, wherein, a model adjusting module configured to adjust a model parameter of the neural network model based on the loss parameter until the loss parameter is lower than a preset loss threshold. The loss parameter obtaining module is further configured to: compare the sample monitoring result with a preset reference sample monitoring result to obtain a prediction loss; compare the sample content classification result with a preset reference classification result to obtain a classification loss; and determine the loss parameter based on the prediction loss and the classification loss.

34. The apparatus for marking and monitoring of multimedia resources according to claim 33, wherein, The loss parameter obtaining module is further configured to: determine a first weight component obtained by multiplying the prediction loss with a first weight coefficient; determine a second weight component obtained by multiplying the classification loss with a second weight coefficient; and determine the loss parameter as a sum of the first weight component and the second weight component.

35. The multimedia resource marking monitoring apparatus of claim 34, wherein, The loss parameter is a cross-entropy loss. ​ ​ ​ 36. The multimedia resource marking monitoring apparatus of claim 35, wherein, ​ 37. An electronic device, comprising: Comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the multimedia resource labeling monitoring method according to any one of claims 1-18.

38. A computer-readable storage medium, characterized in that, The computer program is executed by the processor to implement the multimedia resource labeling monitoring method according to any one of claims 1-18.

39. A computer program product comprising a computer program, characterised in that, The computer program is executed by the processor to implement the multimedia resource labeling monitoring method according to any one of claims 1-18.

Images