Advertisement data audit method, device, terminal equipment and storage medium

Abstract

The embodiment of the present application is applicable to the technical field of computer vision, and provides a method, apparatus, terminal device and storage medium for auditing advertising data, the method comprising: receiving advertising data to be audited; identifying the business category to which the advertising data belongs based on audio data; dividing the video data into multiple video clips whose contents belong to independent scenes; marking multiple sampling points in the video clips, and extracting image data at each sampling point; inputting the image data into a backbone structure to extract common image features; inputting the common image features into a neck structure to convert them into target image features; inputting the target image features into multiple head structures to generate individual violation values ​​for multiple violation categories; generating violation levels for the video clips according to the audit plan based on the multiple individual violation values; and displaying the violation levels of the video clips on the timeline of the video data. The accuracy of content auditing is maintained while reducing the frequency of updating the content auditing network.

Description

Technical Field

[0001] The embodiments of the present application belong to the technical field of computer vision, and in particular, relate to a method, apparatus, terminal device, and storage medium for reviewing advertising data. Background Art

[0002] Placing advertising data on Internet websites and applications is one of the main ways for companies to promote their business. In order to ensure that the content of the advertising data complies with regulations, the content of the advertising data needs to be reviewed before it is placed.

[0003] Some advertising data contains video data. For video data, the current method is to extract frames at certain intervals and use deep learning models to detect whether each frame contains any illegal content.

[0004] The diversity of corporate businesses and advertising creativity has led to a large amount of content in advertising data, which exceeds the learning scope of deep learning models and reduces the accuracy of content review.

[0005] If we continue to increase the structure of the deep learning model based on the content of advertising data and collect relevant samples to train the deep learning model, the cost of maintaining the deep learning model will increase dramatically, as will the resources consumed by the deep learning model's operation. Summary of the invention

[0006] In view of this, embodiments of the present application provide a method, apparatus, terminal device, and storage medium for reviewing advertisement data, so as to improve the accuracy of reviewing video data in advertisement data while maintaining costs and resources.

[0007] A first aspect of an embodiment of the present application provides a method for reviewing advertisement data, comprising:

[0008] When loading the content review network, receiving the advertisement data to be reviewed; the content review network includes a backbone structure, a neck structure and a plurality of head structures, and the advertisement data includes video data and audio data;

[0009] Identify the business category to which the advertisement data belongs based on the audio data; the business category is configured with an audit scheme;

[0010] Under the condition of filtering out the end card and the banner data in the video data, dividing the video data into a plurality of video segments, each of which has contents belonging to an independent scene;

[0011] Marking a plurality of sampling points in the video clip, and extracting image data at each of the sampling points;

[0012] Inputting the image data into the backbone structure to extract common image features;

[0013] Inputting the common image features into the neck structure and converting them into target image features;

[0014] Inputting the target image feature into a plurality of the head structures to generate individual violation values ​​on a plurality of violation categories;

[0015] generating a violation level for the video clip according to the review scheme based on the plurality of individual violation values;

[0016] The violation level of the video segment is displayed on a time axis of the video data.

[0017] A second aspect of an embodiment of the present application provides an advertisement data review device, including:

[0018] A pre-processing module, used for receiving advertisement data to be reviewed when loading a content review network; the content review network includes a backbone structure, a neck structure and a plurality of head structures, and the advertisement data includes video data and audio data;

[0019] A business category identification module, used to identify the business category to which the advertisement data belongs based on the audio data; the business category is configured with an audit scheme;

[0020] A video segment segmentation module, used for segmenting the video data into a plurality of video segments each of which belongs to an independent scene, under the condition of filtering out the end card and banner data in the video data;

[0021] An image data extraction module, used to mark a plurality of sampling points in the video clip and extract image data at each of the sampling points;

[0022] A backbone structure calling module, used for inputting the image data into the backbone structure to extract common image features;

[0023] A neck structure calling module, used for inputting the common image features into the neck structure and converting them into target image features;

[0024] A header structure calling module, used for inputting the target image features into a plurality of the header structures to generate individual violation values ​​for a plurality of violation categories;

[0025] A violation level generating module, configured to generate a violation level for the video clip according to the plurality of individual violation values ​​and the audit scheme;

[0026] The audit result display module is used to display the violation level of the video clip on the time axis of the video data.

[0027] A third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for reviewing advertising data as described in the first aspect above is implemented.

[0028] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for reviewing advertising data as described in the first aspect above is implemented.

[0029] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the advertising data review method described in the first aspect.

[0030] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0031] In the embodiment of the present application, the interference of the end card and the banner data is filtered out from the video data in the advertisement data, so that the content of the video data is more concise and effective, and the video data in the advertisement data is divided into video clips of multiple independent scenes, so as to improve the content concentration in each video clip, which is beneficial to the subsequent assessment of the violation level of the video clip, and the backbone structure is used on the content review network to extract the common image features from the image data in the video data and use the neck structure to convert it into the target image features, so as to provide high-quality and reusable features for each head structure, reduce the operation of extracting features, and improve the efficiency of detection, and use the audio data in the advertisement data to identify the business category. The audio data is synchronized with the video data, which can reflect the content of the video data to a certain extent, and the overall operation of the audio data is relatively simple, so as to ensure the accuracy of the business category. The tendency of illegal content in different business categories is different, and the results of the content review network are corrected according to the review plan of the business category, so as to maintain the accuracy of the content review while reducing the frequency of updating the content review network. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 is a schematic diagram of an advertising data review method provided in an embodiment of the present application;

[0034] Figure 2is a schematic diagram of a content review network provided in an embodiment of the present application;

[0035] Figure 3 It is a schematic diagram showing the audit results provided by an embodiment of the present application;

[0036] Figure 4 is a schematic diagram of an advertising data review device provided in an embodiment of the present application;

[0037] Figure 5 It is a schematic diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0039] The technical solution of the present application is described below through specific embodiments.

[0040] Reference Figure 1 , shows a schematic diagram of a method for reviewing advertising data provided by an embodiment of the present application, which may specifically include the following steps:

[0041] Step 101: When loading the content review network, receive the advertisement data to be reviewed.

[0042] In an embodiment of the present application, a content review network can be constructed and trained in advance for the task of reviewing video data in advertising data. For given advertising data, the content review network can identify individual violation values ​​of the video in the advertising data based on multiple preset violation categories.

[0043] In order to adapt to the diverse content of advertising data, the violation categories are generally more generalized categories, such as inappropriate items (such as cigarettes), intimate behaviors (such as kissing), violence, exposure of body parts, etc.

[0044] In the specific implementation, Figure 2 As shown, the content review network includes a backbone structure Backbone, a neck structure Neck and multiple head structures Head.

[0045] Among them, the backbone structure Backbone is responsible for extracting features from image data, and the neck structure Neck is responsible for multiple fusions of the features of image data to better utilize the features extracted by the backbone structure Backbone. The head structure Head has a one-to-one correspondence with the violation category, that is, one head structure Head is responsible for the detection of one violation category.

[0046] In many cases, advertisers independently or hire third parties to produce advertising data based on business needs, and entrust advertising alliances to deliver the advertising data in their channels. The same content in the advertising data may appear in multiple violation categories. For example, in game advertising data, exposure of body parts and violence may appear at the same time, inappropriate items and violence may appear at the same time, and in gift advertising data, intimate behavior and exposure of body parts may appear at the same time, and so on.

[0047] Multiple head structures Head independently evaluate violation categories in different violation categories, which can more comprehensively evaluate the degree of violation of advertising data.

[0048] Generally speaking, the content review network assists appraisers in reviewing the compliance of advertising data before it is released. When the appraiser logs in to the client and starts reviewing the compliance of advertising data, the content review network can be loaded into the memory for operation, and multiple instances can be generated in a multi-threaded manner for use by different appraisers. At this time, the client receives advertising data waiting for compliance review.

[0049] The advertisement data includes video data and audio data. The advertisement data is unpacked to separate the advertisement data into independent video data and independent audio data.

[0050] Step 102: Identify the business category to which the advertisement data belongs based on the audio data.

[0051] Advertising data is usually used to promote the advertiser's business. Different businesses differ in products, user groups, etc., which results in different tendencies in the types of violations that appear in the advertising data of different businesses. For example, the gaming business tends to have other violation categories such as exposed body parts, violence, and inappropriate items; the gift business tends to have other violation categories such as intimate behavior and exposed body parts, and so on.

[0052] Therefore, the embodiments of the present application can summarize the characteristics of the businesses of many advertisers in advance and divide them into multiple business categories, such as games, shopping, gifts, short videos, live broadcasts, mobile phones, cars, etc., and configure audit plans for the tendency of violation categories that appear in these business categories. That is, the business categories are configured with audit plans, and the audit plans are a way of auditing the video data of the advertising data in the corresponding business categories.

[0053] Furthermore, the audit plan can be updated regularly or irregularly offline through optimization algorithms such as Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) to improve the adaptability of the audit plan to the business category.

[0054] Generally speaking, there are a large number of image data in video data, and there are many elements in each frame of video data, so it is difficult to completely identify the semantics of video data.

[0055] Since the content of the video data is synchronized with the content of the audio data, the audio data can reflect the content of the video data to a certain extent. In addition, the data structure of the audio data is simpler than that of the video data, and the semantics of the audio data can be quickly and accurately identified.

[0056] Then, in the embodiment of the present application, the semantics of the audio data can be identified, and the business category to which the advertisement data belongs can be identified based on the semantics of the audio data.

[0057] In a specific implementation, the audio data can be input into an end-to-end speech recognition network, such as DFCNN (Deep Fully Convolutional Neural Network), to perform speech recognition (Automatic Speech Recognition, ASR) on the audio data to obtain text information.

[0058] The information of the advertisement data is queried in the local database, and the advertiser who has placed the advertisement data is identified in the information (indicated by information such as ID); wherein, a plurality of business tags are marked for the advertiser, and the business tags indicate the business that the advertiser has carried out, such as shopping platforms, fresh food supermarkets, payment tools, corporate attendance tools, etc.

[0059] Processing text information and labels belongs to natural language processing (NLP). The embodiment of the present application can preset a language model (LM), especially a large language model (LLM), for example, an autoencoder-based model, a recursive neural network model, and the like.

[0060] Since text information and labels are universal information, third-party pre-trained language models can be used directly.

[0061] For the language model, a prompt template may be pre-set, in which spaces for text information and labels are reserved.

[0062] For example, the template for the prompt is as follows:

[0063] You are an advertiser. Your existing business includes [business label]. Your task now is to evaluate which [business category] the advertisement with the content [text information] intends to promote.

[0064] A plurality of business tags and text information are written into the blanks in the template, thereby being converted into a prompt word Prompt adapted to a preset language model, and the prompt word Prompt is input into the language model to identify the business category to which the advertisement data belongs.

[0065] Step 103 : Under the condition of filtering out the end card and banner data in the video data, the video data is divided into a plurality of video segments each of which has contents belonging to an independent scene.

[0066] In actual applications, the video data in the advertising data usually sets a banner data Banner and an Ending Card (EC) due to the need to promote the business. The banner data is generally rectangular information, located at the top and bottom, or, left and right sides of the video data, and is used to record information such as the manufacturer, product, etc., such as the manufacturer's name, manufacturer's Logo (icon), gift code, etc. The Ending Card is generally located a few seconds at the end of the video data, and is used to record information such as the manufacturer, product name, application name and Logo, etc.

[0067] Since banner data and end cards are mostly business content, they interfere with the review of the content of video data to a certain extent. Therefore, banner data and end cards can be filtered out from the video data.

[0068] Generally, there is no banner data in the end card, so the end card and the banner data can be filtered out from the video data in turn.

[0069] Specifically, based on the characteristics of the end card (such as a still picture, a transition to a solid color picture, etc.), the frame difference method, color statistics method, etc. are used at the end of the video data to detect and delete the end card. If the end card has been deleted, the banner data can be detected and deleted in each frame of the image data based on the characteristics of the banner data (such as shape, position, etc.).

[0070] If the banner data has been deleted, multiple video clips each belonging to an independent scene can be detected and segmented from the video data based on the characteristics of the scene (such as the background of the picture changing slowly, transitioning to a solid color picture, etc.).

[0071] Step 104: Mark multiple sampling points in the video clip, and extract image data at each sampling point.

[0072] In the embodiment of the present application, uniform sampling, random sampling, etc. can be used to mark multiple sampling points in each video clip, and image data is extracted at each sampling point.

[0073] Furthermore, on the time axis of the video data, each sampling point has a time point.

[0074] Step 105: Input the image data into the backbone structure to extract common image features.

[0075] like Figure 2 As shown, each frame image data is input into the backbone structure Backbone, and the backbone structure Backbone extracts common image features from each frame image data.

[0076] In one embodiment of the present application, Figure 2 As shown, the backbone structure Backbone includes a first convolutional layer Conv_1, a second convolutional layer Conv_2, a third convolutional layer Conv_3, a fourth convolutional layer Conv_4, a first bottleneck layer Bottleneck_1, a second bottleneck layer Bottleneck_2, a first depthwise separable convolutional layer DWConv_1, a second depthwise separable convolutional layer DWConv_2 and a third depthwise separable convolutional layer DWConv_3.

[0077] Among them, the first convolution layer Conv_1, the second convolution layer Conv_2, the third convolution layer Conv_3, and the fourth convolution layer Conv_4 all belong to the convolution layer. The convolution layer provides a convolution operation, that is, the inner product (that is, element-by-element multiplication and summation) of the input image data and the filter matrix (constant filter fiter) is performed to extract local features in the image data.

[0078] The first bottleneck layer Bottleneck_1 and the second bottleneck layer Bottleneck_2 both belong to the bottleneck layer Bottleneck. The bottleneck layer Bottleneck is composed of three convolutional layers of size 1×1, 3×3, and 1×1. The function of the 1×1 convolutional layer is to first reduce and then restore the dimension, and the 3×3 convolutional layer is the bottleneck of smaller input / output dimensions. Therefore, the bottleneck layer Bottleneck can be used for feature dimensionality reduction, reducing the number of layers of feature maps, reducing the number of parameters and thus reducing the amount of calculation.

[0079] The first depthwise separable convolution layer DWConv_1, the second depthwise separable convolution layer DWConv_2 and the third depthwise separable convolution layer DWConv_3 all belong to the depthwise separable convolution layer (DWConv). In the depthwise separable convolution, the convolution operation is divided into two sub-operations: depthwise convolution and pointwise convolution. The depthwise convolution uses a convolution kernel for each channel of the input feature map, and the outputs of all convolution kernels are concatenated to obtain the output. Because the number of output channels of the convolution operation is equal to the number of convolution kernels, and only one convolution kernel is used for each channel in the depthwise convolution, the number of output channels of a single channel after the convolution operation is also 1. Pointwise convolution is actually a 1×1 convolution, which allows the depthwise separable convolution layer to freely change the number of output channels, and to perform channel fusion on the feature map output by the depthwise convolution.

[0080] In the embodiments of the present application, Figure 2 As shown, the image data is input into the first convolutional layer Conv_1 to extract the first backbone features.

[0081] The first backbone features are input into the second convolutional layer Conv_2 to extract the second backbone features.

[0082] The second backbone feature is input into the first bottleneck layer Bottleneck_1 to extract the third backbone feature.

[0083] The third backbone feature is input into the second bottleneck layer Bottleneck_2 to extract the fourth backbone feature.

[0084] The fourth backbone feature is input into the third convolutional layer Conv_3 to extract the fifth backbone feature as the common image feature.

[0085] The fifth backbone feature is input into the first depth-wise separable convolutional layer DWConv_1 to extract the sixth backbone feature.

[0086] The sixth backbone feature is input into the second depth-wise separable convolutional layer DWConv_2 to extract the seventh backbone feature.

[0087] The seventh backbone feature is input into the third depth-wise separable convolutional layer DWConv_3 to extract the eighth backbone feature.

[0088] The fifth backbone feature, the sixth backbone feature, the seventh backbone feature and the eighth backbone feature are concatenated to form the ninth backbone feature.

[0089] The ninth backbone feature is input into the fourth convolutional layer Conv_4 to extract the tenth backbone feature as the common image feature.

[0090] In this backbone structure Backbone, the idea of ​​extraction and diversion is used, and at the same time combined with the residual structure, the main branch gradient is a depth-separable convolutional layer, and multiple depth-separable convolutional layers are stacked to reduce the gradient and realize multi-scale feature fusion, thereby improving the quality of features.

[0091] Step 106: Input the common image features into the neck structure and convert them into target image features.

[0092] like Figure 2 As shown, the common image features are input into the neck structure Neck, and the neck structure Neck performs multi-scale fusion on the common image features to obtain the target image features.

[0093] In one embodiment of the present application, Figure 2 As shown, the neck structure Neck includes a fifth convolutional layer Conv_5, a sixth convolutional layer Conv_6, a seventh convolutional layer Conv_7 and a third bottleneck layer Bottleneck_3; wherein, the fifth convolutional layer Conv_5, the sixth convolutional layer Conv_6, and the seventh convolutional layer Conv_7 are all convolutional layers, and the third bottleneck layer Bottleneck_3 belongs to the bottleneck layer.

[0094] In the embodiments of the present application, Figure 2 As shown, upsample is performed on the tenth backbone feature to obtain the first bottleneck feature.

[0095] The first bottleneck feature is input into the fifth convolutional layer Conv_5 to extract the second bottleneck feature.

[0096] The second bottleneck feature and the fifth backbone feature are concatenated into the third bottleneck feature.

[0097] The third bottleneck feature is input into the sixth convolutional layer Conv_6 to extract the fourth bottleneck feature.

[0098] The fourth bottleneck feature is input into the third bottleneck layer Bottleneck_3 to extract the fifth bottleneck feature.

[0099] The tenth backbone feature and the fifth bottleneck feature are concatenated into the sixth bottleneck feature.

[0100] The sixth bottleneck feature is input into the seventh convolutional layer Conv_7 to extract the target image features.

[0101] Step 107: Input the target image features into multiple head structures to generate individual violation values ​​for multiple violation categories.

[0102] like Figure 2 As shown, the target image features output by the neck structure Neck are split into multiple target image features using a chunk function or the like, so that the number of target image features is the same as the number of the head structure Head. At this time, each target image feature is input into each head structure Head respectively.

[0103] In each head structure Head, a single violation value of the image data in a specified violation category is generated according to the target image feature, so that the single violation value represents the violation probability of the image data in the specified violation category.

[0104] Generally speaking, the larger the single violation value is, the higher the probability that the image data has violated the specified violation category is. Conversely, the smaller the single violation value is, the lower the probability that the image data has violated the specified violation category is.

[0105] In one embodiment of the present application, Figure 2 As shown, each head structure Head includes an eighth convolution block Conv_8, a fourth depth-separable convolution layer DWConv_4 and a fully connected layer (FC).

[0106] Among them, the eighth convolution block Conv_8 belongs to the convolution layer, the fourth depth-wise separable convolution layer DWConv_4 belongs to the depth-wise separable convolution layer, and the fully connected layer flattens the output of the previous layer (usually the convolution layer or the pooling layer) and connects it to the output layer, thereby converting the high-dimensional feature representation into a one-dimensional vector to realize the operation of the monomer violation value.

[0107] In the embodiments of the present application, Figure 2 As shown, in each head structure Head, the target image feature is input into the eighth convolution block Conv_8 to extract the first head feature.

[0108] In each head structure Head, the first head feature is input into the fourth depth-separable convolutional layer DWConv_4 to extract the second head feature.

[0109] In each head structure Head, the second head feature is input into the fully connected layer FC and mapped to the monomer violation value of the specified violation category.

[0110] Step 108: Generate a violation level for the video clip according to the multiple individual violation values ​​in accordance with the review plan.

[0111] In each video clip, following the audit scheme corresponding to the current business category, statistical analysis is performed on each individual violation value of each sampling point in the video clip, thereby generating a violation level for the video clip.

[0112] In one embodiment of the present application, step 108 may include the following steps:

[0113] Step 1081: Read sampling rules and fragment rules from the audit plan.

[0114] In the embodiment of the present application, the audit scheme has two rules, namely the sampling rule and the fragment rule. When generating the violation level, the sampling rule and the fragment rule are read from the audit scheme respectively according to the ID and other identifiers.

[0115] The sampling rule is a rule for fusing individual violation values ​​for a single sampling point, and the segment rule is a rule for fusing overall violation values ​​for a single video segment.

[0116] Furthermore, the sampling rule has individual weights configured for multiple violation categories. The sampling rule can be expressed as Among them, W α is the set of monomer weights, is the monomer weight of the i-th violation category, i∈n, and n is the number of violation categories.

[0117] Generally speaking, the individual weights of violation categories that tend to appear in the current business category are higher, and the individual weights of violation categories that do not tend to appear in the current business category are lower. In this way, additional individual weights are used to adjust the detection results of the content review network (that is, the individual violation values ​​on multiple violation categories), so as to correct the detection results to meet business needs. This can maintain the accuracy of detection while reducing the frequency of updating the content review network.

[0118] Step 1082: For the same violation category, calculate the product of the single violation value and the single weight to obtain the weighted violation value.

[0119] In the embodiment of the present application, for the same sampling point, each violation category is traversed, and for each violation category, the product between its single violation value and the corresponding single weight is calculated, which is recorded as the weighted violation value.

[0120] Step 1083: Add the weighted violation values ​​of all violation categories to obtain the overall violation value of the sampling point.

[0121] In the embodiment of the present application, for the same sampling point, the weighted violation values ​​of all violation categories are added together to obtain the overall violation value of the sampling point.

[0122] Then, the overall violation value can be expressed as:

[0123]

[0124] Among them, Score w is the overall violation value, s i The individual violation value of the i-th violation category, is the monomer weight of the i-th violation category, i∈n, and n is the number of violation categories.

[0125] Step 1084: For the same video segment, use the overall violation value to generate a violation level for the video segment according to the segment rule.

[0126] In each video segment, following the segment rule corresponding to the current service category, the overall violation value of each sampling point in the video segment is used to generate a violation level for the video segment.

[0127] In a specific implementation, content review of video data in advertising data usually constitutes a one-vote veto, that is, if the video data in the advertising data contains content of a violation category once, the entire advertising data will be deemed to be in violation.

[0128] Therefore, when assessing the violation level, we mainly focus on the content with higher overall violation values. Then, we regard the sampling points in the entire video data as discrete points, and use polynomial functions and other methods to fit the discrete points into a curve. In the same video clip, we screen out the overall violation value belonging to the peak in the curve as a candidate violation value.

[0129] The average value of all candidate violation values ​​is amplified to obtain a violation threshold, that is, the product of the candidate violation value and a preset amplification factor is calculated as the violation threshold, wherein the amplification factor is greater than 1 and is generally less than 2, such as 1.1, 1.2, etc.

[0130] Each candidate violation value is compared with the violation threshold, and candidate violation values ​​greater than or equal to the violation threshold are screened out to obtain a typical violation value, thereby realizing adaptive screening of content with a higher overall violation value.

[0131] The fragment rules corresponding to each violation category contain multiple arrays, each array records a different number of fusion weights, and the fusion weights in each array are not numerically the same.

[0132] Then, the fragment rule can be expressed as:

[0133]

[0134]

[0135] …

[0136]

[0137] in, is the mth array, is the jth fusion weight, j∈m+1, m+1 is the number of fusion weights, m is a positive integer, m≥1.

[0138] Among them, W α is the set of monomer weights, is the monomer weight of the i-th violation category, i∈n, and n is the number of violation categories.

[0139] Then, the number of typical violation values ​​in the same video clip can be counted, and the fusion weights adapted to the typical violation values ​​can be queried in the clip rules; wherein the number of fusion weights is the same as the number of typical violation values.

[0140] Fusion weights are assigned to typical violation values ​​so that the typical violation values ​​are positively correlated with the fusion weights, i.e., the fusion weight with the largest value is assigned to the typical violation value with the highest value, the fusion weight with the second largest value is assigned to the typical violation value with the second highest value, and so on.

[0141] The target violation value is obtained by summing the products between the typical violation value and the fusion weight.

[0142] At this point, the target violation value is expressed as:

[0143]

[0144] Among them, Score t Score w is the overall violation value, is the jth fusion weight, j∈m+1, m+1 is the number of overall violation values ​​and the number of fusion weights, m is a positive integer, m≥1.

[0145] In an embodiment of the present application, multiple violation ranges may be pre-divided, and the endpoints between two adjacent violation ranges intersect, and each violation range is configured with a violation level, such as low risk, medium risk, high risk, etc.

[0146] The target violation value is compared with a plurality of preset violation ranges, and the violation range in which the target violation value is located is searched in the plurality of preset violation ranges to obtain the target range, thereby configuring the violation level corresponding to the target range to the video clip.

[0147] Step 109: Display the violation level of the video clip on the time axis of the video data.

[0148] In the embodiments of the present application, Figure 3 As shown, a timeline can be drawn for the video data, and positioning operations on the timeline can be applied to the video data. At this time, the violation levels of the video clips are distinguished by text, background color, etc. on the timeline of the video data, to assist appraisers in reviewing the advertising data, so that appraisers can concentrate when the violation level is high and reduce the probability of missing illegal content.

[0149] Furthermore, if Figure 3 As shown, the violation level of the video clip and the overall violation value of the sampling point are displayed simultaneously on the timeline of the video data, wherein the overall violation value of the sampling point can be displayed in the form of discrete points, curves, broken lines, etc., so that the appraiser can use fast forwarding, dragging, etc. to locate the content with a higher probability of violation.

[0150] In the embodiment of the present application, the interference of the end card and the banner data is filtered out from the video data in the advertisement data, so that the content of the video data is more concise and effective, and the video data in the advertisement data is divided into video clips of multiple independent scenes, so as to improve the content concentration in each video clip, which is beneficial to the subsequent assessment of the violation level of the video clip, and the backbone structure is used on the content review network to extract the common image features from the image data in the video data and use the neck structure to convert it into the target image features, so as to provide high-quality and reusable features for each head structure, reduce the operation of extracting features, and improve the efficiency of detection, and use the audio data in the advertisement data to identify the business category. The audio data is synchronized with the video data, which can reflect the content of the video data to a certain extent, and the overall operation of the audio data is relatively simple, so as to ensure the accuracy of the business category. The tendency of illegal content in different business categories is different, and the results of the content review network are corrected according to the review plan of the business category, so as to maintain the accuracy of the content review while reducing the frequency of updating the content review network.

[0151] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0152] Reference Figure 4 , showing a schematic diagram of an advertising data review device provided in an embodiment of the present application, which may specifically include:

[0153] The pre-processing module 401 is used to receive advertisement data to be reviewed when loading the content review network; the content review network includes a backbone structure, a neck structure and multiple head structures, and the advertisement data includes video data and audio data;

[0154] A business category identification module 402 is used to identify the business category to which the advertisement data belongs based on the audio data; the business category is configured with an audit scheme;

[0155] The video segment segmentation module 403 is used to segment the video data into a plurality of video segments each of which belongs to an independent scene, under the condition of filtering out the end card and the banner data in the video data;

[0156] An image data extraction module 404 is used to mark a plurality of sampling points in the video clip and extract image data at each of the sampling points;

[0157] A backbone structure calling module 405, used for inputting the image data into the backbone structure to extract common image features;

[0158] A neck structure calling module 406 is used to convert the common image features into target image features by inputting the common image features into the neck structure;

[0159] A header structure calling module 407, used for inputting the target image feature into a plurality of the header structures to generate individual violation values ​​for a plurality of violation categories;

[0160] A violation level generating module 408, configured to generate a violation level for the video clip according to the plurality of individual violation values ​​and the audit scheme;

[0161] The review result display module 409 is used to display the violation level of the video clip on the time axis of the video data.

[0162] In one embodiment of the present application, the service category identification module 402 is further used to:

[0163] Performing speech recognition on the audio data to obtain text information;

[0164] Identifying an advertiser who places the advertisement data; the advertiser is marked with multiple business tags;

[0165] Converting the plurality of service tags and the text information into prompts adapted to a preset language model;

[0166] The prompt is input into the language model to identify the business category to which the advertisement data belongs.

[0167] In one embodiment of the present application, the backbone structure includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first bottleneck layer, a second bottleneck layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer, and a third depth-separable convolutional layer;

[0168] The backbone structure calling module 405 is also used for:

[0169] Inputting the image data into the first convolutional layer to extract the first backbone features;

[0170] Inputting the first backbone features into the second convolutional layer to extract second backbone features;

[0171] Inputting the second backbone features into the first bottleneck layer to extract third backbone features;

[0172] Inputting the third backbone feature into the second bottleneck layer to extract the fourth backbone feature;

[0173] Inputting the fourth backbone feature into the third convolutional layer to extract the fifth backbone feature as a common image feature;

[0174] Inputting the fifth backbone feature into the first depthwise separable convolutional layer to extract the sixth backbone feature;

[0175] Inputting the sixth backbone feature into the second depthwise separable convolutional layer to extract the seventh backbone feature;

[0176] Inputting the seventh backbone feature into the third depthwise separable convolutional layer to extract the eighth backbone feature;

[0177] splicing the fifth backbone feature, the sixth backbone feature, the seventh backbone feature and the eighth backbone feature into a ninth backbone feature;

[0178] The ninth backbone feature is input into the fourth convolutional layer to extract the tenth backbone feature as a common image feature.

[0179] In one embodiment of the present application, the neck structure includes a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer and a third bottleneck layer;

[0180] The neck structure calling module 406 is also used for:

[0181] Performing upsampling on the tenth backbone feature to obtain a first bottleneck feature;

[0182] Inputting the first bottleneck feature into the fifth convolutional layer to extract the second bottleneck feature;

[0183] splicing the second bottleneck feature and the fifth backbone feature into a third bottleneck feature;

[0184] Inputting the third bottleneck feature into the sixth convolutional layer to extract a fourth bottleneck feature;

[0185] Inputting the fourth bottleneck feature into the third bottleneck layer to extract the fifth bottleneck feature;

[0186] splicing the tenth backbone feature and the fifth bottleneck feature into a sixth bottleneck feature;

[0187] The sixth bottleneck feature is input into the seventh convolutional layer to extract the target image feature.

[0188] In one embodiment of the present application, each of the head structures includes an eighth convolutional block, a fourth depthwise separable convolutional layer and a fully connected layer;

[0189] The header structure calling module 407 is also used for:

[0190] In each of the head structures, the target image feature is input into the eighth convolution block to extract the first head feature;

[0191] In each of the head structures, the first head feature is input into the fourth depthwise separable convolutional layer to extract the second head feature;

[0192] In each of the head structures, the second head features are input into the fully connected layer and mapped into individual violation values ​​of a specified violation category.

[0193] In one embodiment of the present application, the violation level generation module 408 is further configured to:

[0194] Reading sampling rules and fragment rules from the audit scheme; the sampling rules have monomer weights configured for multiple violation categories;

[0195] For the same violation category, the product of the monomer violation value and the monomer weight is calculated to obtain a weighted violation value;

[0196] Adding the weighted violation values ​​of all the violation categories to obtain an overall violation value of the sampling point;

[0197] For the same video segment, using the overall violation value to generate a violation level for the video segment according to the segment rule;

[0198] The displaying of the violation level of the video clip on the time axis of the video data includes:

[0199] The violation level of the video segment and the overall violation value of the sampling point are simultaneously displayed on the time axis of the video data.

[0200] In one embodiment of the present application, the violation level generation module 408 is further configured to:

[0201] In the same video clip, the overall violation value belonging to the peak is screened out as a candidate violation value;

[0202] Amplifying the average value of all the candidate violation values ​​to obtain a violation threshold;

[0203] Screening out the candidate violation values ​​that are greater than or equal to the violation threshold to obtain a typical violation value;

[0204] Searching the fragment rule for a fusion weight that matches the typical violation value; the number of the fusion weights is the same as the number of the typical violation values;

[0205] assigning the fusion weight to the typical violation value so that the typical violation value is positively correlated with the fusion weight;

[0206] summing the product of the typical violation value and the fusion weight to obtain a target violation value;

[0207] Querying the violation range where the target violation value is located in a plurality of preset violation ranges to obtain a target range; each violation range is configured with a violation level;

[0208] The violation level corresponding to the target range is configured to the video segment.

[0209] An embodiment of the present application provides an advertisement data auditing device. By using the advertisement data auditing device, each step in the aforementioned method embodiments can be implemented.

[0210] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0211] Reference Figure 5 , shows a schematic diagram of a terminal device provided in an embodiment of the present application. Figure 5 As shown, the terminal device 500 in the embodiment of the present application includes: a processor 510, a memory 520, and a computer program 521 stored in the memory 520 and executable on the processor 510. When the processor 510 executes the computer program 521, the steps in each embodiment of the above-mentioned advertising data audit method are implemented, such as Figure 1 Alternatively, when the processor 510 executes the computer program 521, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 4 Functions of modules 401 to 409 are shown.

[0212] Exemplarily, the computer program 521 can be divided into one or more modules / units, and the one or more modules / units are stored in the memory 520 and executed by the processor 510 to complete the present application. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments can be used to describe the execution process of the computer program 521 in the terminal device 500. For example, the computer program 521 can be divided into a pre-processing module, a business category identification module, a video segment segmentation module, an image data extraction module, a backbone structure call module, a neck structure call module, a head structure call module, a violation level generation module and an audit result display module. The specific functions of each module are as follows:

[0213] A pre-processing module, used for receiving advertisement data to be reviewed when loading a content review network; the content review network includes a backbone structure, a neck structure and a plurality of head structures, and the advertisement data includes video data and audio data;

[0214] A business category identification module, used to identify the business category to which the advertisement data belongs based on the audio data; the business category is configured with an audit scheme;

[0215] A video segment segmentation module, used for segmenting the video data into a plurality of video segments each of which belongs to an independent scene, under the condition of filtering out the end card and banner data in the video data;

[0216] An image data extraction module, used to mark a plurality of sampling points in the video clip and extract image data at each of the sampling points;

[0217] A backbone structure calling module, used for inputting the image data into the backbone structure to extract common image features;

[0218] A neck structure calling module, used for inputting the common image features into the neck structure and converting them into target image features;

[0219] A header structure calling module, used for inputting the target image features into a plurality of the header structures to generate individual violation values ​​for a plurality of violation categories;

[0220] A violation level generating module, configured to generate a violation level for the video clip according to the plurality of individual violation values ​​and the audit scheme;

[0221] The audit result display module is used to display the violation level of the video clip on the time axis of the video data.

[0222] The terminal device 500 may be the advertising data auditing device in the aforementioned embodiments, and the advertising data auditing device may be a computing device such as a desktop computer, a cloud server, etc. The terminal device 500 may include, but is not limited to, a processor 510 and a memory 520. Those skilled in the art will appreciate that Figure 5 It is only an example of the terminal device 500 and does not constitute a limitation of the terminal device 500. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 500 may also include input and output devices, network access devices, buses, etc.

[0223] The processor 510 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0224] The memory 520 may be an internal storage unit of the terminal device 500, such as a hard disk or memory of the terminal device 500. The memory 520 may also be an external storage device of the terminal device 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 500. Further, the memory 520 may also include both an internal storage unit of the terminal device 500 and an external storage device. The memory 520 is used to store the computer program 521 and other programs and data required by the terminal device 500. The memory 520 may also be used to temporarily store data that has been output or is to be output.

[0225] An embodiment of the present application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the advertising data review method as described in the above-mentioned embodiments when executing the computer program.

[0226] The embodiments of the present application further disclose a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the advertising data review method as described in the above embodiments is implemented.

[0227] The embodiments of the present application further disclose a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the advertising data auditing method described in the above embodiments.

[0228] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for auditing advertising data, characterized in that: include: When loading the content review network, receiving the advertisement data to be reviewed; The content review network includes a backbone structure, a neck structure and multiple head structures, and the advertisement data includes video data and audio data; Identify the business category to which the advertisement data belongs based on the audio data; the business category is configured with an audit scheme; Under the condition of filtering out the end card and the banner data in the video data, the video data is divided into a plurality of video segments, each of which has contents belonging to independent scenes; Marking a plurality of sampling points in the video clip, and extracting image data at each of the sampling points; Inputting the image data into the backbone structure to extract common image features; Inputting the common image features into the neck structure and converting them into target image features; Inputting the target image feature into a plurality of the head structures to generate individual violation values ​​on a plurality of violation categories; generating a violation level for the video clip according to the review scheme based on the plurality of individual violation values; Displaying the violation level of the video clip on the time axis of the video data; The backbone structure includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first bottleneck layer, a second bottleneck layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer and a third depth-separable convolutional layer; The step of inputting the image data into the backbone structure to extract common image features comprises: Inputting the image data into the first convolutional layer to extract the first backbone features; Inputting the first backbone features into the second convolutional layer to extract second backbone features; Inputting the second backbone features into the first bottleneck layer to extract third backbone features; Inputting the third backbone feature into the second bottleneck layer to extract the fourth backbone feature; Inputting the fourth backbone feature into the third convolutional layer to extract the fifth backbone feature as a common image feature; Inputting the fifth backbone feature into the first depthwise separable convolutional layer to extract the sixth backbone feature; Inputting the sixth backbone feature into the second depthwise separable convolutional layer to extract the seventh backbone feature; Inputting the seventh backbone feature into the third depthwise separable convolutional layer to extract the eighth backbone feature; splicing the fifth backbone feature, the sixth backbone feature, the seventh backbone feature and the eighth backbone feature into a ninth backbone feature; Inputting the ninth backbone feature into the fourth convolutional layer to extract the tenth backbone feature as a common image feature; The neck structure includes a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer and a third bottleneck layer; The inputting the common image features into the neck structure to convert them into target image features comprises: Performing upsampling on the tenth backbone feature to obtain a first bottleneck feature; Inputting the first bottleneck feature into the fifth convolutional layer to extract the second bottleneck feature; splicing the second bottleneck feature and the fifth backbone feature into a third bottleneck feature; Inputting the third bottleneck feature into the sixth convolutional layer to extract a fourth bottleneck feature; Inputting the fourth bottleneck feature into the third bottleneck layer to extract the fifth bottleneck feature; splicing the tenth backbone feature and the fifth bottleneck feature into a sixth bottleneck feature; The sixth bottleneck feature is input into the seventh convolutional layer to extract the target image feature.

2. The method according to claim 1, characterized in that The identifying, based on the audio data, the business category to which the advertisement data belongs includes: Performing speech recognition on the audio data to obtain text information; Identifying an advertiser who places the advertisement data; the advertiser is marked with multiple business tags; Converting the plurality of service tags and the text information into prompts adapted to a preset language model; The prompt is input into the language model to identify the business category to which the advertisement data belongs.

3. The method according to claim 1, characterized in that: Each of the head structures includes an eighth convolutional block, a fourth depthwise separable convolutional layer and a fully connected layer; The step of inputting the target image feature into a plurality of the head structures to generate individual violation values ​​for a plurality of violation categories comprises: In each of the head structures, the target image feature is input into the eighth convolution block to extract the first head feature; In each of the head structures, the first head feature is input into the fourth depthwise separable convolutional layer to extract the second head feature; In each of the head structures, the second head features are input into the fully connected layer and mapped into individual violation values ​​of a specified violation category.

4. The method according to any one of claims 1 to 3, characterized in that Generating a violation level for the video clip according to the audit scheme based on the plurality of individual violation values ​​includes: Reading sampling rules and fragment rules from the audit scheme; the sampling rules have monomer weights configured for multiple violation categories; For the same violation category, the product of the monomer violation value and the monomer weight is calculated to obtain a weighted violation value; Adding the weighted violation values ​​of all the violation categories to obtain an overall violation value of the sampling point; For the same video segment, using the overall violation value to generate a violation level for the video segment according to the segment rule; The displaying of the violation level of the video clip on the time axis of the video data includes: The violation level of the video segment and the overall violation value of the sampling point are simultaneously displayed on the time axis of the video data.

5. The method according to claim 4, characterized in that For the same video segment, using the overall violation value to generate a violation level for the video segment according to the segment rule includes: In the same video clip, the overall violation value belonging to the peak is screened out as a candidate violation value; Amplifying the average value of all the candidate violation values ​​to obtain a violation threshold; Screening out the candidate violation values ​​that are greater than or equal to the violation threshold to obtain a typical violation value; Searching the fragment rule for a fusion weight that matches the typical violation value; the number of the fusion weights is the same as the number of the typical violation values; assigning the fusion weight to the typical violation value so that the typical violation value is positively correlated with the fusion weight; summing the product of the typical violation value and the fusion weight to obtain a target violation value; Querying the violation range where the target violation value is located in a plurality of preset violation ranges to obtain a target range; each violation range is configured with a violation level; The violation level corresponding to the target range is configured to the video segment.

6. An advertising data auditing device, characterized in that: include: A pre-processing module, used to receive advertisement data to be reviewed when loading the content review network; The content review network includes a backbone structure, a neck structure and multiple head structures, and the advertisement data includes video data and audio data; A business category identification module, used to identify the business category to which the advertisement data belongs based on the audio data; the business category is configured with an audit scheme; A video segment segmentation module, used for segmenting the video data into a plurality of video segments each of which belongs to an independent scene, under the condition of filtering out the end card and banner data in the video data; An image data extraction module, used to mark a plurality of sampling points in the video clip and extract image data at each of the sampling points; A backbone structure calling module, used for inputting the image data into the backbone structure to extract common image features; A neck structure calling module, used for inputting the common image features into the neck structure and converting them into target image features; A header structure calling module, used for inputting the target image features into a plurality of the header structures to generate individual violation values ​​for a plurality of violation categories; A violation level generating module, configured to generate a violation level for the video clip according to the plurality of individual violation values ​​and the audit scheme; An audit result display module, used to display the violation level of the video clip on the time axis of the video data; The backbone structure includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first bottleneck layer, a second bottleneck layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer and a third depth-separable convolutional layer; The backbone structure calling module is also used for: Inputting the image data into the first convolutional layer to extract the first backbone features; Inputting the first backbone features into the second convolutional layer to extract second backbone features; Inputting the second backbone features into the first bottleneck layer to extract third backbone features; Inputting the third backbone feature into the second bottleneck layer to extract the fourth backbone feature; Inputting the fourth backbone feature into the third convolutional layer to extract the fifth backbone feature as a common image feature; Inputting the fifth backbone feature into the first depthwise separable convolutional layer to extract the sixth backbone feature; Inputting the sixth backbone feature into the second depthwise separable convolutional layer to extract the seventh backbone feature; Inputting the seventh backbone feature into the third depthwise separable convolutional layer to extract the eighth backbone feature; splicing the fifth backbone feature, the sixth backbone feature, the seventh backbone feature and the eighth backbone feature into a ninth backbone feature; Inputting the ninth backbone feature into the fourth convolutional layer to extract the tenth backbone feature as a common image feature; The neck structure includes a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer and a third bottleneck layer; The neck structure calling module is also used for: Performing upsampling on the tenth backbone feature to obtain a first bottleneck feature; Inputting the first bottleneck feature into the fifth convolutional layer to extract the second bottleneck feature; splicing the second bottleneck feature and the fifth backbone feature into a third bottleneck feature; Inputting the third bottleneck feature into the sixth convolutional layer to extract a fourth bottleneck feature; Inputting the fourth bottleneck feature into the third bottleneck layer to extract the fifth bottleneck feature; splicing the tenth backbone feature and the fifth bottleneck feature into a sixth bottleneck feature; The sixth bottleneck feature is input into the seventh convolutional layer to extract the target image feature.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for reviewing advertising data according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the advertising data auditing method according to any one of claims 1 to 5 is implemented.

Images