Multimedia information recognition method, device, equipment, storage medium and program product

Abstract

Embodiments of the application disclose a multimedia information identification method, device, equipment, storage medium and program product. The method comprises: acquiring multimedia information to be identified and basic information associated with the multimedia information; obtaining an identification result by performing preset identification on the multimedia information and the basic information through a preset content filtering module. The content filtering module comprises a multi-size identification detection model and a classifier cascaded with the multi-size identification detection model. The training process of the classifier comprises at least three model training stages. In the case that the identification result is a false identification, user identity information associated with the multimedia information is confirmed. The scheme can efficiently and accurately identify multimedia information.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a multimedia information recognition method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of the internet and multimedia information technology, the dissemination of multimedia information via the internet has become one of the main ways of information interaction, as exemplified by the rise of short video and live streaming services. However, in the process of disseminating multimedia information, some false and fabricated information may emerge. Therefore, how to efficiently identify multimedia information has become an urgent problem to be solved.

[0003] In related technologies, when identifying multimedia information, a common approach is to set up a reporting channel to receive reports from users. Reviewers then use this channel to identify the corresponding user in the system database, ultimately obtaining the identification result for the multimedia information. However, this method requires significant manpower and the identification results are subject to delays. Summary of the Invention

[0004] This application provides a multimedia information recognition method, apparatus, device, storage medium, and program product, which solves the problems of high labor costs and poor real-time performance of recognition results in related technologies, and enables efficient and accurate multimedia information recognition.

[0005] In a first aspect, embodiments of this application provide a multimedia information identification method, the method comprising:

[0006] Obtain the multimedia information to be identified and the basic information associated with the multimedia information;

[0007] The multimedia information and the basic information are identified by a preset content filtering module to obtain the identification result. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with the multi-size identifier detection model. The training process of the classifier includes at least three model training stages.

[0008] If the identification result is a false identifier, the user identity information associated with the multimedia information is verified.

[0009] Secondly, embodiments of this application also provide a multimedia information recognition device, including:

[0010] The information acquisition module is configured to acquire multimedia information to be identified and basic information associated with the multimedia information;

[0011] The recognition result determination module is configured to identify the multimedia information and the basic information by performing preset identification on the preset identifiers through a preset content filtering module to obtain the recognition result. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with the multi-size identifier detection model. The training process of the classifier includes at least three model training stages.

[0012] The information determination module is configured to confirm the user identity information associated with the multimedia information if the identification result is a false identifier.

[0013] Thirdly, embodiments of this application also provide a multimedia information recognition device, the device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the multimedia information recognition method described in the embodiments of this application.

[0017] Fourthly, embodiments of this application also provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the multimedia information recognition method described in embodiments of this application.

[0018] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the multimedia information recognition method described in embodiments of this application.

[0019] In this embodiment, the multimedia information to be identified and the basic information associated with it are obtained. A preset content filtering module identifies the multimedia information and basic information using preset identifiers to obtain the identification result. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with the multi-size identifier detection model. The classifier's training process includes at least three model training stages. If the identification result is a false identifier, the user identity information associated with the multimedia information is verified. This multimedia information identification method achieves efficient identification of multimedia information through the cascaded multi-size identifier detection model and classifier. The classifier training is based on multiple different model training stages, which improves the accuracy of multimedia information identification. Attached Figure Description

[0020] Figure 1A flowchart illustrating a multimedia information recognition method provided in this application embodiment;

[0021] Figure 2 A schematic diagram of the model structure of an exemplary multi-size identifier detection model provided in this application embodiment;

[0022] Figure 3 This is a schematic diagram of the structure of a feature extraction layer provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the internal structure of each module of the feature extraction layer provided in an embodiment of this application;

[0024] Figure 5 A structural diagram of an adaptive anchor frame candidate network provided in an embodiment of this application;

[0025] Figure 6 This application provides a schematic diagram of the internal structure of each module of an adaptive anchor frame candidate network.

[0026] Figure 7a A schematic diagram of the model structure for the first training stage of a classifier provided in an embodiment of this application;

[0027] Figure 7b A schematic diagram of the model structure for the second training stage of a classifier provided in an embodiment of this application;

[0028] Figure 7c A schematic diagram of the model structure for the third training stage of a classifier provided in an embodiment of this application;

[0029] Figure 8 A flowchart illustrating a data augmentation processing method for multi-image fusion coefficients and multi-dimensional image arrangement provided in this application embodiment;

[0030] Figure 9 A flowchart illustrating how an adaptive identifier scaling method is used to obtain a dataset, as provided in this application embodiment;

[0031] Figure 10 This is a schematic diagram of the structure of a face recognition model provided in an embodiment of this application;

[0032] Figure 11 A schematic diagram of model training for a face recognition model provided in an embodiment of this application;

[0033] Figure 12 A structural block diagram of a multimedia information recognition device provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of a multimedia information recognition device provided in an embodiment of this application. Detailed Implementation

[0035] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, only the parts relevant to the embodiments of this application are shown in the accompanying drawings, not the entire structure.

[0036] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0037] The multimedia information identification method provided in this application embodiment can be applied to situations where it is necessary to identify multimedia information to determine whether it belongs to a preset type of information. This preset type of information includes false information, illegal information, and information stolen from others. Taking the live streaming and short video industries as an example, it can identify whether a user's posted video is copied or stolen from others, and screen the user who posted the video to determine whether they are a fake user.

[0038] Figure 1 A flowchart of a multimedia information recognition method provided in this application embodiment specifically includes the following steps:

[0039] Step S101: Obtain the multimedia information to be identified and the basic information associated with the multimedia information.

[0040] In one embodiment, the multimedia information can be user-posted information such as images or videos. Taking the short video and live streaming industries as examples, it could be screenshots from user-posted short videos or live streams. This multimedia information is associated with corresponding basic information. In one embodiment, this basic information could be information containing visual elements such as a user's avatar or cover image. That is, the process of multimedia information recognition includes both the multimedia information itself and the associated basic information.

[0041] Step S102: The multimedia information and the basic information are identified by a preset identifier through a preset content filtering module to obtain the identification result. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with the multi-size identifier detection model.

[0042] In one embodiment, after acquiring multimedia information and associated basic information, it is input into a preset content filtering module for identification. This content filtering module includes a multi-size identifier detection model and a classifier cascaded with it. The multi-size identifier detection model and classifier are generated through pre-training. The classifier training process includes at least three model training stages. The preset content filtering module outputs the identification results of preset identifiers corresponding to the currently acquired multimedia information and associated basic information. These preset identifiers can be preset watermarks or V-shaped identifiers. These watermarks and V-shaped identifiers can be platform-set identifiers used for user authentication. By identifying these preset identifiers, it is determined whether they are fake, and then the results are used to further confirm the user identity information associated with the multimedia information, ultimately identifying whether the multimedia information is fake.

[0043] Optionally, the multi-size identifier detection model in the content filtering module is mainly used to identify preset identifiers in multimedia information and provide the location detection results and identifier classification results of the preset identifiers. The location detection results record the position of the preset identifiers in the multimedia information, and the identifier classification results include the identification results of real preset identifiers and fake preset identifiers.

[0044] In one embodiment, the multi-scale label detection model includes a feature extraction layer, a bidirectional multi-scale pyramid module, and an adaptive anchor box candidate network. This multi-scale label detection model outputs multi-scale location detection results and label classification results. The location detection results and label classification results are normalized coordinates of the feature map after downsampling of the multimedia information. After obtaining the output results, the model further includes mapping the coordinates of the boxes containing the preset labels in the original multimedia information based on the receptive field corresponding to the output results, and using a non-maximum suppression method based on box center distance and overlap area to remove redundant identified boxes.

[0045] In one embodiment, the input to the multi-size label detection model is multiple patches obtained after dividing the image into grids. It uses a single-stage training method, regressing the detection results of each patch against preset labels, thereby achieving a lightweight multi-size label detection model at a lower cost. Optionally, the feature extraction layer includes a CBM convolutional layer, a CBL convolutional layer, and a CSP module. The activation function corresponding to the CBM convolutional layer is the Mish activation function, and the activation function corresponding to the CBL convolutional layer is the LeakyReLU activation function. The reason for using the Mish and LeakyReLU activation functions is that these two activation functions have the ability to activate negative inputs, and compared with the conventional ReLU activation function, they are less likely to deactivate neurons, thus allowing the detection network to achieve stronger fitting ability with fewer weight parameters. The bidirectional multi-scale pyramid module includes a spatial pyramid network layer, a deconvolution layer, and a CSP module. Unlike conventional pyramid modules that use bilinear interpolation for upsampling, this embodiment uses a deconvolution layer instead of bilinear interpolation to expand and restore the feature map, reducing information loss during the feature map resolution restoration process and allowing the upsampling process to be refined in a supervised manner, thus enhancing the generalization ability when facing complex data.

[0046] In one embodiment, the multi-size marker detection model includes an adaptive anchor box candidate network. Conventional single-stage detection networks use pre-defined anchor boxes as the initial anchor boxes for each grid cell, leading to poor performance with data exhibiting significant shape and size variations. This embodiment incorporates an adaptive anchor box candidate network before the detection regression outputs at multiple scales to adaptively provide initial anchor boxes for each grid cell. Compared to the general method of adaptively calculating initial anchor boxes based on statistical information from the training set, the adaptive anchor box candidate network in this embodiment closely links anchor box coordinates with prediction results and more fully utilizes labeled data to guide the detection network convergence. By outputting the location detection results and marker classification results of preset markers at multiple different scales, it can accurately identify large targets while reducing the omission of small targets. Figure 2 A schematic diagram of the model structure of an exemplary multi-size identifier detection model provided in this application embodiment is shown below. Figure 2As shown, it includes a feature extraction layer, an adaptive anchor box candidate network, and a bidirectional multi-scale pyramid module. The adaptive anchor box candidate networks (RPN_like-big, RPN_like-mid, and RPN_like-small) output anchor box offsets at different scales. Based on these offsets and the original coordinates of the anchor boxes, the final predicted bounding boxes are obtained as the location detection results with preset labels. The deconvolution layer in the bidirectional multi-scale pyramid module, denoted as DConv, replaces the bilinear interpolation upsampling method to expand and restore the feature map.

[0047] like Figure 2 The model structure shown uses Spatial Pyramid Pooling (SPP) to transform features before they enter the network for detection and regression, thereby reducing noise interference on the feature map. Unlike ordinary SPP layers, SPP here is combined with padding operations so that the size of the feature map does not change before and after transformation. At the same time, in order to avoid information loss due to excessive pooling, pooling kernels of various sizes are used here to preserve information fidelity.

[0048] In one embodiment, the CSP module included in the feature extraction layer is constructed based on the CBM convolutional layer, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of a feature extraction layer provided in an embodiment of this application, including a CBM convolutional layer, a CBL convolutional layer, and a CSP module, wherein the CSP module is constructed based on the CBM convolutional layer. Optionally, the specific construction of the CSP module is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of the internal structure of each module of the feature extraction layer provided in an embodiment of this application. Figure 3 and Figure 4As can be seen, the feature extraction layer is mainly constructed using the CSP0 module, with a transition using multi-scale dilated convolution (MdiCBL) before each downsampling operation. CSP0 constructs all sub-modules based on CBM layers because the Mish activation function is more powerful than LeakyReLU in non-linear activation, making it more suitable for feature extraction in shallow networks. CSP0 stacks X residual blocks in the main network layer, deepening the feature extraction network and further enhancing its expressive power. At the exit of each CSP0 module, a depthwise separating convolutional layer (DWCBM) is used instead of a regular convolutional layer to complete the fitting operation, further reducing the number of weight parameters and thus further reducing computation, as the computational cost of a depthwise separating convolutional layer is about one-third that of a regular convolutional layer for the same fitting capability. Simultaneously, before entering the next illustration module, the model performs a channel shuffle operation to enhance information exchange between different channels. Before the downsampling operation, a multi-scale dilated convolution is used as a transition. Without increasing the convolution kernel parameters, this operation greatly increases the receptive field of the feature values ​​and reduces redundant feature information learning caused by the approximation of information in adjacent feature maps, providing sufficiently rich information for the subsequent downsampling operation.

[0049] In one embodiment, the adaptive anchor box candidate network includes grouped convolutional layers, a first CSP module, a second CSP module, a CBM convolutional layer, and a CBL convolutional layer, wherein the first CSP module is constructed based on residual modules, and the second CSP module is constructed based on a stacked network layer structure. For example, Figure 5 This document presents a structural diagram of an adaptive anchor box candidate network provided in an embodiment of this application. It includes two connected grouped convolutional layers with two output branches, used to learn the initial coordinate information of each grid anchor box and whether the anchor box contains a target. Each coordinate is represented by a tuple [x, y, w, h], while the corresponding target inclusion information is represented by [obj_score, no_obj_score]. Optionally, each grid will predict the coordinates of nine initial anchor boxes of different sizes and aspect ratios. Neurons corresponding to anchor box coordinates with target confidence scores below a threshold do not participate in subsequent prediction calculations and backpropagation updates; subsequent detection regression only needs to correct the offset based on the initial anchor box coordinate information. Figure 5As shown, the input undergoes further nonlinear transformation after two grouped convolutional layers (GConvCBL). The grouped convolutional layers are designed to reduce inference computation and improve the network's real-time performance. To avoid isolation of information from different channels by the grouped convolutions, a channel shuffle operation is also added here. Afterward, the network passes through a large bypass branch structure consisting of a first CSP module (CSP1) and a second CSP module (CSP2). The difference between CSP1 and CSP2 lies in the use of residual modules. CSP1, as the main branch, uses residual modules as building blocks for relay network layers to construct deeper network layers, while CSP2 uses a conventional network layer stacking method. Specifically, as shown... Figure 6 As shown, Figure 6 This is a schematic diagram of the internal structure of each module of an adaptive anchor frame candidate network provided in an embodiment of this application. Figure 6 The residual module shown uses LeakyReLU as the activation function to reduce floating-point operations. At the exits of CSP1 and CSP2, depthwise separable convolutional layers (DWCBM) are used instead of ordinary convolutional layers to complete the fitting operation, further reducing the network's weight.

[0050] In one embodiment, although the multi-size label detection model detects the location of the suspicious label and roughly provides a classification result for the preset label, using the classification result of the detection model as the final judgment criterion is often inaccurate because the receptive field used for classification during the training of the detection model is relatively small and does not consider the correlation influence of neighboring feature information. Therefore, a combined classifier is cascaded to improve the recognition accuracy. Optionally, the training process of the classifier includes multiple stages, specifically three training stages, referred to as the first training stage, the second training stage, and the third training stage.

[0051] Optional, a schematic diagram of the model structure for the three training phases is shown below. Figures 7a to 7c As shown, where, Figure 7a This is a schematic diagram of the model structure for the first training stage of a classifier, provided in an embodiment of this application. Figure 7b This is a schematic diagram of the model structure for the second training stage of a classifier, provided in an embodiment of this application. Figure 7cThis is a schematic diagram of the model structure for the third training stage of a classifier provided in an embodiment of this application. The dataset used in the first training stage is obtained through a data augmentation process using multi-image fusion coefficients and multi-dimensional image permutations, with the corresponding loss function being the binary cross-entropy loss function. Optionally, the model's feature extraction layer adopts the classic EfficientNetB3 network structure to improve inference efficiency. The dataset used in the first training stage uses preset labels as the main visual content, enabling the network to learn the ability to initially distinguish between two patterns. For example, the negative sample label is (0, 0). Since real and non-real images of the preset labels may appear in the same image, they are not absolutely mutually exclusive.

[0052] The dataset used in the second training phase was obtained through an adaptive label scaling method. The model used in this phase included a second feature extraction layer and a multi-scale pooling feature module. The multi-scale pooling feature module extracted feature maps from the first, second, and third feature layers of the second feature extraction layer, and then calculated feature vectors from these feature maps using a CSPP module. The dataset obtained using the adaptive label scaling method closely approximates the visual distribution of real-world data, where embedded labels appear only as foreground targets while the background occupies the majority of the content. Therefore, the second training phase needs to consider changes in target size, and an additional multi-scale pooling feature module is used to assist in this phase.

[0053] The third training phase uses the mixed data from the first two phases to fine-tune the model, also based on multi-scale information. As shown in Figure 7, the third training phase incorporates a bidirectional spatial pyramid structure to assist training. Compared to the unidirectional spatial pyramid, the bidirectional spatial pyramid supplements the information lost by the unidirectional spatial pyramid from bottom to top, and more significantly reflects the characteristics of the target when performing multi-scale information fusion. The dataset used in the third training phase is an imbalanced dataset with more negative samples than positive samples. Focal Loss-based BCE is used for training to address the long-tail distribution problem. After three training phases, only the backbone is selected as the final classification network.

[0054] In one embodiment, a specific data augmentation processing method for multi-image fusion coefficients and multi-dimensional image arrangement is provided, such as... Figure 8 As shown, Figure 8 A flowchart of a data augmentation processing method for multi-image fusion coefficients and multi-dimensional image arrangement provided in this application embodiment includes:

[0055] Step S1021: Randomly obtain a preset number of images from the original dataset. The preset number of images includes positive sample images and negative sample images. The positive sample images are images that contain a preset identifier and are the main body of the image. The negative sample images are images that do not contain the preset identifier.

[0056] For example, 2 to 4 images are randomly selected from the original dataset. If the visual content of these images does not contain the preset label (i.e., all are negative samples), the selection process is repeated until at least one image contains the preset label.

[0057] Step S1022: Perform random scaling on the positive sample image to obtain a positive sample enhanced image, and perform random scaling and random cropping on the negative sample image to obtain a negative sample enhanced image.

[0058] Optionally, positive sample images are randomly scaled, while negative sample images are randomly scaled and cropped. Since positive samples mainly contain the main subject and generally appear as a whole, cropping would destroy their effectiveness. Therefore, cropping is not included in the processing of positive sample images.

[0059] Step S1023: Randomly arrange the positive sample enhancement image and the negative sample enhancement image to obtain a combined image.

[0060] After obtaining the positive and negative sample augmented images, they are randomly arranged to generate a combined image. For example, a combined image can be generated using a random grid arrangement.

[0061] Step S1024: Generate the dataset corresponding to the first training stage based on the original dataset and the combined images.

[0062] Optionally, when generating the dataset corresponding to the first training phase, the images from the original dataset are combined with the composite images to obtain the final dataset. This can be achieved by: combining the original dataset and the composite images to obtain a hybrid dataset; randomly selecting two images from the hybrid dataset; weighted summing of the pixel values ​​at the same locations in the two images to obtain a fused image; and using the fused image as the dataset corresponding to the first training phase when the number of fused images reaches a preset number.

[0063] In one embodiment, a specific adaptive identifier scaling method for obtaining the dataset is given, such as... Figure 9 As shown, Figure 9 A flowchart of obtaining a dataset through an adaptive identifier scaling processing method provided in this application embodiment includes:

[0064] Step S1025: Randomly select images of preset label types from the original dataset as foreground images, and scale the foreground images according to the set scaling factor.

[0065] For example, there can be 2 to 10 preset label types. That is, 2 to 10 preset label images are randomly selected from the original dataset as foreground images. A scaling factor is randomly set for each type of preset label. For example, the scaling factor can be 0.3 to 3. Accordingly, the length and width values ​​of each image are calculated based on the set scaling factor.

[0066] In one embodiment, to ensure that the scaled foreground image is not distorted and that the number of fill pixels is minimized, an adaptive scaling method that maintains the aspect ratio is adopted. That is, based on the length and width values ​​before and after scaling, the scaling ratio of the length and width is calculated respectively, the smaller scaling ratio is selected for scaling, and black borders are filled into the scaled image to maintain the aspect ratio of the original image. The alpha channel position corresponding to the black border pixels is set to 0. When embedding, the black border in the foreground image will be made transparent, and the pixel values ​​of the background image will be retained at the corresponding positions.

[0067] Step S1026: Obtain an image that does not contain a preset identifier as a background image.

[0068] Optionally, any image without a preset identifier can be obtained from the original dataset and used as the background image.

[0069] Step S1027: If the sum of the areas of the foreground images and the area of ​​the background image satisfy the image embedding conditions, the foreground images are embedded into the background images to obtain an embedded image.

[0070] Optionally, before performing the specific embedding operation, the set embedding conditions must be met. Optionally, the total area of ​​the foreground image must be less than the area of ​​the background image to avoid the preset identifier obscuring the entire background.

[0071] Optionally, the dataset generated above can be used simultaneously for training both the multi-size label detection model and the classifier.

[0072] In one embodiment, during the training of the multi-size label detection model, different loss functions are selected for training based on the preset label size. Generally, the loss function used to train the detection model comprehensively considers the inter-box distance, the overlapping area of ​​two boxes, and the difference in length and width between the two boxes, such as DIOU loss and CIOU loss. In practice, these three factors are indeed more important for large target recognition, but for small targets, the difference in length and width between the two boxes is less important; usually, only the first two factors need to be considered to obtain relatively accurate results. In this embodiment, targets of different sizes are distinguished based on their relative area, and an appropriate loss function is selected for calculation during training based on the size of the target, as exemplified below:

[0073]

[0074]

[0075] As mentioned above, when the target is small (norm_Areagt is less than a certain threshold), the loss calculation considers the IOU (intersection area of ​​the boxes divided by the union area of ​​the boxes) between the predicted and ground truth boxes, the proportion of the difference between the minimum bounding rectangle and the two boxes, and the standard center distance between the two boxes. The latter two comprehensively measure the distance between the predicted and ground truth boxes. When the target is large, in addition to considering IOU and the distance between boxes, the loss calculation also incorporates an aspect ratio factor v to characterize the difference in length and width between the predicted and ground truth boxes. The formula for calculating v restricts the difference in length and width to the range [0, 1], ensuring that the loss function is differentiable.

[0076] Step S103: If the identification result is a false identifier, confirm the user identity information associated with the multimedia information.

[0077] In one embodiment, when a preset identifier contained in multimedia information is identified as a fake identifier, the user identity information associated with the multimedia information is further verified. Optionally, for cases where the user is confirmed to be a legitimate user, the fake identifier contained in the multimedia information can be replaced with a genuine identifier through a fake identifier content processing module. If a preset identifier is a watermark, the watermark can be removed.

[0078] As described above, by acquiring the multimedia information to be identified and the basic information associated with it, a preset content filtering module identifies the multimedia information and basic information using preset identifiers to obtain the identification result. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with it. The classifier's training process includes at least three model training stages. In cases where the identification result is a false identifier, the user identity information associated with the multimedia information is verified. This multimedia information identification method achieves efficient identification of multimedia information through the cascaded multi-size identifier detection model and classifier. The classifier training is based on multiple different model training stages, which improves the accuracy of multimedia information identification.

[0079] Based on the above technical solution, for cases where multimedia information contains faces, the solution further includes: performing face detection using a preset face recognition model, and matching the face detection results with the face images of authenticated users recorded in the database to obtain a matching result. The face recognition model includes a feature extraction layer and a one-way pyramid module. The one-way pyramid module uses bilinear interpolation for upsampling. The feature extraction layer includes a CBM convolutional layer, a CBL convolutional layer, and a CSP module. The activation function corresponding to the CBM convolutional layer is the Mish activation function, and the activation function corresponding to the CBL convolutional layer is the LeakyReLU activation function. The two-way multi-scale pyramid module includes a spatial pyramid network layer, a deconvolutional layer, and a CSP module. For example, as shown... Figure 10 and Figure 11 As shown, Figure 10 This is a schematic diagram of the structure of a face recognition model provided in an embodiment of this application. Figure 11This is a schematic diagram illustrating the training of a face recognition model provided in this application embodiment. Compared to a multi-size marker detection model, it eliminates the detection regression module, retaining only the one with the highest resolution. Since the aspect ratio and size variation range of a face are relatively small, and extreme differences in target size are rare, eliminating the other two detection regression modules with lower resolutions helps reduce computational overhead while ensuring that the performance does not significantly decrease. Furthermore, an adaptive anchor box candidate network is no longer used to generate the initial coordinates of the anchor boxes; instead, a method of manually setting anchor boxes is used to assist in prediction. This is also due to the relatively fixed shape of the face bounding box, and the anchor box settings provided by existing experience are sufficient. The feature pyramid of this face recognition model adopts a unidirectional structure, and a convolutional layer is added before referencing the feature map with higher resolution to alleviate the underfitting of shallow networks. When restoring the size of the feature pyramid, a parameter-independent bilinear interpolation method is used for upsampling, isolating the upsampling process to accelerate the convergence speed of model training. Furthermore, when training the face recognition model, this embodiment uses CIOU loss, which is the loss function for large targets in the loss formula explained in step S1027 above, and no longer distinguishes based on the size of the regression target. This face recognition model does not fit to a specific category, but adds a classification prediction of whether each prediction box contains a face object. The loss function uses ArcFace loss to train the model to handle the distinction of a large number of clusters. After training the face recognition model, its feature extraction layer is copied and used to train the face feature extraction model. The network structure of this feature extraction layer is consistent with the structure of the feature extraction layer in the aforementioned multi-size identifier detection model. The copied feature extraction layer is connected to several lightweight CSP modules and an SPP module for transition, enhancing the fitting of features. The final output feature map group undergoes max pooling operation channel by channel to obtain the image feature vector. Optionally, metric learning is used for loss calculation here. For each input pair of face images, the model performs two forward propagations followed by one backward propagation. Each forward propagation calculates the feature vector of one image, and the feature vectors of the image pair are used to calculate the L2 loss between them. Image pairs with close feature distances have smaller loss values, and vice versa. The purpose of this method is to ensure that the face embeddings extracted by the model are as close as possible to faces of the same type and as far away as possible from dissimilar faces in the feature space. Furthermore, before being combined into face image pairs, each face image undergoes 60 simple data augmentations to improve its compatibility with the original image. Figure 1The dataset is composed of 60 image pairs representing positive samples, image pairs consisting of different faces of the same person are also considered positive samples, and image pairs consisting of other faces of different people are considered negative samples. This data augmentation constructs an imbalanced dataset, therefore the L2 loss used to train the model is organized in the form of Focal Loss. The trained face recognition model takes a single face image as input and outputs a face feature vector. In practical use, the detection result of the face recognition model is used as input. After obtaining the feature vector, it is compared with the face features in the face feature database based on cosine similarity. If the cosine similarity is greater than a certain threshold, it is considered to be the same face, providing support for subsequent user identity verification.

[0080] Figure 12 This is a structural block diagram of a multimedia information recognition device provided in an embodiment of this application. The device is used to execute the multimedia information recognition method provided in the above embodiments, and has corresponding functional modules and beneficial effects for executing the method. For example... Figure 12 As shown, the device specifically includes: an information acquisition module 101, a recognition result determination module 102, and an information determination module 103, wherein,

[0081] The information acquisition module 101 is configured to acquire multimedia information to be identified and basic information associated with the multimedia information;

[0082] The recognition result determination module 102 is configured to identify the multimedia information and the basic information by performing preset identification on the preset identifiers through a preset content filtering module to obtain the recognition result. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with the multi-size identifier detection model. The training process of the classifier includes at least three model training stages.

[0083] The information determination module 103 is configured to confirm the user identity information associated with the multimedia information when the identification result is a false identifier.

[0084] As described above, by acquiring the multimedia information to be identified and the basic information associated with it, a pre-defined content filtering module identifies the multimedia information and basic information using pre-defined identifiers to obtain the identification result. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with it. The classifier's training process includes at least three model training stages. In cases where the identification result is a false identifier, the user identity information associated with the multimedia information is verified. This multimedia information identification method achieves efficient identification of multimedia information through the cascaded multi-size identifier detection model and classifier. The classifier training is based on multiple different model training stages, which improves the accuracy of multimedia information identification.

[0085] In one possible embodiment, the multi-size marker detection model includes a feature extraction layer, a bidirectional multi-scale pyramid module, and an adaptive anchor box candidate network. The recognition results include the location detection results and marker classification results of the preset markers at multiple different scales.

[0086] In one possible embodiment, the feature extraction layer includes a CBM convolutional layer, a CBL convolutional layer, and a CSP module. The activation function corresponding to the CBM convolutional layer is the Mish activation function, the activation function corresponding to the CBL convolutional layer is the LeakyReLU activation function, and the bidirectional multi-scale pyramid module includes a spatial pyramid network layer, a deconvolutional layer, and a CSP module.

[0087] In one possible embodiment, the adaptive anchor box candidate network includes a grouped convolutional layer, a first CSP module, a second CSP module, a CBM convolutional layer, and a CBL convolutional layer, wherein the first CSP module is constructed based on a residual module, and the second CSP module is constructed based on a network layer stack.

[0088] In one possible embodiment, the training process of the classifier includes a first training phase, wherein the dataset used in the first training phase is obtained by a data augmentation method using a set multi-image fusion coefficient and a multi-dimensional image arrangement, and the corresponding loss function is a binary cross-entropy loss function.

[0089] In one possible embodiment, the device further includes a model training module 104 configured as follows:

[0090] A preset number of images are randomly selected from the original dataset. The preset number of images includes positive sample images and negative sample images. The positive sample images are images that contain a preset identifier and are the main body of the image. The negative sample images are images that do not contain the preset identifier.

[0091] The positive sample image is obtained by randomly scaling the positive sample image, and the negative sample image is obtained by randomly scaling and cropping the negative sample image.

[0092] The positive sample enhanced image and the negative sample enhanced image are randomly arranged to obtain a combined image;

[0093] The dataset corresponding to the first training phase is generated based on the original dataset and the combined images.

[0094] In one possible embodiment, the training process of the classifier includes a second training phase. The dataset used in the second training phase is obtained through a set adaptive label scaling processing method. The model used in the second training phase includes a second feature extraction layer and a multi-scale pooling feature module. The multi-scale pooling feature module is used to extract feature map groups from the first feature layer, the second feature layer and the third feature layer in the second feature extraction layer, respectively, and to calculate feature vectors from the feature map groups through a set CSPP module.

[0095] In one possible embodiment, the adaptive identifier scaling processing method includes:

[0096] Images with preset label categories are randomly selected from the original dataset as foreground images, and the foreground images are scaled according to the set scaling factor.

[0097] Get an image that does not contain a preset identifier as the background image;

[0098] If the sum of the areas of the foreground images and the area of ​​the background image satisfy the image embedding conditions, the foreground images are embedded into the background images to obtain an embedded image.

[0099] In one possible embodiment, the training process of the classifier includes a third training phase, the dataset used in the third training phase includes an imbalanced dataset, in which negative sample data is greater than positive sample data, and the model structure used in the third training phase includes a bidirectional spatial pyramid structure.

[0100] In one possible embodiment, the device further includes an identification result determination module 102, configured to:

[0101] After the multimedia information and the basic information are identified by a preset identifier through a preset content filtering module to obtain the identification result, if the multimedia information contains a face, a face detection is performed by a preset face recognition model, and the face detection result is matched with the face image of the authenticated user recorded in the database to obtain the matching result.

[0102] In one possible embodiment, the face recognition model includes a feature extraction layer and a one-way pyramid module. The one-way pyramid module uses a bilinear interpolation algorithm for upsampling. The feature extraction layer includes a CBM convolutional layer, a CBL convolutional layer, and a CSP module. The activation function corresponding to the CBM convolutional layer is the Mish activation function, and the activation function corresponding to the CBL convolutional layer is the LeakyReLU activation function. The bidirectional multi-scale pyramid module includes a spatial pyramid network layer, a deconvolutional layer, and a CSP module.

[0103] Figure 13 This is a schematic diagram of the structure of a multimedia information recognition device provided in an embodiment of this application, such as... Figure 13 As shown, the device includes a processor 201, a memory 202, an input device 203, and an output device 204; the number of processors 201 in the device can be one or more. Figure 13 Taking a processor 201 as an example; the processor 201, memory 202, input device 203, and output device 204 in the device can be connected via a bus or other means. Figure 13 Taking a bus connection as an example, the memory 202, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimedia information recognition method in this embodiment. The processor 201 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 302, thereby realizing the aforementioned multimedia information recognition method. The input device 203 can be used to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 204 may include a display screen or other display device.

[0104] This application also provides a non-volatile storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a multimedia information recognition method described in the above embodiments, wherein the method includes:

[0105] Obtain the multimedia information to be identified and the basic information associated with the multimedia information;

[0106] The multimedia information and the basic information are identified by a preset content filtering module to obtain the identification result. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with the multi-size identifier detection model. The training process of the classifier includes at least three model training stages.

[0107] If the identification result is a false identifier, the user identity information associated with the multimedia information is verified.

[0108] It is worth noting that in the above embodiments of the multimedia information recognition device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.

[0109] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device can execute the multimedia information recognition method described in the embodiments of this application. The program product can be implemented using any combination of one or more readable media.

Claims

1. A multimedia information recognition method, characterized in that, include: Obtain the multimedia information to be identified and the basic information associated with the multimedia information; The multimedia information and the basic information are identified by a preset content filtering module, which includes a multi-size identifier detection model and a classifier cascaded with the multi-size identifier detection model. The classifier training process includes at least three model training stages. The multi-size identifier detection model includes a bidirectional multi-scale pyramid module, which has deconvolution layers to expand and restore the feature maps. The classifier training process includes a first training stage, where the dataset used in the first training stage is obtained through a data augmentation process using set multi-image fusion coefficients and multi-dimensional image permutations. A data augmentation processing method for arranging numbers and multidimensional images includes: randomly obtaining a preset number of images from the original dataset, the preset number of images including positive sample images and negative sample images, the positive sample images being images containing a preset identifier and being the main body of the image, and the negative sample images being images not containing the preset identifier; randomly scaling the positive sample images to obtain positive sample augmented images; randomly scaling and randomly cropping the negative sample images to obtain negative sample augmented images; randomly arranging the positive sample augmented images and the negative sample augmented images to obtain combined images; and generating a dataset corresponding to the first training stage based on the original dataset and the combined images. If the identification result is a false identifier, the user identity information associated with the multimedia information is verified.

2. The multimedia information recognition method according to claim 1, characterized in that, The multi-size marker detection model also includes a feature extraction layer and an adaptive anchor box candidate network. The recognition results include the location detection results and marker classification results of the preset markers at multiple different scales.

3. The multimedia information recognition method according to claim 2, characterized in that, The feature extraction layer includes a CBM convolutional layer, a CBL convolutional layer, and a CSP module. The activation function corresponding to the CBM convolutional layer is the Mish activation function, and the activation function corresponding to the CBL convolutional layer is the LeakyReLU activation function. The bidirectional multi-scale pyramid module includes a spatial pyramid network layer, a deconvolutional layer, and a CSP module.

4. The multimedia information recognition method according to claim 2, characterized in that, The adaptive anchor box candidate network includes a grouped convolutional layer, a first CSP module, a second CSP module, a CBM convolutional layer, and a CBL convolutional layer. The first CSP module is constructed based on a residual module, and the second CSP module is constructed based on a stacked network layer structure.

5. The multimedia information recognition method according to claim 1, characterized in that, The loss function corresponding to the first training phase is the binary cross-entropy loss function.

6. The multimedia information recognition method according to claim 1, characterized in that, The model used in the second training phase includes a second feature extraction layer and a multi-scale pooling feature module. The multi-scale pooling feature module is used to extract feature map groups from the first feature layer, the second feature layer and the third feature layer in the second feature extraction layer, and to calculate feature vectors from the feature map groups through the set CSPP module.

7. The multimedia information recognition method according to claim 1, characterized in that, The training process of the classifier includes a third training phase. The dataset used in the third training phase includes an imbalanced dataset, in which negative samples are greater than positive samples. The model structure used in the third training phase includes a bidirectional spatial pyramid structure.

8. The multimedia information recognition method according to any one of claims 1-7, characterized in that, After obtaining the recognition result by using a preset content filtering module to identify the multimedia information and the basic information using preset identifiers, the method further includes: When the multimedia information contains a face, a face is detected using a preset face recognition model, and the face detection result is matched with the face image of the authenticated user recorded in the database to obtain a matching result.

9. The multimedia information recognition method according to claim 8, characterized in that, The face recognition model includes a feature extraction layer and a one-way pyramid module. The one-way pyramid module uses a bilinear interpolation algorithm for upsampling. The feature extraction layer includes a CBM convolutional layer, a CBL convolutional layer, and a CSP module. The activation function corresponding to the CBM convolutional layer is the Mish activation function, and the activation function corresponding to the CBL convolutional layer is the LeakyReLU activation function. The bidirectional multi-scale pyramid module includes a spatial pyramid network layer, a deconvolutional layer, and a CSP module.

10. A multimedia information recognition device, characterized in that, include: The information acquisition module is configured to acquire multimedia information to be identified and basic information associated with the multimedia information; The recognition result determination module is configured to obtain a recognition result by recognizing preset identifiers on the multimedia information and the basic information through a preset content filtering module. The content filtering module includes a multi-size identifier detection model and a classifier cascaded with the multi-size identifier detection model. The training process of the classifier includes at least three model training stages. The multi-size identifier detection model includes a bidirectional multi-scale pyramid module, which contains deconvolution layers to expand and restore the feature maps. The training process of the classifier includes a first training stage, where the dataset used in the first training stage is obtained through a data augmentation process using set multi-image fusion coefficients and multi-dimensional image permutations. The data augmentation processing method for multi-image fusion coefficients and multi-dimensional image arrangement includes: randomly obtaining a preset number of images from the original dataset, the preset number of images including positive sample images and negative sample images, the positive sample images being images containing a preset identifier and being the main body of the image, and the negative sample images being images not containing the preset identifier; randomly scaling the positive sample images to obtain positive sample augmented images; randomly scaling and randomly cropping the negative sample images to obtain negative sample augmented images; randomly arranging the positive sample augmented images and the negative sample augmented images to obtain combined images; and generating a dataset corresponding to the first training stage based on the original dataset and the combined images. The information determination module is configured to confirm the user identity information associated with the multimedia information if the identification result is a false identifier.

11. A multimedia information recognition device, the device comprising: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the multimedia information recognition method according to any one of claims 1-9.

12. A non-volatile storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform the multimedia information recognition method according to any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimedia information recognition method according to any one of claims 1-9.

Images