A method and device for identifying a logo pattern in media content

Abstract

This application discloses an artificial intelligence-based method for training a recognition model to identify symbols in media content, a media content recognition method, and related apparatus. For the symbols of the object to be identified, training samples can be constructed through addition. Since the addition method can identify the location where the symbol is added to the image, or the way the content of the symbol is changed when it is added, the training samples constructed in this way can encompass various possible forms of adding the symbol to the image. The generation of training samples reduces the dependence on web crawling and manual labeling, and can quickly generate a large number of high-quality training samples based on the addition method, all of which include accurate labels. Therefore, the recognition model obtained by supervised training with these training samples can accurately identify various variations of the symbol in media content, even for newly generated symbols from the network.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a recognition model training method for identifying identification patterns in media content, a media content recognition method and related devices. Background Art

[0002] In the Internet age, a large amount of media content is uploaded to the Internet through various channels for dissemination. Due to the dissemination of media content, in some cases, the generator or uploader of media content will use media content to promote certain objects such as marketing accounts, products, software, etc. to increase the exposure of the objects.

[0003] A common way to promote an object is to add a logo pattern of the object to be promoted in the media content, such as a QR code, software icon, marketing account logo, etc. pointing to the object. Through different adding forms, the logo pattern is displayed during the playback and display of the media content.

[0004] In order to avoid spreading bad content through logo patterns, it is necessary to accurately identify logo patterns of media content in the network to identify media content carrying logo patterns. However, since logo patterns have many variations, it is very difficult to identify logo patterns. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a method and device for identifying identification patterns in media content, so that the training samples include as many training samples as possible in which the identification patterns exist in various deformed forms, as well as training samples in which the identification patterns are in various added positions, thereby improving the accuracy of the recognition model trained with the training samples in identifying the identification patterns in media content.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a recognition model training method for recognizing a logo pattern in media content, the method comprising:

[0008] Obtaining a logo pattern of an object to be identified;

[0009] Adding the identification pattern to the image in different adding forms as training samples;

[0010] Training a recognition model using the training samples, the recognition model being used to recognize identification patterns in media content;

[0011] The addition form is used to identify any one or more combinations of the following:

[0012] Adding a location for adding the identification pattern in the image;

[0013] The content of the identification pattern is changed.

[0014] On the other hand, an embodiment of the present application provides a method for identifying media content, the method comprising:

[0015] Selecting a plurality of image frames from the media content to be identified as a frame sequence;

[0016] respectively determining position information of identification patterns in the plurality of image frames;

[0017] determining, according to position information of adjacent image frames in the frame sequence, whether identification patterns in the adjacent image frames partially overlap;

[0018] If the partial overlap ratio of the identification patterns of the target adjacent image frames meets the threshold, it is determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

[0019] On the other hand, an embodiment of the present application provides a recognition model training device for recognizing identification patterns in media content, the device comprising an acquisition unit, an adding unit and a training unit:

[0020] The acquisition unit is used to acquire the identification pattern of the object to be identified;

[0021] The adding unit is used to add the identification pattern to the image in different adding forms as a training sample;

[0022] The training unit is used to train a recognition model using the training samples, wherein the recognition model is used to recognize identification patterns in media content;

[0023] The addition form is used to identify any one or more combinations of the following:

[0024] Adding a location for adding the identification pattern in the image;

[0025] The content of the identification pattern is changed.

[0026] On the other hand, an embodiment of the present application provides a media content recognition device, the device comprising a selection unit, a first determination unit, a second determination unit and a third determination unit:

[0027] The selection unit is used to select a plurality of image frames from the media content to be identified as a frame sequence;

[0028] The first determining unit is used to respectively determine the position information of the identification patterns in the multiple image frames;

[0029] The second determining unit is used to determine whether the identification patterns in the adjacent image frames partially overlap according to the position information of the adjacent image frames in the frame sequence;

[0030] The third determination unit is configured to determine that an image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern if a partial overlap ratio of identification patterns of target adjacent image frames meets a threshold.

[0031] On the other hand, an embodiment of the present application provides a device, the device comprising a processor and a memory:

[0032] The memory is used to store program code and transmit the program code to the processor;

[0033] The processor is used to execute the above-mentioned recognition model training method or the above-mentioned media content recognition method according to the instructions in the program code.

[0034] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the above-mentioned recognition model training method or the above-mentioned media content recognition method.

[0035] It can be seen from the above technical solution that for the identification pattern of the object to be identified, the training sample can be constructed by adding. Since the adding form can identify the adding position of the identification pattern in the image, or the way to change the content of the identification pattern when adding the identification pattern, the training sample constructed in this way can include various possible forms of adding the identification pattern in the image. The generation of training samples has reduced dependence on network crawling and manual identification, and a large number of high-quality training samples can be quickly generated according to the adding form, and all include accurate labels, so that the recognition model obtained by supervised training with this training sample can accurately identify the identification pattern in various changes in the media content even for the identification pattern newly generated in the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 A schematic diagram showing a media content with a logo of "AA Video Software" added to it;

[0038] Figure 2 A schematic diagram of an application scenario of a recognition model training method provided in an embodiment of the present application;

[0039] Figure 3 A flow chart of a recognition model training method for recognizing identification patterns in media content provided in an embodiment of the present application;

[0040] Figure 4 An example diagram of a training sample for generating a training sample based on a content change method provided in an embodiment of the present application;

[0041] Figure 5 A flow chart of a media content identification method provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of a scene for identifying a dynamic identification pattern provided in an embodiment of the present application;

[0043] Figure 7 A structural diagram of a watermark recognition system provided in an embodiment of the present application;

[0044] Figure 8 A structural diagram of a recognition model training device provided in an embodiment of the present application;

[0045] Fig. 9 A structural diagram of a media content identification device provided in an embodiment of the present application;

[0046] Fig.10 A data processing device structure diagram is provided for an embodiment of the present application;

[0047] Fig.11 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0049] At present, it is common to add an object identification pattern such as a QR code, software icon, marketing account logo, etc. to media content so that the identification pattern can be displayed and disseminated during the playback and display of the media content. Figure 1 The schematic diagram shown is a display diagram in which a logo pattern of the object "AA video software" is added to the media content.

[0050] In order to avoid spreading bad content through logo patterns, it is necessary to identify logo patterns on media content. However, some media content generators and uploaders will change the logo patterns to be added in various forms in order to avoid identification, which makes it very difficult to identify the logo patterns.

[0051] The recognition model training method provided in the embodiment of the present application and the media content recognition method introduced later can be implemented based on artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0052] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0053] In the embodiments of the present application, the artificial intelligence technologies mainly involved include the above-mentioned computer vision technology, natural language processing technology, machine learning and other directions.

[0054] For example, it may involve image processing (Image Processing), image semantic understanding (Image Semantic Understanding, ISU), video processing (Video Processing), video semantic understanding (Video Semantic Understanding, VSU), 3D object reconstruction (3D object reconstruction), face recognition, etc. in computer vision (Computer Vision).

[0055] For example, it may involve text preprocessing and semantic understanding in natural language processing (NLP), including word / sentence segmentation, word tagging, word / sentence classification, etc.

[0056] For example, it may involve deep learning in machine learning (ML), including various types of artificial neural networks.

[0057] To this end, an embodiment of the present application provides a recognition model training method, and the recognition model obtained by training through this method can accurately recognize various forms of identification patterns in image frames of media content.

[0058] The recognition model training method provided in the present application can be executed by a data processing device, which can be a terminal device or a server. Among them, the terminal device can be, for example, a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, a point of sales (POS), a car computer and other devices. The server can be an independent server or a server in a cluster.

[0059] The data processing equipment can have the ability to implement computer vision technology. Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify targets, trace traces and measure machine vision, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0060] In an embodiment of the present application, a data processing device may use computer vision technology to obtain image frames in a media content such as a video, identify pattern content embodied by an identification pattern in an image frame, and identify pattern content embodied by an identification pattern from an image frame of the media content.

[0061] The data processing device may also have the ability to implement natural language processing, which is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes technologies such as text processing and semantic understanding.

[0062] In the embodiment of the present application, the data processing device implements the above-mentioned NLP technology to determine the corresponding information type of the identification pattern in the media content.

[0063] The data processing device may have machine learning (ML) capabilities. ML is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks.

[0064] In an embodiment of the present application, the data processing device can identify identification patterns in media content image frames, and expand training samples by means of data enhancement, etc.

[0065] In order to facilitate understanding of the technical solution of the present application, the recognition model training method provided in the embodiment of the present application is introduced below with the server as the execution entity and combined with the actual application scenario.

[0066] See also Figure 2 , which shows a schematic diagram of an application scenario of a recognition model training method provided by an embodiment of the present application. Figure 2 As shown, the scene includes a server 201, and the server 201 executes the recognition model training method. In the embodiment of the present application, the server 201 can obtain the identification pattern of the object to be identified, and quickly and automatically generate a large number of training samples according to different addition forms. Since the addition form identifies the location of the added identification pattern and the content change method, the training samples determined based on the addition form all have training labels related to the identification pattern. Based on this type of training samples, a high-quality recognition model for the identification pattern can be trained.

[0067] The identification pattern involved in the embodiment of the present application may be a new one that appears on the network. Due to the short propagation time of the identification pattern, the number of training samples that can be crawled and detected is very small, resulting in insufficient training samples to train the recognition model in a targeted manner, making the identification pattern less accurate when recognized by the original recognition model. Based on this situation, the solution provided by the present application can automatically generate sufficient training samples based on the recognition pattern to train the recognition model, so that the newly appeared identification pattern can be accurately recognized in a short time.

[0068] The identification pattern involved in the embodiment of the present application may not be newly appeared on the network. In this case, although sufficient training samples can be obtained for the identification pattern by crawling the network and manually annotating, it will consume a lot of manpower and network resources. Based on this situation, the solution provided by the present application can automatically generate sufficient training samples based on the recognition pattern to train the recognition model, without the need to determine the actual recognition results from the network as training samples, thereby improving the model training efficiency while ensuring the recognition accuracy.

[0069] In the embodiments of the present application, the objects to be identified may be the aforementioned objects involving inappropriate content, or may be objects unrelated to the media content platform, or may be objects that need to be specified for identification, etc.

[0070] exist Figure 2 In the scenario shown, the server 201 may actively obtain the identification pattern of the object to be identified under the above requirements, or may report the identification pattern of the object to be identified to the server 201 after the terminal device detects it.

[0071] The identification pattern of the object to be identified has the function of uniquely or clearly identifying the object to be identified, and has various possible forms of expression, such as the logo of the object to be identified, a QR code, a software icon, a marketing account logo, a website, a user ID, etc. It is understandable that the identification pattern can be text or an icon, or a combination of text and an icon.

[0072] For example, Figure 2 As shown, the object to be identified is, for example, "AA video software", and the identification pattern of the object to be identified "AA video software" can be a pattern composed of the text "AA video" and a "gray circle" icon.

[0073] The server 201 may automatically construct training samples according to the addition form based on the acquired identification pattern of the object to be identified.

[0074] In actual scenarios, the party that generates or uploads media content usually adds the logo pattern of the object to be promoted or disseminated in various ways to avoid recognition. Generally speaking, the ways to avoid recognition of the added logo pattern mainly include: adding the logo pattern to various positions in the media content image, and adding the logo pattern to the media content after processing it in various content-changing ways.

[0075] Therefore, in order to ensure that the training samples can include identification patterns that exist in various possible forms, when constructing the training samples, the server 201 can add identification patterns to the images in various different adding forms, and add labels related to the identification patterns to the images to generate training samples. The identification patterns in the training samples can exist in the training samples in the form of digital watermarks.

[0076] In an embodiment of the present application, the adding form can be used to identify various forms of adding identification patterns to an image, and the adding form can identify any one or more combinations of the following: the adding position of the identification pattern in the image, and the way the content of the identification pattern is changed.

[0077] When adding a logo pattern to an image, if the adding form identifies the adding position of the logo pattern in the image, the logo pattern can be added to the image according to the adding position indicated by the adding form. If the adding form identifies the content changing mode of the logo pattern, the content display form of the logo pattern can be changed according to the content changing mode indicated by the adding form to obtain logo patterns displayed in various forms, and various logo patterns deformed by the content changing mode can be added to the image.

[0078] It should be noted that the content change method in the embodiment of the present application can be a change in the content display form of the logo pattern, and will not change the substantial content embodied by the logo pattern. In other words, no matter what content change form is used to add the logo pattern, the object to be identified by the logo pattern will not change. It should be noted that if the addition form only includes the content change form, then when adding the logo pattern to the image according to the addition form, it can be added to a random position in the image.

[0079] The training samples constructed in the above manner are guaranteed to cover as many training samples as possible in which the logo pattern exists in various deformation forms, and the training samples in which the logo pattern is in various added positions.

[0080] For example, see Figure 2, the server 201 can adopt different adding positions to add the identification pattern of the object to be identified "AA video software" to the upper left position M and the lower right position N of the image respectively, and add training labels of the adding positions of the identification pattern in the image, i.e., the upper left position M and the lower right position N, and training labels related to the content (such as the substantial content "AA video" embodied by the identification pattern) to the image to generate training samples. In addition, the identification pattern of the object to be identified "AA video software" can be changed in content such as display color, and the changed identification pattern can be added to the image, and training labels related to the content (such as the substantial content "AA video" embodied by the identification pattern) of the identification pattern can be added to the image to generate training samples.

[0081] Finally, the server 201 may apply the training samples constructed by the above method to perform model training to obtain a recognition model.

[0082] In this method, training samples are constructed by adding different identification patterns to ensure that various possible forms of adding identification patterns to images are covered, and the generation of training samples has reduced dependence on web crawling and manual identification. A large number of high-quality training samples can be quickly generated based on the adding form, and the training samples all include accurate labels, so that the recognition model obtained by supervised training using these training samples can accurately determine the identification patterns that exist in various forms in the media content.

[0083] In addition, if the newly generated identification pattern in the network is used to construct training samples, the training samples obtained by the above method also include training samples added in various forms of the newly generated identification pattern. In this way, it is ensured that the trained recognition model can accurately recognize such newly generated identification patterns in various forms in the media content.

[0084] Next, the recognition model training method provided in the embodiment of the present application will be introduced using the server as the execution entity.

[0085] See also Figure 3 , which shows a flow chart of a recognition model training method for recognizing identification patterns in media content provided by an embodiment of the present application, and the method may include:

[0086] S301: Acquire the identification pattern of the object to be identified.

[0087] In an embodiment of the present application, according to actual scenarios and application requirements, the server can determine objects that need to be identified from objects promoted and disseminated in the network as objects to be identified, and obtain identification patterns of the objects to be identified to construct training samples.

[0088] It should be noted that the embodiment of the present application does not limit the method for obtaining the identification pattern, such as the identification pattern can be collected on the network. In a specific implementation, the server can obtain an image that carries the identification pattern and includes an alpha channel, and the image carrying the identification pattern can be an image in a portable network graphics (PNG) format. In this way, when adding the identification pattern to the image in S302, the alpha channel of these images carrying the identification pattern can be adjusted to correspond to a transparent state, and then these images carrying the identification pattern are added to the image for generating the training sample, thereby ensuring that the training sample only includes the identification pattern, and does not include the rest of the image carrying the identification pattern.

[0089] S302: adding logo patterns to the image in different adding forms as training samples.

[0090] In actual scenarios, the party that generates or uploads media content usually adds logo patterns to the media content in various ways to avoid identifying the logo patterns of the object to be promoted or disseminated. The ways to avoid identifying the added logo patterns mainly include: adding the logo patterns to various positions in the media content image, and adding the logo patterns to the media content after processing them in various content-changing ways.

[0091] In an embodiment of the present application, in order to ensure that the constructed training samples include identification patterns in various forms as much as possible, thereby improving the identification pattern recognition accuracy of the trained recognition model, when constructing the training samples, different addition forms can be used to add identification patterns to the image as training samples.

[0092] The adding form may be used to identify any one or more of the following combinations: the adding position of the identification pattern in the image, and the way of changing the content of the identification pattern.

[0093] That is, when adding a logo pattern to an image, if the adding form identifies the adding position of the logo pattern in the image, the logo pattern can be added to the image according to the various adding positions indicated by the adding form. If the adding form identifies the content changing mode of the logo pattern, the content display form of the logo pattern can be changed according to the various content changing modes indicated by the adding form to obtain the logo pattern displayed in various forms, and the logo patterns deformed by the various content changing modes can be added to the image.

[0094] In the embodiment of the present application, in order to ensure that the training samples include as many logo patterns as possible with various content change modes, thereby improving the accuracy of the recognition model in identifying logo patterns processed with various content change modes. In a possible implementation, when constructing the training samples, the content change mode identified by the addition form may include any one or more of the following combinations: a logo pattern clarity change mode; a logo pattern size change mode; a logo pattern font change mode; and a logo pattern color change mode.

[0095] That is, when adding the identification pattern of the object to be identified to the image based on the content change method, the clarity, size, font, and color of all or part of the content in the identification pattern can be changed, and then the changed identification pattern is added to the image. Figure 4 , which shows an example diagram of a training sample for generating training samples based on a content change method provided by an embodiment of the present application, such as Figure 4 As shown, Figure 4 The training sample in a is constructed by reducing the definition of the logo pattern; Figure 4 The training sample in b is constructed by increasing the size of the logo pattern in the height direction; Figure 4 The training samples in c are constructed by changing the font type in the logo pattern; Figure 4 The training samples in d are constructed by changing the color of the logo pattern.

[0096] Need to explain, for Figure 4 The training samples in the above are generated by adding the logo patterns deformed by different content change methods to the same adding position in the image, that is, only based on the content change method, without involving the adding position. In actual scenarios, when constructing training samples, training samples can be constructed for both the content change method and the adding position. For example, the logo patterns deformed by different content change methods can be added to different adding positions in the image to generate training samples.

[0097] In this way, the training samples include the training samples after the logo pattern is processed in different content change modes, thereby ensuring that the recognition model accurately recognizes the logo pattern in different content presentation forms in the media content.

[0098] In addition, in some scenarios, the technician needs to determine the information type of the identified identification pattern and perform subsequent corresponding processing operations according to the information type. To this end, in a possible implementation, when constructing a training sample in S302, the training sample also includes the information type of the added identification pattern. Among them, the information type can be a type determined based on a certain aspect of information embodied by the identification pattern (such as information embodied based on the content of the added identification pattern itself). For example, the information type may include the diversion type for guiding users to consume and use the object identified by the identification pattern, etc.

[0099] In actual scenarios, the information type of the identification pattern can be flexibly determined according to specific scenarios or application requirements and added to the training samples. For example, if it is necessary to identify and eliminate the identification pattern of the diversion type in the media content, the information type of the identification pattern can be determined as the diversion type and the non-diversion type when constructing the training sample, so as to ensure that the recognition model trained by this training sample can accurately identify whether the identification pattern in the media content is of the diversion type.

[0100] It should be noted that in actual scenarios, different addition forms may be used to construct training samples for the acquired identification patterns. Based on the different addition forms, the information types of the identification patterns added in the training samples may be different. For example, when the identification pattern is added to the image after reducing its clarity to obtain a training sample, the training sample may be added with the information type of "blurred identification pattern". When the identification pattern with a changed font color is added to the image to obtain a training sample, the training sample may be added with the information type of "changed identification pattern font color". Therefore, the training samples constructed based on different addition forms for the same identification pattern in the training samples may have the same or different identification pattern information types according to actual needs.

[0101] By determining the corresponding information type for the added identification pattern in the training sample, it is ensured that the recognition model trained by the training sample can also recognize the information type of the identification pattern in the media content, which facilitates relevant technical personnel to perform further subsequent processing of the media content according to the information type of the identified identification pattern.

[0102] In a specific implementation, images in the ImageNet network test set can be used as images for constructing training samples. The ImageNet network test set is a large visualization database that can be used to study visual objects, which includes a large number of high-quality images covering various types. By using the images in the ImageNet network test set to construct training samples to train the recognition model, the generalization ability of the model training can be improved.

[0103] In addition, when the server uses different adding forms to add the logo pattern to the image, when the adding form identifies the adding position of the logo pattern in the image, the probability of adding the logo pattern in each corner of the image (adding position) can be increased, and when the adding form identifies the content change method of the logo pattern, the logo pattern can be changed such as expanding the size (Scale) of the logo pattern, changing the font of the text in the logo pattern, changing the color of the logo pattern, and changing such as reducing the clarity of the logo pattern to make the logo pattern blurred (Blur), and then adding the logo pattern changed by the content change method to the image. In addition, the number of training samples can be increased by data augmentation.

[0104] S303: training a recognition model through training samples, where the recognition model is used to recognize identification patterns in image frames of media content.

[0105] In the embodiment of the present application, the media content that needs to be recognized by the recognition model can be content that conveys information to users by display, such as videos, images, and documents. Based on the feature that media content can convey information to users, the generator or uploader of the media content can add the identification pattern of the object that needs to be promoted and disseminated to the media content to realize the promotion and dissemination function.

[0106] It should be noted that the identification pattern in the media content may be added during the process of producing the media content, or added after the production of the media content is completed, or even exists in the media content itself. Regardless of how the identification pattern exists in the media content, the recognition model in the embodiment of the present application can accurately recognize it.

[0107] Among them, when the recognition model recognizes media content, for video media content, the recognition model can be used to recognize the identification pattern in each video frame of the video; for image media content, the recognition model can be used to recognize the identification pattern in each image; for document media content, the recognition model can be used to recognize the animated images, images included in the document, and the identification pattern in the background of the document, etc.

[0108] It can be seen from the above technical solution that for the identification pattern of the object to be identified, the training sample can be constructed by adding. Since the adding form can identify the adding position of the identification pattern in the image, or the way to change the content of the identification pattern when adding the identification pattern, the training sample constructed in this way can include various possible forms of adding the identification pattern in the image. The generation of training samples has reduced dependence on network crawling and manual identification, and a large number of high-quality training samples can be quickly generated according to the adding form, and all include accurate labels, so that the recognition model obtained by supervised training with this training sample can accurately identify the identification pattern in various changes in the media content even for the identification pattern newly generated in the network. At the same time, the recall rate of identification pattern recognition is also improved.

[0109] It is understandable that new identification patterns will continuously appear in the network to promote the identified objects. In order to improve the recognition accuracy of the recognition model for such identification patterns, in a possible implementation, the method may also include:

[0110] The recognition model is trained iteratively using training samples constructed from recognized logo patterns.

[0111] The identified identification pattern may include any one or more of the following combinations:

[0112] The identification pattern that is identified and reported, and the identification pattern that is identified by the recognition model during the media content recognition process.

[0113] The identified and reported identification pattern may be an identification pattern obtained by being identified and reported in various ways. For example, a user reports (reports) after seeing an identification pattern in the displayed media content. The identified and reported identification pattern may be obtained from a failure instance (Bad Case) reported by the user, or may be detected and fed back (reported) by professionals from media content disseminated on the Internet, etc., which will not be elaborated.

[0114] It should be noted that the embodiment of the present application does not limit the method for obtaining the identified and reported identification pattern, and a suitable method can be selected according to the actual scenario. For example, if the Bad Case reported by the user is a video frame, the identification pattern in the video frame can be obtained from the video frame.

[0115] As for the identification pattern recognized by the recognition model in the process of media content recognition, it can be understood as the identification pattern recognized by the recognition model in the process of recognizing various media contents after the training of the recognition model is completed.

[0116] Therefore, training samples can be constructed using the above-mentioned recognized identification patterns to perform training iterations on the recognition model.

[0117] It should be noted that the embodiments of the present application are not limited to the method of constructing training samples through the above-mentioned recognized identification patterns. Preferably, the training samples can be constructed through the above-mentioned S301-S302 methods to ensure as much as possible that rich and high-quality training samples are constructed, that is, the constructed training samples include the above-mentioned recognized identification patterns in various variations.

[0118] In this way, by training the recognition model iteratively using the training samples constructed by the recognized identification pattern, the recognition accuracy of the iteratively trained recognition model for various identification patterns in the network can be improved.

[0119] It should also be noted that, since the timing of obtaining the above-mentioned recognized identification pattern is random and uncertain, the identification pattern that is recognized and reported may be obtained by irregular reporting by the user.

[0120] Therefore, the recognized identification patterns can be obtained according to a specific iteration cycle, and training samples can be constructed based on them, and then the recognition model can be trained and iterated based on the constructed training samples.

[0121] The embodiment of the present application does not limit the iteration cycle, and the iteration cycle can be determined according to actual scenarios and different needs. For example, in order to facilitate iteration, the iteration cycle can be a preset time period, so that a training sample can be constructed based on the recognized identification patterns obtained in each time period, and then the recognition model can be trained and iterated. For another example, in order to perform iterative training after obtaining enough recognized identification patterns, the iteration cycle can be determined by a preset threshold value of the number of recognized identification patterns, and each time the number of recognized identification patterns obtained reaches a preset threshold number of recognized identification patterns, a training sample is constructed and training iterations are performed. And so on, no further elaboration is given.

[0122] In some scenarios, in order to fully publicize and spread the object to be promoted, the logo pattern produced for the desired promotional object includes not only the icon of the object, but also text that can fully reflect the information of the object, so that users who see the logo pattern can accurately identify the desired promotional object. In other logo patterns, only text that reflects the information of the desired promotional object may be included.

[0123] Therefore, in order to improve the recognition accuracy of the identification model for the identification pattern, in a possible implementation method, if the addition form in the above S301 is at least used to identify the adding position of the identification pattern in the image, that is, according to the adding position indicated by the addition form, the identification pattern is added to the image to construct a training sample, and the training sample includes a training label for identifying the adding position of the identification pattern in the image. Then, the recognition model trained in S303 may include a position recognition sub-model for identifying the position of the identification pattern, and a content recognition sub-model for identifying the content of the identification pattern. The method of training the recognition model through training samples in the above S303 may include the following two training methods:

[0124] The first training method includes: training the location recognition sub-model and the content recognition sub-model in parallel through training samples.

[0125] The position identification sub-model can identify the position information of the identification pattern in the media content, and the content identification sub-model can identify the content embodied by the identification pattern in the media content.

[0126] Since the training samples include training labels for identifying the added position of the identification pattern in the image and training labels for identifying the content embodied by the identification pattern, the training samples can be used to separately train the position recognition sub-model and the content recognition sub-model.

[0127] In an embodiment of the present application, the location recognition sub-model and the content recognition sub-model may be trained in parallel through training samples.

[0128] In this way, the location recognition sub-model and the content recognition sub-model can be trained simultaneously through training samples, which reduces the model training time and improves the model training efficiency.

[0129] The second training method includes: training the location recognition sub-model through training samples, and training the content recognition sub-model according to the model output of the location recognition sub-model and the corresponding training samples during the training process.

[0130] It can be understood that the training samples are obtained by adding identification patterns to the images, that is, the training samples include both the images and the identification patterns. Therefore, in order to avoid the content recognition sub-model being affected by the images in the training samples when recognizing the content of the identification patterns in the training samples, the position recognition sub-model can be trained through the training samples. During the training process, the content recognition sub-model is trained based on the model output of the position recognition sub-model, that is, the position information of the identification pattern in the training samples and the corresponding training samples.

[0131] By performing model training in this way, since the position recognition sub-model can identify the position information of the identification pattern in the training sample, when training the content recognition sub-model, the content recognition sub-model can be trained to recognize the content embodied by the identification pattern at the position information in the training sample based on the model output of the position recognition sub-model, that is, the position information of the identification pattern in the training sample, thereby improving the recognition accuracy of the content recognition sub-model.

[0132] In a specific implementation, the location recognition sub-model may be a Single Shot MultiBox Detector (SSD) model, and the content recognition sub-model may be an Optical Character Recognition (OCR) model. SSD may be used to detect the location information of a target object. OCR may be a technology that determines the shape of an object by detecting dark and light patterns, and then translates the shape into computer text using a character recognition method.

[0133] It can be understood that when constructing training samples, if the addition form identifies the content change method, various sizes of logo patterns can be present in the constructed training samples. In order to enable the training recognition model to accurately identify the logo patterns of various sizes in the training samples, when applying the SSD model to identify the added position of the logo pattern in the training sample, the position recognition can be performed in a multi-scale manner. By analyzing and counting the size data of historical logo patterns, the prior knowledge of the logo pattern size is obtained, and prior frames of different sizes are constructed to adapt to logo patterns of various sizes. Among them, the prior frame can be used to capture the area in the image.

[0134] A dynamic identification pattern that can change dynamically is propagated in the network. For example, an identification pattern whose position can change dynamically. The dynamic identification pattern usually achieves dynamic change based on multiple consecutive image frames by making the positions of the identification patterns in adjacent image frames overlap at a high ratio and displaying the multiple image frames continuously.

[0135] In actual scenarios, such dynamic identification patterns are not specifically identified, but are identified one by one based on the identification patterns in each image frame, thereby being unable to determine the relevance of the identification patterns in the image frames of the media content.

[0136] To this end, the present application embodiment provides a method for identifying media content, see Figure 5 , which shows a flow chart of a media content identification method provided by an embodiment of the present application, such as Figure 5 As shown, the method may include:

[0137] S501: Selecting a plurality of image frames from the media content to be identified as a frame sequence.

[0138] It can be understood that since the dynamic logo pattern in the media content has the characteristics of time sequence and continuity, in multiple continuous image frames of the media content, the logo patterns in adjacent image frames have a high overlap ratio. Further, based on this characteristic of the dynamic logo pattern, that is, the logo patterns in adjacent image frames have a high overlap ratio, it can be identified whether the media content includes the dynamic logo pattern.

[0139] In the embodiment of the present application, in order to improve the efficiency of identifying the dynamic identification pattern, it is not necessary to identify the complete media content, but only multiple image frames can be selected from the media content as a frame sequence, and the frame sequence can be identified to see whether it includes the dynamic identification pattern. The frame sequence can be a sequence of multiple image frames arranged based on temporal characteristics.

[0140] It should be noted that the embodiment of the present application does not limit the method of selecting multiple image frames from the media content to be identified as a frame sequence, and a suitable method can be selected according to different actual scenarios and application requirements.

[0141] In a possible implementation, the method of selecting a plurality of image frames from the media content to be identified as a frame sequence in S501 includes:

[0142] A plurality of image frames are selected from the media content as a frame sequence by means of interval frame extraction.

[0143] The interval frame extraction method may refer to a method of extracting frames according to specific intervals such as time intervals, frame number intervals, etc. The interval frame extraction method may be equal intervals or unequal intervals.

[0144] In the embodiment of the present application, multiple image frames can be conveniently selected from the media content as a frame sequence by this simple and reliable interval frame extraction method. For example, frame extraction can be performed by this method when the media content is a video and the duration of the video is short.

[0145] Alternatively, multiple image frames may be selected from the media content as a frame sequence according to key frames in the media content.

[0146] Among them, the key frame can be an image frame that embodies key or important information or content in the media content. For example, for the media content of video, the key frame can be a scene switching frame in which the brightness of the video changes significantly.

[0147] Since key frames reflect key information of media content and are easier to be viewed by users, key frames of media content are more likely to have identification patterns. Thus, when selecting image frames from media content as a frame sequence, multiple image frames can be selected as a frame sequence according to the key frames in the media content.

[0148] In addition, when the media content is a video and the video is long, determining the frame sequence by interval frame extraction will result in an excessive number of image frames in the frame sequence, increasing the amount of calculation for determining the dynamic identification pattern based on the frame sequence. Therefore, in this case, it is preferred to determine the frame sequence by selecting image frames through key frames.

[0149] By using this method of determining the frame sequence through key frames, it can be ensured that the frame sequence is more likely to contain the identification pattern, thereby improving the accuracy of subsequent recognition of the dynamic identification pattern.

[0150] S502: Determine position information of identification patterns in a plurality of image frames respectively.

[0151] The position information of the identification pattern in the image frame can identify the area covered by the identification pattern in the image frame.

[0152] S503: Determine whether identification patterns in adjacent image frames partially overlap according to position information of adjacent image frames in the frame sequence.

[0153] In the embodiment of the present application, adjacent image frames in a frame sequence may be determined from the frame sequence according to the temporal characteristics of the image frames.

[0154] It can be understood that, based on the timing and continuity characteristics of the dynamic identification pattern, if the frame sequence includes a dynamic identification pattern, the identification patterns in adjacent image frames of the frame sequence should overlap at least partially. Thus, in the embodiment of the present application, in order to identify whether the frame sequence includes a dynamic identification pattern, it can be determined whether the identification patterns in the adjacent image frames in the frame sequence partially overlap based on the identification patterns of the adjacent image frames.

[0155] It should be noted that the partial overlap of identification patterns in adjacent image frames in the frame sequence may refer to the partial or complete overlap of areas between identification patterns in adjacent image frames.

[0156] S504: If the partial overlap ratio of the identification patterns of the target adjacent image frames meets the threshold, it is determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

[0157] In the embodiment of the present application, any adjacent image frame in the frame sequence can be used as the target image frame, and the partial overlap ratio of the identification pattern of the target adjacent image frame can be determined. The partial overlap ratio of the identification pattern of the target adjacent image frame can be understood as the ratio of the overlap area size of the identification pattern in the target adjacent image frame and the coverage area size of the identification pattern in the target adjacent image frame (any of the image frames), and the partial overlap ratio of the identification pattern of the target adjacent image frame can be the ratio of the former to the latter.

[0158] It should be noted that adjacent image frames in a frame sequence are not necessarily adjacent image frames in the media content. For example, the media content includes 50 image frames, and the frame sequence determined by equally spaced frame extraction with a frame interval of 10 frames includes the 1st image frame, the 11th image frame, the 21st image frame, the 31st image frame, and the 41st image frame. In this example, the i+1th image frame and the i+11th image frame are adjacent image frames in the frame sequence, but are not adjacent image frames in the media content, i=0,1,2,3.

[0159] The threshold value may be used to measure whether the partial overlap ratio of the identification patterns of adjacent image frames in the frame sequence meets the partial overlap ratio range of the dynamic identification patterns in the adjacent image frames.

[0160] In actual scenarios, the threshold can be flexibly set according to the method of determining the frame sequence. For example, when the frame sequence is determined by a method of equally spaced frames with a small interval, a higher threshold can be set, and when the frame sequence is determined by a method of equally spaced frames with a large interval, a lower threshold can be set.

[0161] Then, it can be determined whether the partial overlap ratio of the identification patterns of the target adjacent image frames meets the threshold. If the partial overlap ratio of the identification patterns of the target adjacent image frames meets the threshold, it can be determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

[0162] The image frame interval corresponding to the target adjacent image frames in the media content can be understood as all image frames between (including) the target adjacent image frames in the media content. For example, based on the above example, if the target adjacent image frames are the 1st image frame and the 11th image frame, the image frame interval corresponding to the target adjacent image frames in the media content can be all image frames between the 1st image frame and the 11th image frame in the media content.

[0163] In this method, multiple image frames can be selected from the media content to be identified as a frame sequence. Then, the position information of the identification pattern in the multiple image frames is determined respectively. Based on the continuity and timing characteristics of the dynamic identification pattern, the identification patterns in adjacent image frames in the frame sequence will partially overlap, thereby determining whether the identification patterns of adjacent image frames in the frame sequence partially overlap. If for any adjacent image frame in the frame sequence, that is, the target adjacent image frame, the partial overlap ratio of its identification pattern meets the threshold, that is, it meets the characteristics of the dynamic identification pattern, then it can be determined that the image frame interval corresponding to the target adjacent image frame in the media content has a dynamic identification pattern.

[0164] It can be seen that by executing this method, the dynamic identification pattern and the image frame interval including the dynamic identification pattern can be determined from the media content. In this way, it is convenient to process the dynamic identification pattern in the image frame interval at the same time, without processing frame by frame, thereby improving the processing efficiency of the identification pattern.

[0165] In the embodiment of the present application, in order to further improve the recognition accuracy of the dynamic identification pattern, in a possible implementation manner, the method further includes:

[0166] S601: Determine the pattern contents of the identification patterns in the target adjacent image frames respectively.

[0167] The pattern content of the logo pattern can identify the information displayed by each component of the logo pattern. For example, if the logo pattern includes two components, an icon and text, the identification information of the logo pattern can identify the information displayed by the icon and text.

[0168] In order to improve the efficiency and accuracy of determining the pattern content of the identification pattern in the target adjacent image frame in S601, in a possible implementation, the pattern content of the identification pattern may be determined by using a recognition model.

[0169] In the embodiment of the present application, the recognition model may include a position recognition sub-model for recognizing the position of the identification pattern and a content recognition sub-model for recognizing the content of the identification pattern. The training methods of the position recognition sub-model and the content recognition sub-model are as described above and will not be repeated here. The method for respectively determining the position information of the identification pattern in the multiple image frames in S502 may include:

[0170] The position information of the identification pattern in the multiple image frames is determined respectively by the position recognition sub-model.

[0171] Then, the method of respectively determining the pattern contents of the identification patterns in the target adjacent image frames in S601 may include:

[0172] According to the position information determined by the position recognition sub-model and the target adjacent image frame, the pattern contents of the identification pattern in the target adjacent image frame are respectively determined by the content recognition sub-model.

[0173] For the target adjacent image frame, if the position recognition sub-model recognizes the position information of the identification pattern for it respectively, the target adjacent image frame and the corresponding position information can be input into the content recognition sub-model so that the content recognition sub-model can recognize the pattern content of the identification pattern in the target adjacent image frame respectively.

[0174] The method of identifying the pattern content of the identification pattern by using a recognition model can improve the accuracy and efficiency of identifying the pattern content of the identification pattern.

[0175] S602: Determine whether identification patterns in target adjacent image frames belong to the same target identification pattern according to pattern content.

[0176] Since the pattern content of the dynamic identification image in multiple image frames is the same, after determining the pattern content of the identification pattern in the target adjacent image frame, it can be determined based on the pattern content whether the identification patterns in the target adjacent image frame belong to the same identification pattern, namely the target identification pattern.

[0177] Then, the method for determining in S504 that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern may include:

[0178] S603: If the identification patterns in the target adjacent image frames belong to the same target identification pattern and the partial overlap ratio of the identification patterns in the target adjacent image frames meets a threshold, it is determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

[0179] In a specific implementation, the media content identification method can be executed by a data processing device such as a terminal device or a server, see Figure 6 , which shows a schematic diagram of a scene for identifying a dynamic identification pattern provided by an embodiment of the present application. Figure 6 As shown, for the media content of the video, a frame sequence can be determined therefrom, and the frame sequence includes image frame 1 (the first image frame), image frame 3 (the third image frame), and image frame 5 (the fifth image frame) in the video. Then, the position information 1 (corresponding to the dotted frame area in image frame 1) and the position information 3 (corresponding to the dotted frame area in image frame 3) of the identification pattern in image frame 1 and image frame 3 are determined respectively. Among them, image frame 5 does not include the identification pattern, and thus the position information is not identified therefrom.

[0180] Next, for the target adjacent image frames including image frame 1 and image frame 3, the pattern content 1 "AA video and circular icon" and pattern content 3 "AA video and circular icon" of their identification patterns can be determined respectively. Finally, based on position information 1 and position information 3, it can be determined that the partial overlap ratio of the identification patterns in the target adjacent image frames meets the threshold, and based on pattern content 1 and pattern content 3, it can be determined that the identification patterns in the target adjacent image frames belong to the same target identification pattern. Thus, it is determined that the image frame interval corresponding to the target adjacent image frames in the video (i.e., the first to third image frames in the video) has a dynamic identification pattern.

[0181] Based on the position condition of the dynamic identification pattern in adjacent image frames in the frame sequence, that is, the partial overlapping ratio of the identification pattern in the target adjacent image frames meets the threshold, and the content condition, that is, the identification patterns in the target adjacent image frames belong to the same target identification pattern, the frame sequence is double-recognized to ensure the accuracy of identifying the dynamic identification pattern.

[0182] In the embodiment of the present application, the identification pattern may be identified for any image frame in the media content, that is, the target image frame. Therefore, the method may further include:

[0183] S701: Determine target position information of a marking pattern in a target image frame by using a position recognition sub-model.

[0184] S702: Determine the target pattern content of the identification pattern in the target image frame through the content recognition sub-model according to the target position information and the target image frame.

[0185] The target position information and the target pattern content may be position information and pattern content determined for the target image frame.

[0186] The method of determining the target position information and target pattern content for the target image frame through the position recognition sub-model and the content recognition sub-model is the same as the aforementioned method of determining the position information and pattern content for the target adjacent image frame through the position recognition sub-model and the content recognition sub-model, and will not be repeated here.

[0187] S703: Determine a recognition result corresponding to the target image frame according to the target position information and the target pattern content.

[0188] In a specific implementation, the recognition result may include that the image frame includes the identification pattern, the image frame does not include the identification pattern, and it is uncertain whether the image frame includes the identification pattern.

[0189] By detecting identification patterns of media content using a recognition model including a position recognition sub-model and a content recognition sub-model, the recognition efficiency and accuracy of the identification patterns are improved.

[0190] In a specific implementation, the identification pattern may be an identification pattern in the form of a digital watermark in the media content. Digital watermark is a technology used to protect digital products. The digital watermark is embedded in the digital product to represent the information of the copyright owner to prevent other teams from misappropriating the digital product.

[0191] Digital watermarks include spatial watermarks, which can be added by directly changing the pixels of the original image. This type of watermark is visible. Generally speaking, spatial watermarks are used more frequently, such as in common images.

[0192] In addition, the media content identified by the application recognition model can be articles, videos, images, etc. The article can be, for example, an article recommended by the application software (Application, APP) for users to read. The article can include animated pictures, images, etc. The article is usually an article that the self-media platform user actively edits and publishes on the network after registration. The video can be, for example, a video recommended by the APP for users to watch. The video can include short videos in vertical and horizontal versions. The media content can exist in the form of a message source (Feeds) stream.

[0193] Among them, the short videos mentioned above are a form of Internet content dissemination, which are generally disseminated on new Internet media and are video dissemination content with a duration of less than 5 minutes. With the popularization of mobile terminals and the acceleration of the Internet, short, flat, fast and high-traffic content has gradually gained the favor of major platforms, fans and capital.

[0194] Among them, the above-mentioned Feeds are also translated as source material, feed, information provision, feed, summary, source, news subscription, network source, etc. It is a data format through which websites can spread the latest information to users. When recommending information, websites usually arrange it in a timeline. Timeline is the most primitive and basic display form of Feeds. The prerequisite for users to subscribe to a website may be that the website provides Feeds and uses the software for aggregation, namely the aggregator, to converge the Feeds in one place to achieve aggregation. For users, the aggregator can be software specifically used to subscribe to websites, generally called information aggregation (Really Simple Syndication, RSS) readers, feed readers, news readers, etc.

[0195] Next, a watermark recognition system provided by an embodiment of the present application is introduced by taking a video as the media content and a logo pattern in the form of a digital watermark in the video as an example. For the convenience of description, the logo pattern in the form of a digital watermark is referred to as a watermark in the following. Figure 7, which shows a watermark recognition system structure diagram provided by an embodiment of the present application, such as Figure 7 As shown, the watermark recognition system 700 may include a watermark recognition service 701, a watermark recognition model 702, a frame extraction service 703, a video deduplication service 704, a transcoding module 705, a scheduling center service 706, a manual review system 707, a video content storage service 708, a download file system 709, a content database 710, an upstream and downstream content interface service 711, a content distribution export service 712, a video content production end 713 and a video content consumption end 714.

[0196] The watermark recognition system is introduced in detail below.

[0197] The watermark recognition service 701 can be used to realize the service of the watermark recognition model 702 (i.e., the aforementioned recognition model) in the system, and can be used to be scheduled by the scheduling center service 706 on the video processing link to perform watermark recognition, and then call the transcoding module 705 for subsequent video processing according to the product strategy.

[0198] For the watermark recognition model 702, the training samples for watermark recognition can be collected and constructed according to the aforementioned training sample construction method, and data enhancement of the training samples can be performed. By integrating the SSD model and the OCR model, a recognition model for identifying watermarks can be constructed. In addition, the training samples can be updated based on the Bad Cases of various watermarks reported and fed back by users of the video content consumption end 714, so as to iteratively train the watermark recognition model 702.

[0199] For the frame extraction service 703, it is to perform primary processing of the features of the video files downloaded from the video content storage service 708 by the download file system 709. In addition, for videos of different durations, it is impossible to use the equal-interval frame extraction strategy to determine the frame sequence with the same number of frames, and when the number of image frames in the frame sequence is too large, the burden and calculation amount of frame extraction will be increased, and the calculation cost will increase sharply. Therefore, based on the multiple key frames included in the video, the frame extraction service 703 can extract the scene switching frames with obvious brightness changes in the video, i.e., the key frames, as the frame sequence, so as to perform dynamic watermark recognition in the video.

[0200] For the video deduplication service 704, it can extract video features and audio features from the video file to construct the fingerprint features of the video, and perform video deduplication through the fingerprint features to reduce the amount of subsequent video processing. For duplicate videos, they can be directly deleted from the video storage content service 708.

[0201] The transcoding module 705 can communicate with the watermark identification service 701 to obtain the location information and information type of the watermark. Then, the video content can be re-encoded, the watermark at the location of the watermark can be removed, and other watermarks or blanks can be re-written at the location of the watermark according to business needs, and the processed video content can be written into the video content storage service 708.

[0202] The dispatch center service 706 can be used to perform the entire dispatch process of video content flow, receive the video content stored in the content database 710 through the uplink and downlink content interface service 711, and then obtain the meta information of the video from the content database 710. It can also be used to dispatch the manual review system 707 and the machine processing system to control the order and priority of the dispatch. For video content, it can first communicate with the video deduplication service 704 so that the video deduplication service 704 can process the different bit rates, different resolutions, different sizes, partial black screens, filters, watermarks, inserting advertising content in similar video content, and cutting the opening and ending of the video accordingly, so as to effectively generate video content entering the manual review process. Dispatch the video content entering the manual review system 707 for manual review to filter out duplicate videos. Enable the content distribution export service 712 for the video content that passes the manual review system 707, so as to provide it to the terminal video content consumers 714 through the recommendation engine, search engine, and display pages directly related to the operation. In addition, the watermark identification service 701 is called to identify various watermarks in the video. According to the actual product strategy, the identified watermark can be directly filtered without subsequent circulation. For example, for videos with high quality and only watermark problems, the transcoding module can be called to blur the location of the identified watermark, remove the watermark, and enable the content distribution export service 712 for subsequent distribution process.

[0203] The manual review system 707 can be a complex system developed based on a network database, which can read the original information of the video content itself in the content database 710, and can manually conduct a preliminary review and filter out pornographic, gambling, and sensitive content in the video. On the basis of the preliminary review, the video content is reviewed twice, and the secondary review is mainly to classify, label or confirm the video. Since the method of classifying and labeling video content completely through machine learning, such as deep learning, is not yet fully mature, it is necessary to conduct two manual review processes on the basis of machine processing, and improve the accuracy of the video labeling itself through human-machine collaboration.

[0204] It is understandable that, based on the premise of a substantial increase in the amount of video uploads, in order to ensure the security of videos, it is necessary to complete the review of videos in a short period of time, such as whether the content involves pornography, gambling, drugs, sensitivity, and quality and safety identification and processing. The current main way is to use a large amount of manpower and assist with machine algorithm capabilities. The following introduces the current distribution process of short videos. The process of short videos from the beginning of uploading to successful uploading and then successfully entering user consumption includes: shooting videos through terminal shooting tools, and then uploading through terminals or public account platforms. In the process of video uploading, the video will be re-transcoded, the video will be standardized, the video metadata will be saved, and the video playback compatibility on various platforms will be improved. Then, the video will be manually reviewed. At the same time as the manual review, the machine will also use algorithms to obtain some auxiliary features of the video, such as classification, labeling, etc.; then, based on the machine algorithm processing, manual standardized annotation will be performed to fill the video with relevant information, such as video labels, categories, celebrity information, etc. This is the video standardization process.

[0205] The video content storage service 708 is usually a group of storage servers that are widely distributed and very close to the users. Usually, a content delivery network (CDN) acceleration server is also included in the periphery for distributed cache acceleration. The video content uploaded by the video content production end 713 can be saved through the upstream and downstream content interface service 711. After obtaining the index information of the video, the video content consumption end 714 can also directly access the video content storage service 708 to download the corresponding video. In addition to being a data source for external services, it can also be used as a data source for internal services, for the download file system 709 to obtain the original video content for related processing. Among them, the paths of the internal and external data sources are usually deployed separately to avoid mutual influence.

[0206] For the download file system 709, the original video is downloaded and obtained from the video content storage service 708, and the download speed and progress are controlled, which usually includes a group of parallel servers with related task scheduling and distribution clusters. The frame extraction service 703 can obtain the necessary key frames from the video source file for the completed downloaded video as the subsequent construction of the watermark recognition model 702 and training samples.

[0207] For the content database 710, it is the core database of video content, and the meta information of all video content production terminals 713 released videos is stored in this business database. The meta information file size, cover image link, bit rate, file format, title, release time, author, and the classification and label of the video during the manual review process (which can include one, two, and three level classifications and labels) of the video are stored. After the upstream and downstream content interface service 711 stores the released video on the video content storage service 708, the video content storage service 708 will perform standard transcoding operations on the video. After the transcoding is completed, the meta information of the video returned asynchronously mainly includes file size, bit rate, specifications, and cover screenshots. These meta information will be stored in the content database 710. The information in the content database 710 will be read during the manual review process, and the results and status of the manual review will also be returned to the content database. The processing of the video by the scheduling center service 706 mainly includes machine processing and manual review processing. Here, the machine processing is to call the video deduplication service 704, and the result of deduplication will be written into the content database 710. Completely repeated videos will not be repeated at the manual review place for secondary processing.

[0208] After the video is reviewed, it will enter the content database 710 of the video platform. In this process, the machine algorithm model will capture the cover image of the video, evaluate and calculate the video quality and clarity. Finally, it will be directly distributed to the operator to the external network or to the recommendation engine. Based on the user's portrait features, recommendation algorithms such as collaborative recommendation, matrix decomposition, supervised learning algorithm logistic regression (Logistic Regression) model, deep learning-based model, factorization machine (Factorization Machine, FM) and gradient boosting decision tree (Gradient Boosting Decision Tree, GBDT) are used. At the same time, in order to encourage the creation of videos, each short video platform has relevant subsidies and incentive mechanisms for videos. At the same time, the threshold for video creation itself will be very high, requiring good creativity and shooting. In order to increase their own income, video creators will upload and move a large number of similar (simple editing and modification of videos, such as video titles, watermarks or editing and cropping, adding advertising titles and endings, modifying audio, etc.) or repeated videos. Since the videos need to be manually reviewed, manual review requires a lot of cost on the one hand, and the efficiency is not enough on the other hand. With the rapid increase in the amount of videos, the costs are very high.

[0209] The uplink and downlink content interface service 711 can communicate directly with the video content production end 713, wherein the video of the video content production end 713 is usually the title, publisher, summary, cover image, and release time of the video, or the shot video directly enters the service end through the server, and the video is stored in the video content storage service 708. The meta information of the video, such as the video file size, cover image link, bit rate, file format, title, release time, and author, is written into the content database 710. The uploaded file is submitted to the scheduling center service 706 for subsequent video processing and circulation.

[0210] The content distribution export service 712 is used to provide the video content to the terminal consumers 714 through the recommendation engine, search engine and display pages directly related to the operation according to the video activation instructions of the manual review system 707.

[0211] For the video content production end 713, that is, the content producers of User Generated Content (UGC), Professional Generated Content (PGC), Professional User Generated Content (PUGC) or Multi-Channel Network (MCN), they provide local or filmed videos through the application programming interface (API) system of the mobile terminal or the back-end interface. These are the main video sources for distributing content. Through communication with the uplink and downlink content interface service 711, the upload server interface address is first obtained, and then the local file is uploaded. During the shooting process, the local video can choose matching music, filter templates and video beautification functions, etc.

[0212] Among them, PGC is used to refer to content with personalized content, diversified perspectives, and virtualized social relationships. It is also called professionally produced content (PPC).

[0213] PUGC is professional audio content produced in the form of UGC that is relatively close to PGC.

[0214] MCN is a product form of a multi-channel network that unites PGCs and, with the strong support of capital, ensures the continuous output of content, thereby ultimately achieving stable commercial monetization.

[0215] In actual scenarios, short videos, as a type of video, integrate themes such as skill sharing, humor, fashion trends, social hot spots, street interviews, public welfare education, advertising creativity, and commercial customization. Because the video is short, it can be a separate film or a series of columns. Unlike micro-films and live broadcasts, short video production does not have specific expression forms and team configuration requirements like micro-films. It has the characteristics of simple production process, low production threshold, strong participation, etc., and has more communication value than live broadcasts. The ultra-short production cycle and interesting content pose certain challenges to the copywriting and planning skills of the short video production team. Excellent short video production teams usually rely on mature self-media or Internet Protocol (IP). In addition to high-frequency and stable content output, they also have powerful fan channels; the emergence of short videos has enriched the form of new media native advertising.

[0216] At present, short videos have been rising from the initial UGC, PGC, and PUGC video content production end 713 upload, to organizations specializing in short video production, to MCN, and to professional short video apps and many other traffic platforms. Short videos have become one of the important ways of dissemination for content entrepreneurship and social media platforms. While short videos have triggered a carnival among video entrepreneurs and impacted video media platforms, their influence has been further upgraded, and major information platforms have also launched a battle around short videos. Therefore, various short videos are becoming more and more numerous and richer. Both producers and consumers of short videos have become a huge group.

[0217] For the video content consumption end 714, it communicates with the upstream and downstream content interface service 711 to obtain the index information for accessing the video file, and then communicates with the video content storage service 708 to download the corresponding streaming media file and play it through the local player. At the same time, the user's playback behavior data, freezes, loading time, playback clicks, etc. during the upload and download process are reported to the server. The video content consumption end 714 usually browses video data through the Feeds stream, so duplicate video files will have a great impact on the user experience. If there are many watermarks or watermark diversions in the video, it will greatly affect the viewing experience, and the corresponding Bad Case will be reported or fed back to the watermark recognition system 700.

[0218] Based on the recognition model training method provided in the above embodiment, the present application embodiment provides a recognition model training device for recognizing identification patterns in media content, see Figure 8 , which shows a structural diagram of a recognition model training device 800 provided in an embodiment of the present application, wherein the device includes an acquisition unit 801, an adding unit 802, and a training unit 803:

[0219] The acquisition unit 801 is used to acquire the identification pattern of the object to be identified;

[0220] The adding unit 802 is used to add the identification pattern to the image in different adding forms as a training sample;

[0221] The training unit 803 is used to train a recognition model using the training samples, wherein the recognition model is used to recognize identification patterns in media content;

[0222] The addition form is used to identify any one or more combinations of the following:

[0223] Adding a location for adding the identification pattern in the image;

[0224] The content of the identification pattern is changed.

[0225] In a possible implementation, the training unit 803 is specifically configured to:

[0226] If the adding form is at least used to identify the adding position of the identification pattern in the image, the recognition model includes a position recognition sub-model for identifying the position of the identification pattern and a content recognition sub-model for identifying the content of the identification pattern, and the position recognition sub-model and the content recognition sub-model are trained in parallel through the training samples; or

[0227] The location recognition sub-model is trained by the training samples, and the content recognition sub-model is trained according to the model output of the location recognition sub-model and the corresponding training samples during the training process.

[0228] In a possible implementation manner, the training sample further includes information type of the added identification pattern.

[0229] In a possible implementation, the content changing manner includes any one or more combinations of the following:

[0230] the way in which the clarity of the identification pattern is changed;

[0231] The size change method of the identification pattern;

[0232] A method of changing the font of the logo pattern;

[0233] The color change mode of the identification pattern.

[0234] In a possible implementation, the training unit 803 is specifically configured to:

[0235] The recognition model is trained iteratively through training samples constructed by the recognized identification pattern; the recognized identification pattern includes any one or more combinations of the following:

[0236] The identification pattern that is recognized and reported;

[0237] The identification pattern is identified by the recognition model during the media content recognition process.

[0238] Based on the media content identification method provided in the above embodiment, the present application embodiment provides a media content identification device, see Fig. 9 , which shows a structural diagram of a media content identification device 900 provided in an embodiment of the present application, the device includes a selection unit 901, a first determination unit 902, a second determination unit 903 and a third determination unit 904:

[0239] The selection unit 901 is used to select a plurality of image frames from the media content to be identified as a frame sequence;

[0240] The first determining unit 902 is used to respectively determine the position information of the identification patterns in the multiple image frames;

[0241] The second determining unit 903 is used to determine whether the identification patterns in the adjacent image frames partially overlap according to the position information of the adjacent image frames in the frame sequence;

[0242] The third determining unit 904 is configured to determine that an image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern if a partial overlap ratio of identification patterns of target adjacent image frames meets a threshold.

[0243] In a possible implementation manner, the first determining unit 902 is further configured to:

[0244] respectively determining pattern contents of identification patterns in the target adjacent image frames;

[0245] Determining, according to the pattern content, whether the identification patterns in the target adjacent image frames belong to the same target identification pattern;

[0246] The third determining unit 904 is further specifically configured to:

[0247] If the identification patterns in the target adjacent image frames belong to the same target identification pattern and the partial overlap ratio of the identification patterns of the target adjacent image frames meets a threshold, it is determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

[0248] In a possible implementation manner, the first determining unit 902 is specifically configured to:

[0249] The recognition model includes a position recognition sub-model for recognizing the position of the identification pattern and a content recognition sub-model for recognizing the content of the identification pattern, and the position information of the identification pattern in the multiple image frames is respectively determined by the position recognition sub-model;

[0250] According to the position information determined by the position recognition sub-model and the target adjacent image frame, the pattern content of the identification pattern in the target adjacent image frame is determined respectively by the content recognition sub-model.

[0251] In a possible implementation manner, the first determining unit 902 is specifically configured to:

[0252] The target image frame is any image frame in the media content, and the target position information of the identification pattern in the target image frame is determined by the position recognition sub-model;

[0253] Determining the target pattern content of the identification pattern in the target image frame by the content recognition sub-model according to the target position information and the target image frame;

[0254] The third determining unit 904 is further specifically configured to:

[0255] According to the target position information and the target pattern content, a recognition result corresponding to the target image frame is determined.

[0256] In a possible implementation, the selection unit 901 is specifically configured to:

[0257] Selecting a plurality of image frames from the media content as a frame sequence by means of interval frame extraction; or,

[0258] A plurality of image frames are selected from the media content as a frame sequence according to key frames in the media content.

[0259] It can be seen from the above technical solution that for the identification pattern of the object to be identified, the training sample can be constructed by adding. Since the adding form can identify the adding position of the identification pattern in the image, or the way to change the content of the identification pattern when adding the identification pattern, the training sample constructed in this way can include various possible forms of adding the identification pattern in the image. The generation of training samples has reduced dependence on network crawling and manual identification, and a large number of high-quality training samples can be quickly generated according to the adding form, and all include accurate labels, so that the recognition model obtained by supervised training with this training sample can accurately identify the identification pattern in various changes in the media content even for the identification pattern newly generated in the network.

[0260] The present application also provides a data processing device, which is described below in conjunction with the accompanying drawings. Fig.10 As shown, the embodiment of the present application provides a structural diagram of a data processing device 1000, and the device 1000 may also be a terminal device. For example, the terminal device is a mobile phone:

[0261] Fig.10 FIG. 1 is a block diagram of a portion of the structure of a mobile phone provided in an embodiment of the present application. Fig.10 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Fig.10 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0262] Combine the following Fig.10 A detailed introduction to the various components of the mobile phone:

[0263] The RF circuit 1010 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Usually, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, referred to as LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (Global System of Mobile communication, referred to as GSM), General Packet Radio Service (General Packet Radio Service, referred to as GPRS), Code Division Multiple Access (Code Division Multiple Access, referred to as CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, referred to as WCDMA), Long Term Evolution (Long Term Evolution, referred to as LTE), email, Short Messaging Service (SMS), etc.

[0264] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0265] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0266] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Fig.10 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0267] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0268] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0269] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Fig.10 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0270] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0271] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0272] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0273] In this embodiment, the processor 1080 included in the terminal device also has the following functions:

[0274] Obtaining an identification pattern of an object to be identified;

[0275] Adding the identification pattern to the image in different adding forms as training samples;

[0276] Training a recognition model using the training samples, the recognition model being used to recognize identification patterns in media content;

[0277] The addition form is used to identify any one or more combinations of the following:

[0278] Adding a location for adding the identification pattern in the image;

[0279] The content of the identification pattern is changed.

[0280] Alternatively, in this embodiment, the processor 1080 included in the terminal device further has the following functions:

[0281] Selecting a plurality of image frames from the media content to be identified as a frame sequence;

[0282] respectively determining position information of identification patterns in the plurality of image frames;

[0283] determining, according to position information of adjacent image frames in the frame sequence, whether identification patterns in the adjacent image frames partially overlap;

[0284] If the partial overlap ratio of the identification patterns of the target adjacent image frames meets the threshold, it is determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

[0285] The device for resource display provided in the embodiment of the present application may be a server. Fig.11 As shown, Fig.11 The structural diagram of the server 1100 provided in the embodiment of the present application, the server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1122 (for example, one or more processors) and memory 1132, one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1122 can be configured to communicate with the storage medium 1130 and execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0286] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0287] The steps in the above embodiment may also be executed by a server, which may be based on the Fig.11 The server structure shown.

[0288] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the methods described in the aforementioned embodiments.

[0289] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0290] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0291] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0292] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0293] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0294] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0295] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0296] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0297] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the above-mentioned storage medium can be at least one of the following media: read-only memory (English: read-only memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0298] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0299] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A recognition model training method for identifying a logo pattern in media content, characterized in that: The recognition model includes a position recognition sub-model for recognizing the position of the identification pattern and a content recognition sub-model for recognizing the content of the identification pattern; the method includes: Obtaining an identification pattern of an object to be identified; the identification pattern includes text and icons; The identification pattern is added to the image in different adding forms as training samples; the adding forms are used to identify the adding position of the identification pattern in the image and the content change method of the identification pattern; the training samples include identification patterns of different sizes, training labels for identifying the adding position and training labels for the content embodied by the identification pattern; The position recognition sub-model is trained by the training samples so that the position recognition sub-model outputs the position information of the identification pattern in the training samples, wherein the prior knowledge of the size of the identification pattern is obtained based on the size data of the historical identification pattern, and prior frames of different sizes are constructed based on the prior knowledge to perform position recognition on identification patterns of different sizes; According to the position information of the identification pattern in the training sample output by the position identification submodule during the training process and the corresponding training sample, the content identification submodel is trained to identify the content embodied by the identification pattern at the position information in the training sample, and the identification model is used to identify the identification pattern in the media content.

2. The method according to claim 1, characterized in that: The training sample also includes information type of the added identification pattern.

3. The method according to claim 1, characterized in that The content change method includes any one or more combinations of the following: the way in which the clarity of the identification pattern is changed; The size change method of the identification pattern; A method of changing the font of the logo pattern; The color change mode of the identification pattern.

4. The method according to claim 1, characterized in that: The method further comprises: The recognition model is trained iteratively through training samples constructed by the recognized identification pattern; the recognized identification pattern includes any one or more combinations of the following: The identification pattern that is recognized and reported; The identification pattern is identified by the recognition model during the media content recognition process.

5. A media content identification method, characterized in that: The method comprises: Selecting a plurality of image frames from the media content to be identified as a frame sequence; Respectively determine the position information of the identification pattern in the plurality of image frames; the identification pattern is obtained by identification through a recognition model, the recognition model is obtained by training using the recognition model training method according to any one of claims 1 to 4; the identification pattern includes text and an icon; determining, according to position information of adjacent image frames in the frame sequence, whether identification patterns in the adjacent image frames partially overlap; If the partial overlap ratio of the identification patterns of the target adjacent image frames satisfies the partial overlap ratio range, it is determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

6. The method according to claim 5, characterized in that The method further comprises: respectively determining pattern contents of identification patterns in the target adjacent image frames; Determining, according to the pattern content, whether the identification patterns in the target adjacent image frames belong to the same target identification pattern; The determining that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern includes: If the identification patterns in the target adjacent image frames belong to the same target identification pattern, and the partial overlap ratio of the identification patterns of the target adjacent image frames satisfies the partial overlap ratio range, it is determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

7. The method according to claim 6, characterized in that The recognition model includes a position recognition sub-model for recognizing the position of the identification pattern and a content recognition sub-model for recognizing the content of the identification pattern. The respectively determining the position information of the identification pattern in the plurality of image frames includes: Determine the position information of the identification patterns in the multiple image frames respectively by using the position recognition sub-model; The respectively determining the pattern contents of the identification patterns in the target adjacent image frames comprises: According to the position information determined by the position recognition sub-model and the target adjacent image frame, the pattern content of the identification pattern in the target adjacent image frame is determined respectively by the content recognition sub-model.

8. The method according to claim 7, characterized in that The target image frame is any image frame in the media content, and the method further includes: Determine the target position information of the identification pattern in the target image frame by using the position recognition sub-model; Determining the target pattern content of the identification pattern in the target image frame by the content recognition sub-model according to the target position information and the target image frame; According to the target position information and the target pattern content, a recognition result corresponding to the target image frame is determined.

9. The method according to claim 5, characterized in that The step of selecting a plurality of image frames from the media content to be identified as a frame sequence includes: Selecting a plurality of image frames from the media content as a frame sequence by means of interval frame extraction; or, According to the key frames in the media content, a plurality of image frames are selected from the media content as a frame sequence.

10. A recognition model training device for identifying logo patterns in media content, characterized in that: The recognition model includes a position recognition sub-model for recognizing the position of the identification pattern and a content recognition sub-model for recognizing the content of the identification pattern; the device includes an acquisition unit, an adding unit and a training unit: The acquisition unit is used to acquire the identification pattern of the object to be identified; the identification pattern includes text and icons; The adding unit is used to add the identification pattern to the image in different adding forms as a training sample; The adding form is used to identify the adding position of the identification pattern in the image and the content change mode of the identification pattern; the training samples include identification patterns of different sizes, training labels for identifying the adding position and training labels for the content embodied by the identification pattern; The training unit is used to train the position recognition sub-model through the training samples, so that the position recognition sub-module outputs the position information of the identification pattern in the training samples, wherein the prior knowledge of the size of the identification pattern is obtained based on the size data of the historical identification pattern, and the prior frames of different sizes are constructed based on the prior knowledge to perform position recognition on the identification patterns of different sizes; According to the position information of the identification pattern in the training sample output by the position identification submodule during the training process and the corresponding training sample, the content identification submodel is trained to identify the content embodied by the identification pattern at the position information in the training sample, and the identification model is used to identify the identification pattern in the media content.

11. A media content recognition device, characterized in that: The device comprises a selection unit, a first determination unit, a second determination unit and a third determination unit: The selection unit is used to select a plurality of image frames from the media content to be identified as a frame sequence; The first determining unit is used to respectively determine the position information of the identification patterns in the multiple image frames; The identification pattern is obtained by recognition through a recognition model, and the recognition model is obtained by training using the recognition model training method according to any one of claims 1 to 4; the identification pattern includes text and icons; The second determining unit is used to determine whether the identification patterns in the adjacent image frames partially overlap according to the position information of the adjacent image frames in the frame sequence; The third determination unit is configured to determine that an image frame interval corresponding to the target adjacent image frame in the media content has a dynamic identification pattern if the partial overlap ratio of the identification patterns of the target adjacent image frames satisfies a partial overlap ratio range.

12. The device according to claim 11, characterized in that The first determining unit is further configured to: respectively determining pattern contents of identification patterns in the target adjacent image frames; Determining, according to the pattern content, whether the identification patterns in the target adjacent image frames belong to the same target identification pattern; The third determining unit is further specifically configured to: If the identification patterns in the target adjacent image frames belong to the same target identification pattern, and the partial overlap ratio of the identification patterns of the target adjacent image frames satisfies the partial overlap ratio range, it is determined that the image frame interval corresponding to the target adjacent image frames in the media content has a dynamic identification pattern.

13. A data processing device, characterized in that: The data processing device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the recognition model training method described in any one of claims 1-4 or the media content recognition method described in any one of claims 5-9 according to the instructions in the program code.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the recognition model training method described in any one of claims 1-4 or the media content recognition method described in any one of claims 5-9.

15. A computer program product comprising instructions, characterized in that When it runs on a computer, it enables the computer to execute the recognition model training method described in any one of claims 1 to 4 or the media content recognition method described in any one of claims 5 to 9.

Images