Methods, apparatus, devices, and computer-readable storage media for identifying vulgar images

Abstract

This application provides a method, apparatus, device, and computer-readable storage medium for identifying vulgar images. The method includes: acquiring an image to be identified and context information associated with the image to be identified; extracting features from the image to be identified and the context information to obtain image features and context features; classifying the image to be identified based on the image features and context features to obtain the category to which the image to be identified belongs; determining the image scene of the image to be identified; and determining the vulgarity identification result of the image to be identified based on the category and the image scene. This application can flexibly handle vulgar image identification in different image scenes, improving the efficiency and accuracy of vulgar image identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a method and device for identifying vulgar images, equipment and a computer readable storage medium. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results.

[0003] Computer vision (CV) is one of the important directions of artificial intelligence software technology, which refers to the use of cameras and computers to replace human eyes to identify, detect and measure targets and further process images to make the computer processing more suitable for human observation or transmission to instrument detection.

[0004] In the identification of vulgar images, the related art usually identifies whether an image is vulgar through manual review and marking or through a classification model. The manual review and marking method is highly dependent on manpower, resulting in low efficiency. The method of determining whether an image is vulgar through a classification model cannot flexibly cope with different image scenarios. SUMMARY

[0005] The embodiments of the present application provide a method and device for identifying vulgar images, equipment and a computer readable storage medium, which can improve the efficiency and accuracy of vulgar image identification.

[0006] The technical solution of the embodiments of the present application is as follows:

[0007] The embodiments of the present application provide a method for identifying vulgar images, comprising:

[0008] obtaining a to-be-identified image and context information associated with the to-be-identified image;

[0009] performing feature extraction on the to-be-identified image and the context information respectively to obtain image features and context features;

[0010] classifying the to-be-identified image based on the image features and the context features to obtain a category to which the to-be-identified image belongs;

[0011] determining an image scenario of the to-be-identified image and determining a vulgar identification result of the to-be-identified image based on the category and the image scenario.

[0012] In the above solution, the to-be-identified image is obtained, comprising

[0013] obtaining a video to be identified;

[0014] extracting at least one key frame from the video to be identified as an image to be identified;

[0015] Correspondingly, after determining the obscene identification result of the image to be identified based on the category and the business scenario, the method further comprises:

[0016] taking the obscene identification result of the image to be identified as the obscene identification result of the video to be identified.

[0017] In the above solution, the extracting of at least one key frame from the video content as an image to be identified comprises:

[0018] obtaining a brightness change value between adjacent video frames in the video to be identified;

[0019] determining a scene switching frame of the video to be identified based on the brightness change value between adjacent video frames, and taking the scene switching frame as a key frame;

[0020] extracting at least one scene switching frame as a key frame as an image to be identified.

[0021] In the above solution, the classifying of the image to be identified based on the image feature and the context feature to obtain the category to which the image to be identified belongs comprises:

[0022] classifying the image to be identified based on the image feature to obtain a first category;

[0023] classifying the image to be identified based on the context feature to obtain a second category;

[0024] taking the union set of the first category and the second category as the category to which the image to be identified belongs.

[0025] In the above solution, the determining of the obscene identification result of the image to be identified based on the category and the business scenario comprises:

[0026] when the category to which the image to be identified belongs includes at least two categories, and the at least two categories include a sensitive category, determining the obscene identification result of the image to be identified based on the sensitive category and the business scenario.

[0027] Embodiments of the present application provide an obscene image identification device, comprising:

[0028] an obtaining module configured to obtain an image to be identified and context information associated with the image to be identified;

[0029] extracting a feature of the to-be-identified image and context information to obtain an image feature and a context feature;

[0030] classifying the to-be-identified image based on the image feature and the context feature to obtain a category to which the to-be-identified image belongs;

[0031] determining an image scene of the to-be-identified image, and determining a pornography identification result of the to-be-identified image based on the category and the image scene.

[0032] In the foregoing solution, the obtaining module is further configured to obtain a to-be-identified video.

[0033] extract at least one key frame from the to-be-identified video as a to-be-identified image;

[0034] Correspondingly, the determining module is further configured to take the pornography identification result of the to-be-identified image as a pornography identification result of the to-be-identified video.

[0035] In the foregoing solution, the obtaining module is further configured to obtain a brightness change value between adjacent video frames in the to-be-identified video.

[0036] determine a scene switching frame of the to-be-identified video based on the brightness change value between the adjacent video frames, and take the scene switching frame as a key frame;

[0037] extract at least one scene switching frame taken as a key frame as a to-be-identified image.

[0038] In the foregoing solution, the obtaining module is further configured to obtain text information associated with the to-be-identified image, the text information including at least one of title information, tag information, and text description information of the to-be-identified image.

[0039] take the obtained text information associated with the to-be-identified image as context information associated with the to-be-identified image.

[0040] In the foregoing solution, the obtaining module is further configured to obtain content description information of a video when the to-be-identified image is a frame image in the video.

[0041] take the content description information of the video as context information of the to-be-identified image.

[0042] In the foregoing solution, the extracting module is further configured to obtain an image classification model pre-trained to classify the to-be-identified image.

[0043] In the process of acquiring the image classification model to classify the to-be-identified image, a feature vector output by an intermediate layer of the image classification model is taken as an image feature of the to-be-identified image.

[0044] In the scheme, the extraction module is further configured to perform word segmentation on the text to obtain at least two words when the context information is the text.

[0045] The at least two words are encoded by a word vector model to obtain an encoding feature corresponding to each word.

[0046] The encoding features corresponding to the words are spliced to obtain the context feature.

[0047] In the scheme, the extraction module is further configured to perform feature extraction on each to-be-identified image to obtain a feature corresponding to each to-be-identified image when the number of to-be-identified images is at least two.

[0048] The features of the at least two to-be-identified images are fused to obtain the image feature.

[0049] In the scheme, the classification module is further configured to splice the image feature and the context feature to obtain a spliced feature.

[0050] Based on the spliced feature, the to-be-identified image is subjected to multi-label classification to obtain at least one category to which the to-be-identified image belongs.

[0051] In the scheme, the classification module is further configured to splice the image feature and the context feature to obtain a spliced feature, and input the spliced feature into a multi-label classification model.

[0052] The spliced feature is encoded by an encoding layer of the multi-label classification model to obtain a corresponding encoding feature.

[0053] The encoding feature is subjected to classification prediction by a classification prediction layer of the multi-label classification model to obtain at least one category to which the to-be-identified image belongs.

[0054] In the scheme, the classification module is further configured to encode the image feature and the context feature by an encoding layer of a multi-label classification model to obtain a first encoding feature and a second encoding feature.

[0055] The first encoding feature and the second encoding feature are subjected to feature splicing by a feature splicing layer of the multi-label classification model to obtain a spliced encoding feature.

[0056] The spliced features are classified and predicted through a classification prediction layer of the multi-label classification model, so as to obtain at least one category to which the image to be recognized belongs.

[0057] In the scheme, the extraction module is further configured to extract features of the image to be recognized and the context information respectively through a feature extraction layer of the multi-label classification model, so as to obtain image features and context features.

[0058] Correspondingly, the classification module is further configured to splice the image features and the context features through a feature splicing layer of the multi-label classification model, so as to obtain spliced features.

[0059] The spliced features are classified and predicted through a classification prediction layer of the multi-label classification model, so as to obtain at least one category to which the image to be recognized belongs.

[0060] In the scheme, the classification module is further configured to classify the image to be recognized based on the image features, so as to obtain a first category.

[0061] The image to be recognized is classified based on the context features, so as to obtain a second category.

[0062] A union of the first category and the second category is taken as a category to which the image to be recognized belongs.

[0063] In the scheme, the determination module is further configured to, when the category to which the image to be recognized belongs includes at least two categories and the at least two categories include a sensitive category, determine a vulgar recognition result of the image to be recognized based on the sensitive category and the image scene.

[0064] In the scheme, the determination module is further configured to obtain a vulgar recognition rule corresponding to the image scene of the image to be recognized.

[0065] The vulgar recognition result of the image to be recognized is determined based on the category and the vulgar recognition rule corresponding to the image scene of the image to be recognized.

[0066] An embodiment of the present application provides a computer device, comprising:

[0067] a memory configured to store executable instructions;

[0068] a processor configured to execute the executable instructions stored in the memory, so as to implement the method for recognizing vulgar images provided in the embodiments of the present application.

[0069] An embodiment of the present application provides a computer readable storage medium storing executable instructions, which are configured to cause a processor to execute, so as to implement the method for recognizing vulgar images provided in the embodiments of the present application.

[0070] The embodiments of this application have the following beneficial effects:

[0071] This application obtains an image to be identified and its associated context information; extracts features from the image to be identified and the context information to obtain image features and context features; classifies the image to be identified based on the image features and context features to obtain the category to which the image to be identified belongs; determines the image scene of the image to be identified, and determines the vulgarity recognition result of the image to be identified based on the category and the image scene; thus, by combining the image to be identified and its associated context information, the classification result of the image to be identified can be more accurate, and by combining the image scene when determining the vulgarity recognition result of the image to be identified, the vulgarity recognition result can flexibly cope with different image scenes, thereby improving the efficiency and accuracy of vulgarity image recognition. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the architecture of the vulgar image recognition system 100 provided in an embodiment of this application;

[0073] Figure 2 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application;

[0074] Figure 3 This is a flowchart illustrating the method for identifying vulgar images provided in an embodiment of this application;

[0075] Figures 4A-4B This is a schematic diagram of the category label division provided in the embodiments of this application;

[0076] Figures 5A-5C This is a schematic diagram of the image to be identified provided in an embodiment of this application;

[0077] Figure 6 This is a schematic diagram of the multi-label classification model provided in the embodiments of this application;

[0078] Figure 7 This is a schematic diagram of the multi-label classification model provided in the embodiments of this application;

[0079] Figure 8 This is a schematic diagram of the multi-label classification model provided in the embodiments of this application;

[0080] Figures 9A-9C This is a schematic diagram of the image to be identified provided in an embodiment of this application;

[0081] Figure 10 This is a schematic diagram of the image recognition system provided in an embodiment of this application;

[0082] Figure 11 This is a schematic diagram of the multi-label classification model provided in the embodiments of this application;

[0083] Figure 12 This is a schematic diagram of the standard cross-entropy loss provided in the embodiments of this application;

[0084] Figure 13 This is a schematic diagram of the composition structure of the vulgar image recognition device provided in the embodiments of this application. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0086] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0087] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0088] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0089] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0090] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0091] 1) Content: Content recommended by the client to users can include videos, pictures, and text. The content is usually actively edited and published by self-media accounts after they register a public account.

[0092] 2) Terminal: An electronic device used by a user that can receive messages and has an intelligent operating system installed on it.

[0093] 3) Terminal programs are applications that run on a terminal and can receive various messages and information streams.

[0094] 4) Server-side: A server program deployed on multiple servers, specifically designed to provide remote network services for terminal programs.

[0095] 5) Image gallery, a style that aggregates and displays image content.

[0096] 6) Highlights: A built-in feature of mobile social networking clients, which can recommend multiple articles and videos for users to read.

[0097] 7) Information stream, or news source, is a data format through which websites disseminate the latest information to users. It is usually arranged in a timeline manner. The timeline is the most original, intuitive, and basic form of information stream display. The prerequisite for users to subscribe to a website is that the website provides news sources. The convergence of information sources in one place is called aggregation, and the software used for aggregation is called an aggregator. For end users, an aggregator is software specifically used to subscribe to websites and is generally also called a reader.

[0098] 8) Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that include convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning. CNNs have representation learning capabilities and can perform shift-invariant classification of input information according to their hierarchical structure. Therefore, they are also called "Shift-Invariant Artificial Neural Networks (SIANNs)".

[0099] 9) The ImageNet database is a large visualization database used for research on visual object recognition software. More than 14 million Uniform Resource Locators (URLs) for images are manually annotated by ImageNet to indicate objects in the images; bounding boxes are also provided for at least one million images.

[0100] 10) Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). It was introduced into machine learning to bring it closer to its original goal—AI. Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to recognize data such as text, images, and sound. It is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies. Deep learning has made significant progress in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as sight, hearing, and thinking, solving many complex pattern recognition problems and leading to significant advancements in artificial intelligence-related technologies.

[0101] 11) Image scene, in this article, refers to the scene corresponding to the image content, that is, the environment / atmosphere indicated by the background information of the image. It can be obtained by image recognition or by scene classification prediction of the image through an image classification model. For example, the image scene can be a street photography scene, a fashion show scene, etc.

[0102] Social networks originated from online social networking, which began with email. The internet is essentially a network of computers. Early email solved the problem of remote email transmission and remains the most widespread application on the internet today. It was also the starting point for online social networking. Forums further normalized "mass messaging" and "forwarding," theoretically enabling the release of information and discussion of topics to everyone, thus becoming an early platform for spontaneous content generation on the internet.

[0103] Forums have taken online social networking a step further, advancing from simple point-to-point communication to point-to-multipoint communication. Instant messaging and blogs are like upgraded versions of the previous two social tools. The former improves the immediacy and parallel communication capabilities, while the latter's information publishing nodes begin to reflect a stronger sense of individuality. This is because information scattered across time can be aggregated, thus becoming the "image" and "personality" of the information publishing node. As online social networking quietly evolves, the individual's online image becomes more complete, giving birth to social networks. Social networks encompass all forms of online services centered on human social interaction. The internet is an interactive platform that enables mutual exchange, communication, and participation. Social networks have expanded the internet from research departments, schools, governments, and commercial application platforms into a tool for human social interaction. Online social networking has further extended its scope to the mobile terminal platform domain. Leveraging the ubiquity of mobile terminals and the application of wireless networks, and utilizing various dating / instant messaging / email software, mobile terminals have become the carriers of new social networks. Social networks, in essence, mean "network + social," connecting people through the network to form groups with specific characteristics.

[0104] In the era of self-media, diverse voices emerge from all directions. Each individual makes judgments based on independently acquired information. Self-media is an information dissemination activity led by the general public, transforming the "point-to-surface" dissemination of related technologies into a "point-to-point" reciprocal dissemination concept. Simultaneously, it refers to an information dissemination method that provides individuals with both privacy and public access to information production, accumulation, sharing, and dissemination. Self-media content is typically displayed in a news feed format for users to quickly refresh. The news feed on the homepage of a news app can be seen as a new type of aggregator, with subscription sources being friends of the mobile terminal's logged-in account or followed public accounts, and the content being publicly released by them. The dynamics of a news feed are such that when you have a large number of active friends, you can receive constantly updated content. This is the most common form of news feed. Time is the ultimate dimension that news feeds follow, because content updates are the result of constantly sending requests to the server. The timeline is the most primitive, intuitive, and basic display form of news feed. Users can subscribe to the various images and texts created by different social network accounts. Then, when there are updates to the images and texts, the corresponding information is pushed to the user through a message flow and displayed on the news feed interface. Users can also actively refresh the news feed to get the latest information. Images and text in the middle of the news feed have become a mainstream mode for users to consume information and news.

[0105] In implementing the embodiments of this application, the applicant discovered that the sources of content on the Internet are very extensive, including articles, images, videos, etc., and the sources and quality of this content vary greatly. Faced with a massive amount of complex content, some online content contains vulgar images that are sensitive to certain user groups, such as exposed breasts, buttocks, underwear, and exposed parts of the body, which is detrimental to purifying social atmosphere, especially for a large number of underage users. Related technologies typically involve manual review and labeling of images to filter out vulgar images. However, with a large number of images, manual review and labeling are inefficient. Related technologies also provide an auxiliary algorithm for image recognition, which works very well for images with very obvious features (such as prohibited images). However, for vulgar and sensitive images, due to the involvement of personal values ​​and scenarios (such as swimwear, fashion shows, large-scale street photography, etc., involving different parts of the human body exposed, and varying areas of exposure), the recognition effect of this method is not good.

[0106] In real-world image scenarios, the user groups for information flow content distribution are diverse. Teenagers and adults, men and women have different perceptions and influences on vulgar content. In large-scale image scenarios, such as information flow content, text and images, image galleries, and video cover images, there has always been this problem: vulgar images have different standards and requirements for different users and different image scenarios. A unified image classification model (i.e., directly judging whether an image is vulgar through an image classification model) cannot flexibly cope with different image scenarios, resulting in the complexity and difficulty in defining vulgar images.

[0107] Based on this, embodiments of this application provide a method, apparatus, computer device, and computer-readable storage medium for identifying vulgar images, which can flexibly cope with vulgar image recognition in different image scenarios and improve the efficiency and accuracy of vulgar image recognition. The exemplary application of the computer device provided in the embodiments of this application is described below. The computer device provided in the embodiments of this application can be implemented as a server. The exemplary application when the device is implemented as a server will be described below.

[0108] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the vulgar image recognition system 100 provided in this application embodiment. To support an exemplary application, terminals (including terminal 400-1 and terminal 400-2) are connected to server 200 through network 300. Network 300 can be a wide area network, a local area network, or a combination of both. Among them, terminal 400-1 is the information flow producer, and terminal 400-2 is the information flow consumer.

[0109] Terminal 400-2 receives the graphic and text content created by the user and sends the received graphic and text content to server 200;

[0110] Server 200 obtains the image to be identified and its associated context information from the text and image content; performs feature extraction on the image to be identified and the context information respectively to obtain image features and context features; classifies the image to be identified based on the image features and context features to obtain the category to which the image to be identified belongs; determines the image scene of the image to be identified, and determines the vulgarity identification result of the image to be identified based on the category and the image scene; when the vulgarity identification result is normal, the text and image content is distributed.

[0111] Terminal 400-1 retrieves the distributed text and image content and displays it for users to browse.

[0112] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0113] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application, taking an electronic device as a server as an example. Figure 2 The server 200 shown includes at least one processor 210, memory 250, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0114] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0115] User interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0116] Memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. Memory 450 may optionally include one or more storage devices physically located away from processor 210.

[0117] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0118] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0119] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0120] The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0121] Presentation module 253 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 associated with user interface 230 (e.g., a display screen, a speaker, etc.).

[0122] The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0123] In some embodiments, the vulgar image recognition device provided in this application can be implemented in software. Figure 2 A device 255 for recognizing vulgar images stored in memory 250 is shown. It may be software in the form of programs and plug-ins, including the following software modules: acquisition module 2551, extraction module 2552, classification module 2553 and determination module 2554. These modules are logically connected and can therefore be arbitrarily combined or further split according to the functions implemented.

[0124] The functions of each module will be explained below.

[0125] In other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the vulgar image recognition method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0126] The image recognition method provided in this application will be described in conjunction with exemplary applications and implementations of the terminals provided in the embodiments of this application.

[0127] See Figure 3 , Figure 3 This is a flowchart illustrating the method for identifying vulgar images provided in this application, which will be combined with... Figure 3 The steps shown are explained.

[0128] Step 301: The server obtains the image to be recognized and the context information associated with the image to be recognized.

[0129] This application does not limit the source and type of the image to be identified. The server can obtain the image to be identified from online business flow, database or other data sources. The image to be identified can be an image in an article published on a public account, a chat image in an instant messaging process, or an advertising image published by an e-commerce platform.

[0130] In the real world, a target cannot exist in isolation; it will always have some relationship with other objects or the environment around it. The objects that relate to the image to be identified are the context information. For example, text information describing the image to be identified, and other images on the same page as the image to be identified, can all serve as context information. That is, in the embodiments of this application, the context information associated with the image to be identified can be either text information or image information; no limitation is made here.

[0131] This application obtains the image to be identified and the context information associated with the image to be identified, and applies the image to be identified and the context information associated with the image to be identified for vulgarity recognition. Because the context information is combined, the vulgarity recognition result is more accurate.

[0132] In some embodiments, the server may obtain the image to be identified by: obtaining the video to be identified; extracting at least one keyframe from the video to be identified as the image to be identified; accordingly, after determining the vulgarity identification result of the image to be identified based on the category and image scene, the server may use the vulgarity identification result of the image to be identified as the vulgarity identification result of the video to be identified.

[0133] The vulgar image recognition method of this application embodiment can be used to identify vulgar content in videos. In actual implementation, the vulgarity recognition result of the video to be identified can be determined by performing vulgarity recognition on key frames in the video to be identified.

[0134] In some embodiments, the server may determine the image to be identified by: obtaining the brightness change value between adjacent video frames in the video to be identified; determining the scene switching frames of the video to be identified based on the brightness change value between adjacent video frames, and using the scene switching frames as key frames; and extracting at least one scene switching frame as a key frame as the image to be identified.

[0135] Here, due to the varying durations of different videos, a uniform frame extraction strategy would result in insufficient sampling frequency, while simultaneously increasing the burden and computational load of frame extraction, drastically increasing computational costs and expanding the comparison space. Therefore, frame extraction is performed based on the brightness change values ​​between adjacent video frames. When the brightness change value between a video frame and its preceding adjacent video frame reaches a threshold, a scene transition is considered to have occurred, and this video frame is then considered the video transition frame.

[0136] In some embodiments, to ensure that the number of video frames extracted from each video is the same, after extracting scene transition frames, frame extraction can be performed at intervals to fill in the gaps. That is, between two scene transition frames, frame extraction is performed at intervals to ensure that the number of extracted video frames is the same as the preset number.

[0137] In some embodiments, when a user uploads a video, they may specify a cover image. If the user specifies a cover image, the server can obtain the cover image and use it as the image to be identified. In this way, the vulgarity identification result of the video to be identified can be determined based on the vulgarity identification result of the cover image.

[0138] In some embodiments, when a cover image associated with the video to be identified exists, keyframes in both the cover image and the video to be identified can be acquired simultaneously, so that the cover image and keyframes are used together as the image to be identified, thereby making the vulgarity identification results of the video to be identified more accurate.

[0139] In some embodiments, the vulgar image recognition method of the present application can be used for vulgar recognition of graphic content (including image sets). In actual implementation, since the graphic content already contains images, the images in the graphic content can be obtained as images to be recognized, so as to determine the vulgar recognition result of the graphic content based on the vulgar recognition result of the images to be recognized.

[0140] In some embodiments, the server may obtain context information associated with the image to be identified by: obtaining text information associated with the image to be identified, wherein the text information includes at least one of the title information, label information, and text description information of the image to be identified; and using the obtained text information associated with the image to be identified as context information associated with the image to be identified.

[0141] In practice, the contextual information can be text information, which can be one or more of the following: title information, tag information, and text description information of the image to be recognized. This text information can be submitted by the user when submitting the image to be recognized; it can also be entered manually during image content review; or it can be tag information assigned to the image to be recognized through artificial intelligence algorithms. The source of the text information is not limited here.

[0142] In some embodiments, the server may obtain the context information associated with the image to be identified in the following ways: when the image to be identified is a frame image in a video, obtain the content description information of the video; and use the content description information of the video as the context information of the image to be identified.

[0143] In practice, the video's content description information can be a brief summary of the video, submitted by the submitter; or it can be the video's comment information, which will also include the video's content description information.

[0144] Step 302: Extract features from the image to be recognized and the context information respectively to obtain image features and context features.

[0145] Here, before feature extraction from the image to be recognized, the image to be recognized can be preprocessed, such as resizing the image to be recognized, for example, making the image size 286×286, and then normalizing the pixel values ​​of the image to be recognized, such as normalizing to between -1 and 1. In actual implementation, other image preprocessing can also be performed as needed, such as random cropping, flipping, etc., which are not limited in this application embodiment.

[0146] In some embodiments, the server can extract features from the image to be recognized in the following ways to obtain image features: obtain a pre-trained image classification model for classifying the image to be recognized; obtain the feature vector output by the intermediate layer of the image classification model during the classification process of the image to be recognized, and use it as the image features of the image to be recognized.

[0147] In practice, image classification models typically include a feature extraction layer and a classification prediction layer. The intermediate layer is the feature extraction layer. The image to be identified is input into the image classification model, and the feature extraction layer extracts features from the image. Since there can usually be multiple feature extraction layers, with the last layer being the classification prediction layer, the output of any feature extraction layer can be used as the image feature of the image to be identified. For example, the feature vector output from the penultimate layer of the image classification model can be extracted as the image feature of the image to be identified.

[0148] Here, the image classification model is pre-trained. In actual implementation, the server builds the image classification model, such as a convolutional neural network model, and trains it in the following way: First, image samples are acquired and labeled with categories. This can be done by labeling the image samples with categories based on the high-frequency sampling labels of the titles corresponding to the image samples, such as the primary categories of images: society, sports, games, animation, etc., to obtain image samples labeled with target categories. Then, the image samples labeled with target categories are input into the image classification model to predict the category of the image samples. The predicted category is compared with the standard target category. Based on the difference between the predicted category and the standard target category, the value of the loss function of the image classification model is determined. The value of the loss function is backpropagated to each layer of the image classification model, and the model parameters of each layer are updated by stochastic gradient descent (SGD) to achieve model training.

[0149] In some embodiments, to accelerate model learning efficiency, a model pre-trained on large-scale data is fine-tuned on new tasks and new data. For example, a model pre-trained on ImageNet can be used to initialize an image classification task on the dataset. Here, Xception is used to initialize the image classification model.

[0150] In some embodiments, the server can extract features from the context information in the following ways to obtain context features: when the context information is text, the text is segmented to obtain at least two words; the at least two words are encoded using a word vector model to obtain the encoding features corresponding to each word; the encoding features corresponding to each word are concatenated to obtain context features.

[0151] In practice, when encoding at least two words using a word vector model, the word embedding vectors corresponding to each word can be obtained. Then, based on the word embedding vectors corresponding to each word and the relationship between each word, at least two words can be encoded to obtain the semantic representation of the context information and obtain the context features.

[0152] As an example, contextual features can be obtained using a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model. When extracting features from text vectors, the BERT model needs to obtain word embeddings, positional embeddings, and segmentation embeddings of contextual information. These embeddings are then input into the BERT model, which extracts semantic features from the contextual information to obtain the contextual features. Word embeddings divide words into a finite set of common sub-word units, achieving a trade-off between word effectiveness and character flexibility. Positional embeddings encode the positional information of words into feature vectors; they are a crucial step in introducing word positional relationships into the model.

[0153] In some embodiments, the server may extract features from the image to be identified in the following ways: when there are at least two images to be identified, feature extraction is performed on each image to be identified to obtain the features of each image to be identified; feature fusion is performed on the features of at least two images to be identified to obtain image features.

[0154] In practical implementation, when there are at least two images to be identified, such as when identifying vulgar content in a video, multiple video frames can be obtained as images to be identified. In this case, features need to be extracted from each image to be identified separately, and then the features of the multiple images to be identified are fused. For example, features can be extracted from each image to be identified through an image classification model, and then the features of the multiple images to be identified are fused by feature concatenation.

[0155] Step 303: Based on image features and contextual features, classify the image to be identified to obtain the category to which the image to be identified belongs.

[0156] In some embodiments, the server may classify the image to be identified by: splicing image features and context features to obtain spliced ​​features; and performing multi-label classification on the image to be identified based on the spliced ​​features to obtain at least one category to which the image to be identified belongs.

[0157] In practice, image features and context features are represented by vectors. Here, image features can be concatenated to the beginning and end of context features to obtain concatenated features. Based on the concatenated features, multi-label classification is performed on the image to be identified to obtain at least one category to which the image to be identified belongs.

[0158] In some embodiments, the image to be identified can be divided into at least two categories, such as the image to be identified belonging to both the human body category and the adult category. This application divides the image to be identified into multiple categories, which is more conducive to combining different image scenarios for vulgarity identification. For example, in different image scenarios, vulgarity identification can be performed based on different categories of at least two categories; and by combining image features and contextual features, the classification results are more accurate when performing multi-label classification.

[0159] In practice, when judging whether an image is vulgar, the core reasons for its perceived vulgarity are usually the presence of specific body parts, actions, clothing, or objects; conversely, the reasons for its perceived lack of vulgarity are typically related to content type requirements or context. Based on this, the category labeling system proposed in this application is used for classification. Figures 4A-4B This is a schematic diagram of the category label division provided in the embodiments of this application. See also: Figure 4A This application categorizes vulgar images into human figures, actions, clothing, animals, objects, and scenes, with each category further subdivided into multiple subcategories. See [link / reference] Figure 4BThis application categorizes human bodies into body parts, age groups, gender, and human body types. Among these, body parts are further divided into face, chest, waist and abdomen, back, buttocks, groin, legs, and feet; the youth stage is divided into infant, child, adolescent, adult, and elderly; gender is divided into male, female, and unknown; and human body types are divided into real person, virtual, and artwork.

[0160] Figures 5A-5C This is a schematic diagram of the image to be identified provided in an embodiment of this application. See also... Figures 5A-5C ,according to Figures 4A-4B The category labeling method in the text, for Figures 5A-5B Multi-label classification of the identified images. Figure 5A It can be tagged as "Human Body_Body Part (Normal Chest)_Age (Adult)_Gender (Female)_Human Body Type (Real Person)". Figure 5B It can be tagged as "Human Body_Body Part (Nude Chest)_Age (Adult)_Gender (Female)_Human Body Type (Real Person)". Figure 5C It can be tagged as "Human Body_Body Part (Prominent Chest)_Age (Adult)_Gender (Female)_Human Body Type (Real Person)".

[0161] It should be noted that the specific label granularity can be defined by the image scene target.

[0162] In some embodiments, the server may classify the image to be identified by: concatenating image features and contextual features to obtain concatenated features, and inputting the concatenated features into a multi-label classification model; encoding the concatenated features through the encoding layer of the multi-label classification model to obtain corresponding encoded features; and performing classification prediction on the encoded features through the classification prediction layer of the multi-label classification model to obtain at least one category to which the image to be identified belongs.

[0163] Figure 6 This is a schematic diagram of the multi-label classification model provided in the embodiments of this application. See also: Figure 6 The multi-label classification model includes an encoding layer and a classification prediction layer. It concatenates the image features and context features to obtain concatenated features 601. Then, the concatenated features are input into the encoding layer 602. The encoding layer 602 encodes the concatenated features to obtain the corresponding encoded features. The classification prediction layer 603 of the multi-label classification model performs classification prediction on the encoded features to obtain the multiple categories to which the image to be identified belongs.

[0164] In practical implementation, image samples can be labeled with category tags based on the above-mentioned category labeling method. Then, a multi-label classification model can be trained based on the labeled image samples. Specifically, first, the image features and contextual features of the image samples are obtained. For example, the image features of the image samples can be obtained through the image classification model, the label information of the image samples can be obtained, and the features of the label information can be obtained as contextual features through the word vector model. The image features of the image samples and the contextual features of the image samples are concatenated end to end to obtain the concatenated features. Then, the concatenated features are input into the encoding layer, and the concatenated features are encoded through the encoding layer to obtain the corresponding encoded features. Through the classification prediction layer of the multi-label classification model, the encoded features are classified and predicted to obtain the multiple categories to which the image to be identified belongs, such as human body category_body part (breast protrusion)_age (adult)_gender (female)_human body type (real person), including five category labels. The predicted categories are compared with the labeled ones to determine the value of the loss function. Based on the value of the loss function, the error signal of the multi-label classification model is determined. The error signal is backpropagated in the multi-label classification model, so that the model parameters of each layer in the multi-label classification model are updated during the backpropagation of the error signal. For example, the error signal can be backpropagated in a multi-label classification model using stochastic gradient descent. During the backpropagation process, the model parameters of the multi-label classification model can be updated and optimized.

[0165] Here, local binary cross-entropy (Focal Sigmoid Cross Entropy) can be used as the loss function, i.e. , where a and b are hyperparameters.

[0166] In some embodiments, the server may classify the image to be identified by: encoding the image features and context features respectively through the encoding layer of a multi-label classification model to obtain a first encoded feature and a second encoded feature; concatenating the first encoded feature and the second encoded feature through the feature concatenation layer of the multi-label classification model to obtain a concatenated encoded feature; and performing classification prediction on the concatenated encoded feature through the classification prediction layer of the multi-label classification model to obtain at least one category to which the image to be identified belongs.

[0167] In practice, image features and contextual features can be directly input into a multi-label classification model, and the image features and contextual features can be fused within the multi-label classification model. Figure 7 This is a schematic diagram of the multi-label classification model provided in the embodiments of this application. See also: Figure 7The multi-label classification model includes an encoding layer 701, a feature concatenation layer 702, and a classification prediction layer 703. Image features and contextual features are input into the encoding layer 701 of the multi-label classification model, which outputs a first encoded feature and a second encoded feature. Then, the first encoded feature and the second encoded feature are input into the feature concatenation layer 702, which outputs a concatenated feature. Finally, the concatenated feature is input into the classification prediction layer 703, which outputs multiple category labels.

[0168] In some embodiments, the server can extract features from the image to be identified and the context information in the following ways: by extracting features from the image to be identified and the context information separately through the feature extraction layer of a multi-label classification model, to obtain image features and context features; correspondingly, the server can classify the image to be identified in the following ways: by concatenating the image features and context features through the feature concatenation layer of a multi-label classification model, to obtain concatenated features; and by performing classification prediction on the concatenated features through the classification prediction layer of a multi-label classification model, to obtain at least one category to which the image to be identified belongs.

[0169] In practice, multi-label classification models can include a feature extraction layer. That is, features can be extracted directly from the image to be identified and the context information through the feature extraction layer of the multi-label classification model, without the need to extract features from the image to be identified and the context information in advance through other means. This simplifies the process of multi-label classification prediction.

[0170] Figure 8 This is a schematic diagram of the multi-label classification model provided in the embodiments of this application. See also: Figure 8 The multi-label classification model provided in this application includes a feature extraction layer 801, a feature splicing layer 802, and a classification prediction layer 803. Here, the image to be identified and the context information are directly input to the feature extraction layer, which outputs image features and context features. Then, the image features and context features are input to the feature splicing layer 802, which outputs spliced ​​features. Then, the spliced ​​features are input to the classification prediction layer 803, which outputs multiple category labels.

[0171] In some embodiments, the server may classify the image to be identified in the following ways: classify the image to be identified based on image features to obtain a first category; classify the image to be identified based on context features to obtain a second category; and take the union of the first category and the second category as the category to which the image to be identified belongs.

[0172] In practice, classification can be performed based on image features and contextual features respectively. That is, the classification results of contextual features are used as a supplement to the results of image feature classification. For example, the first category obtained by image features is human body, body part (normal chest), and gender (female). By classifying the image to be recognized using contextual features, the second category is age (adult) and human body type (real person). By merging the first and second categories, a more complete category can be obtained, which is more conducive to the subsequent vulgar recognition of the image to be recognized.

[0173] Step 304: Determine the image scene of the image to be identified, and determine the vulgarity recognition result of the image to be identified based on the category and the image scene.

[0174] In practice, the standards and requirements for vulgar images vary depending on the specific image context, such as... Figures 9A-9C This is a schematic diagram of the image to be identified provided in an embodiment of this application. See also... Figure 9A Some people believe Figure 9A It's considered vulgar because of the close-up of the buttocks; some argue it's not vulgar because it's common in street photography and people dress like this in everyday life. See also Figures 9B-9C If identification is based solely on the content of an image, it will be classified as vulgar. However, if the image is a normal fashion show, it will be considered normal. Similarly, for images containing swimwear or shirtless men, if the scene is on a beach, the image can be considered normal; if the scene is on a street, it can be considered vulgar. Therefore, a one-size-fits-all classification model (judging whether something is vulgar) cannot flexibly handle different situations. Based on this, this application combines classification results with image scenes to enable vulgarity recognition results to adapt to different image scenarios.

[0175] The image scene here can be the scene corresponding to the image, such as a fashion show scene or a street photography scene, or it can be the user that the image is intended for, such as a user watching a fashion show, so that the vulgar image recognition method provided in this application embodiment can meet the needs of different scenes and users.

[0176] In some embodiments, the server may determine the vulgarity identification result of the image to be identified in the following ways: when the image to be identified belongs to at least two categories, and at least two categories include a sensitive category, the server determines the vulgarity identification result of the image to be identified based on the sensitive category and the image scene.

[0177] In practice, when the categories of the image to be identified include at least two, the vulgarity identification result of the image to be identified can be determined based on all categories and the image scene; alternatively, the vulgarity identification result of the image to be identified can be determined solely based on the sensitive category and the image scene. For example, if the obtained categories include face (normal) and chest (nude), then when determining the vulgarity identification result of the image to be identified, it is only necessary to rely on "chest (nude)" and the image scene to determine the vulgarity identification result of the image to be identified.

[0178] In some embodiments, the server may determine the vulgarity recognition result of the image to be recognized by: obtaining the vulgarity recognition rule corresponding to the image scene of the image to be recognized; and determining the vulgarity recognition result of the image to be recognized based on the category and the vulgarity recognition rule corresponding to the image scene of the image to be recognized.

[0179] In practical implementation, corresponding vulgarity recognition results can be pre-set for the image scene of the image to be identified. Here, the vulgarity recognition rules should correspond to the categories, that is, it is necessary to set the correspondence between the categories and the vulgarity recognition results. For example, in a fashion show scene, breasts (normal) and breasts (naked) can be identified as normal images, while breasts (protruding) and breasts (exposed) can be identified as vulgar images. When the category of the image to be identified is breasts (naked), the image to be identified is determined to be a normal image. In a street photography scene, breasts (normal) can be identified as normal images, while breasts (naked), breasts (protruding), and breasts (exposed) can be identified as vulgar images. When the category of the image to be identified is breasts (naked), the image to be identified is determined to be a vulgar image.

[0180] This application obtains an image to be identified and its associated context information; extracts features from the image to be identified and the context information to obtain image features and context features; classifies the image to be identified based on the image features and context features to obtain the category to which the image to be identified belongs; determines the image scene of the image to be identified, and determines the vulgarity recognition result of the image to be identified based on the category and the image scene; thus, by combining the image to be identified and its associated context information, the classification result of the image to be identified can be more accurate, and by combining the image scene when determining the vulgarity recognition result of the image to be identified, the vulgarity recognition result can flexibly cope with different image scenes, thereby improving the efficiency and accuracy of vulgarity image recognition.

[0181] The following will describe an exemplary application of the embodiments of this application in a real-world image scene.

[0182] The internet is rife with diverse and abundant content sources, including articles, images, and videos, varying greatly in quality and origin. Faced with this massive and complex volume of content, a small amount contains vulgar images that may be sensitive to certain user groups, such as exposed breasts, buttocks, underwear, or exposed body parts, particularly the groin area. Placing these images within videos or articles, for example as cover images, can be misleading and create a "cover-only" mentality, leading to user disappointment after clicking. This also hinders the purification of social values, especially for the large number of underage users. Current technologies typically involve manual review and labeling of content to filter out sensitive images, but this is inefficient with large volumes. Algorithms are also used to identify content, showing good results for highly suggestive images. However, for the aforementioned vulgar and sensitive images, which involve personal values ​​(such as swimwear, fashion shows, and revealing street photography, involving different body parts and varying areas of exposure), these methods are less effective and heavily rely on human background knowledge and judgment. Human judgment also requires combining multiple dimensions of information about the content. In real-world image scenarios, the user groups for information flow content distribution are diverse. Teenagers and adults, men and women have different perceptions and influences on vulgar content. In large-scale image scenarios, such as information flow content, text and images, image galleries, and video cover images, there has always been a problem: vulgar images have different standards and requirements for different users and different image scenarios. A unified image classification model cannot flexibly cope with this. There has always been a problem of the complexity and difficulty in defining vulgar images, requiring a more efficient and granular solution.

[0183] The solutions in related technologies have the following problems:

[0184] (1) It is highly dependent on manpower, inefficient, and slow in processing and response;

[0185] (2) Vulgar and sensitive images involve personal value and scene (such as swimwear, fashion shows, large-scale street photography, etc., which involve different parts of the human body being exposed, and the area of ​​exposure is different, etc.). The method of judgment is not very effective and relies heavily on human background knowledge and feelings. Human judgment also combines multiple dimensions of information in the content.

[0186] (3) Vulgar images have different standards and requirements for different users and different image scenarios. A unified image classification model cannot flexibly cope with this, and there are problems with the complexity and difficulty in defining vulgar images. A more efficient and fine-grained solution is needed.

[0187] Figure 10 This is a schematic diagram of the image recognition system provided in an embodiment of this application. See also: Figure 10The image recognition system includes: content production end, uplink and downlink service interface server, content database, dispatch center server, content deduplication service, content distribution export service, manual review system, multimodal vulgar image content recognition service, reporting and complaint service for vulgar image content, frame extraction service, download file system, content storage service, and content consumption end.

[0188] in, Figure 10 The core functional module is the multimodal vulgar image content recognition service, which is the engineering service corresponding to the multimodal vulgar image content features and models. This multimodal vulgar image content recognition can be viewed as a fine-grained multi-label classification problem based on multimodal information.

[0189] Information in the real world often appears in different modalities, and each source or form of information can be called a modality. For text and image content (including image galleries), the content itself already contains images. Since the cover image and contextual information contain rich information that can capture the viewer's attention, the cover image and contextual information can be combined for fine-grained multi-label classification. Here, the contextual information includes the title and tag information of the text and image content. For video content, since it does not contain images itself, video frames need to be extracted from the video as images for fine-grained multi-label classification. The extracted video frames and contextual information are then combined for fine-grained multi-label classification.

[0190] Multilabel classification is a common problem. For example, a news article might belong to both the finance and internet categories, while a question on Zhihu (a Chinese Q&A website) might belong to multiple topics. In other words, a document can be assigned to multiple labels or categories simultaneously and independently. Multilabel classification is similar to traditional single-class classification in that the total number of labels is constant. However, it differs in that the number of labels in multilabel classification is not fixed; it could be one or more. That is, given a finite set of labels, a subset of labels is assigned to each instance. For example, in this embodiment, the subset of labels includes body parts, type, and applicable scenarios in vulgar images. This multilabel information and degree judgment allow for more targeted strategies in specific scenarios.

[0191] Since vulgar images have different standards and requirements in different image scenarios, a unified classification model (to determine whether something is vulgar) cannot flexibly cope with this. A more granular multi-classification model is needed to identify the scene, type, and degree of the image, so as to create conditions for flexible strategies in actual product applications.

[0192] The core of this application lies in the criteria for judging whether an image is vulgar. The criteria for judging vulgar images differ depending on the image context. The key point here is how to design the labeling system for judging vulgarity, that is, why is an image considered vulgar? Usually, the core reason is the presence of specific body parts / actions / clothing / specific items, etc.; and why is it not always vulgar? Usually, the core reason is due to content type requirements / context, etc.

[0193] Based on this, the classification method of the category labeling system proposed in this application is presented. Figures 4A-4B This is a schematic diagram of the category label division provided in the embodiments of this application. See also: Figure 4A This application categorizes vulgar images into human body, action, clothing, animal, object, and scene categories. Each category can be further divided into multiple subcategories, such as the action category, which can be divided into action category 1, action category 2, action category 3, action category 4, action category 5, and action category 6. See also Figure 4B This application categorizes human bodies into body parts, age groups, gender, and human body types. Among these, body parts are further divided into face, chest, waist and abdomen, back, buttocks, groin, legs, and feet; the youth stage is divided into infant, child, adolescent, adult, and elderly; gender is divided into male, female, and unknown; and human body types are divided into real person, virtual, and artwork.

[0194] See Figures 5A-5C ,according to Figures 4A-4B The category labeling method in the text Figure 5A It can be tagged as "Human Body_Body Part (Normal Chest)_Age (Adult)_Gender (Female)_Human Body Type (Real Person)". Figure 5B It can be tagged as "Human Body_Body Part (Nude Chest)_Age (Adult)_Gender (Female)_Human Body Type (Real Person)". Figure 5C It can be tagged as "Human Body_Body Part (Prominent Chest)_Age (Adult)_Gender (Female)_Human Body Type (Real Person)".

[0195] It should be noted that the specific label granularity can be defined by the image scene target.

[0196] The fine-grained multi-label classification method provided in the embodiments of this application will be described below. The multimodal information here includes: the image to be identified (the cover image for text and image content, and the extracted frames of the video content for video content) + content title vector + content tag vector (the tag information of the main content itself, such as a beach volleyball image including swimwear, beach, sunshine, seaside, volleyball, etc.), and these modal information are used as input.

[0197] Here, we will take the fusion method of concatenating the feature vectors of the image, the title, and the label as an example to illustrate the multi-label classification method.Figure 11 This is a schematic diagram of the multi-label classification model provided in the embodiments of this application. See also: Figure 11 The Inception V3 base model is used to input the image to be recognized and output the feature vector of the image to be recognized. The title of the image to be recognized is input into the BERT model and output the feature vector of the title. The label of the image to be recognized is input into the BERT model and output the feature vector of the label. The feature vectors of the image, the title and the label are concatenated and input into the hidden layer. After passing through the hidden layer, the fully connected layer and the Softmax layer, the different probabilities of the multi-class labels are output.

[0198] It should be noted that this application can also use other methods to fuse the feature vectors of the image, the title, and the label, such as directly adding the feature vectors of the title and the label to the encoder output or to the decoder initialization vector.

[0199] In practice, when vectorizing the images to be recognized, transfer learning is used. This involves fine-tuning a pre-trained model on new tasks and new data to accelerate model learning efficiency. The basic approach is to use a pre-trained model from ImageNet to initialize an image classification task on the dataset. The classification labels are high-frequency abstract labels corresponding to the image's title, such as primary image classifications like society, sports, games, and animation. The pre-trained model uses Xception to initialize the image classification model, and then extracts the penultimate 2048-dimensional vector as the image representation.

[0200] For video content, frame extraction services and strategies are involved. Frame extraction is divided into keyframe extraction and regular frame extraction. A frame is the smallest unit of image in animation, equivalent to each frame on film. A keyframe is equivalent to a key drawing in 2D animation, referring to the frame containing the crucial action of a character or object's movement or change. The animation between keyframes can be created by software, called transition frames or in-between frames. For uniform frame extraction, a frame is typically sampled every 5 seconds based on the video duration.

[0201] To summarize, the specific vectorization model here is as follows:

[0202] (1) Image to be recognized: Inception V3;

[0203] (2) Image title: BERT vectorization. If there are too many sample contents (tens of millions of contents), FastBERT can be used to speed up the process.

[0204] (3) Image label: The semantics are extracted by BERT vectorization after combining the tag text. If the image has no label information, this can be empty.

[0205] BERT stands for Bidirectional Encoder Representation from Transformers, and its core is the bidirectional Transformer Encoder. During implementation, it was found that a 12-layer BERT outperformed a 2-layer BERT, but its training and inference speeds were significantly slower. Considering both model performance and efficiency, this application's implementation chose a 2-layer model to significantly improve inference speed with minimal loss of accuracy.

[0206] Because this application embodiment utilizes the image's title information and tag information to perform multimodal fusion, and fuses multiple results to provide fine-grained low-value tag information for the image, it can significantly improve the overall accuracy.

[0207] This application uses a transformation of cross-entropy loss, namely local binary cross-entropy (Focal sigmoid cross entropy), as the final loss function to train a multi-label classification model.

[0208] The cross-entropy cost function is: Here, cross-entropy is used to evaluate the difference between the probability distribution obtained from the current training and the true distribution. Reducing cross-entropy loss improves the model's prediction accuracy. Here, p(x) represents the probability of the true distribution, and q(x) is the probability estimate calculated by the model from the data.

[0209] The cross-entropy cost function for binary classification is: Here, f(x) can be the Sigmoid function or other activation functions in deep learning, while y(i)∈0,1 is usually used as the cost function for classification problems. For binary classification, Sigmoid + cross-entropy is chosen; for multi-class classification, Softmax + cross-entropy is chosen. This application uses Softmax + cross-entropy.

[0210] Figure 12 This is a schematic diagram of the standard cross-entropy loss provided in the embodiments of this application. See also... Figure 12Here, standard cross-entropy loss is used, which essentially involves adding a sigmoid layer to each output and optimizing it with binary cross-entropy loss (BCE loss). The BCE loss method essentially performs binary classification learning independently for each class. Through the sharing of underlying parameters, this method can indirectly learn the correlation between labels.

[0211] Cross-entropy can be expressed by the following formula:

[0212] ;

[0213] Where f is the Sigmoid function.

[0214] Furthermore, the Sigmoid Cross Entropy loss function can be easily transformed into FocalSigmoid Cross Entropy, which can address the issues of class imbalance and differences in classification difficulty. The formula for FocalSigmoid Cross Entropy is shown below:

[0215] ;

[0216] Where a and b are hyperparameters, a=0.25 and b=2 have the best effect and are used as the loss function in the final model.

[0217] The following is about Figure 10 The following is a detailed explanation of the modules other than the multimodal vulgar image content recognition service:

[0218] Figure 10This includes content production and consumption. Professionally Generated Content (PGC), User Generated Content (UGC), and Multi-Channel Networks (MCNs) are all types of content producers. PGCs, or PUGCs (Professional User Generated Content), provide local or filmed video content, self-written articles, or image sets through mobile devices or backend application programming interfaces (APIs). Authors can choose to actively upload cover images for their content; these are the main sources of distributed content. The content production side communicates with upstream and downstream content interface services to obtain the upload server interface address and then uploads local files. During filming, local video content can be customized with music, filters, templates, and enhancement features. The content consumption side communicates with upstream and downstream content interface services to obtain index information for accessing content. The system processes data, including information, communication with video storage servers to download corresponding streaming media files and play them locally when accessing video content, and communication with edge-deployed CDN services when accessing text and images. Simultaneously, it reports user browsing behavior data during upload and download, such as reading speed, completion rate, reading time, buffering, loading time, and playback clicks, to the server. Consumers typically browse data via Feeds streams. The system provides a direct reporting and feedback portal for vulgar image content encountered by consumers, directly connecting to a manual review system for confirmation and verification. The verification results are stored in a vulgar image content sample library, serving as a data source for subsequent model training.

[0219] Figure 10 This includes upstream and downstream content interface servers that communicate directly with content producers to obtain the content submitted by them. This content typically includes the title, publisher, summary, cover image, and publication date. Alternatively, videos can be directly uploaded to the server and stored in the video content storage service. The server also writes metadata about the video content, such as file size, cover image link, bitrate, file format, title, publication date, author, and author-marked tags, into the content database. Finally, the server submits the uploaded files and content metadata to the scheduling center service for subsequent content processing and distribution.

[0220] Figure 10This includes a content database, which is the core database of content. All metadata about content published by producers is stored in this business database. The focus is on the metadata of the content itself, such as file size, cover image link, bitrate, file format, title, publication time, author, video file size, video format, whether it's original or a first release, and also the content classification during manual review (including first, second, and third level categories and tag information; for example, an article explaining 'x' as a mobile phone might have a first-level category of 'technology,' a second-level category of 'smartphones,' a third-level category of 'domestic mobile phones,' and the tag information 'x' as and model number). During the manual review process, information is read from the content database, and the results and status of the manual review are also sent back to the content database. The scheduling center processes content mainly through machine processing and manual review. Machine processing involves various quality judgments such as low-quality filtering, content tagging (e.g., category, tag information), and content deduplication. Their results are written to the content database; completely duplicate content will not be subject to repeated manual processing. Subsequent tag extraction will read the content's metadata from the content database. The metadata obtained from multimodal prediction of vulgar image content also comes from this database.

[0221] Figure 10 This includes a scheduling center service, responsible for the entire scheduling process of video and text content flow. It receives content from the upstream and downstream content interface servers and then retrieves the content's metadata from the content metadata database. As the actual scheduler and controller of the text and video link operation, it schedules a multimodal vulgar image content recognition service system to process the corresponding content according to its type, directly filtering and tagging content accordingly. It also schedules a manual review system and a machine processing system, controlling the scheduling order and priority. Content is activated through the manual review system and then directly displayed to end-user content consumers through a content distribution service (usually a recommendation engine, search engine, or operations platform), providing the consumer with the content index information.

[0222] Figure 10This includes manual review services and a service for reporting and complaining about vulgar image content. It is typically a web system that receives machine-filtered results, manually confirms and reviews them, and records the results in a content information metadata database. Simultaneously, the manual review results can be used to evaluate the actual effectiveness of machine-based filtering models online. The system also reports detailed logs of the manual review process, including the source of the task, review results, and start and end times, to a statistics server. Furthermore, it interfaces with the user-end complaint and content reporting review system, prioritizing the processing of vulgar image content complaints and reports. Upon confirmation, the vulgar image content is directly added to the vulgar image content seed library. The review results in the vulgar image content library provide a data foundation for the subsequent construction of multimodal, fine-grained vulgar image content.

[0223] Figure 10 This includes content storage services, typically a group of widely distributed storage servers located far from users. These servers are usually accompanied by CDN acceleration servers for distributed caching. Upstream and downstream content interface servers store the video and image content uploaded by content creators. End consumers, after obtaining content index information, can directly access the video content storage server to download the corresponding content. Besides serving as a data source for external services, it also serves as a data source for internal services, allowing the download file system to obtain raw video data for related processing. The pathways for internal and external data sources are usually deployed separately to avoid mutual interference.

[0224] Figure 10 It includes a vulgar image content library, which is used to obtain manually reviewed and marked content from the content metadata and repository as the raw input for creating vulgar image content; vulgar image content is retrieved regularly, usually every day.

[0225] Figure 10 This includes multimodal vulgar image content features and models. Through statistical analysis of manual review and reporting data, it identifies the scenarios and degrees of vulgarity in vulgar image content, reaches a consensus and understandable definition for each type of problem, defines the main features and types of vulgar image content, and labels and collects multi-label data according to the multi-label classification types described above. Then, according to the description of the multimodal vulgar image content model above, it constructs a corresponding machine learning algorithm model by combining shallow and deep multimodal features to achieve fine-grained identification and labeling of vulgar image content.

[0226] Figure 10 This includes a multimodal vulgar image content recognition service, which is used to service-encode the vulgar image content model built above, and build a service that can be called on the link to realize the crackdown and filtering of content, or to mark it, and to realize the distribution according to the scenario and according to the user, thereby improving the experience of key users.

[0227] Figure 10 The system includes a download file system, used to download and retrieve raw video content from the content storage server, controlling the download speed and progress. It is usually composed of a group of parallel servers with related task scheduling and distribution clusters. After the file is downloaded, a frame extraction service is called to extract the necessary video file keyframes from the video source file, which are used to construct video image frames for content recognition.

[0228] Figure 10 The process includes a frame extraction service, which, according to the frame extraction method described above, downloads files from the video content storage service and performs initial processing of video file features—video frame extraction—including keyframes and uniformly extracted frames. The core challenge of frame extraction lies in handling videos of varying durations, making uniform frame extraction unsuitable. Uniform frame extraction leads to insufficient sampling frequency, while extracting all frames increases the burden and computational load, drastically increasing computational costs and expanding the comparison space. This invention provides a modified frame extraction strategy, the core of which is to extract scene transition frames with significant brightness changes, and then fill in the gaps with equally spaced frames before and after, serving as the input for subsequent multimodal frames.

[0229] Figure 10 It includes a statistics server for receiving consumption data reports from content moderation and consumption ends; statistically mining and analyzing the reported data for machine algorithm models to filter and read; and providing scheduling effects for detecting and analyzing content activation rates and content moderation backlog delays. It also analyzes vulgar image content reported by users and receiving negative feedback to obtain impact and effect assessments, providing a basis for prioritizing the types of vulgar image content in the multimodal vulgar image content recognition system.

[0230] The embodiments of this application have the following beneficial effects:

[0231] (1) It can improve the response and processing speed of vulgar image content, while saving a lot of review manpower; according to the results of the vulgar image content recognition model applied in different scenarios, it can avoid selecting highly vulgar image content to the greatest extent.

[0232] (2) Due to the fine-grained classification of vulgar images, the degree of vulgarity caused by content appearing in appropriate scenarios is greatly reduced. In particular, for content that is not suitable for teenagers to see, it can effectively purify the online content consumption environment and meet the content demands and different style control needs of users of different ages and levels.

[0233] The description of the vulgar image recognition device provided in the embodiments of this application will continue. Figure 13 A schematic diagram of the composition structure of the vulgar image recognition device provided in the embodiments of this application is shown below. Figure 13The vulgar image recognition device 255 provided in this application embodiment includes:

[0234] The acquisition module 2551 is used to acquire the image to be identified and the context information associated with the image to be identified;

[0235] Extraction module 2552 is used to extract features from the image to be identified and the context information respectively to obtain image features and context features;

[0236] The classification module 2553 is used to classify the image to be identified based on the image features and context features, so as to obtain the category to which the image to be identified belongs;

[0237] The determination module 2554 is used to determine the image scene of the image to be identified, and to determine the vulgarity recognition result of the image to be identified based on the category and the image scene.

[0238] In some embodiments, the acquisition module 2551 is further configured to acquire the video to be identified;

[0239] Extract at least one keyframe from the video to be identified as the image to be identified;

[0240] Correspondingly, the determining module 2554 is also used to take the vulgarity recognition result of the image to be recognized as the vulgarity recognition result of the video to be recognized.

[0241] In some embodiments, the acquisition module 2551 is further configured to acquire the brightness change value between adjacent video frames in the video to be identified;

[0242] Based on the brightness change values ​​between adjacent video frames, the scene switching frames of the video to be identified are determined, and the scene switching frames are used as key frames.

[0243] At least one of the scene switching frames, which serves as a keyframe, is extracted as the image to be identified.

[0244] In some embodiments, the acquisition module 2551 is further configured to acquire text information associated with the image to be identified, wherein the text information includes at least one of the title information, tag information, and text description information of the image to be identified;

[0245] The text information associated with the image to be identified is used as the context information associated with the image to be identified.

[0246] In some embodiments, the acquisition module 2551 is further configured to acquire the content description information of the video when the image to be identified is a frame image in a video.

[0247] The content description information of the video is used as the context information of the image to be identified.

[0248] In some embodiments, the extraction module 2552 is further configured to obtain a pre-trained image classification model for classifying the image to be identified;

[0249] In the process of obtaining the image classification model for classifying the image to be identified, the feature vector output by the intermediate layer of the image classification model is used as the image feature of the image to be identified.

[0250] In some embodiments, the extraction module 2552 is further configured to perform word segmentation on the text when the context information is text, to obtain at least two words;

[0251] The at least two words are encoded using a word vector model to obtain the encoding features corresponding to each word;

[0252] The context features are obtained by concatenating the encoded features corresponding to each word.

[0253] In some embodiments, the extraction module 2552 is further configured to extract features from each of the images to be identified when there are at least two images to be identified, thereby obtaining features corresponding to each of the images to be identified.

[0254] The features of at least two of the images to be identified are fused to obtain the image features.

[0255] In some embodiments, the classification module 2553 is further configured to concatenate the image features and the context features to obtain concatenated features;

[0256] Based on the splicing features, the image to be identified is classified using multi-label classification to obtain at least one category to which the image to be identified belongs.

[0257] In some embodiments, the classification module 2553 is further configured to concatenate the image features and the context features to obtain concatenated features, and input the concatenated features into a multi-label classification model;

[0258] The concatenated features are encoded through the encoding layer of the multi-label classification model to obtain the corresponding encoded features;

[0259] The coded features are classified and predicted using the classification prediction layer of the multi-label classification model to obtain at least one category to which the image to be identified belongs.

[0260] In some embodiments, the classification module 2553 is further configured to encode the image features and context features respectively through the encoding layer of a multi-label classification model to obtain a first encoded feature and a second encoded feature;

[0261] The first encoded feature and the second encoded feature are concatenated through the feature concatenation layer of the multi-label classification model to obtain the concatenated encoded feature.

[0262] The classification prediction layer of the multi-label classification model is used to classify and predict the spliced ​​encoded features to obtain at least one category to which the image to be identified belongs.

[0263] In some embodiments, the extraction module 2552 is further configured to extract features from the image to be identified and the context information respectively through the feature extraction layer of the multi-label classification model to obtain image features and context features;

[0264] Correspondingly, the classification module 2553 is also used to perform feature splicing on the image features and context features through the feature splicing layer of the multi-label classification model to obtain spliced ​​features;

[0265] The spliced ​​features are classified and predicted using the classification prediction layer of the multi-label classification model to obtain at least one category to which the image to be identified belongs.

[0266] In the above scheme, the classification module 2553 is further used to classify the image to be identified based on the image features to obtain a first category;

[0267] Based on the contextual features, the image to be identified is classified to obtain a second category;

[0268] The union of the first category and the second category is taken as the category to which the image to be identified belongs.

[0269] In some embodiments, the determining module 2554 is further configured to determine the vulgarity recognition result of the image to be recognized based on the sensitive category and the image scene when the category to which the image to be recognized belongs includes at least two categories, and the at least two categories include a sensitive category.

[0270] In some embodiments, the determining module 2554 is further configured to obtain vulgar recognition rules corresponding to the image scene of the image to be recognized;

[0271] Based on the category and the vulgarity recognition rules corresponding to the image scene of the image to be identified, the vulgarity recognition result of the image to be identified is determined.

[0272] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the vulgar image recognition method described above in this application.

[0273] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 3 The method shown.

[0274] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0275] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0276] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0277] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0278] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of recognizing a vulgar image, characterized by, The method comprises: When the brightness change value between a video frame of a video to be identified and a previous adjacent video frame reaches a change threshold, the video frame is taken as a scene switching frame, at least one scene switching frame is extracted as an image to be identified; between two scene switching frames, the extracted scene switching frames are supplemented by means of interval frame extraction, so that the number of extracted scene switching frames is the same as a preset number; Text information used for describing the image to be identified and other images in the same page as the image to be identified are obtained as context information associated with the image to be identified; Image features and context features are obtained by respectively performing feature extraction on the image to be identified and the context information. The image to be identified is classified based on the image features and context features and a preset category label system, to obtain a category to which the image to be identified belongs, wherein the preset category label system divides the image to be identified into at least one of a human body category, an action category, a dressing category or an article category label used for describing the image to be identified, and at least one scene category label used for describing a content type or a scene of the image to be identified, and further divides each category label into a plurality of subcategory labels; An image scene of the image to be identified is determined, and a lowbrow identification result of the image to be identified is determined based on the category and a lowbrow identification rule corresponding to the image scene, and the lowbrow identification result of the image to be identified is taken as a lowbrow identification result of the video to be identified.

2. The method of claim 1, wherein, The text information used for describing the image to be identified and the other images in the same page as the image to be identified are obtained as the context information associated with the image to be identified, and the method comprises: Text information associated with the image to be identified is obtained, and the text information comprises at least one of title information, label information and text description information of the image to be identified; Other images in the same page as the image to be identified are obtained; The obtained text information associated with the image to be identified and the other images are taken as the context information associated with the image to be identified.

3. The method of claim 1, wherein, The text information used for describing the image to be identified and the other images in the same page as the image to be identified are obtained as the context information associated with the image to be identified, and the method comprises: Other images in the same page as the image to be identified are obtained; The content description information of the video and the other images are taken as the context information of the image to be identified.

4. The method of claim 1, wherein, Feature extraction is performed on the image to be identified to obtain image features and context features, and the method comprises: An image classification model used for classifying the image to be identified is obtained by pre-training; During classification of the image to be identified by the image classification model, a feature vector output by an intermediate layer of the image classification model is taken as the image features of the image to be identified.

5. The method of claim 1, wherein, Feature extraction is performed on the context information to obtain context features, and the method comprises: When the context information is text, word segmentation processing is performed on the text to obtain at least two words; encoding the at least two words through a word vector model to obtain an encoding feature corresponding to each word; concatenating the encoding features corresponding to the words to obtain the context feature.

6. The method of claim 1, wherein, extracting features of the to-be-identified images to obtain image features, including: when the number of to-be-identified images is at least two, extracting features of each to-be-identified image to obtain features corresponding to each to-be-identified image; fusing features of at least two to-be-identified images to obtain the image features.

7. The method of claim 1, wherein, classifying the to-be-identified images based on the image features and the context features and a preset category label system to obtain a category to which the to-be-identified images belong, including: concatenating the image features and the context features to obtain concatenated features; performing multi-label classification on the to-be-identified images based on the concatenated features to obtain at least one category to which the to-be-identified images belong in the preset category label system.

8. The method of claim 1, wherein, classifying the to-be-identified images based on the image features and the context features and a preset category label system to obtain a category to which the to-be-identified images belong, including: concatenating the image features and the context features to obtain concatenated features, and inputting the concatenated features into a multi-label classification model; encoding the concatenated features through an encoding layer of the multi-label classification model to obtain corresponding encoding features; performing classification prediction on the encoding features through a classification prediction layer of the multi-label classification model to obtain at least one category to which the to-be-identified images belong in the preset category label system.

9. The method of claim 1, wherein, classifying the to-be-identified images based on the image features and the context features and a preset category label system to obtain a category to which the to-be-identified images belong, including: encoding the image features and the context features through an encoding layer of a multi-label classification model to obtain first encoding features and second encoding features; concatenating the first encoding features and the second encoding features through a feature concatenation layer of the multi-label classification model to obtain concatenated encoding features; performing classification prediction on the concatenated encoding features through a classification prediction layer of the multi-label classification model to obtain at least one category to which the to-be-identified images belong in the preset category label system.

10. The method of claim 1, wherein, extracting features of the to-be-identified images and the context information through a feature extraction layer of a multi-label classification model to obtain image features and context features, including: extracting features of the to-be-identified images and the context information through a feature extraction layer of a multi-label classification model to obtain image features and context features; correspondingly, classifying the to-be-identified images based on the image features and the context features to obtain a category to which the to-be-identified images belong, including: concatenating the image features and the context features through a feature concatenation layer of a multi-label classification model to obtain concatenated features; performing classification prediction on the concatenated features through a classification prediction layer of the multi-label classification model to obtain at least one category to which the to-be-identified images belong.

11. A device for recognizing a vulgar image, characterized by, including: The acquisition module is configured to: when a brightness change value between a video frame of a to-be-identified video and a previous adjacent video frame reaches a change threshold, take the video frame as a scene switching frame, extract at least one scene switching frame as a to-be-identified image; and between two scene switching frames, fill in by means of interval frame extraction, so that the number of extracted scene switching frames is the same as a preset number. The acquisition module is configured to acquire text information describing the to-be-identified image and other images in a same page as the to-be-identified image as context information associated with the to-be-identified image. The extraction module is configured to respectively perform feature extraction on the to-be-identified image and the context information to obtain image features and context features. The classification module is configured to classify the to-be-identified image based on the image features and the context features and a preset category label system to obtain a category to which the to-be-identified image belongs, wherein the preset category label system divides the to-be-identified image into at least one of a human body category, an action category, a dressing category or an article category label for describing the to-be-identified image, and at least one scene category label for describing a content type or a scene of the to-be-identified image, and further divides each category label into a plurality of subcategory labels. The determination module is configured to determine an image scene of the to-be-identified image, and determine a pornography recognition result of the to-be-identified image based on the category and a pornography recognition rule corresponding to the image scene; and take the pornography recognition result of the to-be-identified image as a pornography recognition result of the to-be-identified video.

12. A computer device, comprising: The memory is configured to store executable instructions. The processor is configured to execute the executable instructions stored in the memory to implement the pornography image recognition method in any one of claims 1 to 10. The executable instructions are stored in the memory and are configured to be executed by the processor to implement the pornography image recognition method in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, ​

Images