Image type recognition method and its device, equipment, medium, and product
By combining the binary classification model and the multi-classification model, the advertising images of e-commerce platforms are identified, and the problems of low identification efficiency and poor accuracy in the existing technology are solved, and more efficient and accurate detection of violation images is achieved to ensure that e-commerce platforms are Information security.
Patent Information
- Application Number
- CN202210411298.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-04-19
AI Technical Summary
When identifying illegal content (such as pornographic pictures) in advertising images in the existing technology in e-commerce platforms, the error detection rate and missed detection rate are high, resulting in low efficiency and unrealistic manual review, making it difficult to accurately and efficiently identify specific types of picture.
Using a combination of binary classification model and multi-classification model, the binary classification model and multi-classification model that are pre-trained to converge state are used to determine the type of detection images, obtain their respective confidence, and achieve comprehensive confidence through fusion confidence. , determine whether the image is the target type, and use the global information of the binary classification model and the detailed information of the multi-classification model to achieve more accurate identification.
Improve the recognition accuracy of the type of image being detected, reduce the rate of false detection and missed detection, and provide higher confidence judgment results. It is suitable for information security maintenance of e-commerce platforms to ensure that there are no illegal images in advertising images. .
Smart Images

Figure CN114708539B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to an image type recognition method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] Image recognition using artificial neural network models has become a mainstream technology. For application scenarios such as e-commerce platforms, a large number of advertising pictures or other pictures are generated every day. Due to the restrictions of e-commerce platforms on advertising pictures of sensitive people, it is generally necessary to detect and identify the image content in the advertising pictures uploaded by users, and then further process them according to different image content and needs.
[0003] For application scenarios such as e-commerce platforms, if the advertising images uploaded by merchants for displaying advertising information contain illegal content, such as pornographic images, it is necessary to conduct pre-detection and processing of the relevant images in order to maintain the information security of the platform.
[0004] Faced with tens of thousands of images, it is not realistic to rely entirely on manual review, as it consumes a lot of manpower and has low work efficiency. For certain types of images, traditional technologies, such as commonly used recognition models or detection models, have high false positive rates or high missed positive rates.
[0005] Therefore, how to accurately and efficiently identify specific types of images from images to make the recognition results more accurate has become a technical problem that needs to be solved in this field. Summary of the invention
[0006] The purpose of the present application is to solve at least one of the above problems and to provide an image type recognition method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product.
[0007] To meet the various purposes of this application, the following technical solutions are adopted:
[0008] In one aspect, a method for identifying an image type is provided to meet one of the purposes of the present application, comprising the following steps:
[0009] Acquire the image to be detected;
[0010] Calling a binary classification model to perform type discrimination on the image to be detected, and determining a first confidence level that the image belongs to a target type, wherein the binary classification model is pre-trained to a convergence state;
[0011] Calling a multi-classification model to perform type discrimination on the image to be detected, and determining a second confidence level that the image belongs to a target type, wherein the multi-classification model is pre-trained to a convergence state;
[0012] The first confidence level and the second confidence level are integrated to obtain a comprehensive confidence level. When the comprehensive confidence level is greater than a preset threshold, the image to be detected is confirmed to be of the target type.
[0013] In some embodiments, before the step of calling the binary classification model to perform type discrimination on the image to be detected, the following steps are included:
[0014] The binary classification model is iteratively trained using training samples in the data set until convergence, wherein the training samples include images to be detected that are positive samples labeled as target types and images to be detected that are negative samples labeled as non-target types, and the number of classifications of the classifier of the binary classification model is configured to be greater than two, wherein the two classifications correspond to the target type and the non-target type, respectively.
[0015] In some embodiments, before the step of calling the multi-classification model to perform type discrimination on the image to be detected, the following steps are included:
[0016] Using the binary classification model trained to a convergent state to test the training samples in the data set, and predicting the classification of each training sample;
[0017] Clustering is performed according to the activated neuron combination in the fully connected layer of each training sample before classification, adjusting the labeling information of the training samples of the data set, labeling the training samples with the same neuron combination and original labeling type in the data set as the same type, wherein the two types correspond to the target type and the non-target type respectively, and labeling the training samples with different neuron combinations or original labeling types in the data set as different types accordingly;
[0018] According to the clustering results, the total number of categories is counted and configured as the number of categories of the multi-classification model;
[0019] The multi-classification model is iteratively trained using the training samples in the data set until convergence.
[0020] In some further embodiments, the binary classification model and the multi-classification model include the same network model architecture, and the network model architecture performs the following steps for each image to be detected:
[0021] Extracting image feature information of the image to be detected through an image feature extraction model;
[0022] Performing a full connection operation on the image feature information through a first fully connected layer to obtain comprehensive feature information;
[0023] Performing a full connection operation on the comprehensive feature information through a second fully connected layer to obtain classification feature information, wherein the dimension of the classification feature information is lower than that of the comprehensive feature information;
[0024] The classification feature information is mapped to each classification corresponding to its preset number of classifications through the output layer of the classifier to obtain the corresponding confidence level of each classification.
[0025] In some embodiments, the first confidence level and the second confidence level are integrated to obtain a comprehensive confidence level. When the comprehensive confidence level is greater than a preset threshold, the image to be detected is confirmed to be the target type, including the following steps:
[0026] Matching the first confidence level and the second confidence level with preset weights respectively for weighted summation to obtain a comprehensive confidence level;
[0027] It is determined whether the comprehensive confidence is greater than a preset threshold, and when it is greater than the preset threshold, it is determined that the image to be detected is the target type.
[0028] In some other embodiments, the first confidence level and the second confidence level are integrated to obtain a comprehensive confidence level. When the comprehensive confidence level is greater than a preset threshold, the image to be detected is confirmed to be the target type, including the following steps:
[0029] Inputting the first confidence and the second confidence into a fully connected layer, and mapping the first confidence and the second confidence to two categories of a binary classifier by associating the learned weights by the fully connected layer, respectively obtaining comprehensive confidences corresponding to the two categories, wherein the two categories correspond to the target type and the non-target type, respectively, and the fully connected layer is preliminarily trained using training samples in the data set until a convergence state is obtained to obtain the weights;
[0030] It is determined whether the comprehensive confidence of the classification corresponding to the target type is greater than a preset threshold. When it is greater than the preset threshold, the image to be detected is determined to be the target type.
[0031] In some extended embodiments, after confirming that the image to be detected is of the target type, the following steps are included:
[0032] Suspending the processing flow of the processing business of the image to be detected;
[0033] Constructing a notification message containing the image to be detected and transmitting it to the review interface;
[0034] The audit interface is monitored for instructions indicating approval of the release, and the processing flow is resumed.
[0035] On the other hand, an image type recognition device provided to meet one of the purposes of the present application includes an image acquisition module, a first recognition module, a second recognition module, and a fusion judgment module, wherein: the image acquisition module is used to acquire an image to be detected; the first recognition module is used to call a binary classification model to perform type discrimination on the image to be detected, and determine a first confidence level that it belongs to a target type, and the binary classification model is pre-trained to a convergence state; the second recognition module is used to call a multi-classification model to perform type discrimination on the image to be detected, and determine a second confidence level that it belongs to a target type, and the multi-classification model is pre-trained to a convergence state; the fusion judgment module is used to fuse the first confidence level and the second confidence level to obtain a comprehensive confidence level, and when the comprehensive confidence level is greater than a preset threshold, confirm that the image to be detected is of the target type.
[0036] In some embodiments, prior to the first recognition module, it also includes: a first training module, which is used to use training samples in the data set to perform iterative training on the binary classification model until a convergence state, the training samples include images to be detected that are positive samples marked as target types and images to be detected that are negative samples marked as non-target types, and the number of categories of the classifier of the binary classification model is configured to be greater than two, wherein the two categories correspond to the target type and the non-target type, respectively.
[0037] In some embodiments, prior to the second recognition module, a test implementation module is included, which is used to test the training samples in the data set using the binary classification model trained to a convergent state to predict the classification of each training sample; a clustering evaluation module is used to cluster according to the activated neuron combination in the fully connected layer of each training sample before classification, adjust the labeling information of the training samples in the data set, label the training samples with the same neuron combination and original labeling type in the data set as the same type, wherein the two types correspond to the target type and the non-target type respectively, and label the training samples with different neuron combinations or original labeling types in the data set as different types accordingly; a category setting module is used to count the total number of categories according to the clustering results, and configure the total number of categories as the number of categories of the multi-classification model; a second training module is used to implement iterative training of the multi-classification model using the training samples in the data set to a convergence state.
[0038] In some further embodiments, the binary classification model and the multi-classification model include the same network model architecture, which includes: a feature extraction unit, configured to extract image feature information of the image to be detected through an image feature extraction model; a first connection unit, configured to perform a full connection operation on the image feature information through a first fully connected layer to obtain comprehensive feature information; a second connection unit, configured to perform a full connection operation on the comprehensive feature information through a second fully connected layer to obtain classification feature information, wherein the dimension of the classification feature information is lower than that of the comprehensive feature information; a classification mapping unit, configured to map the classification feature information to each classification corresponding to its preset number of classifications through the output layer of the classifier, and obtain the confidence corresponding to each classification.
[0039] In some embodiments, the fusion judgment module includes: a weighted summation unit, used to match the first confidence and the second confidence with preset weights for weighted summation to obtain a comprehensive confidence; an image judgment unit, used to determine whether the comprehensive confidence is greater than a preset threshold, and when it is greater than the preset threshold, determine that the image to be detected is the target type.
[0040] In some other embodiments, the fusion judgment module includes a connection synthesis unit, which is used to input the first confidence and the second confidence into a fully connected layer, and the fully connected layer associates the learned weights to map the first confidence and the second confidence to two categories of the binary classifier, and obtains the comprehensive confidences corresponding to the two categories, respectively. The two categories correspond to the target type and the non-target type, respectively. The fully connected layer pre-trains using the training samples in the data set to a convergence state to obtain the weights; a classification judgment unit, which is used to determine whether the comprehensive confidence of the category corresponding to the target type is greater than a preset threshold. When it is greater than the preset threshold, it is determined that the image to be detected is the target type.
[0041] In some extended embodiments, after the fusion judgment module, it includes: a process suspension unit, which is used to suspend the processing flow of the processing business of the image to be detected; an interface calling unit, which is used to construct a notification message containing the image to be detected and transmit it to the review interface; a process recovery unit, which is used to listen to the instructions returned by the review interface that represent the consent to be released, and resume the processing flow.
[0042] On the other hand, a computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the image type recognition method described in the present application.
[0043] On the other hand, a computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the image type recognition method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, it executes the steps included in the method.
[0044] On the other hand, a computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.
[0045] Compared with the prior art, this application has many advantages, including at least the following aspects:
[0046] This application uses a binary classification model and a multi-classification model to detect the same image to be detected, respectively, to obtain the confidence of the deep semantic features of the image to be detected output by each model mapped to the target type, and then, fuses the confidences of the two classification models to obtain a comprehensive confidence, and identifies the target type of the image to be detected whose comprehensive confidence is greater than a preset threshold, thereby utilizing the classification capabilities of the two models at different granularities to achieve accurate identification of the type of the image to be detected.
[0047] Since the recognition process of the image to be detected integrates the judgment results of the two models, the binary classification model can focus on the global information of the image to be detected to obtain its classification results, and the multi-classification model can focus on the detailed information of the image to be detected to obtain its classification results. Finally, the confidence corresponding to the target type in the two classification results is used to comprehensively determine whether the image to be detected is an image of the target type, which is more comprehensive and accurate.
[0048] Therefore, the judgment result made by the present application on whether the image to be detected contains images of the target type has a higher confidence level and can be highly trusted. It is suitable for detecting whether there are images of the target type in advertising pictures in application scenarios such as e-commerce platforms. It can provide more accurate judgment results for needs such as pornographic pictures, thereby maintaining the information security of e-commerce platforms and avoiding the appearance of illegal images in their advertising pictures. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0050] Figure 1 A flowchart of a typical embodiment of the image type recognition method of the present application;
[0051] Figure 2 This is a schematic diagram of the structure of the network infrastructure used by the binary classification model and / or multi-classification model in one embodiment of the present application;
[0052] Figure 3 This is a schematic diagram of the structure of the network infrastructure used by the binary classification model and / or multi-classification model in another embodiment of the present application;
[0053] Figure 4 A flowchart of a training process for training the binary classification model and the multi-classification model in one embodiment of the present application;
[0054] Figure 5 In the embodiments of this application, Figure 2 A flowchart of the working process of the neural network model of the network infrastructure shown;
[0055] Figure 6 A flowchart of a process of obtaining a comprehensive confidence corresponding to a target type by fusing the confidence of a target type output by a binary classification model and a multi-classification model using a fully connected layer in an embodiment of the present application;
[0056] Figure 7 A flowchart of a process of controlling a business process according to an image recognition result in an embodiment of the present application;
[0057] Figure 8 This is a principle block diagram of the image type recognition device of the present application;
[0058] Fig. 9 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION
[0059] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0060] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0062] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.
[0063] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.
[0064] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.
[0065] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.
[0066] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.
[0067] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.
[0068] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.
[0069] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.
[0070] An image type recognition method of the present application can be programmed as a computer program product and deployed in a client or server for execution. For example, in the exemplary application scenario of the present application, it can be deployed and implemented in a server of an e-commerce platform, thereby accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface to execute the method.
[0071] See also Figure 1 In a typical embodiment, the image type recognition method of the present application includes the following steps:
[0072] Step S1100: Obtain an image to be detected:
[0073] The images to be detected can have multiple sources. For example, in one method, the images to be detected can be obtained from product images or text images submitted by merchant users or pre-stored, or the platform can use a timing trigger mechanism to call the relevant images of independent sites of relevant online stores in each platform one by one to obtain the corresponding images to be detected. Generally, considering the exemplary e-commerce scenario of this application, relevant detection can be focused on images with advertising promotion properties. Therefore, the images to be detected can be obtained from advertising images submitted by merchant users, mainly online stores, in order to detect the corresponding target type images.
[0074] Of course, the image recognition technology of the present application is not limited to the exemplary e-commerce scenario. In theory, it can be applied to any scenario that requires image type recognition. As long as the technical solution of the present application is used to realize the recognition of specific types of images, it is theoretically covered by the creative spirit of the present application and is not restricted by the source and naming of the carrier image. Technical personnel in this field should be aware of this.
[0075] In order to meet the needs of subsequent standardized processing of different classification models, in one embodiment, a carrier image can be appropriately preprocessed according to the input parameter constraints of each classification model, such as performing image standardization operations such as image scaling and cropping, to obtain the image to be detected required for recognition by each classification model.
[0076] Step S1200: calling a binary classification model to perform type discrimination on the image to be detected, and determining a first confidence level that the image belongs to a target type, wherein the binary classification model is pre-trained to a convergence state:
[0077] The present application prepares a binary classification model. The key application of the binary classification model is to extract features from the image to be detected and then map it to two categories to obtain the confidence of the two categories. The two categories correspond to whether the image to be detected is a target type or not a target type. For the classification that is not a target type, in this application, in order to avoid ambiguity, it is specifically stated that it is named as a specific "non-target type".
[0078] The two-class model can be Figure 2 As shown, any image feature extraction model can be used as the backbone model and connected to the classifier structure, which is trained to convergence using two types of training samples and put into use, the two types of training samples are positive samples and negative samples, wherein the positive samples are images to be detected that are pre-labeled as the target type, and the negative samples are images to be detected that are pre-labeled as the non-target type. Those skilled in the art can use a sufficient amount of the training samples to train the two-classification model according to the conventional neural network model training principle, train it to a convergent state, and obtain the confidence that the input image to be detected belongs to the classification corresponding to the target type.
[0079] The selection of the image feature extraction model includes but is not limited to various mature models based on CNN and RNN, such as Resnet, SwinTrasformer, EffiencyNet, etc., which can be flexibly selected by technical personnel in this field according to the actual performance of each mature model without affecting the embodiment of the creative spirit of this application.
[0080] In one embodiment of the present application, as Figure 3 As shown, when the image feature extraction model 78 is connected to the output layer 703 of the classifier 70, it is fully connected through the two fully connected layers 701 and 702, wherein the first fully connected layer 701 is responsible for extracting sufficiently rich features from the image feature extraction model, has a relatively large number of neurons, and outputs a higher dimensional result, while the second fully connected layer 702 is responsible for allowing the image features of the same type of images to be detected to share the same feature combination before the output layer, correspondingly reducing the number of image feature center points, and thus having a relatively small number of neurons. This will be further revealed later, so it will not be discussed here for the time being.
[0081] After the image to be detected is classified and mapped by the binary classification model, the confidence corresponding to its target type can be obtained, that is, the first confidence referred to in this application, which represents the classification probability that the image to be detected is identified as the target type by the binary classification model.
[0082] Step S1300: calling a multi-classification model to perform type discrimination on the image to be detected, and determining a second confidence level that the image belongs to the target type, wherein the multi-classification model is pre-trained to a convergence state:
[0083] The present application also prepares a multi-classification model, which is similar to the binary classification model in that the image to be detected is mapped to multiple categories after feature extraction to obtain the confidence of each category, wherein there are also two categories corresponding to the target type and non-target type respectively.
[0084] In one embodiment, the multi-classification model and the binary classification model can be completely isomorphic models, but their training processes and the number of classifications of the classifiers can be different. Therefore, it can be understood that the network structure of the multi-classification model and the network structure of the binary classification model can be completely the same. The multi-classification model and the binary classification model are implemented isomorphically, so that useful information in the training process can be shared as much as possible, so that the model trained later converges faster, thereby saving overall training costs.
[0085] In another embodiment, the image feature extraction models used as the backbone models of the multi-classification model and the binary classification model can be different. For example, the binary classification model can use Resnet, while the multi-classification model can use SwinTransformer or EfficientNet. The use of different image feature extraction models can realize the utilization of differentiation capabilities, thereby extracting feature information from multiple aspects of the image to be detected, and providing rich criteria for the classification and discrimination of the image to be detected.
[0086] After the image to be detected is classified and mapped by the multi-classification model, the confidence corresponding to its target type can be obtained, that is, the second confidence referred to in this application, which represents the classification probability that the image to be detected is identified as the target type by the multi-classification model.
[0087] It should be pointed out that, for step S1200 and step S1300, although they are described in chronological order in this embodiment, the present application does not limit their execution order in this way. Therefore, the execution order of the two can be interchanged or they can be executed concurrently. Those skilled in the art should understand this.
[0088] In an embodiment of this workaround, in order to save computing power, step S1200 or step S1300 can be made to first determine whether the image to be detected belongs to the target type. After the first step determines that it belongs to the target type, step S1300 or step S1200 is judged again to ensure that the image to be detected belongs to the target type through double detection. When it is determined that it does not belong to the target type or belongs to the non-target type in the first step, it can be directly confirmed that the image to be detected does not belong to the target type without having to perform another step of judgment. This embodiment fully takes into account the reality that a large number of pictures on the e-commerce platform are non-target type pictures, so it can be expected that the technical solution of this application can save a lot of computing power resources when it is implemented on a computer device.
[0089] Step S1400: The first confidence level and the second confidence level are integrated to obtain a comprehensive confidence level. When the comprehensive confidence level is greater than a preset threshold, it is confirmed that the image to be detected is of the target type.
[0090] After the image to be detected obtains the confidences mapped to the target type, namely the first confidence and the second confidence, respectively, through the binary classification model and the multi-classification model, the first confidence and the second confidence can be fused for joint judgment. An exemplary simple fusion method can be to compare the comprehensive confidence obtained by simply adding the two with a preset threshold. When the comprehensive confidence is greater than the preset threshold, it can be confirmed that the image to be detected is the target type, that is, the original carrier image of the image to be detected is determined to be the target type. Otherwise, the original carrier image can be determined to be the non-target type. It is not difficult to understand that when the target type is a pornographic image type, it means that the original carrier image is a pornographic image. Based on this, the e-commerce platform can make further responses based on its own preset information security business rules.
[0091] In addition to the above simple fusion method, the comprehensive confidence can also be obtained by methods such as weighted fusion, model fusion, etc. Other embodiments of the present application will further reveal other fusion methods, which will not be discussed here for the time being.
[0092] The preset threshold used here to determine the comprehensive confidence level may be an empirical threshold or a measured threshold, and may be flexibly set by those skilled in the art.
[0093] According to the embodiments disclosed herein, it can be understood that, compared with the prior art, the present application has many advantages, including at least the following aspects:
[0094] This application uses a binary classification model and a multi-classification model to detect the same image to be detected, respectively, to obtain the confidence of the deep semantic features of the image to be detected output by each model mapped to the target type, and then, fuses the confidences of the two classification models to obtain a comprehensive confidence, and identifies the target type of the image to be detected whose comprehensive confidence is greater than a preset threshold, thereby utilizing the classification capabilities of the two models at different granularities to achieve accurate identification of the type of the image to be detected.
[0095] Since the recognition process of the image to be detected integrates the judgment results of the two models, the binary classification model can focus on the global information of the image to be detected to obtain its classification results, and the multi-classification model can focus on the detailed information of the image to be detected to obtain its classification results. Finally, the confidence corresponding to the target type in the two classification results is used to comprehensively determine whether the image to be detected is an image of the target type, which is more comprehensive and accurate.
[0096] Therefore, the judgment result made by the present application on whether the image to be detected contains images of the target type has a higher confidence level and can be highly trusted. It is suitable for detecting whether there are images of the target type in advertising pictures in application scenarios such as e-commerce platforms. It can provide more accurate judgment results for needs such as pornographic pictures, thereby maintaining the information security of e-commerce platforms and avoiding the appearance of illegal images in their advertising pictures.
[0097] See also Figure 4 In some embodiments, before the step S1200, calling the binary classification model to distinguish the type of the image to be detected, the following steps are included:
[0098] Step S2100, using the training samples in the data set to perform iterative training on the binary classification model until convergence, the training samples include the images to be detected that are positive samples marked as target types and the images to be detected that are negative samples marked as non-target types, and the number of categories of the classifier of the binary classification model is configured to be greater than two, wherein the two categories correspond to the target type and the non-target type respectively:
[0099] As mentioned above, before the binary classification model is put into use, it is pre-trained to a convergence state using a sufficient number of training samples. Accordingly, before training, a data set is prepared, which may include a large number of training samples, each of which contains an image to be detected. Then, each training sample is labeled according to positive samples and negative samples, that is, for images to be detected that are positive samples, they are labeled as target types; for images to be detected that are negative samples, they are labeled as the non-target types.
[0100] On the basis of the prepared data set, the number of classifications of the classifier of the binary classification model is configured. In this embodiment, the number of classifications of the binary classification model is configured to be greater than two categories, for example, 500. That is, although it is ultimately expected that the binary classification model will be used for binary classification prediction, it is trained according to multi-classification prediction during its training phase. The purpose is to make the confidence of the target type output by the binary classification model more reliable under the supervised training of the positive and negative samples, and to avoid the situation where the confidence of the classification corresponding to the target type and the non-target type is very high when only two classifications are set. Subsequently, after the binary classification model is trained to convergence, when the confidence of the classification corresponding to the target type and the non-target type is required, the corresponding confidence can be obtained, and other classifications may not be adopted. Therefore, after it is put into use, it is essentially still a binary classification model.
[0101] After completing the configuration of the binary classification model, it can be iteratively trained using the training samples in the data set. Each training process may include the following steps:
[0102] First, an image to be detected in a training sample in the data set is input into a binary classification model, and the image feature extraction model therein extracts deep semantic information from it to obtain corresponding image feature information;
[0103] Secondly, the image feature information is fully connected through the fully connected layer in the classifier, and then mapped to the output layer of the classifier, and the classifications corresponding to the number of classifications mapped thereto are calculated to obtain the confidence under each classification;
[0104] Then, according to the label corresponding to the training sample, that is, the target type or the non-target type, a cross entropy loss function is applied to calculate the model loss value for the model output result;
[0105] Finally, whether to continue iterative training is determined based on whether the model loss value reaches a preset threshold. Specifically, when the model loss value reaches the preset threshold, it can be considered that the binary classification model has been trained to a convergence state, so the training of the binary classification model can be terminated. Otherwise, it indicates that the binary classification model has not yet converged. Therefore, the binary classification model is back-propagated according to the model loss value to implement gradient update, and then the next training sample is called from the data set to continue iterative training of the binary classification model until the model is trained to a convergence state.
[0106] The embodiment disclosed herein shows that during the training stage, the binary classification model pre-configures multiple classification numbers and uses two types of training samples for training, so that the confidence obtained for the two classifications of the target type and non-target type is more credible, thereby improving the reliability of the recognition results of the binary classification model in the reasoning stage and improving its accuracy in identifying the type of the image to be detected.
[0107] It should be noted that in the embodiments disclosed herein, there is no need to consider the training method of the multi-classification model, that is, the training method of the multi-classification model does not affect the acquisition of the classification and recognition capability of the binary classification model here, and those skilled in the art should be aware of this.
[0108] Please continue reading Figure 4 In some embodiments, a multi-classification model can be prepared based on the binary classification model trained in the above embodiments. Therefore, before the step S1300, invoking the multi-classification model to distinguish the type of the image to be detected, the following steps are included:
[0109] Step S2200: Use the binary classification model trained to a convergent state to test the training samples in the data set to predict the classification of each training sample:
[0110] After the binary classification model is trained to a convergent state, it is used to test a data set, and each training sample in the data set is tested. The data set referred to here can be the data set used to train the binary classification model, or it can be derived from a subset of the data set, or it can be prepared by technical personnel in the field according to the structure of the data set.
[0111] Each training sample in the data set is input into the binary classification model one by one for classification prediction, and the classification results of each training sample are obtained. In this process, the confidence level of each training sample corresponding to the target type and the non-target type can be obtained, so as to determine whether each training sample belongs to the target type or the non-target type.
[0112] Step S2300: clustering is performed according to the activated neuron combination in the fully connected layer of each training sample before classification, adjusting the labeling information of the training samples of the data set, labeling the training samples with the same neuron combination and original labeling type in the data set as the same type, wherein the two types correspond to the target type and the non-target type respectively, and labeling the training samples with different neuron combinations or original labeling types in the data set as different types accordingly:
[0113] In order to configure the number of classifications of the multi-classification model to be trained, this embodiment analyzes and clusters the neuron activation performance of the last fully connected layer of the binary classification model during the process of predicting the classification of each training sample of the data set using the binary classification model to obtain the number of classifications used to configure the multi-classification model.
[0114] Specifically, the binary classification model is reactivated in the last fully connected layer before the output layer of its classifier. When the value of a neuron in the fully connected layer is greater than a preset threshold, its output value is 1, indicating that the neuron is activated. Otherwise, its output value is 0, indicating that the neuron is not activated. Thus, after each training sample is predicted, the combination of activated neurons obtained in the last fully connected layer is determined.
[0115] Then, the annotation type of each training sample in the data set is adjusted according to the following principles:
[0116] For multiple training samples that were originally labeled as the same type (the same target type or non-target type), when they are predicted by the binary classification model, the activated neuron combinations are consistent. These training samples are retained as the originally labeled types (target type or non-target type) and labeled accordingly.
[0117] For multiple training samples that were originally labeled as different types (target type and non-target type), when they are predicted by the binary classification model and the activated neuron combinations are consistent, these training samples are regarded as different types and labeled accordingly.
[0118] Of course, any training samples with different neuron combinations are considered to be of the same type and can be labeled accordingly.
[0119] According to the type classification principle disclosed above, that is, by clustering the types corresponding to the training samples according to the activated neuron combination of the last fully connected layer in the binary classification model, the corresponding clustering results can be obtained.
[0120] Step S2400: Count the total number of categories according to the clustering results, and configure the total number of categories as the number of categories of the multi-classification model:
[0121] Further, according to the clustering results obtained after aggregation, the total number of categories is counted, and then configured as the number of categories of the multi-classification model to be trained, so as to configure the multi-classification model before training. It can be seen from this that the multi-classification model in this embodiment utilizes the classification prediction ability learned by the binary classification model, clusters according to the activated neuron combination of the last fully connected layer of the binary classification model, configures its own number of categories according to the total number of categories obtained by clustering, and obtains the annotation information guided by the preliminary classification ability learned by the binary classification model, eliminating the time cost of manual annotation, but can promote itself to converge faster when being trained, greatly improving the training speed of the multi-classification model. In particular, for the embodiments in which the binary classification model and the multi-classification model are isomorphic, since the two models are isomorphic, the multi-classification model can realize the corresponding mapping of the feature extraction ability of the binary classification model, so that the multi-classification model essentially inherits part of the ability obtained by the binary classification model training.
[0122] Step S2500: using the training samples in the data set to perform iterative training on the multi-classification model until convergence:
[0123] After completing the configuration of the number of classifications of the multi-classification model and the preparation of its data set, the multi-classification model can be iteratively trained using each training sample in the data set according to the same training process as the binary classification model, so that the multi-classification model can be quickly trained to a convergence state for use in the reasoning of this application. Since the training principle of the multi-classification model is the same as that of the binary classification model, it will not be repeated here.
[0124] In addition, the embodiments disclose that the data set of the multi-classification model of the present application is labeled according to the classification ability learned by the binary classification model, and the corresponding number of classifications is determined by clustering the activated neuron combination of the last fully connected layer before the output layer of the classifier of the binary classification model, so as to complete the labeling of the corresponding training samples, realize a finer-grained division of the target type and non-target type of the training samples in the original data set, cluster the same type of data with the same neuron combination characteristics, and split the different types of data with the same neuron combination characteristics into categories. After multi-classification retraining, the samples that are misclassified or prone to error in the binary classification model are corrected in the multi-classification model, which greatly improves the feature generalization ability of the multi-classification model, thereby making it easier for the multi-classification model to converge quickly and ensure that it obtains accurate classification prediction results, which can improve the classification and accurate recognition ability of the image to be detected.
[0125] See also Figure 5 In some embodiments, the binary classification model and the multi-classification model include the same network model architecture, and the exemplary architecture is as follows: Figure 3 As shown, it includes an image feature extraction model 78 and a classifier 70, wherein the classifier includes two fully connected layers 701, 702 and an output layer 703. The network model architecture performs the following steps for each image to be detected:
[0126] Step S3100: extracting image feature information of the image to be detected through an image feature extraction model:
[0127] As mentioned above, when the image to be detected is input into the binary classification model or the multi-classification model, the image feature extraction model therein extracts deep semantic information from the image to be detected, thereby obtaining its corresponding image feature information. In this embodiment, the image feature extraction model is recommended to use SwinTransformer, which has achieved better performance in the actual measurement of this application, and of course other possible selections are not excluded.
[0128] Step S3200: Perform a full connection operation on the image feature information through the first fully connected layer to obtain comprehensive feature information:
[0129] The image feature extraction model is followed by the first fully connected layer in the classifier, so the image feature extraction model is fully connected through the first fully connected layer, and after performing the fully connected operation, comprehensive feature information is obtained. The number of neurons in the first fully connected layer can be set to a large number, such as 2048, so that the comprehensive feature information can extract sufficiently rich features from the image feature information.
[0130] Step S3300: Perform a full connection operation on the comprehensive feature information through the second fully connected layer to obtain classification feature information, where the dimension of the classification feature information is lower than that of the comprehensive feature information:
[0131] The comprehensive feature information is further input into a second fully connected layer, and the number of neurons in the second fully connected layer is set to a number that is significantly reduced compared to the first fully connected layer, for example, 64, so as to further map the high-dimensional comprehensive feature information into fewer dimensions. Similarly, the second fully connected layer performs a full connection operation on the comprehensive feature information to extract the comprehensive feature information into classification feature information with a lower dimension.
[0132] Step S3400: Map the classification feature information to each category corresponding to its preset number of categories through the output layer of the classifier, and obtain the confidence corresponding to each category:
[0133] Finally, the output layer of the classifier can be constructed using the Softmax function. The output layer calculates the classification probability of each classification corresponding to the number of classifications configured by the corresponding binary classification model or multi-classification model based on the classification feature information. The classification probability of each classification is the confidence corresponding to the classification. When the confidence of the classification corresponding to the target type is the highest, for the corresponding binary classification model or multi-classification model, the image to be detected is the target type.
[0134] According to the embodiments disclosed herein, when the binary classification model and the multi-classification model are isomorphic, for each image to be detected, rich features are extracted through the first fully connected layer, and then the features are classified through the second fully connected layer, and finally the classification prediction is made through the output layer. When both models are trained on the same data set and the multi-classification model adjusts the annotation information of the data set according to the binary classification model, it can be ensured that the confidence obtained by the two models is relatively credible.
[0135] In some embodiments, the step S1400, fusing the first confidence and the second confidence to obtain a comprehensive confidence, and when the comprehensive confidence is greater than a preset threshold, confirming that the image to be detected is the target type, includes the following steps:
[0136] Step S1411: Match the first confidence level and the second confidence level with preset weights respectively for weighted summation to obtain a comprehensive confidence level:
[0137] In this embodiment, taking into account that the binary classification model and the multi-classification model have their own emphases, a weighted summation method can be adopted when calculating the comprehensive confidence, and the first confidence of the target type output corresponding to the binary classification model and the second confidence of the target type output corresponding to the multi-classification model are assigned to match the preset weights respectively, and the two are added on the basis of the matching weights to obtain the comprehensive confidence.
[0138] In one embodiment, a hyperparameter may be used to match the weights of the first confidence and the second confidence to achieve smoothing and normalization, thereby obtaining the comprehensive confidence.
[0139] Step S1412: determine whether the comprehensive confidence is greater than a preset threshold. If it is greater than the preset threshold, determine that the image to be detected is the target type.
[0140] Finally, a preset threshold is used to determine whether the comprehensive confidence is greater than the preset threshold. When it is greater than the preset threshold, it can be confirmed that the image to be detected is a target type, so that the next business operation can be performed accordingly. Similarly, the preset threshold can be an empirical threshold or an experimental threshold, which can be flexibly set by those skilled in the art.
[0141] The embodiment here determines the comprehensive confidence by weighted summing the first confidence and the second confidence, and makes a judgment on whether the image to be detected belongs to the target type based on this. The implementation is simple, but the weight of the binary classification model and the multi-classification model in the image recognition process can be flexibly adjusted by adjusting the weights of the first confidence and the second confidence, which is convenient for adjustment according to actual conditions, thereby improving the flexibility of implementing the present application.
[0142] In some other embodiments, the step S1400, fusing the first confidence and the second confidence to obtain a comprehensive confidence, and when the comprehensive confidence is greater than a preset threshold, confirming that the image to be detected is the target type, includes the following steps:
[0143] Step S1421, input the first confidence and the second confidence into the fully connected layer, and the fully connected layer associates the learned weights to map the first confidence and the second confidence to the two categories of the binary classifier, and obtain the comprehensive confidences corresponding to the two categories, respectively, and the two categories correspond to the target type and the non-target type, respectively. The fully connected layer pre-uses the training samples in the data set to perform training until the convergence state to obtain the weights:
[0144] See also Figure 6 The present application further uses a fully connected layer 901 to synthesize the first confidence and the second confidence matching learnable weights, so that they are mapped to two categories through the output layer, and the two categories correspond to the target type and the non-target type respectively. Thus, the fully connected layer 901 actually constructs a binary classifier 90, thereby obtaining Figure 6Based on the network model shown, it is not difficult to understand that the binary classifier 90 can be connected to the binary classification model 91 and the multi-classification model 92 to implement joint training, so that the fully connected layer 901 of the binary classifier 90 obtains the weights after training, and then when put into reasoning use, for the same image to be detected, the first confidence output by the binary classification model 91 and the second confidence output by the multi-classification model 92 are matched with the learned weights respectively, and after classification mapping, the comprehensive confidence corresponding to the classification of the target type and non-target type is obtained.
[0145] Step S1422: determine whether the comprehensive confidence of the classification corresponding to the target type is greater than a preset threshold. If it is greater than the preset threshold, determine that the image to be detected is the target type.
[0146] Similarly, a preset threshold is used to determine whether the comprehensive confidence corresponding to the target category is greater than the preset threshold. When it is greater than the preset threshold, it can be confirmed that the image to be detected is a target type, so that the next business operation can be performed accordingly. The preset threshold can be an empirical threshold or an experimental threshold, which can be flexibly set by those skilled in the art.
[0147] The embodiments herein disclose that the fusion judgment of the first confidence level and the second confidence level can be realized through a jointly trained classifier, thereby making the category recognition of the image to be detected more intelligent and smooth, and it is expected that the accuracy of the category recognition of the image to be detected can be more reliable.
[0148] See also Figure 7 In combination with the exemplary e-commerce advertising application scenario of the present application, in some extended embodiments, after confirming that the image to be detected is the target type, the following steps are included:
[0149] Step S4100: suspend the processing of the image to be detected.
[0150] The image to be detected is derived from an advertisement picture published by a merchant user. When a merchant user promotes the advertisement picture, it is submitted to the server, expecting to perform the processing business corresponding to the advertisement publishing. After the server obtains the advertisement picture, it uses it as the original carrier picture and obtains the image to be detected after image preprocessing.
[0151] Then, the method of any embodiment of the present application is applied to perform image recognition on the image to be detected. When it is confirmed through identification that the image to be detected is the target type, the processing flow of the processing business is suspended and the information security audit process is called instead.
[0152] Step S4200: construct a notification message containing the image to be detected and transmit it to the review interface:
[0153] In order to call the information security audit process, the advertising image and / or its image to be detected, the target type, and the characteristic information of the merchant user and / or its online store are encapsulated as a notification message and transmitted to a preset audit interface. The audit interface can be a manual audit interface, for example, through an e-mail or a dedicated data storage address, or a dedicated interface function, and can eventually be received and processed by a manual user, and the manual user's corresponding instruction indicating consent or refusal to publish is returned.
[0154] Step S4300: monitor the instruction returned by the review interface indicating the approval of the release, and resume the processing flow:
[0155] After submitting the notification message to the review interface, the server starts to listen to the instructions returned by the review interface. When the review interface returns an instruction indicating a refusal to publish, the processing flow of the processing business can be interrupted, and the corresponding notification message is returned to the corresponding merchant user; when the review interface returns an instruction indicating an agreement to publish, it indicates that after manual review, the image to be detected is not a violation type, and the processing flow of the processing business can be resumed to realize advertisement publishing.
[0156] The embodiments disclosed herein exemplarily improve the business process of the present application in a specific application scenario. It can be seen that by further providing an audit interface audit link and further checking and supplementing the results identified by the model with the help of manual detection results, the health of the advertising images of the e-commerce platform can be ensured, thereby maintaining the information ecology of the e-commerce platform. Since the technical solution of the present application can be expected to achieve accurate identification results, calling the audit interface is a low-probability event, but it plays a bottom-line role, and it can be expected to achieve accurate security detection results for massive image information on the e-commerce platform at a lower labor cost.
[0157] See also Figure 8 , an image type recognition device provided to meet one of the purposes of the present application is a functional embodiment of the image type recognition method of the present application, the device includes: an image acquisition module 1100, a first recognition module 1200, a second recognition module 1300, and a fusion judgment module 1400, wherein: the image acquisition module 1100 is used to acquire the image to be detected; the first recognition module 1200 is used to call a binary classification model to perform type discrimination on the image to be detected, and determine a first confidence that it belongs to the target type, and the binary classification model is pre-trained to a convergence state; the second recognition module 1300 is used to call a multi-classification model to perform type discrimination on the image to be detected, and determine a second confidence that it belongs to the target type, and the multi-classification model is pre-trained to a convergence state; the fusion judgment module 1400 is used to fuse the first confidence and the second confidence to obtain a comprehensive confidence, and when the comprehensive confidence is greater than a preset threshold, it is confirmed that the image to be detected is the target type.
[0158] In some embodiments, prior to the first recognition module 1200, it also includes: a first training module, which is used to use training samples in the data set to perform iterative training on the binary classification model until a convergence state, the training samples include images to be detected that are positive samples marked as target types and images to be detected that are negative samples marked as non-target types, and the number of categories of the classifier of the binary classification model is configured to be greater than two, wherein the two categories correspond to the target type and the non-target type, respectively.
[0159] In some embodiments, prior to the second recognition module 1300, a test implementation module is included, which is used to use the binary classification model trained to a convergent state to test the training samples in the data set and predict the classification of each training sample; a clustering evaluation module is used to cluster according to the activated neuron combination in the fully connected layer of each training sample before classification, adjust the labeling information of the training samples in the data set, label the training samples with the same neuron combination and original labeling type in the data set as the same type, wherein the two types correspond to the target type and the non-target type respectively, and label the training samples with different neuron combinations or original labeling types in the data set as different types; a category setting module is used to count the total number of categories according to the clustering results, and configure the total number of categories as the number of categories of the multi-classification model; a second training module is used to use the training samples in the data set to implement iterative training on the multi-classification model until convergence.
[0160] In some further embodiments, the binary classification model and the multi-classification model include the same network model architecture, which includes: a feature extraction unit, configured to extract image feature information of the image to be detected through an image feature extraction model; a first connection unit, configured to perform a full connection operation on the image feature information through a first fully connected layer to obtain comprehensive feature information; a second connection unit, configured to perform a full connection operation on the comprehensive feature information through a second fully connected layer to obtain classification feature information, wherein the dimension of the classification feature information is lower than that of the comprehensive feature information; a classification mapping unit, configured to map the classification feature information to each classification corresponding to its preset number of classifications through the output layer of the classifier, and obtain the confidence corresponding to each classification.
[0161] In some embodiments, the fusion judgment module 1400 includes: a weighted summation unit, used to match the first confidence and the second confidence with preset weights for weighted summation to obtain a comprehensive confidence; an image judgment unit, used to determine whether the comprehensive confidence is greater than a preset threshold, and when it is greater than the preset threshold, determine that the image to be detected is the target type.
[0162] In some other embodiments, the fusion judgment module 1400 includes a connection synthesis unit, which is used to input the first confidence and the second confidence into a fully connected layer, and the fully connected layer associates the learned weights to map the first confidence and the second confidence to two categories of the binary classifier, and obtains the comprehensive confidences corresponding to the two categories, respectively. The two categories correspond to the target type and the non-target type, respectively. The fully connected layer pre-trains using the training samples in the data set to a convergence state to obtain the weights; a classification judgment unit, which is used to determine whether the comprehensive confidence of the category corresponding to the target type is greater than a preset threshold. When it is greater than the preset threshold, it is determined that the image to be detected is the target type.
[0163] In some extended embodiments, after the fusion judgment module 1400, it includes: a process suspension unit, used to suspend the processing flow of the processing business of the image to be detected; an interface calling unit, used to construct a notification message containing the image to be detected and transmit it to the review interface; a process recovery unit, used to listen to the instructions returned by the review interface that represent the consent to be released, and resume the processing flow.
[0164] In order to solve the above technical problems, the present application also provides a computer device. Fig. 9 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor may implement an image type recognition method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may execute the image type recognition method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0165] In this embodiment, the processor is used to execute Figure 8The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the image type recognition device of the present application, and the server can call the program code and data of the server to execute the functions of all submodules.
[0166] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the image type recognition method of any embodiment of the present application.
[0167] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.
[0168] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0169] To sum up, the judgment result made by the present application on whether the detected image contains images of the target type has a higher confidence level and can be highly trusted. It is suitable for detecting whether there are images of the target type in advertising pictures in application scenarios such as e-commerce platforms. It can provide more accurate judgment results for needs such as pornographic pictures, thereby maintaining the information security of e-commerce platforms and avoiding the appearance of illegal images in their advertising pictures.
[0170] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.
[0171] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for identifying image types, It is characterized in that The steps include: Iteratively training the binary classification model using training samples in the data set until convergence, wherein the training samples include images to be detected that are positive samples labeled as target types and images to be detected that are negative samples labeled as non-target types, and the number of classifications of the classifier of the binary classification model is configured to be greater than two, wherein the two classifications correspond to the target type and the non-target type, respectively; Using the binary classification model trained to a convergent state to test the training samples in the data set, and predicting the classification of each training sample; Clustering is performed according to the activated neuron combination in the fully connected layer of each training sample before classification, adjusting the labeling information of the training samples of the data set, labeling the training samples with the same neuron combination and original labeling type in the data set as the same type, wherein the two types correspond to the target type and the non-target type respectively, and labeling the training samples with different neuron combinations or original labeling types in the data set as different types accordingly; According to the clustering results, the total number of categories is counted and configured as the number of categories of the multi-classification model; Using the training samples in the data set to perform iterative training on the multi-classification model until convergence; Acquire the image to be detected; Calling a binary classification model to perform type discrimination on the image to be detected, and determining a first confidence level that the image belongs to a target type, wherein the binary classification model is pre-trained to a convergence state; Calling a multi-classification model to perform type discrimination on the image to be detected, and determining a second confidence level that the image belongs to a target type, wherein the multi-classification model is pre-trained to a convergence state; The first confidence level and the second confidence level are integrated to obtain a comprehensive confidence level. When the comprehensive confidence level is greater than a preset threshold, the image to be detected is confirmed to be of the target type.
2. The image type recognition method according to claim 1, It is characterized in that The binary classification model and the multi-classification model include the same network model architecture, which performs the following steps for each image to be detected: Extracting image feature information of the image to be detected through an image feature extraction model; Performing a full connection operation on the image feature information through a first fully connected layer to obtain comprehensive feature information; Performing a full connection operation on the comprehensive feature information through a second fully connected layer to obtain classification feature information, wherein the dimension of the classification feature information is lower than that of the comprehensive feature information; The classification feature information is mapped to each classification corresponding to its preset number of classifications through the output layer of the classifier to obtain the corresponding confidence level of each classification.
3. The image type recognition method according to claim 1, It is characterized in that The first confidence level and the second confidence level are integrated to obtain a comprehensive confidence level. When the comprehensive confidence level is greater than a preset threshold, the image to be detected is confirmed to be the target type, including the following steps: Matching the first confidence level and the second confidence level with preset weights respectively for weighted summation to obtain a comprehensive confidence level; It is determined whether the comprehensive confidence is greater than a preset threshold, and when it is greater than the preset threshold, it is determined that the image to be detected is the target type.
4. The image type recognition method according to claim 1, It is characterized in that The first confidence level and the second confidence level are integrated to obtain a comprehensive confidence level. When the comprehensive confidence level is greater than a preset threshold, the image to be detected is confirmed to be the target type, including the following steps: Inputting the first confidence and the second confidence into a fully connected layer, and mapping the first confidence and the second confidence to two categories of a binary classifier by associating the learned weights by the fully connected layer, respectively obtaining comprehensive confidences corresponding to the two categories, the two categories corresponding to the target type and the non-target type, respectively, and the fully connected layer preliminarily adopts the training samples in the data set to perform training until a convergence state to obtain the weights; It is determined whether the comprehensive confidence of the classification corresponding to the target type is greater than a preset threshold. When it is greater than the preset threshold, the image to be detected is determined to be the target type.
5. The image type recognition method according to claim 1, It is characterized in that After confirming that the image to be detected is of the target type, the following steps are included: Suspending the processing flow of the processing business of the image to be detected; Constructing a notification message containing the image to be detected and transmitting it to the review interface; The audit interface is monitored for instructions indicating approval of the release, and the processing flow is resumed.
6. An image type recognition device, It is characterized in that include: An image acquisition module, used to implement iterative training on the binary classification model using training samples in the data set until convergence, wherein the training samples include images to be detected that are positive samples marked as target types and images to be detected that are negative samples marked as non-target types, and the number of categories of the classifier of the binary classification model is configured to be greater than two, wherein the two categories correspond to the target type and the non-target type, respectively; The binary classification model trained to a convergent state is used to test the training samples in the data set to predict the classification of each training sample; clustering is performed according to the activated neuron combination in the fully connected layer of each training sample before classification, and the labeling information of the training samples in the data set is adjusted, and the training samples with the same neuron combination and original labeling type in the data set are labeled as the same type, wherein the two types correspond to the target type and the non-target type respectively, and the training samples with different neuron combinations or original labeling types in the data set are labeled as different types accordingly; the total number of categories is counted according to the clustering results, and the total number of categories is configured as the number of categories of the multi-classification model; the multi-classification model is iteratively trained to a convergent state using the training samples in the data set; and an image to be detected is obtained; A first recognition module is used to call a binary classification model to perform type discrimination on the image to be detected and determine a first confidence level that the image belongs to a target type, wherein the binary classification model is pre-trained to a convergence state; A second recognition module is used to call a multi-classification model to perform type discrimination on the image to be detected and determine a second confidence level that the image belongs to a target type, wherein the multi-classification model is pre-trained to a convergence state; The fusion judgment module is used to fuse the first confidence and the second confidence to obtain a comprehensive confidence. When the comprehensive confidence is greater than a preset threshold, it is confirmed that the image to be detected is the target type.
7. A computer device comprising a central processing unit and a memory, It is characterized in that The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.
8. A computer-readable storage medium, It is characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 5 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.
