A picture multi-label classification processing method and related apparatus

Abstract

Embodiments of the present application disclose a picture multi-label classification processing method and related device, at least related to artificial intelligence and other technologies, reducing unnecessary algorithm repeated modeling in the field of picture quality evaluation, reducing research and development cost investment, and quickly responding to business needs, suitable for various application scenarios. The method comprises: obtaining a to-be-predicted picture and a preset label classification result, the preset label classification result comprising at least two marked labels; performing feature extraction processing on the to-be-predicted picture based on a preset self-supervised model to obtain picture embedding features of the to-be-predicted picture; performing feature extraction on each of the marked labels based on a preset label model to obtain label embedding features of the marked labels; and performing fusion processing on the picture embedding features and the label embedding features based on a preset multi-label classification model to obtain a multi-label classification result of the to-be-predicted picture.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of picture recognition, and in particular to a picture multi-label classification processing method and related device. BACKGROUND

[0002] With the rapid development of the Internet and the lowering of the threshold of content production, the release of various contents is growing at an exponential rate, and people can access all kinds of information anytime and anywhere. Whether it is text content or video content, the core factor for the object to see the content is the title, cover picture and self-media account name of the content. For example, in the text content of the consumption scene, the quality of the cover picture greatly affects the object's desire to click and browse. The quality of the cover picture can be seen from two aspects. On the one hand, the quality of the picture, such as whether the picture is clear and beautiful, etc. On the other hand, the information conveyed by the picture, such as whether the content of the picture is effective and consistent with the theme. In addition, picture label extraction and label classification are also very important for picture processing business in information flow. A large number of pictures are involved in the business processing process, such as picture selection, picture cropping, picture-text relevance, similar and repeated picture calculation, etc. In the processing process of related information flow picture content, the most typical scene is the cover picture processing scene, which mainly involves picture selection and cropping. Figure 2 In each process, multiple dimensional feature labels of the picture need to be used, such as physical dimension, quality dimension, information dimension and perception dimension, etc. Moreover, in these processing processes, a large number of picture features are involved, and different thresholds or combinations are set for these picture features, for example, when outputting a cover picture, the quality of multiple cover pictures needs to be judged comprehensively, and the picture with the highest comprehensive rating is selected as the high-quality cover and output.

[0003] However, in the process of picture label classification of the cover picture, the existing label classification model usually takes the extracted picture and the context text information corresponding to the picture as input, and then predicts the label result corresponding to the picture. Due to the very extensive source of picture content, the label classification model needs to use different processing models to process the picture and the context text information of the picture separately in the process of determining the label of the picture from different angles, such as determining the clarity, beauty and watermark recognition of the picture, which respectively need a processing model to process the picture and the context text information of the picture according to different picture features, thereby causing these processing models to be unable to be reused, and the cost of separate calculation is very high. Moreover, in different application scenarios, due to the difference in the context text information of pictures of different categories, different picture features need to be combined and applied or different thresholds need to be set, and multiple processing models need to be upgraded and updated at the same time, which causes the response speed of business processing to be slow. SUMMARY

[0004] The embodiment of the present application provides a picture multi-label classification processing method and a related device, which can effectively reduce unnecessary algorithm repeated modeling in the picture quality evaluation field, reduce the research and development cost investment, improve the research and development efficiency, quickly respond to the business demand, and be suitable for various application scenarios.

[0005] In a first aspect, the embodiment of the present application provides a picture multi-label classification processing method. The method comprises the following steps: obtaining a to-be-predicted picture and a preset label classification result, wherein the preset label classification result comprises at least two marked labels, and each marked label is used to indicate a preset picture classification condition of the to-be-predicted picture; performing feature extraction processing on the to-be-predicted picture based on a preset self-supervised model to obtain picture embedding features of the to-be-predicted picture; performing feature extraction on each marked label based on a preset label model to obtain label embedding features of the marked label; and performing fusion processing on the picture embedding features and the label embedding features based on a preset multi-label classification model to obtain a multi-label classification result of the to-be-predicted picture, wherein the multi-label classification result is used to indicate a target picture category matched by the to-be-predicted picture, and the preset multi-label classification model is a machine learning model obtained by iterative training with a multi-label classification result of a sample picture as a training target and with picture embedding features of the sample picture and label embedding features corresponding to the marked label of the sample picture as training data.

[0006] In a second aspect, the embodiment of the present application provides a multi-label classification processing device. The multi-label classification processing device comprises, but is not limited to, a terminal device, a server and the like. The multi-label classification processing device comprises an obtaining unit and a processing unit. The obtaining unit is configured to obtain a to-be-predicted picture and a preset label classification result, wherein the preset label classification result comprises at least two marked labels, and each marked label is used to indicate a preset picture classification condition of the to-be-predicted picture. The processing unit is configured to perform feature extraction processing on the to-be-predicted picture based on a preset self-supervised model to obtain picture embedding features of the to-be-predicted picture, perform feature extraction on each marked label based on a preset label model to obtain label embedding features of the marked label, and perform fusion processing on the picture embedding features and the label embedding features based on a preset multi-label classification model to obtain a multi-label classification result of the to-be-predicted picture, wherein the multi-label classification result is used to indicate a target picture category matched by the to-be-predicted picture, and the preset multi-label classification model is a machine learning model obtained by iterative training with a multi-label classification result of a sample picture as a training target and with picture embedding features of the sample picture and label embedding features corresponding to the marked label of the sample picture as training data.

[0007] In some optional examples, the processing unit is configured to: input the picture embedding feature and the label embedding feature into the preset multi-label classification model to obtain a predicted classification probability value of each marked label of the picture to be predicted; and determine a multi-label classification result of the picture to be predicted based on the predicted classification probability value of each marked label.

[0008] In some optional examples, the preset multi-label classification model comprises a first Transform sub-model and a second Transform sub-model; and the processing unit is configured to: perform encoding processing on the picture embedding feature and the label embedding feature based on the first Transform sub-model to obtain an encoded feature vector; and perform fusion processing on the encoded feature vector and the picture embedding feature based on the second first Transform sub-model to obtain the predicted classification probability value of each marked label of the picture to be predicted.

[0009] In some optional examples, the obtaining unit is further configured to obtain a target loss value based on a difference between each marked label and the corresponding predicted classification probability value; and the processing unit is further configured to update a model parameter of the preset multi-label classification model based on the target loss value to obtain a trained preset multi-label classification model.

[0010] In some optional examples, the processing unit is configured to: calculate a difference between each marked label and the corresponding predicted classification probability value to obtain a loss value between each marked label and the corresponding predicted classification probability value; and perform average processing on the loss value between each marked label and the corresponding predicted classification probability value to obtain a target loss value.

[0011] In some optional examples, the processing unit is further configured to, after determining the multi-label classification result of the picture to be predicted based on the predicted classification probability value of each marked label, determine a business policy corresponding to each label in the multi-label classification result based on each label, the business policy being used to indicate a picture processing policy adopted when the picture is processed; and perform business processing on the picture to be predicted based on the label and the business policy.

[0012] In some optional examples, the obtaining unit is further configured to: obtain a training sample set, the training sample set comprising at least two picture samples; and the processing unit is configured to process a sample label and a sample feature similarity based on a preset contrast loss model to obtain a contrast loss value, wherein the sample label is used to indicate a matching condition of each two picture samples in the at least two picture samples, and the sample similarity is a similarity between sample features of each two picture samples; and update a model parameter of the preset self-supervised model based on the contrast loss value to obtain a trained preset self-supervised model.

[0013] In some optional examples, the label tags include one or more of picture content, picture geometry, picture style, picture quality, shooting skill, and object feeling to the picture to be predicted of the picture to be predicted.

[0014] The third aspect of the embodiments of the present application provides a multi-label classification processing apparatus, comprising a memory, an input / output (I / O) interface and a processor. The memory is configured to store program instructions. The processor is configured to execute the program instructions in the memory to perform the picture multi-label classification processing method corresponding to the implementation manners of the first aspect.

[0015] The fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores instructions. When the instructions are run on a computer, the computer is caused to perform the method corresponding to the implementation manners of the first aspect.

[0016] The fifth aspect of the embodiments of the present application provides a computer program product containing instructions, which, when run on a computer or a processor, causes the computer or the processor to perform the method corresponding to the implementation manners of the first aspect.

[0017] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0018] In this embodiment, since the preset label classification result includes at least two labels, each label indicating the preset image classification of the image to be predicted, after obtaining the image to be predicted, feature extraction processing can be performed on the image to be predicted using a preset self-supervised model to obtain the image embedding features of the image to be predicted. After obtaining the preset label classification result, feature processing is performed on each label using a preset label model to obtain the label embedding features of the label. Then, the image embedding features and label embedding features are fused using a preset multi-label classification model to obtain the multi-label classification result of the image to be predicted. This multi-label classification result indicates the target image category matched by the image to be predicted. The above method effectively extracts image embedding features using a pre-defined self-supervised model. The pre-defined multi-label classification model is trained iteratively using the multi-label classification results of sample images as the training objective and the image embedding features and corresponding label embedding features of the sample images as training data. Therefore, using the image embedding features and label embedding features of the image to be predicted as input to this pre-defined multi-label classification model allows for prediction of the image's multi-label classification result. This effectively reduces unnecessary repetitive modeling in image quality assessment, lowers R&D costs, and improves R&D efficiency. Furthermore, it eliminates the need to consider the contextual text information of the image to be predicted, thus avoiding the need to combine different image features. Instead, it combines multiple labels from the final predicted multi-label classification result to solve different business problems, enabling rapid response to business needs and applicability to various application scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This illustration shows a schematic diagram of a multi-tag image classification processing flow framework provided in an embodiment of this application;

[0021] Figure 2 A flowchart of an image multi-tag classification processing method provided in an embodiment of this application is shown;

[0022] Figure 3 This illustration shows a schematic diagram of the training structure of a self-supervised model provided in an embodiment of this application;

[0023] Figure 4Another flow diagram of the picture multi-label classification processing method provided by the embodiments of the present application is shown.

[0024] Figure 5 A system flow diagram of an application scenario provided by the embodiments of the present application is shown.

[0025] Figure 6 A structural diagram of the multi-label classification processing apparatus provided by the embodiments of the present application is shown.

[0026] Figure 7 A hardware structural diagram of the multi-label classification processing apparatus provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0027] The embodiments of the present application provide a picture multi-label classification processing method and related apparatus, which can effectively reduce unnecessary algorithm repeated modeling in the field of picture quality evaluation, reduce the research and development cost investment, improve the research and development efficiency, and quickly respond to the demand of business, and is suitable for various application scenarios.

[0028] It can be understood that in the specific embodiments of the present application, the data related to user information and the like are involved, and when the above embodiments of the present application are applied to specific products or technologies, the user permission or consent needs to be obtained, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the country and region.

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0030] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0031] With the research and progress of artificial intelligence (AI) technology, artificial intelligence technology is researched and applied in multiple fields, such as common smart home, smart wearable device, virtual assistant, smart sound box, unmanned driving, automatic driving, unmanned aerial vehicle, robot, smart medical treatment, smart customer service and the like. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important value.

[0032] The embodiment of the present application provides a picture multi-label classification processing method. The picture multi-label classification processing method provided by the embodiment of the present application is realized based on artificial intelligence. Artificial intelligence is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0033] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware level technology and software level technology. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech technology, natural language processing technology and machine learning / deep learning, etc.

[0034] In the embodiment of the present application, the artificial intelligence technology mainly involved includes the above-mentioned computer vision (CV) technology, machine learning and the like. For example, it can involve image semantic understanding (ISU) and the like in computer vision technology, including image feature extraction, image classification and the like; it can also involve deep learning and the like in machine learning (ML).

[0035] The picture multi-label classification processing method provided in the application can be applied to a multi-label classification processing device with data processing capability, such as a terminal device, a server, and the like. The terminal device can include, but is not limited to, a smart phone, a desktop computer, a notebook computer, a tablet computer, a smart speaker, a vehicle-mounted device, a smart watch, a wearable smart device, a smart voice interaction device, a smart household appliance, a flying device, and the like. The server can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud computing services, and the like, which are not limited in the application. In addition, the terminal device and the server mentioned can be directly connected or indirectly connected through wired communication or wireless communication, which are not limited in the application. The embodiments of the application can be applied to various scenes, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving.

[0036] The multi-label classification processing device mentioned above can have the ability to implement the computer vision technology mentioned above. The computer vision technology mentioned refers to a science that studies how to make a machine "see". Further, it refers to using a camera and a computer to replace the human eye to identify, trace, and measure the target machine vision, and further performs image processing, so that the computer processing becomes an image more suitable for human eye observation or transmission to an instrument detection. As a scientific discipline, computer vision researches related theories and technologies, and tries to establish an artificial intelligence system that can obtain information from images or multidimensional data. The computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and the like. It also includes common face recognition, fingerprint recognition, and other biometric identification technologies.

[0037] In addition, the multi-label classification processing device can also have machine learning capability. Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and other disciplines. It is a subject that studies how a computer simulates or implements human learning behavior to obtain new knowledge or skills, and reorganizes the existing knowledge structure to continuously improve its performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its application is widespread in various fields of artificial intelligence. Machine learning and deep learning usually include neural network technology.

[0038] In the picture multi-label classification processing method provided in the embodiments of the application, the artificial intelligence model is mainly related to the application of a neural network, which realizes the prediction processing of the label classification of the picture through the neural network.

[0039] For the convenience of understanding the technical solutions of the present application, the picture multi-label classification processing method provided by the embodiments of the present application will be introduced below in combination with a framework flowchart. Figure 1 A picture multi-label classification processing flowchart framework diagram provided by an embodiment of the present application is shown.

[0040] As shown in Figure 1 After obtaining the to-be-predicted pictures marked with different marking labels, such as pictures marked with animation and natural scenery labels, pictures marked with background blur labels, pictures marked with mosaic labels, and pictures marked with splicing and glass labels, the picture embedding features of the to-be-predicted pictures can be extracted through a preset self-supervised model, and the label embedding features of the marking labels can be extracted through a preset label model (such as a single-layer Transform network model). Finally, the picture embedding features and the label embedding features are processed through a first Transform sub-model in a preset multi-label classification model, and the encoding feature vector output by the first Transform sub-model and the picture embedding features are processed using a second Transform sub-model in the preset multi-label classification model, and finally the prediction probability value of each marking label is output through a softmax activation function, thereby obtaining the multi-label classification result.

[0041] It should be noted that the specific processing process of the described preset self-supervised model, preset label model, and preset multi-label classification model can be understood with reference to the content described in steps 201 to 204 in the subsequent Figure 2 description, which will not be repeated here.

[0042] The picture multi-label classification processing method provided by an embodiment of the present application will be introduced below in combination with the accompanying drawings. Figure 2 A picture multi-label classification processing method provided by an embodiment of the present application is shown. As shown in Figure 2 , the picture multi-label classification processing method can include the following steps:

[0043] 201, obtaining a to-be-predicted picture and a preset label classification result, the preset label classification result including at least two marking labels, each marking label being used to indicate a preset picture classification condition of the to-be-predicted picture.

[0044] In this example, the picture to be predicted can be understood as a picture in a picture set, a picture-text, and / or a video cover. The picture to be predicted can be labeled by means of a preset picture sub-tag, and at least two labeled tag labels of the picture to be predicted can be obtained. Each tag label can be used to indicate a preset picture classification of the picture to be predicted. In other words, for the picture to be predicted, multi-label classification processing is needed, so that the picture to be predicted belongs to multiple target picture categories. The multi-label classification mentioned above is compared with single-label classification, which can be understood as a sample belonging to only one category. However, the single-label classification method sometimes does not conform to the actual situation. Multi-label classification can be understood as a sample belonging to multiple categories, such as a movie belonging to an action movie and a comedy movie, etc. Therefore, for the picture to be predicted provided in the embodiments of the present application, multi-label classification processing can be performed on the picture to be predicted, and then a multi-label classification result of the picture to be predicted is determined.

[0045] It should be noted that the described label tags include one or more of the picture content, picture geometry, picture style, picture quality, shooting skill, and the feeling of the object to the picture to be predicted. In actual application, the label tags of the picture to be predicted are not limited to the picture content, picture geometry, picture style, and the like mentioned above, and the embodiments of the present application are not limited to specific description. In addition, for picture geometry tags, high-quality tags such as wide-angle, 90° rotation, 180° rotation, stretching, compression, and low-quality tags such as center composition, horizontal line composition, vertical line composition, symmetric composition, diagonal line composition, and guide line composition can be included. For picture content tags, high-quality tags such as magazine cover, news report, promotional poster, web screenshot, comic face, game screen, live screen, natural scenery, artistic painting, photographic work, posed composition, city close-up, building close-up, pet close-up, food close-up, single person close-up, group photo, dense crowd, reasonable content distribution but not prominent subject, large area of text without subject, content ratio is small, and picture text overlap, and low-quality tags such as character special effect, agriculture, red theme, science fiction work, concert, and picture target clutter can be included. Similarly, for picture style tags, high-quality tags such as monochrome, color clutter, black and white, and dim, and low-quality tags such as excessive cold color and excessive warm color can be included. In addition, for picture quality tags, high-quality tags such as black and white edges, lace, one-sided frosted glass, asymmetric frosted glass, large area frosted glass, mosaic, QR code, collage, comic, picture-in-picture, fresh and pure picture, and picture ghosting, and low-quality tags such as image processing (photoshop, PS) synthesis can be included. Similarly, for shooting skill tags, high-quality tags such as blur can be included, and low-quality tags such as defocus, overexposure, and underexposure can also be included. For object feeling tags of the picture to be predicted, high-quality tags such as incomplete, disgusting, vulgar, sensitive image, solid color, and screen of flowers can be included, and the like. For the high-quality tags, low-quality tags, and the like in the above-mentioned various tags, it is only an illustrative description, and other tags can also be included in actual application, and the embodiments of the present application are not limited to specific description. For example, for the acquisition of high-quality tags, the object can obtain the tags by self-defining. For the acquisition of low-quality tags, the tags can be obtained based on the feedback or report of the object, and the specific description is not limited.

[0046] 202. Perform feature extraction processing on the picture to be predicted based on a preset self-supervised model to obtain picture embedding features of the picture to be predicted.

[0047] In the example, after obtaining the to-be-predicted picture, the to-be-predicted picture can be taken as an input of a preset self-supervised model, and then the to-be-predicted picture is subjected to feature extraction processing by the preset self-supervised model, so as to obtain picture embedding features of the to-be-predicted picture. Illustratively, the preset self-supervised model is a machine learning model obtained after iterative training of taking extraction of picture embedding features of a picture as a training target and taking a picture, a frame picture of video content that has been subjected to label marking as training data.

[0048] Illustratively, in some embodiments, the picture multi-label classification processing method can further include: obtaining a training sample set, the training sample set including at least two picture samples; processing sample labels and sample feature similarities based on a preset contrast loss model to obtain a contrast loss value, wherein the sample labels are used to indicate matching conditions of each two picture samples in the at least two picture samples, and the sample similarities are similarities between sample features of the each two picture samples; and updating model parameters of the preset self-supervised model based on the contrast loss value to obtain a trained preset self-supervised model.

[0049] Figure 3 A training structure schematic diagram of a self-supervised model provided by an embodiment of the present application is shown. As shown in Figure 3 the model structure is composed of a Swin Transformer submodule and a contrast learning (Contrastive Learning) submodule. It should be noted that the Swin Transformer submodule is optional in the embodiment of the present application and can be used as a BackBone. The structure of the Swin Transformer submodule can not only generate feature representations, but also reduce the time complexity to be linearly related to the image size, and the training time can also be less. The core part is the introduction of the window to reduce the complexity, and the introduction of the patch fusion and the shifted window can extract multi-dimensional features. In addition, the model size can be reduced by model distillation, which is beneficial to deployment and model acceleration and more in line with the needs of actual business.

[0050] In addition, in the training process of the self-supervised model, a contrast learning technology is also used, and a Contrastive Learning loss function in the Contrastive Learning submodule is used for contrast learning to measure the similarity between pictures. In other words, the Contrastive Learning loss function can effectively process the similarity between paired data in a neural network. It should be noted that the preset contrast loss model mentioned above can be understood as Figure 3The Contrastive Learning loss function is used in this context. For example, the expression for the Contrastive Learning loss function is as follows: Where d=||a n -b n ||2 represents two image samples a n and b n The Euclidean distance between the two image samples, y, can be understood as a sample label, which can be used to indicate the difference between the two image samples a. n and b n The matching situation. For example, when y=1, it means a n and b n These two image samples are similar or match; when y=0, it means a. n and b n These two image samples do not match. Additionally, the margin is a preset threshold.

[0051] Therefore, after obtaining the training sample set, the Euclidean distance between any two image samples can be determined, and this Euclidean distance can then be used to determine the sample feature similarity between the two image samples. This also determines the matching status of each pair of image samples, thus obtaining the sample label. In this way, the sample label and sample feature similarity are used as the basis for the... Figure 3 The contrastive learning loss function is input to calculate the contrastive loss value. This contrastive loss value can then be used to update the model parameters of the pre-defined self-supervised model, for example, to update... Figure 3 The parameters in the Swin Transformer submodule and the model distillation submodule are used to obtain a pre-trained self-supervised model.

[0052] In addition, from Figure 3It can also be seen that, in the training process of the preset self-supervised model, the source of the training data in the training sample set can include but is not limited to frame extraction pictures of video content, picture data in the business link field, and crawled Internet public data. For training data derived from frame extraction pictures of video content, for example, for a certain video content, adjacent frame extraction pictures can be considered similar. After deduplication processing of the frame extraction pictures, frame extraction pictures of different video contents and non-adjacent video frames are not similar. Specifically, negative sample pairs can be randomly generated from frame extraction pictures of different video contents, while using the existing video deduplication relationship chain to avoid duplicate videos. Moreover, for an information flow video content library, using the enabled videos in the information content video library, in each video content, adjacent or similar video frames are extracted, and similar video images are used as positive samples. For example, in the cover picture scenario, there are a large number of positive cover pictures selected by the host and labels, and the title of the content. The relevance between the title and the entity information of the cover picture is very strong, and the pictures with similar titles and entity information can be considered as similar cover pictures. These labels can be considered as weak supervision information, which do not need special manual labeling and can be automatically collected from the business. In addition, for picture data derived from the business link field, pictures labeled with the same tag can be considered as similar pictures, and pictures not belonging to the same tag category are not similar pictures. Similarly, for Internet public data, the entity words corresponding to the content tags can be counted through the information flow distribution content as tags (Query), and the content in the search application and vertical website is searched to collect a large amount of picture data and picture description field data, such as vulgar and various small data in the field as the pre-training data source in the information flow business, which can better cope with quality tags.

[0053] It should be noted that, for the source of the training data in the preset self-supervised model, it can also be obtained from other angles in actual application, which is not limited in the embodiments of the present application.

[0054] In this way, after obtaining the trained preset self-supervised model, the picture to be predicted obtained in step 201 can be input into the trained preset self-supervised model, so as to extract the picture embedding features of the picture to be predicted based on the Swin Transformer sub-module in the trained preset self-supervised model.

[0055] 203, feature extraction of each labeled tag based on the preset label model to obtain label embedding features of the labeled tag.

[0056] In the example, after obtaining the to-be-predicted picture, the preset label model can be used to perform feature extraction on each label tag corresponding to the to-be-predicted picture, and then the label embedding features of each label tag are obtained. It should be noted that the described preset label model can be understood as a single-layer Transform network model, which can ensure the performance of the calculation.

[0057] 204. The picture embedding features and the label embedding features are fused based on a preset multi-label classification model to obtain a multi-label classification result of the to-be-predicted picture, and the multi-label classification result is used to indicate a target picture category matched by the to-be-predicted picture. The preset multi-label classification model is a machine learning model obtained by iterative training with the multi-label classification result of a sample picture as a training target and the picture embedding features of the sample picture and the label embedding features corresponding to the labeled tags of the sample picture as training data.

[0058] In the example, the described preset multi-label classification model can be understood as a double-layer Transform network model, that is, the preset multi-label classification model includes a first Transform sub-model and a second Transform sub-model. In some examples, the picture embedding features and the label embedding features are input into the preset multi-label classification model to obtain the predicted classification probability value of each labeled tag of the to-be-predicted picture, which can be realized by the following manner: the picture embedding features and the label embedding features are encoded by the first Transform sub-model to obtain an encoded feature vector. Then, the encoded feature vector and the picture embedding features are fused by the second Transform sub-model to obtain the predicted classification probability value of each labeled tag of the to-be-predicted picture. Exemplarily, the predicted classification probability value of each labeled tag can be output by a softmax activation function.

[0059] It should be noted that, due to the correlation between the labels, in order to fully utilize the correlation between the labels, the picture embedding features and the label embedding features can be processed by the double-layer Transform network model mentioned above, the mutual relationship between different modalities can be extracted, and the corresponding relationship between the picture and the label can be better extracted. In addition, the preset multi-label classification model is a machine learning model obtained by iterative training of the multi-label classification result of the sample picture as the training target, and the picture embedding features of the sample picture and the label embedding features corresponding to the labeled label of the sample picture as the training data. The training data of the preset multi-label classification model can include, but is not limited to, video frame extraction pictures (including key frames and uniformly extracted frames, etc.) obtained by processing the video file features of the video files downloaded from the video content storage service by the download file system; can also include pictures with quality problems reported by the artificial review content consumption end and the content consumption end; or can also include content related to picture quality problems in the consumption flow content reported by the content review end and the content consumption end, and other training data that can be obtained are more targeted and consistent with the distribution of the picture and text quality problems in the business. The specific application is not limited to the description. In other words, since the preset picture classification situation of the to-be-predicted picture, i.e., the labeled label, is known in advance in the embodiment of the application, the multi-label classification problem of the to-be-predicted picture is converted into cross-modal fusion matching of the picture embedding features and the corresponding label embedding features by the preset multi-label classification model, without considering the context text information of the to-be-predicted picture, so as to not need to combine and apply different picture features, but fully consider the mutual matching between the picture embedding features and the label embedding features, so as to predict and match the final multi-label classification result.

[0060] In some possible embodiments, the target loss value can also be obtained based on the difference between each labeled label and the corresponding predicted classification probability value. Then, the model parameters in the preset multi-label classification model are updated according to the target loss value, and a trained preset multi-label classification model is obtained. Exemplarily, the target loss value can be obtained by the following method, that is, the difference between each labeled label and the corresponding predicted classification probability value is calculated to obtain the loss value between each labeled label and the corresponding predicted classification probability value, and then the loss value between each labeled label and the corresponding predicted classification probability value is averaged to obtain the target loss value.

[0061] In this example, in order to fully use the advantages of the Transform, the inter-relationship between the labels can be indirectly used to improve the loss function. For example, in the embodiments of the present application, the asymmetric parameter and probability drift loss (ASL) function is used to replace the ordinary binary cross entropy (BCE) loss function, which can overcome the positive and negative sample imbalance problem in multi-label classification. Exemplarily, the ASL loss function is specifically expressed as follows:

[0062]

[0063] wherein L is a loss value, p m = max(p-m, 0).

[0064] In this example, for the multi-label classification task, the difference between each labeled label of the to-be-predicted picture and the corresponding predicted classification probability value is calculated to obtain the loss value between each labeled label and the corresponding predicted classification probability value (the output layer uses the sigmoid function). Then, the average value is obtained by averaging the loss values between each labeled label and the corresponding predicted classification probability value, that is, the target loss value is obtained. Similarly, when calculating the accuracy, the accuracy of each label of the to-be-predicted picture can also be calculated, and then the average value is obtained. In other words, since the labels are not mutually exclusive, the multi-label classification problem can be finally converted into a binary classification problem, that is, the prediction process of each label label obtained finally is treated as a binary classification problem for processing, and then in the calculation process of the prediction probability value and the loss value of the corresponding labeled label, the calculation method of binary classification can be used. For example, for a to-be-predicted picture x, assuming that the to-be-predicted picture x corresponds to 5 labeled labels, i.e., y1 to y5. Then, for the given picture sample {x, y = [y1, y2, y3, y4, y5]}, the output prediction probability value of the preset multi-label classification model is y = [p1, p2, p3, p4, p5], wherein each p value is the prediction probability value of each labeled label calculated as described above. At this time, the loss value between each labeled label and the corresponding prediction probability value can also be calculated using the following sigmod cross entropy function, that is:

[0065] cost(h θ (x), y) = -y i log(h θ (x)) - (1-y i )log(1-h θ (x))

[0066] wherein x is a to-be-predicted picture, y ih is a label tag of the to-be-predicted picture θ (·) h is a preset multi-label classification model θ (x) is a prediction probability value of the label tag.

[0067] It should be noted that, for the preset multi-label classification model obtained by training as described above, in order to avoid model degradation, the model parameters of the preset multi-label classification model can be updated in the module library periodically, for example, a period of one week, one day, etc., which is not limited in particular. In addition, the preset multi-label classification model can also be serviced, that is, a service that can be called on a business link is constructed to implement prediction of the label of the to-be-predicted picture and writing and saving of the predicted label in the multi-label result library.

[0068] In this way, after obtaining the picture embedding feature and the label embedding feature, the picture embedding feature and the label embedding feature can be fused and processed based on the preset multi-label classification model obtained by training as described above, and then a multi-label classification result of the to-be-predicted picture is obtained. Illustratively, the picture embedding feature and the label embedding feature can be input into the preset multi-label classification model to obtain a prediction classification probability value of each label tag of the to-be-predicted picture. Then, the multi-label classification result of the to-be-predicted picture can be determined based on the prediction classification probability value of each label tag. It should be noted that the multi-label classification result described can be used to indicate a target picture category matched by the to-be-predicted picture. Moreover, each label in the multi-label classification result can reflect, to some extent, that the image quality of the to-be-predicted picture is excellent when matching different picture categories.

[0069] In some possible implementations, after obtaining the multi-label classification result of the to-be-predicted picture through the operation process in Figure 2 , a related business processing strategy can also be determined based on the multi-label classification result. For example, the to-be-predicted picture can be processed by shunting on a business flow link through the label in the multi-label classification result, and the like. Figure 4 Another flowchart of a picture multi-label classification processing method provided by an embodiment of the present application is shown. As shown in Figure 4 , the picture multi-label classification processing method can include the following steps:

[0070] 401. Obtain a to-be-predicted picture, the to-be-predicted picture corresponding to at least two label tags, each label tag being used to indicate an image quality situation of the to-be-predicted picture.

[0071] In this example, the content described in step 401 can be understood with reference to the content of step 201 in the foregoing Figure 2 , which will not be described herein.

[0072] 402. Perform feature extraction processing on the to-be-predicted picture based on the preset self-supervised model to obtain picture embedding features of the to-be-predicted picture.

[0073] In this example, the described preset self-supervised model can be understood with reference to the foregoing Figure 3 described model, which will not be described here in detail. In addition, the content described in step 402 can also be understood with reference to the content of step 202 in the foregoing Figure 2 , which will not be described here in detail.

[0074] 403. Perform feature extraction on each labeled label based on a preset label model to obtain label embedding features of the labeled label.

[0075] In this example, the content described in step 403 can be specifically understood with reference to the content of step 203 in the foregoing Figure 2 , which will not be described here in detail.

[0076] 404. Perform fusion processing on the picture embedding features and the label embedding features based on a preset multi-label classification model to obtain a multi-label classification result of the to-be-predicted picture, the multi-label classification result being used to indicate a target picture category matched by the to-be-predicted picture, wherein the preset multi-label classification model is a machine learning model obtained by iterative training with a multi-label classification result of a sample picture as a training target and with picture embedding features of the sample picture and label embedding features corresponding to labeled labels of the sample picture as training data.

[0077] In this example, the content described in step 404 can be specifically understood with reference to the content of step 204 in the foregoing Figure 2 , which will not be described here in detail.

[0078] 405. Determine a business policy corresponding to each label in the multi-label classification result based on each label, the business policy being used to indicate a picture processing policy adopted when performing business processing on the picture.

[0079] In this example, after obtaining the multi-label classification result of the to-be-predicted picture, multiple labels of the to-be-predicted picture can be known, i.e., it can be predicted which target picture categories the to-be-predicted picture belongs to. The classification of the labels here can also be understood with reference to the classification of the labeled labels in step 201 in the foregoing Figure 2 , which will not be described here in detail. In this way, after obtaining each label in the multi-label classification result, a business policy corresponding to each label can be determined, and the business policy can indicate a picture processing policy adopted when performing business processing on the picture. In other words, through the business policy, the to-be-predicted picture can be processed in a shunt manner in a business flow, and the processing efficiency is improved.

[0080] In some examples, storage and query services of the multi-label classification result can also be provided in the subsequent business processing process, which will not be described in detail.

[0081] 406. Business processing is performed on the to-be-predicted picture based on the label and the business policy.

[0082] In this example, after the business policy is determined, the to-be-predicted picture can be processed based on the label and the business policy. For example, in the processing scenario of the cover picture, the picture processing can be performed based on the label and the business policy of the cover picture, such as the picture selection process, quality review, and selection of high-quality pictures. For example, in the process of business processing, the dispatch of the center service can also be accepted, and the processing of the picture can be completed.

[0083] The above-described picture multi-label classification processing method is applied to the professional generated content (PGC) and user generated content (UGC) scenarios, Figure 2 and Figure 4 The system flowchart of an application scenario provided by the embodiment of the present application is shown. Figure 5

[0084] As Figure 5 shown, the system flowchart mainly involves a preset multi-label classification model, a multi-label classification service module, a content consumption end, a content storage server, a label result library, a business policy, and a business service module. For example, a content generation end, a dispatch center server, an uplink and downlink content interface server, and a content distribution outlet service module can also be included.

[0085] In this example, the content generation end can be understood as a content production object of PGC, UGC, multi-channel network (MCN), or professional user generated content (PUGC), which mainly provides local or photographed video content or written WeMedia articles, picture sets, etc. through a mobile end or a back-end interface API system. The object can actively upload the cover picture corresponding to the content. In addition, the content generation end communicates with the uplink and downlink content interface server to upload the locally saved picture content and video content. The described picture content includes but is not limited to the title, abstract, cover picture, and publication time of the picture. The video content includes but is not limited to the video file size, cover picture connection, code rate, and file format.

[0086] ​The content consumer end usually browses and consumes data in the form of Feeds through the content distribution outlet service module, provides an entry for direct reporting and feedback for low-quality picture content, video content, etc. uploaded by the content generation end, and interfaces with the artificial review system for confirmation and review by artificial personnel, and then saves the finally confirmed picture related to the picture quality problem in the picture sample library. In addition, the content storage server can also save the video content and picture content uploaded by the content production end through the uplink and downlink content interface server, and after processing the saved video and picture content through the download file system, such as frame extraction, the processed picture is marked with a label and written into the picture sample library. In this way, after obtaining the picture, the marked label and other content from the picture sample library, the preset multi-label classification model can be trained.

[0087] Then, the multi-dimensional labels of the picture and the probability values of the corresponding labels can be obtained through the preset multi-label classification model. In addition, the preset multi-label classification model can be serviced to build a service called on the business link to realize the service of label prediction and writing of the picture and the label result library, and then save the multi-dimensional labels and the probability values of the corresponding labels in the label result library through the multi-label classification service module.

[0088] Through the business strategy and business service module, the labels are obtained from the label result library, and the business strategy corresponding to the labels is combined to complete the business processing flow of the picture under the dispatch of the dispatch center server, such as cover picture selection, picture and text quality review, video review, picture cropping, and high-quality picture screening.

[0089] Exemplarily, the dispatch center server can also be responsible for the entire dispatch process of video content and picture content flow, receive video content and picture content through the uplink and downlink content interface server, and then obtain the meta information of the content from the content meta information database. In addition, it can also be an actual dispatch controller of the picture and video link operation, and then dispatch the multi-modal unsuitable content identification service system to process the corresponding content according to the type of the picture content in the link, directly filter and label the content. The content processing of the dispatch center processor mainly includes machine processing and artificial review processing, so the dispatch center processor can also dispatch the artificial review system and the machine processing system to control the order and priority of the dispatch. In addition, when the artificial review system is called, the dispatch center server can provide the display page to the content consumer end through the content outlet distribution service module, that is, provide the label and other information to the content consumer end.

[0090] In the embodiments of the present application, since the preset label classification result includes at least two marked labels, each of which is used to indicate the preset picture classification condition of the picture to be predicted, after the picture to be predicted is obtained, the preset self-supervised model can be used to perform feature extraction processing on the picture to be predicted to obtain picture embedding features of the picture to be predicted, and after the preset label classification result is obtained, the preset label model is used to perform feature processing on each marked label to obtain label embedding features of the marked label. Then, the preset multi-label classification model is used to perform fusion processing on the picture embedding features and the label embedding features, so as to obtain the multi-label classification result of the picture to be predicted, which is used to indicate the target picture category matched by the picture to be predicted. Through the above manner, the picture embedding features are better extracted by the preset self-supervised model, and the preset multi-label classification model is a machine learning model obtained by taking the multi-label classification result of the sample picture as a training target and iteratively training the picture embedding features of the sample picture and the label embedding features corresponding to the marked label of the sample picture as training data. Therefore, the picture embedding features and the label embedding features of the picture to be predicted are taken as inputs of the preset multi-label classification model, so as to predict the multi-label classification result of the picture by the preset multi-label classification model, which can effectively reduce unnecessary algorithm repeated modeling in the field of picture quality evaluation, reduce the research and development cost investment, and improve the research and development efficiency. Moreover, since the preset picture classification condition of the picture to be predicted, i.e., the marked label, is known in advance in the embodiments of the present application, the multi-label classification problem of the picture to be predicted is converted into cross-modal fusion matching of the picture embedding features and the corresponding label embedding features, without considering the context text information of the picture to be predicted, so that different picture features do not need to be combined and applied, but the multiple labels in the multi-label classification result obtained by the final prediction and matching are combined and applied to solve different business problems, which can quickly respond to the needs of the business and is suitable for various application scenarios.

[0091] The above mainly describes the scheme provided by the embodiments of the present application from the perspective of the method. It can be understood that in order to implement the above functions, the corresponding hardware structure and / or software module for executing each function are included. Those skilled in the art should easily realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0092] The embodiments of the present application can divide the functions of the device according to the above method examples, for example, each function module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be realized in the form of hardware or in the form of a software function module. It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is only a logical function division. When actually implemented, another division method can be used.

[0093] The multi-label classification processing device in the embodiments of the present application will be described in detail below, Figure 6 An embodiment of the multi-label classification processing device provided in the embodiments of the present application is shown in the figure. As Figure 6 shown, the multi-label classification processing device can include an acquisition unit 601 and a processing unit 602.

[0094] The acquisition unit 601 is configured to acquire a to-be-predicted picture and a preset label classification result. The preset label classification result includes at least two marked labels, and each marked label is used to indicate a preset picture classification condition of the to-be-predicted picture. The processing unit 602 is configured to perform feature extraction processing on the to-be-predicted picture based on a preset self-supervised model to obtain picture embedding features of the to-be-predicted picture; perform feature extraction on each marked label based on a preset label model to obtain label embedding features of the marked label; and perform fusion processing on the picture embedding features and the label embedding features based on a preset multi-label classification model to obtain a multi-label classification result of the to-be-predicted picture. The multi-label classification result is used to indicate a target picture category matched by the to-be-predicted picture. The preset multi-label classification model is a machine learning model obtained by iterative training with a multi-label classification result of a sample picture as a training target and with picture embedding features of the sample picture and label embedding features corresponding to the marked label of the sample picture as training data.

[0095] In some optional examples, the processing unit 602 is configured to input the picture embedding features and the label embedding features into the preset multi-label classification model to obtain a predicted classification probability value of each marked label of the to-be-predicted picture; and determine the multi-label classification result of the to-be-predicted picture based on the predicted classification probability value of each marked label.

[0096] In some optional examples, the preset multi-label classification model includes a first Transform sub-model and a second Transform sub-model. The processing unit 602 is configured to perform encoding processing on the picture embedding features and the label embedding features based on the first Transform sub-model to obtain an encoded feature vector; and perform fusion processing on the encoded feature vector and the picture embedding features based on the second first Transform sub-model to obtain the predicted classification probability value of each marked label of the to-be-predicted picture.

[0097] In some optional examples, the obtaining unit 601 is further configured to obtain a target loss value based on a difference between each of the mark labels and the corresponding predicted classification probability value; and the processing unit 602 is further configured to update the model parameters of the preset multi-label classification model based on the target loss value to obtain the trained preset multi-label classification model.

[0098] In some optional examples, the processing unit 602 is configured to: calculate a difference between each of the mark labels and the corresponding predicted classification probability value to obtain a loss value between each of the mark labels and the corresponding predicted classification probability value; and perform average processing on the loss value between each of the mark labels and the corresponding predicted classification probability value to obtain the target loss value.

[0099] In some optional examples, the processing unit 602 is further configured to, after determining the multi-label classification result of the to-be-predicted picture based on the predicted classification probability value of each of the mark labels, determine, based on each label in the multi-label classification result, a service policy corresponding to the label, the service policy being used to indicate a picture processing strategy adopted when the picture is processed; and perform service processing on the to-be-predicted picture based on the label and the service policy.

[0100] In some optional examples, the obtaining unit 601 is further configured to: obtain a training sample set, the training sample set including at least two picture samples; and the processing unit 602 is configured to process the sample label and the sample feature similarity based on the preset contrast loss model to obtain a contrast loss value, wherein the sample label is used to indicate a matching condition of each of the at least two picture samples, and the sample similarity is a similarity between sample features of each of the at least two picture samples; and update the model parameters of the preset self-supervised model based on the contrast loss value to obtain the trained preset self-supervised model.

[0101] In some optional examples, the mark label includes one or more of a picture content, a picture geometry, a picture style, a picture quality, a shooting skill, and a feeling of an object to the to-be-predicted picture.

[0102] The multi-label classification processing apparatus in the embodiments of the present application is described from the perspective of modular functional entities above, and is described from the perspective of hardware processing below. Figure 7 FIG. 1 is a hardware structure schematic diagram of a multi-label classification processing apparatus provided by an embodiment of the present application. The multi-label classification processing apparatus can have great differences due to different configurations or performances. The multi-label classification processing apparatus can include at least one processor 701, a communication line 707, a memory 703, and at least one communication interface 704.

[0103] The processor 701 can be a general central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of program instructions for the schemes described herein.

[0104] The communication line 707 can include a path for transmitting information between the above-mentioned components.

[0105] The communication interface 704 can be used to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like, using any transceiver-type device.

[0106] The memory 703 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, and the like. The memory can be present either independently or in combination with the processor, connected to the processor through the communication line 707. The memory can also be integrated with the processor.

[0107] The memory 703 is configured to store computer-executable instructions for implementing the schemes described herein, and the processor 701 is configured to execute the computer-executable instructions stored in the memory 703. The processor 701 is configured to execute the computer-executable instructions stored in the memory 703, thereby implementing the picture multi-label classification processing method provided by the embodiments described herein.

[0108] Optionally, the computer-executable instructions in the embodiments described herein can also be referred to as application program codes, which are not specifically limited in the embodiments described herein.

[0109] In a specific implementation, as an embodiment, the multi-label classification processing apparatus can include multiple processors, such as the processor 701 and the processor 702 in the apparatus 700. Figure 7 Each of the processors can be a single-CPU processor or a multi-CPU processor. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0110] In a particular implementation, as an example, the multi-label classification processing apparatus can further include an output device 705 and an input device 706. The output device 705 is in communication with the processor 701 and can display information in a variety of ways. The input device 706 is in communication with the processor 701 and can receive input of the target object in a variety of ways. For example, the input device 706 can be a mouse, a touch screen device, a sensor device, or the like.

[0111] The multi-label classification processing apparatus described above can be a general-purpose device or a special-purpose device. In a particular implementation, the multi-label classification processing apparatus can be a server, a terminal, or the like, or a device having a similar structure. The embodiments of the present application do not limit the type of the multi-label classification processing apparatus. Figure 7 In a particular implementation, the multi-label classification processing apparatus can be a server, a terminal, or the like, or a device having a similar structure. The embodiments of the present application do not limit the type of the multi-label classification processing apparatus.

[0112] It should be noted that the processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703. Figure 7 The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703. Figures 2 to 5 The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703. The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703.

[0113] The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703. Figure 6 The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703. Figure 7 The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703. Figure 6 The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703. Figure 7 The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703. The processor 701 in the multi-label classification processing apparatus can cause the multi-label classification processing apparatus to perform the method in the corresponding method embodiment by invoking the computer-executed instructions stored in the memory 703.

[0114] In the above-described embodiments, the system, apparatus, and unit described above can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, the system, apparatus, and unit described above can be implemented in whole or in part in the form of a computer program product.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, apparatus, and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0116] In the several embodiments provided by the present application, it should be understood that the disclosed system, apparatus, and method can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, and the division of the units is merely a logical function division. In actual implementation, another division mode can be used, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices, or units, and can be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0118] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0119] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0120] The above embodiments can be realized in whole or in part by software, hardware, firmware or any combination thereof, and when realized by software, can be realized in whole or in part in the form of a computer program product.

[0121] The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computer, the flow or function according to the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that the computer can store or be integrated into a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, an SSD), etc.

[0122] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A processing method of picture multi-label classification, characterized in that, The method comprises the following steps: acquiring a to-be-predicted picture and a preset label classification result, the preset label classification result comprising at least two marked labels, each of the marked labels being used to indicate a preset picture classification condition of the to-be-predicted picture; performing feature extraction processing on the to-be-predicted picture based on a preset self-supervised model to obtain picture embedding features of the to-be-predicted picture, wherein the preset self-supervised model is trained in the following manner: acquiring a training sample set, the training sample set comprising at least two picture samples; processing sample labels and sample feature similarities based on a preset contrast loss model to obtain a contrast loss value, wherein the sample labels are used to indicate a matching condition of each two picture samples in the at least two picture samples, and the sample feature similarity is a similarity between sample features of the each two picture samples; updating model parameters of the preset self-supervised model based on the contrast loss value to obtain a trained preset self-supervised model; performing feature extraction on each of the marked labels based on a preset label model to obtain label embedding features of the marked labels; performing fusion processing on the picture embedding features and the label embedding features based on a preset multi-label classification model to obtain a multi-label classification result of the to-be-predicted picture, wherein the fusion processing comprises: inputting the picture embedding features and the label embedding features into the preset multi-label classification model to obtain a predicted classification probability value of each of the marked labels of the to-be-predicted picture; and determining the multi-label classification result of the to-be-predicted picture based on the predicted classification probability value of each of the marked labels, wherein the multi-label classification result is used to indicate a target picture category matched by the to-be-predicted picture, and the preset multi-label classification model is a machine learning model obtained by iterative training of training data, wherein the training data comprises a multi-label classification result of a sample picture, picture embedding features of the sample picture, and label embedding features corresponding to marked labels of the sample picture; wherein the preset multi-label classification model comprises a first Transform sub-model and a second Transform sub-model; and the predicted classification probability value of each of the marked labels of the to-be-predicted picture is obtained by inputting the picture embedding features and the label embedding features into the preset multi-label classification model, comprising: performing encoding processing on the picture embedding features and the label embedding features based on the first Transform sub-model to obtain an encoded feature vector; performing fusion processing on the encoded feature vector and the picture embedding features based on the second Transform sub-model to obtain the predicted classification probability value of each of the marked labels of the to-be-predicted picture.

2. The method of claim 1, wherein, The method further comprises: acquiring a target loss value based on a difference between each of the marked labels and a corresponding predicted classification probability value; updating model parameters of the preset multi-label classification model based on the target loss value to obtain a trained preset multi-label classification model.

3. The method of claim 2, wherein, The acquiring of the target loss value based on the difference between each of the marked labels and a corresponding predicted classification probability value comprises: calculate a difference between each label and a corresponding predicted classification probability value, to obtain a loss value between each label and a corresponding predicted classification probability value; average the loss value between each label and a corresponding predicted classification probability value, to obtain a target loss value.

4. The method of claim 1, wherein, After determining the multi-label classification result of the to-be-predicted picture based on the predicted classification probability value of each label, the method further comprises: determining, based on each label in the multi-label classification result, a business policy corresponding to the label, the business policy being used to indicate a picture processing strategy adopted when the picture is processed; processing the to-be-predicted picture based on the label and the business policy.

5. The method of claim 1, wherein, The label includes one or more of picture content, picture geometry, picture style, picture quality, shooting skill, and object feeling of the to-be-predicted picture.

6. A multi-label classification processing apparatus characterized by comprising: comprises: an acquisition unit, configured to acquire a to-be-predicted picture and a preset label classification result, the preset label classification result comprising at least two labels, each label being used to indicate a preset picture classification situation of the to-be-predicted picture; a processing unit, configured to perform feature extraction processing on the to-be-predicted picture based on a preset self-supervised model, to obtain picture embedding features of the to-be-predicted picture, wherein the preset self-supervised model is trained in the following manner: a training sample set is acquired, the training sample set comprising at least two picture samples; a sample label and a sample feature similarity are processed based on a preset contrast loss model, to obtain a contrast loss value, wherein the sample label is used to indicate a matching situation of each two picture samples in the at least two picture samples, and the sample feature similarity is a similarity between sample features of the each two picture samples; model parameters of the preset self-supervised model are updated based on the contrast loss value, to obtain the trained preset self-supervised model; the processing unit, configured to perform feature extraction on each label based on a preset label model, to obtain label embedding features of the label; the processing unit, configured to perform fusion processing on the picture embedding features and the label embedding features based on a preset multi-label classification model, to obtain a multi-label classification result of the to-be-predicted picture, wherein the fusion processing comprises: inputting the picture embedding features and the label embedding features into the preset multi-label classification model, to obtain a predicted classification probability value of each label of the to-be-predicted picture; determining the multi-label classification result of the to-be-predicted picture based on the predicted classification probability value of each label; and the multi-label classification result is used to indicate a target picture category matched by the to-be-predicted picture, wherein the preset multi-label classification model is a machine learning model obtained by iterative training with a multi-label classification result of a sample picture as a training target and with picture embedding features of the sample picture and label embedding features corresponding to a label of the sample picture as training data; The preset multi-label classification model includes a first Transform sub-model and a second Transform sub-model, and the processing unit is configured to: perform encoding processing on the picture embedding feature and the label embedding feature based on the first Transform sub-model to obtain an encoded feature vector; perform fusion processing on the encoded feature vector and the picture embedding feature based on the second Transform sub-model to obtain a predicted classification probability value of each marked label of the to-be-predicted picture.

7. The apparatus of claim 6, wherein, The processing unit is configured to: obtain a target loss value based on a difference between each marked label and a corresponding predicted classification probability value; update a model parameter of the preset multi-label classification model based on the target loss value to obtain a trained preset multi-label classification model.

8. The apparatus of claim 7, wherein, The processing unit is configured to: calculate a difference between each marked label and a corresponding predicted classification probability value to obtain a loss value between each marked label and a corresponding predicted classification probability value; perform average processing on the loss value between each marked label and a corresponding predicted classification probability value to obtain a target loss value.

9. The apparatus of claim 6, wherein, The processing unit is further configured to: determine a business policy corresponding to each label in the multi-label classification result based on each label in the multi-label classification result after determining the multi-label classification result of the to-be-predicted picture based on the predicted classification probability value of each marked label, wherein the business policy is used to indicate a picture processing policy adopted when the picture is processed; perform business processing on the to-be-predicted picture based on the label and the business policy.

10. The apparatus of claim 6, wherein, The marked label includes one or more of a picture content, a picture geometry, a picture style, a picture quality, a shooting skill, and a feeling of an object to the to-be-predicted picture.

11. A multi-label classification processing apparatus characterized by comprising: It includes: an input / output (I / O) interface, a processor, and a memory having program instructions stored therein; the processor is configured to execute the program instructions stored in the memory to perform the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer device, cause the computer device to perform the method of any one of claims 1 to 5.

13. A computer program product, characterised in that, The computer program product includes instructions that, when executed on a computer device, cause the computer device to perform the method of any one of claims 1 to 5.

Images