Image creativity type identification method and device, storage medium and program product

Abstract

The invention provides an image creativity type identification method and device, a storage medium and a program product. The identification method comprises the steps that S10, an original image is acquired, the original image comprises the physical attributes of manual pre-marking and the final creative tendency, and the physical features comprise the scene type, the background type, the number of people and the figure integrity; s11, the original image is input into a Vision-Language model for LoRA fine tuning, then the Vision-Language model after fine tuning is used for reasoning massive unlabeled data, an extended label set containing multi-dimensional information is generated to construct a rich semantic data set, and the extended label set comprises predicted physical features and creative category tendencies; s12, carrying out two-stage training by adopting a ConvNeXt v2 network; and S13, creativity type decision making: outputting a final image creativity type identification result according to the multi-dimensional physical label and the creativity type confidence coefficient in combination with a preset logic verification rule.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a method, device, storage medium and program product for image creative type recognition. Background Technology

[0002] In the field of automated image design and compositing, the key to achieving personalized intelligent production lies in selecting suitable images from massive user photo albums for specific design templates. Different design concepts place extremely stringent requirements on images. For example, "cutout" requires the subject to be complete and the background to be easily separated, "blending" requires white space or a simple background texture, and "photo booth" has specific restrictions on the composition of the scene.

[0003] Existing technologies mainly include the following categories: Traditional manual labeling and screening relies on designers manually reviewing images and marking them with suitable template types. Its drawbacks include extremely low efficiency, high labor costs, inconsistent subjective judgment standards among different personnel, and difficulty in scaling up.

[0004] Traditional CNN-based single-label classification directly trains a simple convolutional neural network to determine image categories (such as portraits or landscapes). Its drawback is a lack of understanding of the "compositional semantics" of the image. For example, the network might recognize a "person" in the image but fail to determine if the person's limbs are cut off (leading to an inability to cut out the image), or it may fail to understand complex "creative suitability" (i.e., whether the image is suitable for compositing in a certain artistic style) through simple classification, resulting in low matching rates and high rejection rates. Summary of the Invention

[0005] The purpose of this invention is to solve the above-mentioned problems by providing an image creative type recognition method, device, storage medium, and program product.

[0006] The technical solution of this application is implemented as follows: This invention provides a method for image creative type recognition based on two-stage progressive learning, comprising the following steps: S10, acquire the original image, which includes manually pre-labeled physical attributes and the final creative tendency. The physical features include shot size, background type, number of people, and completeness of the characters. S11, the original image is input into the Vision-Language model for LoRA fine-tuning, and then the fine-tuned Vision-Language model is used to reason about massive amounts of unlabeled data to generate an extended label set containing multi-dimensional information to construct a semantically rich dataset. The extended label set includes predicted physical features and creative category tendencies. S12 uses the ConvNeXt v2 network for two-stage training: Phase 1, Multi-dimensional physical feature learning: Input the image to be processed and learn the physical features in the rich semantic dataset to output multi-dimensional physical labels, wherein the multi-dimensional physical labels include: scene classification, subject integrity score, background complexity score, and number of subjects; Phase 2, Creative Type Adaptation Learning: Using the multidimensional physical labels from Phase 1 and the image to be processed as input, learn the creative type adaptation probability and output confidence scores for at least image blending, image cutout, image splicing, and non-creative creative types. S13. Creative Type Decision: Based on the multidimensional physical labels and creative type confidence, combined with preset logical verification rules, the final image creative type recognition result is output.

[0007] As a further improvement, the learning using physical features from the rich semantic dataset specifically includes: Multi-task learning of edge detection, object segmentation cues, and spatial relationships is performed using the corresponding physical attribute labels in the rich semantic dataset.

[0008] As a further improvement, the multi-task learning of edge detection, object segmentation cues, and spatial relationships using the corresponding physical attribute labels in the rich semantic dataset specifically satisfies the following model: ;in, Used to calculate the predicted probability distribution With real labels The differences between them; Used to calculate predicted scores Compared with real ratings based on VLM The Euclidean distance between them; As an auxiliary loss term; These are the weights of the balanced hyperparameters for each task, used to dynamically adjust the contribution ratio of different tasks in gradient backpropagation.

[0009] As a further improvement, Phase Two further includes: An improved Focal Loss approach is used to focus on hard-to-classify samples.

[0010] As a further improvement, the improved Focal Loss method specifically addresses hard-to-distinguish samples by satisfying the following model: ,in, : Represents the total number of creative categories, : represents the model's predicted probability for the target creative category. : represents the modulation coefficient; : This is a class balance parameter used to adjust for bias caused by a large disparity in the proportion of positive and negative samples.

[0011] As a further improvement, among which, The default value is 2.0.

[0012] The present invention further provides a computer device, the computer device including a memory, a communication interface and a processor, wherein the memory, the communication interface and the processor are interconnected; the memory stores a computer program, and the processor calls the computer program stored in the memory to implement the image creative type recognition method based on two-stage progressive learning as described above.

[0013] The present invention further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image creative type recognition method based on two-stage progressive learning as described above.

[0014] The present invention further provides a computer program product, the computer program product including a computer program stored in a computer storage medium; a processor of a computer device reads the computer program from the computer storage medium, the processor executes the computer program, causing the computer device to perform the image creative type recognition method based on two-stage progressive learning as described above.

[0015] The advantages or beneficial effects of the above technical solutions include at least the following: I. The invention has extremely high recognition dimensionality and semantic depth: By introducing VLM and combining it with LoRA fine-tuning, the invention can not only identify "what it is", but also understand "how it is" (such as whether the background is cluttered, whether the limbs are cut off, etc.), filling the gap in the design of semantic understanding in traditional classification models.

[0016] Second, the method of this invention improves the robustness of the system through a two-stage learning process: Unlike end-to-end black-box training, this invention adopts a progressive learning path of "physical features -> creative adaptation". It first solidifies basic attribute recognition, and then performs advanced logical inference, significantly solving the problem of inaccurate recognition in complex scenarios (such as low light or multiple people).

[0017] III. Balance between computational efficiency and accuracy: Although VLM is used for data augmentation, the lightweight and efficient ConvNeXt v2 network is used in the online inference stage, which ensures both high accuracy of recognition and meets the low latency requirements of industrial-scale real-time processing. Attached Figure Description

[0018] The accompanying drawings illustrate exemplary embodiments of the present application and, together with the description thereof, serve to explain the principles of the present application. These drawings are included to provide a further understanding of the present application and are incorporated in and constitute a part of this specification.

[0019] Figure 1 A flowchart of an image creative type recognition method based on two-stage progressive learning provided by an embodiment of the present invention is shown. Detailed Implementation

[0020] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0021] It should be noted that, where there is no conflict, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0023] Reference Figure 1 This invention provides a method for image creative type recognition based on two-stage progressive learning, comprising the following steps: S10, acquire the original image, which includes manually pre-labeled physical attributes and the final creative tendency. The physical features include shot size, background type, number of people, and completeness of the characters. The final creative tendency includes whether it is suitable for image blending or image cutout. S11, the original image is input into the Vision-Language model for LoRA fine-tuning, and then the fine-tuned Vision-Language model is used to reason about massive amounts of unlabeled data to generate an extended label set containing multi-dimensional information to construct a semantically rich dataset. The extended label set includes predicted physical features and creative category tendencies. S12 uses the ConvNeXt v2 network for two-stage training: Phase 1, Multi-dimensional physical feature learning: Input the image to be processed and learn the physical features in the rich semantic dataset to output multi-dimensional physical labels, wherein the multi-dimensional physical labels include: scene classification, subject integrity score, background complexity score, and number of subjects; Phase 2, Creative Type Adaptation Learning: Using the multidimensional physical labels from Phase 1 and the image to be processed as input, learn the creative type adaptation probability and output confidence scores for at least image blending, image cutout, image splicing, and non-creative creative types. S13. Creative Type Decision: Based on the multidimensional physical labels and creative type confidence, combined with preset logical verification rules, the final image creative type recognition result is output.

[0024] In step S12, the original image is input into the Vision-Language model for LoRA fine-tuning. This is to allow the VLM to understand the proprietary knowledge of the design domain, using a small amount of high-quality designer-annotated data to enable it to output descriptions containing design attributes, such as fine-grained tags like "high subject integrity," "cluttered background suitable for image cutout rather than image blending," and "occlusion of the subject." Additionally, an expanded tag set containing multi-dimensional information is generated to construct a semantically rich dataset, including multi-dimensional information such as: scene type, subject integrity, background complexity, number of people, and creative category preference.

[0025] In step S12, the two-stage training using the ConvNeXt v2 network is a two-stage progressive recognition model training based on ConvNeXt v2. This step is the core of the invention. Through a phased learning strategy, it solves the problem that single-task training is difficult to balance the accuracy of basic features and the matching of abstract creative ideas. The ConvNeXt v2 model architecture is chosen because it uses a fully convolutional Masked Autoencoder pre-training framework, which has a strong ability to perceive the global structure and local details of images, making it very suitable for judging compositional features such as truncation and occlusion, as detailed below.

[0026] The primary objective of multi-dimensional physical feature learning in Phase 1 is to enable the model to "see" the basic structure of the image. Specifically, as a further improvement, the learning using physical features from the rich semantic dataset includes: Multi-task learning of edge detection, object segmentation cues, and spatial relationships is performed using the corresponding physical attribute labels in the rich semantic dataset.

[0027] As a further improvement, the multi-task learning of edge detection, object segmentation cues, and spatial relationships using the corresponding physical attribute labels in the rich semantic dataset specifically satisfies the following model: ;in, Cross-Entropy Loss is used to calculate the predicted probability distribution. With real labels The differences between them are used to ensure that the model can accurately distinguish the basic composition type of the image. MeanSquared Error Loss is used to calculate the predicted score. Compared with real ratings based on VLM The Euclidean distance between them enables the model to have quantitative perception capabilities; (Pairwise Ranking Loss) is an auxiliary loss term that forces the model to learn the relative relationship that "the score of a suitable image is higher than the score of an unsuitable image" for images within the same batch, rather than simply fitting absolute values. These are the weights of the balanced hyperparameters for each task, used to dynamically adjust the contribution ratio of different tasks in gradient backpropagation.

[0028] The main objective of creative type adaptation learning in Phase Two is to enable the model to "understand" the design matching logic. The training strategy involves freezing the first few layers of the ConvNeXt v2 backbone network and focusing on training the high-level semantic head. For example, physical features learned in Phase One are used as prior knowledge and mapped to specific creative categories. This includes learning the implicit logic that "simple background + complete subject = high image matting adaptation."

[0029] In the creative adaptation stage, addressing the problem of extreme imbalance between creative and non-creative samples (i.e., high-quality images suitable for a specific template account for a very small percentage of users' albums), this invention abandons the traditional cross-entropy loss. As a further improvement, in stage two, it further includes: An improved Focal Loss approach is used to focus on hard-to-classify samples.

[0030] As a further improvement, the improved Focal Loss method specifically addresses hard-to-distinguish samples by satisfying the following model: ,in, : Represents the total number of creative categories (such as image blending, image cutout, and non-creative). : represents the model's predicted probability for the target creative category. : is the modulation coefficient, which is used when the sample is accurately classified ( When the coefficient approaches 1, it approaches 0, thus reducing the weight of simple samples; when samples are difficult to distinguish, the weight increases. : This is a class balance parameter used to adjust for bias caused by a large disparity in the proportion of positive and negative samples.

[0031] As a further improvement, among which, The default value is 2.0, which reflects the ability of the pre-built model to identify seemingly ambiguous edge samples.

[0032] In step S13, the model output is not a single result, but rather all intermediate labels and creative category labels. These are integrated through post-processing, receiving the confidence scores of each creative type from the model's second-stage output. This is combined with the physical labels from the first stage for logical circuit breaking verification to adapt to changing business needs. In one embodiment, the verification example is as follows: if the confidence score for "cutout creative" is high, but the "subject integrity" label detects severe limb truncation, then the priority of "cutout creative" is forcibly reduced, and the "image blending" or "non-creative" mode is recommended.

[0033] This invention further provides a computer device, which includes a memory, a communication interface, and a processor, wherein the memory, the communication interface, and the processor are interconnected; the memory stores a computer program, and the processor calls the computer program stored in the memory to implement the image creative type recognition method based on two-stage progressive learning as described above.

[0034] This invention further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image creative type recognition method based on two-stage progressive learning as described above.

[0035] This invention further provides a computer program product, which includes a computer program stored in a computer storage medium; a processor of a computer device reads the computer program from the computer storage medium and executes the computer program, causing the computer device to perform the image creative type recognition method based on two-stage progressive learning as described above.

[0036] The present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0037] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0038] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0039] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0040] Those skilled in the art should understand that the above embodiments are merely for illustrative purposes and are not intended to limit the scope of this application. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of this application.

Claims

1. An image creative type recognition method based on two-stage progressive learning, characterized in that, The method comprises the following steps: S10, obtaining an original image, wherein the original image comprises artificial pre-labeled physical attributes and a final creative tendency, and the physical features comprise scene type, background type, number of people, and completeness of the characters; S11, inputting the original image into a Vision-Language model for LoRA fine-tuning, and then using the fine-tuned Vision-Language model to infer a large amount of unlabeled data to generate an extended label set containing multi-dimensional information to construct a rich semantic data set, wherein the extended label set comprises predicted physical features and creative category tendencies; S12, performing two-stage training using a ConvNeXt v2 network: Stage one, multi-dimensional physical feature learning: inputting an image to be processed and learning using the physical features in the rich semantic data set to output multi-dimensional physical labels, wherein the multi-dimensional physical labels comprise scene classification, subject integrity score, background complexity score, and subject quantity; Stage two, creative type adaptation learning: taking the multi-dimensional physical labels of stage 1 and the image to be processed as input, learning creative type adaptation probability, and outputting at least a confidence score including a dissolve creative, a cutout creative, a splice creative, and a non-creative; S13, creative type decision: outputting a final image creative type recognition result according to the multi-dimensional physical labels and the creative type confidence score, and combining a preset logical verification rule. 2.The image creative type recognition method based on two-stage progressive learning according to claim 1, characterized in that: The learning using the physical features in the rich semantic data set specifically comprises: Using the corresponding physical attribute labels in the rich semantic data set for multi-task learning of edge detection, object segmentation suggestion, and spatial relationship. 3.The image creative type recognition method based on two-stage progressive learning according to claim 2, characterized in that: The multi-task learning of edge detection, object segmentation suggestion, and spatial relationship using the corresponding physical attribute labels in the rich semantic data set specifically satisfies the following model: ; wherein, , for computing a predictive probability distribution the difference between the predicted probability distribution and the true label ​ , for computing a prediction score and a real score based on VLM expansion between them; is an auxiliary loss term; are balanced hyper-parameter weights for each task, respectively, for dynamically adjusting the contribution ratio of different tasks in gradient backpropagation. 4.The image creative type recognition method based on two-stage progressive learning according to claim 1, characterized in that: In stage two, further comprising: Using an improved Focal Loss to focus on difficult-to-classify samples.

5. The image creative type recognition method based on two-stage progressive learning according to claim 4, characterized in that: The improved Focal Loss for focusing on difficult-to-classify samples specifically satisfies the following model: wherein, : represents the total number of creative categories, : is the predicted probability of the model for the target creative category, : is the modulation coefficient; : is the category balancing parameter, used to adjust the bias caused by the large proportion of positive and negative samples. 6.The image creative type recognition method based on two-stage progressive learning according to claim 5, characterized in that: wherein, The default value is 2.

0.

7. A computer device, comprising: The computer device comprises a memory, a communication interface, and a processor, wherein the memory, the communication interface, and the processor are connected to each other; the memory stores a computer program, and the processor invokes the computer program stored in the memory to implement the image creative type recognition method based on two-stage progressive learning according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is executed by the processor to implement the image creative type recognition method based on two-stage progressive learning according to any one of claims 1 to 6.

9. A computer program product, characterised in that, The computer program product comprises a computer program stored in a computer storage medium; the processor of the computer device reads the computer program from the computer storage medium, and the processor executes the computer program to enable the computer device to execute the image creative type recognition method based on two-stage progressive learning according to any one of claims 1 to 6.

Images