A method, system, terminal and medium for image emotion editing based on large model
Through the image emotion editing method based on the big model, using the emotion data set and visual language model to extract and cluster emotional factors, build an emotional factor tree for editing, which solves the problem that image emotion editing cannot be accurately processed in the existing technology, and realizes emotion Editing accuracy and structural integrity.
Patent Information
- Application Number
- CN202410911543.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-09
AI Technical Summary
The prior art cannot accurately process emotional changes in images, especially when dealing with more abstract emotions.
Using a large-model-based image emotion editing method, visual semantics are extracted through the emotion dataset, emotional factors are clustered, and pre-constructed emotional factor tree is used for editing, and the best editing results are finally determined through the sorting strategy.
Accurate editing of image emotions is achieved, which can accurately evoke target emotions while maintaining the integrity of image structure, improving the accuracy and consistency of emotional editing.
Smart Images

Figure CN118505841B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a large model-based image emotion editing method, system, terminal and medium. Background Art
[0002] Image emotion editing aims to evoke specific emotional responses by modifying user-provided images. This process not only needs to evoke the expected emotions, but also needs to preserve the structure of the original image, which is a complex task that needs to balance the two goals.
[0003] Image processing covers a variety of tasks, including image colorization, object addition / removal, background replacement, action change, and style transfer. Existing image emotion editing methods mainly achieve editing by adjusting the color and style of the image, but they often fail to induce precise and significant changes in emotional style, and therefore still face severe challenges when processing more abstract emotions. Summary of the invention
[0004] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide an image emotion editing method, system, terminal and medium based on a large model, aiming to solve the problem that the image emotion editing method in the prior art cannot accurately process the emotional changes of the image.
[0005] In order to achieve the above object, the present invention provides a first aspect of a large model-based image emotion editing method, comprising:
[0006] Get the image to be processed;
[0007] Performing emotion recognition on the content of the image to be processed according to the target emotion category to obtain a number of emotion factors;
[0008] Using the pre-constructed emotion factor tree to edit each emotion factor respectively, to obtain the editing result of each emotion factor;
[0009] All the editing results are sorted using a preset sorting strategy to determine the best editing result.
[0010] In one embodiment, the process of constructing the emotion factor tree includes:
[0011] Get the sentiment dataset;
[0012] Extracting features of each emotion included in the emotion data set to obtain visual semantics of each emotion;
[0013] Clustering the visual semantics based on the correlation between the emotional factors and the visual semantics to obtain a number of emotional factors;
[0014] A preset visual language model is used to assign content and color to each of the emotional factors, and based on the content and the color, an emotional factor tree corresponding to each of the emotional factors is constructed.
[0015] In one embodiment, clustering the visual semantics based on the correlation between the emotional factors and the visual semantics to obtain several emotional factors includes:
[0016] Based on the correlation between the emotional factors and the visual semantics, a number of clusters are initialized using the emotional factors corresponding to the visual semantics;
[0017] Filtering the clusters based on a preset cluster filtering rule to obtain valid clusters;
[0018] Based on the emotion categories included in the effective clusters, the emotion factors corresponding to each emotion category are obtained.
[0019] In one implementation, filtering the clusters based on a preset cluster filtering rule to obtain valid clusters includes:
[0020] Performing initial filtering on the clusters based on the number of images contained in the clusters to obtain initial valid clusters;
[0021] The similarity and the weighted average sentiment score between any two of the initial valid clusters are calculated, and the initial valid clusters are filtered based on the similarity and the weighted average sentiment score to obtain valid clusters.
[0022] In one implementation, the editing of each of the emotional factors using the pre-constructed emotional factor tree to obtain the editing result of each of the emotional factors includes:
[0023] Extracting the content and color of each of the emotional factors based on a pre-constructed emotional factor tree, and determining a number of editing instructions;
[0024] The image to be processed is edited using each of the editing instructions to obtain an editing result of each of the emotional factors.
[0025] In one embodiment, the method of sorting all the editing results using a preset sorting strategy to determine the best editing result includes:
[0026] Obtaining several preset sorting strategies and the priority of each of the sorting strategies;
[0027] All the editing results are divided into emotion categories in descending order of priority of the sorting strategy to determine the best editing result.
[0028] A second aspect of the present invention provides an image emotion editing system based on a large model, the system comprising:
[0029] The module for acquiring the image to be processed is used to acquire the image to be processed;
[0030] An emotional factor recognition module is used to perform emotional recognition on the content of the image to be processed according to the target emotional category to obtain a number of emotional factors;
[0031] An emotion editing module, used to edit each of the emotion factors respectively using a pre-built emotion factor tree to obtain an editing result of each of the emotion factors;
[0032] The best editing result screening module is used to sort all the editing results using a preset sorting strategy to determine the best editing result.
[0033] In one embodiment, the emotion editing module includes a data set acquisition unit, an emotion feature extraction unit, a clustering unit and an emotion factor tree construction unit, wherein the data set acquisition unit is used to acquire an emotion data set; the emotion feature extraction unit is used to extract features for each emotion included in the emotion data set to obtain visual semantics for each emotion; the clustering unit is used to cluster the visual semantics based on the correlation between the emotion factors and the visual semantics to obtain a number of emotion factors; and the emotion factor tree construction unit is used to assign content and color to each emotion factor using a preset visual language model, and to construct an emotion factor tree corresponding to each emotion factor based on the content and the color.
[0034] A third aspect of the present invention provides a terminal, comprising a memory, a processor, and a large model-based image emotion editing program stored in the memory and executable on the processor, wherein the large model-based image emotion editing program, when executed by the processor, implements any one of the steps of the above-mentioned large model-based image emotion editing method.
[0035] A fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a large model-based image emotion editing program, which, when executed by a processor, implements any step of the above-mentioned large model-based image emotion editing method.
[0036] Compared with the prior art, the beneficial effects of this solution are as follows:
[0037] The large-model-based image emotion editing method proposed in the present invention includes two-stage processing: emotion attribution and image editing. In the emotion attribution stage, based on the correlation between emotion and visual semantics, a visual language model is used to create a hierarchy of semantic factors representing abstract emotions; in the image editing stage, a visual language model is used to identify the factors most relevant to the provided image, and emotion recognition and screening are performed on the processed image according to the target emotion category to obtain more valuable emotion factors, and a pre-constructed emotion factor tree is used to edit and screen each of the emotion factors to obtain accurate image emotion editing results. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0039] Figure 1 A brief flow chart of the image emotion editing method based on a large model of the present invention;
[0040] Figure 2 A block diagram of image emotion editing based on a large model of the present invention;
[0041] Figure 3 Constructing a flow chart for the emotion factor tree of the present invention;
[0042] Figure 4 It is a schematic diagram of the qualitative evaluation results of the image emotion editing method based on the large model of the present invention;
[0043] Figure 5 It is a schematic diagram of the module of the image emotion editing system based on the large model of the present invention;
[0044] Figure 6 It is a schematic diagram of the terminal structure of the present invention. DETAILED DESCRIPTION
[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.
[0046] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0047] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0048] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0049] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0051] In view of the problem that the image emotion editing methods in the prior art cannot accurately handle the emotion changes of images, the present invention proposes an image emotion editing method (EmoEdit) based on a large model. The method is implemented based on a novel two-stage framework, including emotion attribution and image editing. In the emotion attribution stage, the visual language model (VLM) is used to create a hierarchy of semantic factors representing abstract emotions based on the emotion dataset (EmoSet); in the image editing stage, the visual language model is used to identify the factors most relevant to the provided image, and guide the generation of the editing model to perform emotion modification. A sorting strategy for selecting the best editing result is proposed, which can well balance the factors of emotional consistency and structural integrity.
[0052] The embodiment of the present invention provides an image emotion editing method based on a large model, which is deployed on electronic devices such as computers and servers and is applied to scenarios where emotion editing of images is required. Figure 1 and Figure 2 As shown, the steps of the method in this embodiment include:
[0053] Step S100: obtaining an image to be processed;
[0054] Specifically, an image to be processed is obtained, and preprocessed by normalization or standardization to ensure that the pixel value is within an appropriate range; image enhancement or denoising can also be performed to improve the robustness of subsequent feature extraction. The type of the image to be processed is not specifically limited.
[0055] Step S200: performing emotion recognition on the content of the image to be processed according to the target emotion category to obtain a number of emotion factors;
[0056] Specifically, a large visual language model (such as GPT-4V) is used to perform emotion recognition on the content of each of the images to be processed according to the target emotion category, and the target emotion category that is most suitable for editing the image to be processed is selected, wherein the emotional factors included in the target emotion category may include one or more of a variety of basic emotions such as pleasure, awe, satisfaction, excitement, anger, disgust, fear and sadness, that is, the target emotion category may be a single emotion or a complex compound emotion or emotional state. Since emotional factors may be diverse and ambiguous, an emotional threshold or confidence level can be set for the large visual language model to screen out several emotional factors corresponding to the image to be processed.
[0057] Step S300: using the pre-constructed emotion factor tree to edit each emotion factor respectively, and obtaining the editing result of each emotion factor;
[0058] Specifically, for each emotional factor, determine the type of editing that needs to be performed. For example, it is necessary to modify the intensity of the emotional factor, move the emotional factor to a different position in the tree, add a new sub-emotional factor, or delete unnecessary emotional factors. The structure and content of the pre-constructed emotional factor tree are clarified, and one or more emotional factors matching the editing type corresponding to each emotional factor are found by traversing the nodes of the emotional factor tree, and then the emotional factors are operated accordingly to obtain one or more editing results corresponding to each of the emotional factors, wherein each editing type corresponds to an editing result; the emotional factor tree is used to characterize all emotional factors evoked by each emotion, and map abstract emotions to tangible visual cues. If the intensity of the emotional factor needs to be modified, the weight or score associated with the emotional factor can be adjusted. If the emotional factor needs to be moved to a different position in the tree, the relationship between the emotional factor and the root node or leaf node can be reestablished. If a new emotional factor needs to be added, a new node can be created under the emotional factor. If an emotional factor needs to be deleted, the emotional factor can be removed from the tree and the integrity of the tree is ensured.
[0059] Step S400: sorting all the editing results using a preset sorting strategy to determine the best editing result.
[0060] Specifically, in order to ensure the accuracy of the results of emotional editing of images for each emotional factor, multiple editing results are usually generated for each emotional factor, and all the editing results are sorted in ascending or descending order using a preset sorting strategy, so as to screen out the best editing result from the sorting results, wherein the sorting strategy is generated according to the requirements of the actual application scenario and the evaluation indicators such as rationality, consistency, accuracy, and user preference of the editing results. If rationality is considered, it can be evaluated whether the editing results are logically reasonable and whether they comply with the rules of the emotional factor tree. If consistency is considered, it can be checked whether the editing results are consistent with other emotional factors or the emotional content of the entire image. If accuracy is considered, the accuracy of the editing results can be evaluated based on historical data or user feedback. If user preference is considered, user ratings or feedback can be introduced as part of the rating. In practical applications, the sorting strategy can be flexibly adjusted and optimized according to the specific situation to obtain more accurate and reliable editing results.
[0061] In this embodiment, based on the correlation between emotion and visual semantics, a large visual language model is used to perform emotion recognition and screening on the processed image according to the target emotion category to obtain more valuable emotion factors, and a pre-constructed emotion factor tree is used to edit and screen each of the emotion factors to obtain accurate image emotion editing results.
[0062] like Figure 3As shown, in one embodiment, the process of constructing the emotion factor tree in step S300 includes:
[0063] Step S310: Acquire a sentiment data set;
[0064] Specifically, a large-scale emotion dataset (such as EmoSet, etc.) is obtained, which includes rich labeled emotion attributes to meet the needs of emotion editing.
[0065] Step S320: extracting features of each emotion included in the emotion data set to obtain visual semantics of each emotion;
[0066] Specifically, due to the correlation between emotion and visual semantics, the Contrastive Language–Image Pre-training (CLIP) model can establish the correlation between images and texts, and the Data Intensive Object detection with self-Supervision version 2 (DINOv2) model based on the self-supervised learning strategy is a computer vision model that can generate high-quality visual features. Based on this, this embodiment uses CLIP and DINOv2 to collaboratively generate the visual semantics of each emotion. Specifically, the CLIP model is used to pre-train the sample image and the corresponding emotion label so that the model can learn the association between the image and the emotion, and the pre-trained CLIP model is used to perform emotion recognition on the sample image, and the emotion label that best matches the image is output. At the same time, the visual features of the sample image are extracted using the DINOv2 model to capture the key information in the sample image. Then the visual features extracted by the DINOv2 model are fused with the emotion label output by the CLIP model to output a visual semantic description that expresses the same emotion as the sample image.
[0067] Step S330: clustering the visual semantics based on the correlation between the emotional factors and the visual semantics to obtain a number of emotional factors;
[0068] Step S340: using a preset visual language model to assign content and color to each of the emotional factors, and constructing an emotional factor tree corresponding to each of the emotional factors based on the content and the color.
[0069] Specifically, a visual language model (such as GPT-4V) is used to assign content and color to each of the emotional factors, that is, each emotional factor is represented by content and color, and the content is used to construct objects, scenes, actions and facial expressions to form an emotional factor tree, and then the colors corresponding to the objects, scenes, actions and facial expressions are determined according to the one-to-one correspondence between content and color. For example, if an image shows a person sitting on the lawn playing football on a sunny spring day, the content of the image can be characterized as "playing football", its color is described as "sunlight, warm", and it is classified under the "action" category.
[0070] The construction and editing of the emotion factor tree is an iterative and continuous optimization process. With the deepening of the understanding of emotion factors and the accumulation of data, it may be necessary to continuously update and adjust the structure and content of the emotion factor tree to improve the accuracy and reliability of emotion recognition.
[0071] In this embodiment, the emotion data set is analyzed and processed through the visual language model, content and color are assigned to each emotion factor, and a hierarchical emotion factor tree is constructed based on the content and color to characterize all the emotion factors evoked by each emotion. That is, based on the emotion data set, a visual language model is used to create a hierarchy of semantic factors that can represent abstract emotions, namely, an emotion factor tree, thereby realizing the accurate mapping of abstract emotions to tangible visual semantic descriptions.
[0072] In one implementation, clustering the visual semantics based on the correlation between the emotional factors and the visual semantics in step S330 to obtain a number of emotional factors includes:
[0073] Step S331: based on the correlation between the emotional factors and the visual semantics, using the emotional factors corresponding to the visual semantics, initializing a number of clusters;
[0074] Step S332: filtering the clusters based on a preset cluster filtering rule to obtain valid clusters;
[0075] Specifically, the clustering clusters are initially filtered based on the number of images contained in the clustering clusters to obtain initial valid clustering clusters; if the similarity is greater than a preset similarity threshold, or the similarity is less than the similarity threshold and the weighted average sentiment score between the initial valid clustering clusters corresponding to the similarity meets the preset sentiment score threshold, a valid clustering cluster is obtained.
[0076] Specifically, the operation steps of the sentiment attribution algorithm provided in Algorithm 1 are:
[0077] Input is M Images: ; The output is Clusters: ; Filter out Emotional factors: ;
[0078] Step 1: Semantic Clustering
[0079] Extract CLIP features for each image: ;initialization clusters, each containing one image: ; Calculate the similarity matrix: ;like , then merge the clusters As a new cluster ; Calculate clustering The center point of ; Update the similarity matrix ;until ,return Clusters: ;
[0080] Step 2: Sentiment Filtering
[0081] Initialize the sentiment factor index: ;make ,like , then calculate the average similarity: ;like , then calculate the average sentiment score: ,in is a pre-trained sentiment classifier; if , then clustering Through the filtering process: ; Iterative update , until , and finally return Emotional factors: .
[0082] Due to the unsupervised nature of clustering, clustering results are not always perfect. Therefore, the initialization matrix is iteratively merged according to the similarity between clusters using Algorithm 1 until all cluster similarities are less than 0.89. Clusters with excessive similarity at the pixel level are considered insufficient as generalized semantic factors to eliminate clusters that are not related to emotions. And the emotional factors in clusters containing less than five images are considered to be unimportant emotional factors for the relevant emotional categories. In addition, in order to enhance the efficacy of emotional editing, clusters that cannot evoke emotions are excluded.
[0083] Step S333: based on the emotion categories included in the effective clusters, obtaining the emotion factors corresponding to each emotion category.
[0084] Specifically, since the emotion editing task involves editing images with unknown and variable content, and a large number of potential emotion factors complicate the selection process, matching appropriate emotion factors with sample image content is essential for producing meaningful emotion editing. Based on this, this embodiment uses GPT-4V to identify the emotion factors that best fit the context for the emotion categories contained in the effective clustering clusters corresponding to each sample image. For example, the emotion factors that rank in the top few (such as the top 5, etc.) in correlation with the emotion category can be selected as the emotion factors corresponding to the emotion category. The emotion factors included in each emotion category can include one or more of a variety of basic emotions such as pleasure, awe, satisfaction, excitement, anger, disgust, fear and sadness. It should be stated that when selecting the number of emotion factors with a large correlation with the emotion category, a comprehensive consideration can be made based on the total number of emotion factors screened out, as well as requirements such as calculation accuracy and calculation amount.
[0085] In this embodiment, the proposed emotion attribution algorithm can automatically summarize the relevant influencing factors corresponding to different emotions in the emotion data set.
[0086] In one implementation, the step S300 of editing each of the emotional factors using the pre-constructed emotional factor tree to obtain the editing result of each of the emotional factors includes:
[0087] Step S350: extracting the content and color of each of the emotional factors based on the pre-constructed emotional factor tree, and determining a number of editing instructions;
[0088] Step S360: Use each of the editing instructions to edit the image to be processed respectively to obtain the editing result of each of the emotional factors.
[0089] Specifically, GPT-4V is used to assign content and color to each emotional factor. Since EmoEdit is a training-free method, this embodiment adopts a user-guided image editing method (InstructPix2Pix, IP2P), and generates an editing instruction for content and color respectively. Each editing instruction is used to edit the image to be processed to obtain the desired editing result, that is, to evoke emotions by modifying the image content and color. For example, if the sample image shows a living room and the desired emotion is "satisfaction", GPT-4V may select the factor "comfortable sofa" and generate a content instruction: "Put a comfortable sofa and some pillows on the floor"; for color, a content instruction may be generated: "Adjust the color scheme to warm tones". In addition, the color can be further fine-tuned according to the content changes to improve the accuracy of each modification. For example, "colorful flowers" usually cause satisfaction, while "colorful swimming" may be considered unusual, so the color "colorful" corresponding to "swimming" can be adjusted, highlighting the importance of the matching instruction format.
[0090] In one implementation, the step S400 of sorting all the editing results using a preset sorting strategy to determine the best editing result includes:
[0091] Step S410: obtaining several preset sorting strategies and the priority of each of the sorting strategies;
[0092] Step S420: classify all the editing results into emotion categories in descending order of priority of the sorting strategy to determine the best editing result.
[0093] Specifically, as an emotion-oriented task, EmoEdit aims to elicit specific emotional responses from viewers. At the same time, as an image editing task, the goal of EmoEdit is to maintain the overall structure of the input image. Therefore, this embodiment uses emotional fidelity and structural integrity as criteria to find the best edit. First, based on the hierarchical nature of emotion, a hierarchical emotion filter comprising a first classifier, a second classifier and a score regressor is constructed, wherein the first classifier corresponds to eight emotion categories, that is, emotions such as pleasure, awe, satisfaction, excitement, anger, disgust, fear and sadness can be output. It can be seen that these eight emotion categories characterize the specific emotional color of the editing result in detail; the second classifier corresponds to two polarities, namely positive and negative, that is, the eight emotion categories are classified into positive categories or negative categories. It can be seen that the second classifier can divide the editing results into two categories of positive and negative, and the expression of emotional color is not as specific as the output result of the first classifier; the score regressor is used to score the editing results to determine the ranking of all generated editing results. It can be seen that the score regressor is a numerical evaluation of the editing results, and the emotional category of each of the editing results can be determined by the size of the numerical value, but it is not as good as the first classifier and the second classifier in being able to intuitively observe the emotional color of the editing results. Then, the processing order of the first classifier, the second classifier and the score regressor is used as the priority from high to low, that is, the first classifier is first used to classify all the editing results into emotional categories, and if the first classifier fails to determine the emotional category corresponding to the editing result of the image to be processed, the second classifier is used to classify all the editing results into emotional categories, and if the first classifier fails to determine the emotional category corresponding to the editing result of the image to be processed, the structural similarity (SSIM) score regressor is used to score all the editing results, and the emotional categories are divided according to the scores, so that the editing result corresponding to the emotional classification result is determined as the best editing result. In this process, EmoEdit can reliably generate emotionally resonant, contextually appropriate and structurally complete images.
[0094] In order to verify the effectiveness of the method of the present invention, a data set containing 416 images was formed in the experiment, and the classification results were quantitatively and qualitatively evaluated. The quantitative evaluation results are shown in Table 1. By adopting the method of the present invention and various image editing methods in the prior art, the peak signal-to-noise ratio, structural similarity, learning-perceived image block similarity, image similarity in CLIP space, emotion accuracy, emotion score, etc. are evaluated as evaluation indicators. It can be seen from Table 1 that the method of the present application performs well in these aspects. Some results of the qualitative evaluation are shown in Table 1. Figure 4 As shown, it should be noted that Figure 4 Each edited image in the image has different content and color. Figure 4 It can be seen that each of the three exemplary images can generate edited images under eight emotions using the first classifier. These emotional edited images can not only evoke corresponding emotions, but also maintain basically the same image structure as the image before editing, showing emotion-oriented and semantic-oriented editing, as well as emotion generation editing, emotion change editing, and emotion enhancement editing. In short, the performance of the method of the present invention is better than that of the existing technical solutions.
[0095] Table 1:
[0096]
[0097] like Figure 5 As shown, corresponding to the above-mentioned image emotion editing method based on a large model, an embodiment of the present invention further provides an image emotion editing system based on a large model, and the above-mentioned image emotion editing system based on a large model includes:
[0098] The to-be-processed image acquisition module 510 is used to acquire the to-be-processed image;
[0099] The emotion factor recognition module 520 is used to perform emotion recognition on the content of the image to be processed according to the target emotion category to obtain a number of emotion factors;
[0100] The emotion editing module 530 is used to edit each of the emotion factors using the pre-constructed emotion factor tree to obtain the editing result of each of the emotion factors;
[0101] The best editing result screening module 540 is used to sort all the editing results using a preset sorting strategy to determine the best editing result.
[0102] Furthermore, the emotion editing module 530 in the system includes a data set acquisition unit, an emotion feature extraction unit, a clustering unit and an emotion factor tree construction unit, wherein:
[0103] A data set acquisition unit, used for acquiring a sentiment data set;
[0104] An emotion feature extraction unit, used for extracting features of each emotion included in the emotion data set to obtain visual semantics of each emotion;
[0105] A clustering unit, used for clustering the visual semantics based on the correlation between the emotional factors and the visual semantics to obtain a plurality of emotional factors;
[0106] The emotional factor tree construction unit is used to allocate content and color to each of the emotional factors using a preset visual language model, and to construct an emotional factor tree corresponding to each of the emotional factors based on the content and the color.
[0107] Specifically, in this embodiment, the specific functions of the above-mentioned large-model-based image emotion editing system can also refer to the corresponding description in the above-mentioned large-model-based image emotion editing method, which will not be repeated here.
[0108] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 6 As shown. The terminal includes a processor, a memory, a network interface and a display screen connected via a system bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and an image emotion editing program based on a large model. The internal memory provides an environment for the operation of the operating system and the image emotion editing program based on a large model in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal via a network connection. When the image emotion editing program based on a large model is executed by the processor, the steps of any one of the above-mentioned image emotion editing methods based on a large model are implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.
[0109] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0110] In one embodiment, a terminal is provided, which includes a memory, a processor, and a large model-based image emotion editing program stored in the memory and executable on the processor. When the large model-based image emotion editing program is executed by the processor, the steps of any one of the large model-based image emotion editing methods provided in the embodiments of the present invention are implemented.
[0111] An embodiment of the present invention also provides a computer-readable storage medium, on which a large model-based image emotion editing program is stored. When the large model-based image emotion editing program is executed by a processor, the steps of any one of the large model-based image emotion editing methods provided in the embodiment of the present invention are implemented.
[0112] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0113] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0114] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0115] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0116] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0117] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A method for image emotion editing based on large models, It is characterized in that The following steps are involved: Acquire a preprocessed image to be processed, wherein the preprocessing includes one or more of normalization, standardization, image enhancement or denoising; Performing emotion recognition on the content of the image to be processed according to the target emotion category to obtain a number of emotion factors; Using the pre-constructed emotion factor tree to edit each emotion factor respectively, to obtain the editing result of each emotion factor; Sorting all the editing results using a preset sorting strategy to determine the best editing result; The process of constructing the emotion factor tree includes: Acquire an emotion data set; extract features for each emotion included in the emotion data set to obtain visual semantics for each emotion; cluster the visual semantics based on the correlation between the emotion factors and the visual semantics to obtain a number of emotion factors; use a preset visual language model to assign content and color to each emotion factor, and construct an emotion factor tree corresponding to each emotion based on the content and the color; The method of sorting all the editing results using a preset sorting strategy to determine the best editing result includes: Obtaining several preset sorting strategies and the priority of each sorting strategy; classifying all the editing results into emotion categories according to the order of priority of the sorting strategies from high to low, and determining the best editing result; the sorting strategy is generated according to the actual application scenario requirements, as well as the rationality, consistency, accuracy and user preference evaluation indicators of the editing results; The emotion factor tree is used to characterize all emotion factors evoked by each emotion, and map abstract emotions to tangible visual cues; modify the strength of emotion factors by adjusting weights or scores associated with emotion factors; move emotion factors to different positions of the tree by re-establishing the relationship between emotion factors and root nodes or leaf nodes of the emotion factor tree; add new emotion factors by creating a new node under the emotion factors; delete emotion factors by removing them from the emotion factor tree, and ensure the integrity of the emotion factor tree.
2. The image emotion editing method based on the large model according to claim 1, It is characterized in that The visual semantics are clustered based on the correlation between the emotional factors and the visual semantics to obtain a number of emotional factors, including: Based on the correlation between the emotional factors and the visual semantics, a number of clusters are initialized using the emotional factors corresponding to the visual semantics; Filtering the clusters based on a preset cluster filtering rule to obtain valid clusters; Based on the emotion categories included in the effective clusters, the emotion factors corresponding to each emotion category are obtained.
3. The image emotion editing method based on the large model according to claim 2, It is characterized in that The filtering of the clusters based on the preset cluster filtering rules to obtain valid clusters includes: Performing initial filtering on the clusters based on the number of images contained in the clusters to obtain initial valid clusters; The similarity and the weighted average sentiment score between any two of the initial valid clusters are calculated, and the initial valid clusters are filtered based on the similarity and the weighted average sentiment score to obtain valid clusters.
4. The image emotion editing method based on the large model according to claim 1, It is characterized in that The method of using the pre-constructed emotion factor tree to edit each emotion factor to obtain the editing result of each emotion factor comprises: Extracting the content and color of each of the emotional factors based on a pre-constructed emotional factor tree, and determining a number of editing instructions; The image to be processed is edited using each of the editing instructions to obtain an editing result of each of the emotional factors.
5. Image emotion editing system based on large model, It is characterized in that The system comprises: The module for acquiring the image to be processed is used to acquire the image to be processed after preprocessing, wherein the preprocessing includes one or more of normalization, standardization, image enhancement or denoising; An emotional factor recognition module is used to perform emotional recognition on the content of the image to be processed according to the target emotional category to obtain a number of emotional factors; The emotion editing module is used to edit each emotion factor respectively by using the pre-constructed emotion factor tree to obtain the editing result of each emotion factor; the construction process of the emotion factor tree includes: obtaining an emotion data set; extracting features for each emotion included in the emotion data set to obtain the visual semantics of each emotion; clustering the visual semantics based on the correlation between the emotion factor and the visual semantics to obtain a number of emotion factors; using a preset visual language model to assign content and color to each emotion factor, and based on the content and the color, constructing an emotion factor tree corresponding to each emotion; the emotion factor tree is used to characterize all the emotion factors evoked by each emotion, and map abstract emotions to tangible visual cues; modifying the intensity of the emotion factor by adjusting the weight or score associated with the emotion factor; moving the emotion factor to a different position of the tree by re-establishing the relationship between the emotion factor and the root node or leaf node of the emotion factor tree; adding a new emotion factor by creating a new node under the emotion factor; deleting the emotion factor by removing the emotion factor from the emotion factor tree and ensuring the integrity of the emotion factor tree; The best editing result screening module is used to sort all the editing results using a preset sorting strategy to determine the best editing result; the sorting of all the editing results using a preset sorting strategy to determine the best editing result includes: Obtain several preset sorting strategies and the priority of each sorting strategy; classify all the editing results into emotion categories in descending order of the priority of the sorting strategy to determine the best editing result; the sorting strategy is generated according to the actual application scenario requirements, as well as the rationality, consistency, accuracy and user preference evaluation indicators of the editing results.
6. A terminal, It is characterized in that The terminal includes a memory, a processor, and a large model-based image emotion editing program stored in the memory and executable on the processor. When the large model-based image emotion editing program is executed by the processor, the steps of the large model-based image emotion editing method as described in any one of claims 1 to 4 are implemented.
7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a large-model-based image emotion editing program, which, when executed by a processor, implements the steps of the large-model-based image emotion editing method as described in any one of claims 1-4.
