Visual prompt word generation method and device based on tone style recognition and reconstruction

Abstract

The application discloses a visual prompt word generation method and device based on tone style recognition and reconstruction, which comprises the following steps: receiving target content text and reference visual data, extracting candidate content text from the reference visual data and obtaining content representation vectors through text encoding to form a content subspace; obtaining reference visual representation vectors through image encoding of the reference visual data, calculating the orthogonal projection of the reference visual representation vectors on the content subspace and removing the orthogonal projection to obtain tone residual representation vectors; then, calculating a matching score according to the similarity of the tone attribute representation vectors, the tone residual representation vectors and the content subspace, screening tone attribute candidates, replacing the original bearing objects in the tone attribute candidates with target objects, generating replaced tone descriptions, and constructing visual prompt words together with the target content text. The application can reduce the interference of reference content semantics on a target picture and improve the accuracy of tone style migration.

Description

Technical Field

[0001] This application relates to the field of visual cue word generation technology, specifically a method and apparatus for generating visual cue words based on tonal style recognition and reconstruction. Background Technology

[0002] With the application of image generation models in advertising visuals, film and television storyboards, and product promotional image production, creators often need to use reference images or video clips to determine the tonal style of the target image. Tonal style usually includes visual attributes such as the relationship between light and dark, color warmth and coolness, contrast, highlight shape, graininess, and material reflection. These attributes can be transferred to new creative content to a certain extent, detached from the specific shooting object.

[0003] Existing visual cue word generation methods mostly rely on the overall recognition of reference images or manual descriptions, which easily incorporates the semantic content of objects, scenes, props, and human actions in the reference image into the cue words. For example, when the reference image is a rainy night street, the system may extract content words such as street, pedestrians, and neon signs at the same time and input them into the generation command model, resulting in the appearance of elements unrelated to the user's creative idea in the target image. If the above content words are simply deleted, the system may also lose tonal information with transferable value, such as wet reflections, cool shadows, and local color highlights.

[0004] Furthermore, when the main object in the reference image occupies a large area, the image representation tends to be more biased towards the subject category, appearance outline, or scene structure, while the tonal information such as ambient light, dark hue, and rim light in the background is easily weakened. Existing methods struggle to effectively exclude the semantics of the reference content while preserving the tonal style of the reference, and also struggle to reasonably point the extracted tonal attributes to the objects in the target image, affecting the accuracy and stability of the generated results. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for generating visual cue words based on tonal style recognition and reconstruction, so as to solve the problems mentioned in the background art.

[0006] According to one aspect of this application, a method for generating visual cue words based on tonal style recognition and reconstruction is provided, comprising the following steps: Receive target content text and reference visual data; The candidate content text extracted from the reference visual data is encoded by a text encoder to obtain a content representation vector, and the content representation vector is used to construct a content subspace; The reference visual data is encoded by an image encoder to obtain a reference visual representation vector. The orthogonal projection component of the reference visual representation vector on the content subspace is calculated. The tonal residual representation vector is obtained by subtracting the orthogonal projection component from the reference visual representation vector. Each tone attribute text in the tone attribute text set is encoded by the text encoder to obtain a tone attribute representation vector. A matching score is calculated based on the similarity between each tone attribute representation vector and the tone residual representation vector, as well as the similarity with the content subspace. Candidate tone attributes are then selected based on the matching scores. Replace the original carrier object in each of the tonal attribute candidates with the target object in the target content text to obtain the replaced tonal description; Visual cue words are constructed based on the target content text and the replaced tonal description.

[0007] Preferably, calculating the matching score includes: the cosine similarity between each of the tonal attribute representation vectors and the tonal residual representation vector, minus the product of the balance coefficient and the maximum cosine similarity between each of the tonal attribute representation vectors and each content representation vector in the content subspace; the balance coefficient ranges from 0 to 1.

[0008] Preferably, after obtaining the tonal residual representation vector, the method further includes: dividing the image of the reference visual data into several regions, wherein the regions include at least a subject region and a background region; encoding the image data of each region by the image encoder to obtain a region image representation vector; calculating and subtracting the orthogonal projection component of each region image representation vector on the content subspace to obtain a region tonal residual representation vector; weighting and summing the region tonal residual representation vectors to obtain a region-level tonal residual representation vector; fusing the region-level tonal residual representation vector with the tonal residual representation vector to form a tonal representation vector; and using the tonal representation vector to replace the tonal residual representation vector in the subsequent calculation of the matching score.

[0009] Preferably, the fusion ratio of the tonal residual representation vector and the regional tonal residual representation vector is determined based on the ratio of the area of ​​the main subject region to the area of ​​the entire image in the reference visual data. The larger the proportion of the main subject region area, the higher the fusion weight of the regional tonal residual representation vector.

[0010] Preferably, when replacing the original carrier object with the target object, the replacement object is selected according to the adaptation relationship between the physical rendering conditions of each of the tonal attribute candidates and the surface characteristics of each of the target objects; for tonal attribute candidates that do not depend on a specific surface, their original text is retained and no replacement is performed.

[0011] Preferably, when assembling the visual cue words, the visual cue words are divided into target content segment, light segment, color segment, material segment, and constraint segment for segmented assembly; the content of the target content segment is taken from the target content text; the content of the light segment and the color segment are taken from the parts of the replaced tonal description that correspond to light and color, respectively; the content of the material segment is taken from the part of the replaced tonal description that describes the texture of the target object; the constraint segment is generated based on the candidate content text constituting the content subspace, listing the elements excluded from the target image.

[0012] Preferably, after assembling the visual cue words, the method further includes: encoding the target content text using the text encoder to obtain a target content representation vector, and encoding the visual cue words using the text encoder to obtain cue word representation vectors; calculating the cosine similarity between the cue word representation vector and the target content representation vector, the cosine similarity with the tonal residual representation vector, and the cosine similarity with the content subspace; determining whether the visual cue words need adjustment based on the three sets of cosine similarities, and if adjustment is needed, modifying only the portion of the visual cue words originating from the replaced tonal description, while keeping the portion originating from the target content text unchanged.

[0013] Preferably, determining whether the visual cue word needs adjustment based on three sets of cosine similarity includes: generating multiple candidate visual cue words, where the content originating from the target content text is the same but the content originating from the replaced tonal description is different; calculating the three sets of cosine similarity for each candidate, and selecting, through relative comparison between the candidates, the candidate with high cosine similarity to the target content representation vector and the tonal residual representation vector and low cosine similarity to the content subspace as the final output visual cue word.

[0014] Preferably, when the reference visual data contains multiple representative frames, the process of extracting the candidate content text from the candidate tonal attribute candidate filtering is performed on each representative frame, the frequency of each tonal attribute ranking high in the multiple representative frames is counted, and the tonal attribute whose frequency in most representative frames meets the preset condition is retained.

[0015] In another aspect, this application also provides a visual cue word generation device based on tonal style recognition and reconstruction, comprising: The data receiving module is used to receive target content text and reference visual data; The content subspace construction module is used to encode the candidate content text extracted from the reference visual data into a content representation vector using a text encoder, and to construct a content subspace using the content representation vector. The tonal residual representation acquisition module is used to encode the reference visual data through an image encoder to obtain a reference visual representation vector, calculate the orthogonal projection component of the reference visual representation vector on the content subspace, and subtract the orthogonal projection component from the reference visual representation vector to obtain the tonal residual representation vector. The tone attribute candidate filtering module is used to encode each tone attribute text in the tone attribute text set through the text encoder to obtain tone attribute representation vectors, calculate matching scores based on the similarity between each tone attribute representation vector and the tone residual representation vector and the similarity with the content subspace, and filter tone attribute candidates according to the matching scores. The tone description generation module is used to replace the original carrier object in each of the tone attribute candidates with the target object in the target content text to obtain the replaced tone description. The visual cue word construction module is used to construct visual cue words based on the target content text and the replaced tonal description.

[0016] This application extracts candidate content text from reference visual data and constructs a content subspace. It removes semantic projections related to objects, scenes, and actions from the reference visual representation, resulting in tonal residuals that more centrally represent light, color, and texture. This reduces the risk of reference image content being incorrectly transferred to the target image. Furthermore, by combining regional tonal residual fusion and tonal attribute matching score filtering, it can retain transferable ambient light, dark area hues, and material reflection information even when the subject has a high proportion or the background tone is weak. Simultaneously, by replacing the original carrier of tonal attributes with the corresponding object in the target content and verifying the generated visual cue words, this application can stably reconstruct the reference tonal values ​​into cue words recognizable by the visual generation model while maintaining the target content unchanged. This improves the accuracy of tonal reproduction, reduces the introduction of irrelevant elements, and enhances the consistency and controllability of the target image generation results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1This is a schematic diagram of a visual cue word generation method based on tonal style recognition and reconstruction provided in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram illustrating the calculation process of the tonal residual characterization vector provided in an embodiment of this disclosure.

[0020] Figure 3 This is a schematic diagram of the tonal attribute candidate filtering process provided in an embodiment of this disclosure.

[0021] Figure 4 This is a schematic diagram of a visual cue word generation device based on tonal style recognition and reconstruction, provided in an embodiment of this application.

[0022] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] According to embodiments of this disclosure, the application scenarios targeted by this method include situations where creators, when producing film storyboards, advertising main visuals, or product promotional images, need to reference the tonal style of a certain reference image or video clip, but the subject and scene of the target image are completely different from the reference image. Tonal style is a comprehensive description of visual attributes such as brightness, color warmth, and texture in photography and film production. For example, taking a photograph of a rainy night street as a reference, creators actually focus on tonal features such as low light, cool shadows, high-saturation local highlights, and wet reflections, rather than the street, pedestrians, or neon signs. If the overall description of the reference image is directly used as a prompt, the visual generation model, highly sensitive to nouns, will treat content words such as "street," "pedestrians," and "neon signs" as generation instructions, introducing elements unrelated to the creative concept into the target image. However, simply deleting these content words may result in the loss of valuable tonal information such as wet reflections and local color highlights. Furthermore, when the main subject occupies a large proportion of the reference image, the overall representation encodes more information such as the appearance of people, product outlines, or architectural forms, while the proportion of light and color information actually set by the photographer is reduced. To address this, our method excludes semantic components from the reference visual data, extracts transferable tonal features, and reconstructs these features into visual cues that fit the target image.

[0025] The following detailed description, in conjunction with specific embodiments, illustrates the implementation process of the visual cue word generation method based on tonal style recognition and reconstruction described in this application. It should be noted that these embodiments are merely for explaining this application and are not intended to limit the scope of protection of this application. Any conventional adjustments or substitutions made by those skilled in the art to the steps without departing from the concept of this application should be included within the scope of protection of this application.

[0026] like Figure 1 As shown in the figure, this application discloses a visual cue word generation method based on tonal style recognition and reconstruction, which includes the following method steps: S1 receives the target content text and reference visual data; S2, the candidate content text extracted from the reference visual data is encoded by a text encoder to obtain a content representation vector, and the content representation vector is used to form a content subspace; S3, the reference visual data is encoded by an image encoder to obtain a reference visual representation vector, the orthogonal projection component of the reference visual representation vector on the content subspace is calculated, and the tonal residual representation vector is obtained by subtracting the orthogonal projection component from the reference visual representation vector. S4, the text encoder encodes each tone attribute text in the tone attribute text set to obtain tone attribute representation vectors, calculates matching scores based on the similarity between each tone attribute representation vector and the tone residual representation vector and the similarity with the content subspace, and filters tone attribute candidates according to the matching scores. S5, replace the original carrier object in each of the tonal attribute candidates with the target object in the target content text to obtain the replaced tonal description; S6, construct visual cue words based on the target content text and the replaced tonal description.

[0027] In some embodiments, for step S1, according to the present disclosure, target content text and reference visual data are first received; in optional embodiments, a generation purpose is also received; wherein the target content text is provided by the user, describing the subject and scene of the target image, such as a high-end watch placed on a black stone countertop, and the image has the texture of a luxury advertisement; the reference visual data may be one or more images, or several frames selected from a video; the generation purpose specifies the application type or expression granularity of the output prompt words, such as a single image, an advertiser's visual, or a sequence of scenes; the system preprocesses the reference visual data, including normalizing the image size to the encoder's input specifications; if the reference data is a video, duplicate frames and blurry frames are removed, and representative frames are selected from the candidate frames based on the representativeness of the image sharpness and brightness distribution to participate in subsequent calculations.

[0028] In some embodiments, the text encoder and image encoder in this method belong to the same image-text joint representation model, which maps images and text to representation vectors in the same vector space, so that semantically related images and texts have a high cosine similarity in this space.

[0029] Specifically, the image-text joint representation model can adopt the contrastive language image pre-training model (CLIP), which has been widely used in the field of image-text multimodal processing. The image encoder is based on the visual transducer (ViT) structure: the input image is divided into fixed-size image blocks, each image block is converted into an embedding vector through linear mapping, a learnable classification label is concatenated at the beginning of the sequence, and the position embedding is superimposed and fed into the transducer encoding layer, which is composed of multiple layers of multi-head self-attention and feedforward networks stacked alternately; finally, the output corresponding to the classification label is taken and linearly projected onto the shared representation space as the image representation vector.

[0030] The text encoder is based on a Transformer structure: the input text is segmented into a token sequence, with start and end tokens added, and then fed into a multi-layer transformer encoding layer for processing; the output at the position corresponding to the end token is linearly projected and mapped to the same shared representation space as the image encoder, serving as the text representation vector; this image-text joint representation model is pre-trained on large-scale image-text pair data through contrastive learning, with the training objective being to make matching image-text pairs close in distance in the shared space and mismatched pairs far apart; this method directly uses the pre-trained image-text joint representation model without changing its network structure and parameters, only calling its encoding capabilities to complete subsequent processing.

[0031] In some embodiments, for step S2, the representation vector obtained after encoding the reference visual data by the image encoder simultaneously contains both content information and tonal information of the image. The former involves object categories, scene attributes, and character actions, while the latter involves light distribution, color tendencies, and texture features. Since the two types of information are encoded together in the same vector, when directly extracting the tonal description from this vector, the content information will be extracted along with it, causing the target image to deviate from the user's creative intent after entering the generated prompt words. To address this, this embodiment constructs a subspace representing the semantic direction of the content, identifies and removes the components of the reference visual representation in this subspace direction, and retains the tonal components perpendicular to the content direction.

[0032] Specifically, candidate content text is extracted from reference visual data. Candidate content text refers to text phrases describing specific objects, people, scenes, props, or actions in the image. Its function is to identify semantic directions in the reference image that should not migrate to the target image. The extraction method specifically includes running a general image recognition or object detection model on the reference image and taking the output category labels as candidate content text, such as street, pedestrian, umbrella, neon sign, etc. In some embodiments, if the reference visual data is accompanied by material annotations or video frame recognition results, these labels can also be included in the candidate range. The extraction range of candidate content text is limited to the content objects that actually appear in the reference image and does not include object descriptions in the target image. Object information in the target image is provided by the target content text. The two types of information have different uses: the former is used to construct exclusion directions, and the latter is used to determine replacement targets.

[0033] Each candidate content text is input into a text encoder to obtain a content representation vector corresponding to each candidate content text. Each content representation vector is a fixed-dimensional vector in a shared representation space, representing the semantic position and direction of the content phrase in that space. The system uses all content representation vectors as column vectors to form a content representation matrix, and the column space of this matrix is ​​the content subspace. The content subspace is a vector subspace spanned by multiple content semantic directions, representing the set of semantic directions in the reference image related to object category, scene attributes, and action posture.

[0034] Since the candidate content text is dynamically extracted with each processing of reference visual data, the content subspace is automatically generated accordingly. Different reference images will generate different candidate content text and different numbers of content representation vectors, thus forming content subspaces with different dimensions and directions, so that the system does not need to maintain independent content filtering rules for each image type.

[0035] In some embodiments, for step S3, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the tonal residual representation vector calculation process provided in this embodiment of the disclosure. In S201, the preprocessed reference image is input into the image encoder to obtain the reference visual representation vector. This vector and the aforementioned content representation vector reside in the same shared representation space, allowing for direct vector operations.

[0036] In S202, the orthogonal projection component of the reference visual representation vector on the content subspace is calculated, and the tonal residual representation vector is obtained by subtracting the orthogonal projection component from the reference visual representation vector.

[0037] To separate the tonal information in the reference visual representation vector, the system calculates the orthogonal projection component of the vector in the content subspace. The orthogonal projection finds the point closest to the reference visual representation vector in the subspace spanned by the content representation vector. This point is the orthogonal projection component, representing the semantic component in the reference image representation that can be interpreted by the content text. The difference between the reference visual representation vector and this projection point is the residual component perpendicular to the content subspace. The system subtracts the orthogonal projection component from the reference visual representation vector to obtain the tonal residual representation vector.

[0038] Specifically, let the reference visual representation vector be... The content representation matrix is The tonal residual representation vector Calculated as follows: When The column vectors are linearly independent and When reversible, ;in The content representation vectors of each candidate text are arranged in columns; in In the case of reversibility, The orthogonal projection matrix that constitutes the content subspace, To reference the component of the visual representation vector along the content semantic direction, subtract this component. Perpendicular to the content subspace, the visual information retained is less associated with specific object categories and scene attributes, and mainly reflects tonal characteristics such as light and dark relationships, color tendencies, contrast distribution, highlight shapes, graininess, and reflective texture.

[0039] when When irreversible or numerically unstable, a content representation matrix is ​​used. Moore-Penrose pseudoreverse Calculate the orthogonal projection components, the orthogonal projection components are: The tonal residual representation vector is ,in express Moore-Penrose pseudo-reverse.

[0040] In the vector space of the image-text joint representation model, object categories and scene attributes typically occupy salient directions in the representation because noun tags appear frequently and have strong semantic discriminative power in the image-text pairs encountered by the model during the pre-training phase. Although light features, color tendencies, and texture attributes are also encoded in the representation vector, their directions do not completely coincide with the content directions. Therefore, these information can still be retained in the residuals after projection removal. It can be understood that content information and tonal information may have some correlation in the representation space. Projection removal is an approximate separation method. The subsequent matching score calculation further filters the residual content components through dual-condition screening. The two steps work together to improve the reliability of content exclusion.

[0041] Optionally, when the candidate content texts are semantically similar, causing the column vectors of the content representation matrix to be approximately linearly correlated, the Gram matrix may be numerically unstable. In this case, the above-mentioned pseudo-inverse method can be used to calculate the orthogonal projection, or redundancy removal processing can be performed on the candidate content texts with highly overlapping semantics before constructing the matrix.

[0042] Optionally, in some embodiments, when the main object occupies a large area of ​​the image, the whole image representation tends to be biased towards the category semantics of the main object. Even after projection removal, features related to the appearance of the main object may still remain in the residual. At the same time, the ambient light, dark hue and edge contour light information carried by the background area account for an insufficient proportion in the whole image representation and are not fully reflected. For example, in a product advertisement reference image, the car in the center of the image occupies most of the area. The representation vector generated by the image encoder for the whole image will reflect more the shape and color of the car, while the low saturation hue, fogging effect and edge contour light in the background are suppressed in the whole image representation.

[0043] Therefore, in an optional embodiment, the image of the reference visual data is divided into several regions, including at least a subject region and a background region. Specifically, a general saliency detection method can be used to locate the subject region, and the remaining parts can be classified as background regions. Optionally, highlight concentration regions and shadow concentration regions can be further distinguished based on brightness distribution. For film and television tonal values, these two types of regions often reflect the lighting style set by the photographer better than the category attributes of the subject itself. After extracting the image data of each region through cropping or masking, the system encodes the image data of each region using an image encoder to obtain the image representation vector of each region. Then, the orthogonal projection component of each region's image representation vector in the content subspace is calculated and subtracted to obtain the tonal residual representation vector of each region. The content subspace used for each region is the same as that used in the overall image processing to ensure consistent content exclusion criteria.

[0044] The cosine similarity between the image representation vector of each region and the content representation vector in the content subspace is calculated. The maximum value among the content representation vectors is taken as the semantic strength index of the content of that region: regions with higher cosine similarity indicate that their visual content is more closely associated with a specific object or scene category, and the tonal residual of this region may still contain subject appearance features, so they are assigned lower weights; regions with lower cosine similarity mainly present tonal information such as light, hue, and texture, and are assigned higher weights. After normalizing the weights of all regions, a weighted sum is performed to obtain the region-level tonal residual representation vector. The semantic intensity index of each region is first normalized to between 0 and 1. Then, 1 minus the normalized semantic intensity index is used as the initial weight of the region. The smaller the initial weight, the closer the region is to the content subspace. Subsequently, all initial weights are normalized and weighted summation is performed.

[0045] The regional tonal residual representation vector and the overall tonal residual representation vector are fused into a single tonal representation vector. The fusion is performed using a weighted average method, and the fusion ratio is determined based on the ratio of the area of ​​the subject region to the area of ​​the overall image in the reference visual data: the larger the proportion of the subject region area, the more significant the influence of the subject on the overall image representation, and the higher the fusion weight of the regional tonal residual representation vector; when the proportion of the subject region area is small, the overall tonal residual representation vector can already reflect the tonal characteristics relatively completely, and its fusion weight remains at a high level.

[0046] The ratio of the main subject area to the total image area is used as the fusion weight for the regional tonal residual representation vector. 1 minus this ratio is used as the fusion weight for the overall image tonal residual representation vector. For example, it can be configured to gradually increase the fusion weight of the regional tonal residual representation vector when the main subject area exceeds a preset proportion of the image area. This preset proportion remains constant throughout the same processing task. The fused tonal representation vector replaces the overall image tonal residual representation vector in subsequent matching score calculations. In implementations without regional-level processing, the tonal residual representation vector directly participates in subsequent steps.

[0047] In some embodiments, for step S4, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the candidate screening process for tonal attributes provided in an embodiment of the present disclosure. In S301, each tonal attribute text in the tonal attribute text set is encoded by the text encoder to obtain a tonal attribute representation vector.

[0048] The tonal residual representation vector or the fused tonal representation vector is a numerical vector in vector space and cannot be directly output as a visual cue word to the generative model. Therefore, this embodiment transforms it into readable text described using commonly used terminology in the fields of photography and film. At the same time, residual semantic content is further eliminated through dual-condition filtering. Subsequently, the filtered tonal attributes are reassigned to specific objects in the target image. It is understood that the generative model can be a general generative model, and this disclosure does not impose any limitations.

[0049] Specifically, the system pre-establishes a set of text describing tonal attributes. Each text in the set uses commonly used terms from the fields of film, photography, color grading, and advertising production to describe visual attributes that can be independent of the content of the image. Examples include low-key lighting, high-key lighting, soft key light, hard light edges, cool shadows, warm highlights, low saturation, strong contrast, shallow depth of field, film grain, metallic reflection texture, wet reflection, and foggy background. These terms describe visual attributes at the levels of light, color, and texture. In principle, they do not specify what kind of objects or scenes should appear in the image. The size and coverage of the set can be expanded according to the application field, and this disclosure does not impose any restrictions.

[0050] The system inputs each tone attribute text in the tone attribute text set into the text encoder to obtain each tone attribute representation vector. Each tone attribute representation vector is in the same shared representation space as the aforementioned tone residual representation vector and content representation vector, and the cosine similarity can be directly calculated.

[0051] In S302, a matching score is calculated based on the similarity between each of the tonal attribute representation vectors and the tonal residual representation vectors, as well as the similarity with the content subspace, and tonal attribute candidates are obtained by filtering according to the matching score.

[0052] Specifically, the calculation of the matching score includes, assuming the first... The tonal attribute representation vector of each tonal attribute text is: The tonal residual representation vector or tonal representation vector is The first content subspace Each content representation vector is Then the matching score for:

[0053] in This indicates the calculation of cosine similarity, the first term. This measure assesses the closeness between this tonal attribute and the tonal residual representation; a higher value indicates that the attribute closely matches the visual characteristics of the reference image after content removal. (Second item) The maximum cosine similarity between the tonal attribute representation vector and all content representation vectors in the content subspace is taken to measure the closeness of the attribute to the most closely related one among the excluded content.

[0054] When the text representation of a certain tonal attribute is simultaneously close to both the tonal residual and the content subspace in the vector space... This will result in a significant penalty, thus lowering their total score; among which This is a balancing coefficient that controls the intensity of suppression of tonal attributes in subspaces close to the content space; its value ranges from 0 to 1. The larger the value, the stronger the suppression of tonal attributes related to content semantics, and the more the filtering results favor purely tonal attributes; among them, The working value can be determined by conducting calibration experiments on combinations of several typical reference images and target content, selecting the value that minimizes the content contamination rate in the output prompt words, or it can be configured as a fixed preset value.

[0055] The system sorts texts by matching score from highest to lowest and selects several top-ranked texts as candidates for tonal attributes. The number of candidates can be preset; if no preset number is set, texts with matching scores not lower than a preset score threshold are selected. The preset number and preset score threshold remain constant throughout the same processing task. A high matching score means that two conditions are met simultaneously: the attribute is close to the tonal features of the reference image and has a large semantic distance from the excluded content. Through this filtering, attributes describing visual effects, such as wet reflections, bluish shadows, and localized color highlights, are retained. Phrases like "rainy night street" and "pedestrians with umbrellas," while related to the representation of the reference image, are excluded due to the larger penalty of the second condition. This dual-condition filtering, together with the aforementioned projection removal, forms a two-level content exclusion mechanism: projection removal performs coarse-grained separation at the representation level, while dual-condition filtering performs fine-grained filtering at the text matching level.

[0056] In some embodiments, for step S5, among the selected tonal attribute candidates, some attributes are attached to specific image elements in the reference image; these elements are the original carrier objects. For example, in wet road surface reflection, the original carrier object is the road surface, and the tonal attribute is wet reflection; in neon sign highlight, the original carrier object is the sign, and the tonal attribute is local color highlight. When there is no road surface or sign in the target image, if the prompt retains these original carrier objects, the generation model may generate them as objects in the image, causing reference content pollution. Conversely, if only the carrier objects are deleted, leaving only isolated tonal adjectives, the generation model lacks spatial orientation and cannot determine where and on which object the effect should appear in the image, reducing the controllability of the generation result.

[0057] The system identifies the original carrier object associated with each tonal attribute candidate, extracts the target object existing in the target image from the target content text, and then selects the replacement object according to the adaptation relationship between the physical presentation conditions of each tonal attribute candidate and the surface characteristics of each target object, replacing the existing original carrier object with the target object, and obtaining the replaced tonal description.

[0058] The selection criterion is the matching relationship between the surface conditions required for the physical presentation of the tonal attribute and the actual surface characteristics of the target object. For example, wet reflection requires a smooth surface with a certain reflectivity; in watch advertisements, this can be achieved with a metal case, glass crystal, or polished stone countertop, while in car advertisements, it can be achieved with the car's paint or a water film on the ground. Localized colored highlights require bright areas with clear edge transitions; in portrait images, this can be achieved with hair edge lighting or catchlights, while in product images, it can be achieved with the backlit area of ​​the product outline. Optionally, the system can calculate the cosine similarity between the representation vector of each tonal attribute and the representation vector obtained by the text encoder for each target object, assisting in automated selection.

[0059] For candidate tonal attributes that do not have an original object to replace and do not depend on a specific surface, such as low-saturation hues, strong contrast, or film grain—attributes that belong to the overall image—their original text is retained without replacement and directly incorporated into subsequent prompts. After object replacement, the original wet pavement reflection becomes a slender, soft reflection on a black stone countertop in the watch advertisement scene, and the outline light of the metal railing becomes a cool-toned outline light on the edge of the metal watch case. The replaced tonal description retains the visual characteristics of the reference image while directing the effect to a reasonable location and object in the target image.

[0060] In some embodiments, for step S6, visual cues are assembled based on the target content text and the replaced tonal description. During assembly, the visual cues are divided into target content segments, light segments, color segments, material segments, and constraint segments. Each segment has a different content source and responsibility, and they are generated separately and then assembled in sequence.

[0061] The target content segment is directly taken from the target content text, describing the subject and scene of the target image, and does not change with the reference image. The light and color segments are taken from the parts of the replaced tonal description corresponding to light and color, respectively: the light segment includes descriptions of the main light direction, contrast, and shadow detail, while the color segment includes descriptions of color temperature tendency, saturation level, and dominant color tone. The material segment is taken from the part of the replaced tonal description describing the texture of the target object, such as the slender, soft reflection of a black stone countertop and the cool-toned outline light on the edge of a metal case. The constraint segment is generated based on the candidate content text that constitutes the content subspace, listing elements excluded from the target image, such as avoiding the appearance of streets, pedestrians, and neon signs. In some embodiments, if the target generation model does not support negative cue words, the constraint segment can be rewritten as a positive environment constraint. The system still first determines the exclusion elements based on the candidate content text, and then rewrites the constraint meaning corresponding to the exclusion elements as a positive environment constraint that does not introduce the exclusion elements, such as a closed photography studio environment, a simple background, and the image focusing on the watch and the table. The segments are arranged in order in the final visual cue words, with the target content segment first, followed by the light segment, color segment, and material segment, and the constraint segment at the end. This arrangement allows the generation model to receive the target subject and scene instructions first, then the detailed description of the light and color aspects, and finally the constraint conditions.

[0062] In some embodiments, after assembling visual cues, the system performs representation verification to detect whether excluded content semantics have been re-introduced during the assembly process. The system encodes the target content text using a text encoder to obtain a target content representation vector, and encodes the visual cues using a text encoder to obtain cue word representation vectors. The system then calculates the cosine similarity between the cue word representation vector and the target content representation vector, the cosine similarity with the tonal residual representation vector, and the cosine similarity with the content subspace. The cosine similarity between the cue word representation vector and the content subspace is determined by the maximum cosine similarity between the cue word representation vector and each content representation vector in the content subspace, consistent with the content similarity caliber used in the matching score calculation.

[0063] The first two cosine similarities reflect the degree to which the visual cue words retain the target creative and reference tone, while the third reflects whether the semantics of the excluded content remain. The system determines whether the visual cue words need to be adjusted based on the three sets of cosine similarities. If adjustment is needed, the system only modifies the part of the visual cue words that originates from the tone description after the replacement, such as reselecting tone attribute words or strengthening constraint entries, while keeping the part that originates from the target content text unchanged, thereby avoiding changes to the user's original creative during the verification process.

[0064] In one implementation, the above determination is accomplished through a relative comparison of multiple candidates. Specifically, the system generates multiple candidate visual cue words. Among the candidates, the content originating from the target content text is the same, but the content originating from the replaced tonal description differs in wording and expression. The system calculates three sets of cosine similarities for each candidate. Through relative comparison among the candidates, the system selects the candidate with high cosine similarity to the target content representation vector and the tonal residual representation vector and low cosine similarity to the content subspace as the final output visual cue word. In this selection process, the candidates are relatively sorted within the current candidate set without setting a fixed similarity threshold. That is, the candidates are first sorted from low to high according to the cosine similarity to the content subspace, and the candidates with higher rankings are retained. Then, from the retained candidates, the candidate with the highest combined cosine similarity to the target content representation vector and the tonal residual representation vector is selected as the final output visual cue word.

[0065] Optionally, in one embodiment, when the reference visual data contains multiple representative frames, the system performs all processing from candidate content text extraction to tonal attribute candidate filtering on each representative frame to obtain the tonal attribute and its matching score for each frame. The system counts the frequency of each tonal attribute with the highest matching score in multiple representative frames. This frequency reflects the stability of the corresponding tonal attribute in the time dimension of the reference video. The system retains tonal attributes whose frequency in most representative frames meets a preset condition as final tonal attribute candidates for subsequent carrier object replacement. Here, most representative frames refer to frames with a number greater than half of the total number of representative frames. If the frequency of a certain tonal attribute in the representative frames with the highest matching score is greater than half of the total number of representative frames, it is determined that its frequency meets the preset condition. Tonal attributes that appear only in a few frames and are not significant in other frames are not included in the final candidate. For example, if the dark areas are cool and the low saturation features appear continuously in most representative frames, they are given priority. If a certain hue is significant only in one frame, its priority is reduced.

[0066] This method decomposes the representation of reference visual data into content semantic components and tonal residual components through content subspace construction and orthogonal projection removal. This ensures that subsequent processing is based solely on the tonal components, reducing the incorporation of inappropriate content such as objects, scenes, and props from the reference image into the target cue words from the source. Region-level processing and weighted summarization reduce the interference of the main area on tonal extraction, allowing for a more complete reflection of ambient light, dark hues, and rim light information in the background. Dual-condition filtering in tonal attribute matching further filters residual content semantics at the text matching level. Object replacement reassigns the extracted tonal attributes to reasonable objects in the target image, giving the generation model a clear spatial orientation. Cue word representation verification detects and corrects potentially re-infiltrated content components through re-encoding and similarity comparison of the final cue words. While maintaining the content of the target image, it transmits the tonal style of the reference image to the visual cue words in a way that the target scene can bear, reducing the contamination of the target image by the reference content and improving the accuracy of tonal reproduction and the stability of the generated target image.

[0067] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0068] Please see Figure 4 , Figure 4 This application provides a visual cue word generation device based on tonal style recognition and reconstruction. This device embodiment is similar to... Figure 1 Corresponding to the illustrated method embodiment, this device can be specifically applied to various electronic devices. The device specifically includes: Data receiving module 401 is used to receive target content text and reference visual data; The content subspace construction module 402 is used to encode the candidate content text extracted from the reference visual data into a content representation vector by a text encoder, and to construct a content subspace using the content representation vector. The tonal residual representation acquisition module 403 is used to encode the reference visual data through an image encoder to obtain a reference visual representation vector, calculate the orthogonal projection component of the reference visual representation vector on the content subspace, and subtract the orthogonal projection component from the reference visual representation vector to obtain the tonal residual representation vector. The tone attribute candidate filtering module 404 is used to encode each tone attribute text in the tone attribute text set through the text encoder to obtain tone attribute representation vectors, calculate matching scores based on the similarity between each tone attribute representation vector and the tone residual representation vector and the similarity with the content subspace, and filter tone attribute candidates according to the matching scores. The tone description generation module 405 is used to replace the original carrier object in each of the tone attribute candidates with the target object in the target content text to obtain the replaced tone description. The visual cue word construction module 406 is used to construct visual cue words based on the target content text and the replaced tonal description.

[0069] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0070] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0071] Figure 5 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The electronic device includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 502 or a program loaded from a storage section 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input section 506, an output section 507, a communication section 509, and an input / output (I / O) interface 505 are also connected to the bus 504.

[0072] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, it performs the functions defined in the methods of this application.

[0073] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0074] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of electronic devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0075] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating visual cue words based on tonal style recognition and reconstruction, characterized in that, include: Receive target content text and reference visual data; The candidate content text extracted from the reference visual data is encoded by a text encoder to obtain a content representation vector, and the content representation vector is used to construct a content subspace; The reference visual data is encoded by an image encoder to obtain a reference visual representation vector. The orthogonal projection component of the reference visual representation vector on the content subspace is calculated. The tonal residual representation vector is obtained by subtracting the orthogonal projection component from the reference visual representation vector. Each tone attribute text in the tone attribute text set is encoded by the text encoder to obtain a tone attribute representation vector. A matching score is calculated based on the similarity between each tone attribute representation vector and the tone residual representation vector, as well as the similarity with the content subspace. Candidate tone attributes are then selected based on the matching scores. Replace the original carrier object in each of the tonal attribute candidates with the target object in the target content text to obtain the replaced tonal description; Visual cue words are constructed based on the target content text and the replaced tonal description.

2. The visual cue word generation method based on tonal style recognition and reconstruction according to claim 1, characterized in that, The matching score is calculated by subtracting the cosine similarity between each of the tonal attribute representation vectors and the tonal residual representation vector, and the product of the balance coefficient and the maximum cosine similarity between each of the tonal attribute representation vectors and each content representation vector in the content subspace; the balance coefficient ranges from 0 to 1.

3. The method for generating visual cue words based on tonal style recognition and reconstruction according to claim 1, characterized in that, After obtaining the tonal residual representation vector, the method further includes: dividing the image of the reference visual data into several regions, wherein the regions include at least a subject region and a background region; encoding the image data of each region by the image encoder to obtain the image representation vector of each region; calculating and subtracting the orthogonal projection component of each region image representation vector on the content subspace to obtain the tonal residual representation vector of each region; weighting and summing the tonal residual representation vectors of each region to obtain a region-level tonal residual representation vector; fusing the region-level tonal residual representation vector with the tonal residual representation vector to form a tonal representation vector; and using the tonal representation vector to replace the tonal residual representation vector in the subsequent calculation of the matching score.

4. The method for generating visual cue words based on tonal style recognition and reconstruction according to claim 3, characterized in that, The fusion ratio of the tonal residual representation vector and the regional tonal residual representation vector is determined based on the ratio of the area of ​​the main subject region to the area of ​​the entire image in the reference visual data. The larger the proportion of the main subject region area, the higher the fusion weight of the regional tonal residual representation vector.

5. The method for generating visual cue words based on tonal style recognition and reconstruction according to claim 1, characterized in that, When the original carrier object is replaced with the target object, the replacement object is selected according to the adaptation relationship between the physical rendering conditions of each of the tonal attribute candidates and the surface characteristics of each of the target objects; for tonal attribute candidates that do not depend on a specific surface, their original text is retained and no replacement is performed.

6. The method for generating visual cue words based on tonal style recognition and reconstruction according to claim 1, characterized in that, When assembling the visual cues, the visual cues are divided into target content segment, light segment, color segment, material segment, and constraint segment for segment assembly; the content of the target content segment is taken from the target content text; the content of the light segment and the color segment are taken from the parts of the replaced tonal description that correspond to light and color, respectively; the content of the material segment is taken from the part of the replaced tonal description that describes the texture of the target object; the constraint segment is generated based on the candidate content text that constitutes the content subspace, listing the elements excluded from the target image.

7. The method for generating visual cue words based on tonal style recognition and reconstruction according to claim 1, characterized in that, After assembling the visual cue words, the method further includes: encoding the target content text using the text encoder to obtain a target content representation vector, and encoding the visual cue words using the text encoder to obtain cue word representation vectors; calculating the cosine similarity between the cue word representation vector and the target content representation vector, the cosine similarity with the tonal residual representation vector, and the cosine similarity with the content subspace; determining whether the visual cue words need adjustment based on the three sets of cosine similarities, and if adjustment is needed, modifying only the part of the visual cue words originating from the replaced tonal description, while keeping the part originating from the target content text unchanged.

8. The method for generating visual cue words based on tonal style recognition and reconstruction according to claim 1, characterized in that, Determining whether the visual cue words need adjustment based on three sets of cosine similarity includes: generating multiple candidate visual cue words, where the content originating from the target content text is the same but the content originating from the replaced tonal description is different; calculating the three sets of cosine similarity for each candidate, and selecting, through relative comparison between the candidates, the candidate with high cosine similarity to the target content representation vector and the tonal residual representation vector and low cosine similarity to the content subspace as the final output visual cue word.

9. The method for generating visual cue words based on tonal style recognition and reconstruction according to claim 1, characterized in that, When the reference visual data contains multiple representative frames, the process of extracting the candidate content text from the candidate tonal attribute candidate filtering is performed on each representative frame. The frequency of each tonal attribute ranking high in the matching score in the multiple representative frames is counted, and the tonal attribute whose frequency in most representative frames meets the preset condition is retained.

10. A visual cue word generation device based on tonal style recognition and reconstruction, characterized in that, include: The data receiving module is used to receive target content text and reference visual data; The content subspace construction module is used to encode the candidate content text extracted from the reference visual data into a content representation vector using a text encoder, and to construct a content subspace using the content representation vector. The tonal residual representation acquisition module is used to encode the reference visual data through an image encoder to obtain a reference visual representation vector, calculate the orthogonal projection component of the reference visual representation vector on the content subspace, and subtract the orthogonal projection component from the reference visual representation vector to obtain the tonal residual representation vector. The tone attribute candidate filtering module is used to encode each tone attribute text in the tone attribute text set through the text encoder to obtain tone attribute representation vectors, calculate matching scores based on the similarity between each tone attribute representation vector and the tone residual representation vector and the similarity with the content subspace, and filter tone attribute candidates according to the matching scores. The tone description generation module is used to replace the original carrier object in each of the tone attribute candidates with the target object in the target content text to obtain the replaced tone description. The visual cue word construction module is used to construct visual cue words based on the target content text and the replaced tonal description.

Images