Method and system for processing visual data

Abstract

Embodiments of the present specification provide a visual data processing method and system, the method comprising: performing information extraction on at least one target reference image in a target atmosphere dimension to obtain atmosphere information of the at least one target reference image in each target atmosphere dimension; obtaining a visual relationship set according to the atmosphere information of the at least one target reference image in each target atmosphere dimension; in a case where target visual data is generated based on the visual relationship set, inputting the visual relationship set and a target prompt text into a visual data generation model to generate candidate visual data; performing content verification on the candidate visual data based on the target prompt text, performing atmosphere verification on the candidate visual data based on the visual relationship set, and in a case where the content verification and the atmosphere verification are both passed, taking the candidate visual data as the target visual data; through the visual relationship set, the visual data generation model can clearly determine the visual relationship that needs to be migrated, and the controllability of generating the target visual data is improved.

Description

Technical Field

[0001] This specification relates to the field of visual processing technology, and in particular to a visual data processing method. One or more embodiments of this specification also relate to a visual data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Currently, in the field of visual data processing, users have a need to generate new images based on reference images. In this case, users usually require that there is some kind of relationship between the newly generated image and the reference image. For example, generating an image of a person in the reference image in a new scene, or generating an image of a new scene with a similar atmosphere based on the atmosphere of the reference image.

[0003] However, an image typically contains multiple types of information, such as the identity of the person, the content of the scene, and the atmosphere. When a user wants to generate a new image based on the atmosphere of a reference image, if the reference image is directly input into the visual data generation model, the model will have difficulty determining which information in the reference image belongs to the atmosphere information that needs to be transferred. This may result in the generated image failing to present an atmosphere similar to the reference image. Furthermore, due to the lack of clear and quantifiable atmosphere transfer standards, the generated image cannot be quantitatively evaluated before outputting it, which may result in the output image having an unsatisfactory atmosphere transfer effect, thus affecting the user experience. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a visual data processing method. One or more embodiments of this specification also relate to a visual data processing system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a visual data processing method is provided, comprising: In the target atmosphere dimension, information is extracted from at least one target reference image to obtain the atmosphere information of the at least one target reference image in each target atmosphere dimension; Based on the atmospheric information of the at least one target reference image in each target atmospheric dimension, a set of visual relationships is obtained; The set of visual relationships and the target prompt text are input into the visual data generation model to generate candidate visual data. The candidate visual data is subjected to content verification based on the target prompt text, and the candidate visual data is subjected to atmosphere verification based on the visual relationship set in the target atmosphere dimension. If both the content verification and the atmosphere verification are passed, the candidate visual data is taken as the target visual data.

[0006] According to a second aspect of the embodiments of this specification, a visual data processing system is provided, including a client and a server, wherein... The client is used to send at least one target reference image to the server; The server is used to extract information from at least one target reference image in the target atmosphere dimension to obtain atmosphere information of the at least one target reference image in each target atmosphere dimension; and to obtain a set of visual relationships based on the atmosphere information of the at least one target reference image in each target atmosphere dimension. The client is further configured to send a visual data generation request to the server, wherein the visual data generation request carries a target set identifier corresponding to the visual relationship set; The server is further configured to invoke the visual relationship set, input the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data; perform content verification on the candidate visual data based on the target prompt text, and perform atmosphere verification on the candidate visual data based on the visual relationship set in the target atmosphere dimension; if both the content verification and the atmosphere verification pass, the candidate visual data is used as the target visual data and the target visual data is returned to the client.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described visual data processing method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual data processing method.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual data processing method.

[0010] This specification provides a visual data processing method in one embodiment. In the target atmosphere dimension, information extraction is performed on at least one target reference image to obtain atmosphere information of the at least one target reference image in each target atmosphere dimension. Based on the atmosphere information of each target reference image in each target atmosphere dimension, a visual relationship set is obtained. Since the visual relationship set is obtained by extracting atmosphere information in each target atmosphere dimension, it includes the atmosphere information corresponding to each target atmosphere dimension. Therefore, when the visual relationship set is input into a visual data generation model, the visual data generation model can clearly define the newly generated visual relationship based on the visual relationship set. The visual data needs to include atmospheric information. The visual relationship set and target prompt text are input into the visual data generation model to generate candidate visual data. To ensure the generation effect, the candidate visual data is content-verified based on the target prompt text, and atmosphere-verified based on the visual relationship set in the target atmosphere dimension. If both the content verification and the atmosphere verification pass, the candidate visual data is used as the target visual data. Through content verification and atmosphere verification, quantifiable acceptance criteria are provided for the generation results. Thus, without the need for manual comparison of the target reference image and the generated image, the controllability and acceptance efficiency of generating new images based on the target reference image are significantly improved. Attached Figure Description

[0011] Figure 1 This is a scene diagram illustrating a visual data processing method provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a visual data processing method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating the processing procedure of a visual data processing method provided in one embodiment of this specification. Figure 4 This is a schematic diagram of the structure of a visual data processing device provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0016] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0017] Reference image set: One or more images uploaded by the user to constrain the image to be generated.

[0018] Control Roles: Referencing the image's role in controlling the generation task, including identity, atmosphere, composition, content, or blending roles. Identity defines a specific person, animal, or other subject. Atmosphere provides color, lighting, optics, weather, movement, texture, mood, or artistic design relationships. Composition provides subject scale, image position, negative space, horizon, depth channels, or occlusion levels. Content specifies the people, objects, places, clothing, actions, or events that should appear in the target visual data. Blending contains more than one type of effective control information and needs to be separated and used separately.

[0019] Atmosphere profile: A set of structured visual relationships extracted from reference image sets and decoupled from specific content.

[0020] Consensus information: Visual features that are repeated or consistent in relationships across multiple reference images.

[0021] Range information: Although there are differences between different reference images, they still fall within the acceptable range of variation of the same visual system.

[0022] Conflict information: Different reference diagrams contain structural contradictions that cannot be satisfied simultaneously on the same dimension.

[0023] Excluded content: Occasional people, clothing, objects, text, signs or specific locations that may not be transferred from the reference drawing to the new drawing.

[0024] Visual relationships: the interaction between color, light, objects, lens, material, air, movement, and space, rather than a single style label.

[0025] Style control anchors: control information obtained from atmospheric profile compression that can be received by the visual data generation model.

[0026] Subject-environment coupling: The subject and the environment follow the same physical imaging relationship in terms of light, air, contact, focus, texture, motion, perspective, scale, and edges.

[0027] Targeted retry: Only modify the control channels that fail verification, while keeping the content, identity, composition, or style of the channels that have passed verification unchanged.

[0028] In the custom style section of the video platform, the existing reference image style generation directly inputs the entire image or mixed features into the model, which can easily lead to the mistransfer of people, clothing, objects, text and specific locations, and cause style averaging, identity drift and human-scene separation when multiple images conflict.

[0029] When a user uploads one or more reference images and controls the new image through image cues, style references, adapters, fine-tuning, or multimodal cues, a single reference image typically contains multiple signals, including the identity of the person, content, composition, and atmosphere. If the entire image is directly input, the model may not be able to determine whether the user wants to transfer color tone and lighting, or whether they want to transfer people, clothing, buildings, objects, or composition.

[0030] This specification provides a visual data processing method that can decouple information in a reference image before generating an image.

[0031] This specification provides a visual data processing method, and also relates to a visual data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0032] See Figure 1 , Figure 1A schematic diagram of a scene illustrating a visual data processing method provided in one embodiment of this specification is shown.

[0033] Specifically, the visual data processing method is implemented using an end-side device 102 and a server 104. The end-side device 102 is used to send a visual data generation request to the server 104, and the visual data generation request includes at least one target reference image.

[0034] Server 104 extracts information from at least one target reference image in the target atmosphere dimension to obtain atmosphere information of the at least one target reference image in each target atmosphere dimension; obtains a visual relationship set based on the atmosphere information of the at least one target reference image in each target atmosphere dimension; inputs the visual relationship set and target prompt text into a visual data generation model to generate candidate visual data; performs content verification on the candidate visual data based on the target prompt text, and performs atmosphere verification on the candidate visual data based on the visual relationship set in the target atmosphere dimension; if both the content verification and the atmosphere verification pass, the candidate visual data is used as the target visual data and returned to the end device 102.

[0035] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0036] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0037] It is worth noting that the visual data processing method provided in the embodiments of this specification can be executed by the server 104. In other embodiments of this specification, the visual data processing model can be deployed in the edge device 102, so that the edge device 102 can also have similar functions to the server 104, thereby executing the visual data processing method provided in the embodiments of this specification. In other embodiments, the visual data processing method provided in the embodiments of this specification can also be jointly executed by the edge device 102 and the server 104.

[0038] See Figure 2 , Figure 2 A flowchart of a visual data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0039] Step 202: In the target atmosphere dimension, extract information from at least one target reference image to obtain the atmosphere information of the at least one target reference image in each target atmosphere dimension.

[0040] Among them, the target atmosphere dimension can be understood as the observation direction used to describe the transferable visual relationship in the reference image. It is a specific dimension item in the image decomposition framework. In this embodiment, when decomposing the image, the specific dimension items include color, lighting, composition, optics, atmosphere, motion, material, emotion, and art design.

[0041] A target reference image can be understood as a reference image uploaded or specified by the user in the current task to provide atmosphere information. There can be one or more target reference images, and a single target reference image can simultaneously assume multiple control roles such as identity, atmosphere, composition, content, or blending.

[0042] A control role can be understood as the control purpose that an image undertakes for the current task. When the control role of an image is identity, the image provides identity information for a specific person, animal, or other subject; that is, the image provides identity control information. When the control role of an image is atmosphere, the image provides atmospheric information such as color, lighting, optics, weather, motion, material, mood, or artistic design relationships. In this embodiment, the control role of the target reference image is atmosphere. When the control role of an image object is composition, the image provides compositional information such as subject scale, image position, negative space, horizon, depth channels, or occlusion levels. When the control role of an image object is content, the image provides content information such as people, objects, places, clothing, actions, or events that should appear in the target visual data. When the control role of an image object is mixed, the image can provide at least two of the control information: identity, atmosphere, composition, and content.

[0043] Information extraction can be understood as examining the target reference image item by item according to the target atmosphere dimension, identifying the stable visual relationships, the range of permissible changes, conflicting information that cannot be satisfied simultaneously, and accidental content that cannot be transferred. Atmosphere information can be understood as a set of structured visual relationships decoupled from specific content obtained after information extraction. It can be further organized into consensus information, range information, conflicting information, excluded content, style control anchors, and dimension-level confidence, etc. Dimension-level confidence is used to indicate the reliability of the judgment on a certain dimension and can be marked as high, medium, or low.

[0044] Specifically, in scenarios where the visual atmosphere of at least one target reference image needs to be structurally extracted, information is extracted from at least one target reference image in the target atmosphere dimension to obtain the atmosphere information of each target reference image in each target atmosphere dimension. In this way, the atmosphere features that were originally mixed in the whole reference image are decomposed into control signals that can be observed separately, compared, and transmitted, so as to avoid people, clothing, objects, text or specific locations being mistakenly transferred along with the atmosphere.

[0045] In practice, this embodiment does not aim to copy the original content of the target reference image. Instead, it requires extracting transferable visual relationships from the target reference image across each target atmosphere dimension. For a single target reference image, information extraction can obtain atmosphere information across multiple target atmosphere dimensions. For two or more target reference images, information extraction can further compare the visual relationships between the images within the same target atmosphere dimension framework. By obtaining the atmosphere information of each target reference image across each target atmosphere dimension, during subsequent image generation, style control anchors can be requested based on this atmosphere information to generate images with corresponding atmospheres. This avoids directly inputting the entire reference image or blended features into the model, which could lead to the generated image mistakenly transferring other features from the reference image.

[0046] For example, a user uploads three reference images of a modern urban rainy night scene and requests the generation of a rainy night chase scene. When extracting information along the target atmosphere dimension, the color dimension can extract relationships such as the main color family of cool blue, shadows with a blue tint, controlled highlights, and a small amount of warm-colored real-scene lighting; the lighting dimension can extract the direction of the main light, its softness and hardness, its fill ratio, and the behavior of the rim light; the weather dimension can extract rain, fog, wetland reflection, and particle distance behavior; the motion dimension can extract the main direction of motion, the contrast between movement and stillness, and camera energy; the material dimension can extract humidity, fabric reflection, specular reflection, and graininess; and the emotion dimension can extract relationships such as tension, restraint, or loneliness.

[0047] If the same actor, the same raincoat, or the same station appears in all three reference images, then these faces, clothing, and precise locations should be excluded rather than being transferred as atmospheric information.

[0048] In the presence of a single target reference image, information extraction can also generate corresponding atmosphere information across multiple target atmosphere dimensions. However, multiple target reference images can provide a more reliable basis for comparison of a common atmosphere, while a single target reference image relies more on the confidence level label of the image itself across each target atmosphere dimension.

[0049] By extracting information from at least one target reference image in the target atmosphere dimension, the atmosphere information of each target reference image in each target atmosphere dimension is obtained, so that the atmosphere control information in the target reference image is transformed from a mixed input of the whole image into a structured relationship separated by dimension.

[0050] In one or more embodiments of this specification, before extracting information from at least one target reference image in the target atmosphere dimension, at least one initial reference image and a reference text description are first obtained. The control type corresponding to each initial reference image is determined based on the reference text description. Then, the initial reference image with the control type of atmosphere is determined as the target reference image. This completes the classification and isolation of reference roles before information extraction, preventing reference images used for atmosphere from being mistakenly identified as authoritative figures, compositional basis, or content sources. Specific implementation methods are described below: Before extracting information from at least one target reference image in the target atmosphere dimension to obtain the atmosphere information of the at least one target reference image in each target atmosphere dimension, the method further includes: Obtain at least one initial reference image and a reference text description; Based on the reference text description, determine the control type corresponding to each initial reference image; The initial reference image with the control type of "atmosphere" is determined as the target reference image.

[0051] The initial reference image can be understood as a reference image uploaded or specified by the user that has not yet been classified as a control role. A single initial reference image can have one or more control roles. The reference text description can be understood as a text description provided with the initial reference image, which is used to express the control purpose that a single reference image is intended to play in this task, such as explaining that a certain image is used to refer to the atmosphere, a certain image is used to specify a person, or a certain image is used to specify a location or clothing.

[0052] The control type can be understood as the control purpose that the initial reference image undertakes in the generation task, including identity type, atmosphere type, composition type, content type, or mixed type. This control type is the control role in the above embodiments. The atmosphere type can be understood as the initial reference image being designated to provide control over color, lighting, optics, weather, movement, material, mood, or artistic design relationships, without assigning the responsibility of specifying character identity, specific clothing, specific objects, or precise locations. An image belonging to the atmosphere type means that the control role corresponding to that image is atmosphere. The target reference image is the initial reference image whose control type is determined to be atmosphere type; it is the image of the atmosphere role and is the object used for extracting target atmosphere information.

[0053] It should be noted that the target reference image is an image of the character in the scene. Therefore, even if a clear human face appears in the target reference image, the character's identity will not be automatically obtained, that is, the identity information contained in the target reference image will not be obtained. Similarly, the initial reference image used to provide identity information will not extract content information such as clothing and background in the scene.

[0054] Specifically, the system receives initial reference images and reference text descriptions, and then determines the control role of each initial reference image in this task based on the reference text descriptions. If the reference text description indicates that an initial reference image is used to provide visual relationships such as color, light, weather, material, or emotion, then the control type of that initial reference image is determined to be the atmosphere type, and it is selected as the target reference image for subsequent information extraction in the target atmosphere dimension.

[0055] If the reference text description indicates that an initial reference image is used to specify the appearance of a person, then the initial reference image belongs to the identity type and no information extraction is performed on the atmosphere dimension. If an initial reference image contains multiple valid control information, it belongs to the mixed type and needs to be separated item by item, rather than inherited as a whole. Item-by-item separation can be understood as determining the specific types of control information contained in the initial reference image and extracting the corresponding control information from the initial reference image separately. For example, if the initial reference image is used to provide both atmosphere and content information, it can be determined as an atmosphere-type image, thus extracting information on the target atmosphere dimension to obtain the atmosphere information. Then, the initial reference image can also be determined as a content-type image, thus extracting content information such as people, objects, places, clothing, actions, or events from the initial reference image.

[0056] By classifying control types, the control type of each initial reference image is clarified, and information corresponding to the control type is extracted from each initial reference image. In this embodiment, the initial reference image of the atmosphere type is the target reference image, which is used to extract the atmosphere information contained therein.

[0057] For example, a user uploads three initial reference images along with a text description: "The first image references the atmosphere of a rainy night, the second is character A's face, and the third is the location of this train station." When determining the control type based on the text description, the first initial reference image is classified as "Atmosphere" and is identified as the target reference image. Information will then be extracted from this target reference image along the target atmosphere dimension to obtain atmosphere information. The second initial reference image is classified as "Identity" and is used to analyze character A's appearance; it is not involved in atmosphere dimension extraction. The third initial reference image is classified as "Content" and is used to specify the location as the train station; it is also not involved in atmosphere dimension extraction.

[0058] If a user uploads an initial reference image and describes it as "referring to the cool blue tones, wetland reflections, and tense mood of this image," then the control type for that initial reference image is atmosphere type. Even if a clear human face appears in the image, that face is still excluded from content that cannot be transferred, and only atmosphere information such as color, lighting, material, and mood is extracted.

[0059] The data processing method provided in the embodiments of this specification obtains an initial reference image and a reference text description, determines the control type corresponding to each initial reference image, and determines the initial reference image with the control type of atmosphere as the target reference image. This limits the information extraction objects in the subsequent target atmosphere dimension to the target reference image, thereby avoiding the mislocation of people, clothing, objects, text, logos or precise locations in the target reference image.

[0060] In one or more embodiments of this specification, after determining the control type corresponding to each initial reference image based on the reference text description, control information corresponding to the control type is extracted from each initial reference image according to the information extraction rules corresponding to each control type. Specific implementation methods are as follows: After determining the control type corresponding to each initial reference image based on the reference text description, the process further includes: The information extraction rules corresponding to each control type are determined, and the control information of the corresponding control type is extracted from each initial reference image according to the information extraction rules. The initial reference image of the atmosphere type is used to provide atmosphere information.

[0061] Among them, the information extraction rules can be understood as the extraction method for each type of control, which is pre-set to obtain the control information corresponding to that control type from the initial reference image; the control information can be understood as the control basis for constraining the generation of target visual data, which is extracted from the initial reference image according to the information extraction rules.

[0062] Atmosphere information can be understood as visual relationship content extracted from the initial reference image of the atmosphere type, used to construct a set of visual relationships, which is the atmosphere profile.

[0063] In specific implementation, when the control type is identity type, identity information is extracted from the corresponding initial reference image, and the identity information includes the appearance features and / or body proportion features of the target object; when the control type is composition type, composition information is extracted from the corresponding initial reference image, and the composition information includes the position, scale, horizon position and / or depth relationship of the target object in the picture; when the control type is content type, content information is extracted from the corresponding initial reference image, and the content information includes scene elements, objects, clothing and / or action postures.

[0064] Among them, identity information can be understood as the control basis used to define the identity of a specific person, animal or other subject, including the appearance characteristics and / or body proportion characteristics of the target object; composition information can be understood as the control basis used to define the position, scale, horizon position and / or depth relationship of the target object in the picture, and the target object can be understood as a specific person, animal or other subject in the initial reference image that serves the purpose of identity control; content information can be understood as the reference content used to specify the people, objects, places, clothing, actions or events that should appear in the target visual data, including scene elements, objects, clothing and / or action postures.

[0065] In this embodiment, after the control type is divided, an independent information extraction channel is further established for each type of control role, so that control information such as identity, atmosphere, composition and content are extracted from the corresponding images according to rules, avoiding mutual contamination of information from different control channels during the extraction stage.

[0066] Specifically, after the control type is determined, control information corresponding to the control type is extracted from each initial reference image according to the correspondence between the control type and the information extraction rules. It should be noted that for initial reference images of mixed types, the various types of control information contained therein are first broken down item by item, and then the corresponding control information is extracted according to the information extraction rules of identity type, atmosphere type, composition type, and content type. Among them, the atmosphere information extracted from the initial reference image of atmosphere type is used to construct a visual relationship set, i.e., atmosphere profile, while identity information, composition information, and content information enter their respective control channels and do not participate in the construction of atmosphere profile.

[0067] Control information corresponding to each control type is extracted separately. Subsequent generation requests can organize channels such as identity, body proportion, atmosphere, and human-scene fusion according to generation requirements, instead of directly inputting the mixed features of the entire reference image into the model. That is, this application can not only decouple atmosphere information from reference images, but also determine the control type of each reference image according to needs, and then extract the control information of the corresponding control type from the reference images, thereby achieving the decoupling of different types of information such as identity, composition, and scene content.

[0068] For example, a user uploads three initial reference images along with a reference text description: "The first image references the atmosphere of a rainy night, the second is the face of character A, and the third is the location of this train station." After determining the control type based on the reference text description, the control type of the first initial reference image is the atmosphere type. According to the information extraction rules corresponding to the atmosphere type, information is extracted in multiple target atmosphere dimensions to obtain the dimensional information of each target atmosphere dimension as atmosphere information. This atmosphere information is used to construct a set of visual relationships.

[0069] The second initial reference image is controlled by the identity type. According to the information extraction rules corresponding to the identity type, the appearance features and / or body proportion features of character A are extracted from it as identity information. The third initial reference image is controlled by the content type. According to the information extraction rules corresponding to the content type, the station scene element is extracted from it as content information.

[0070] The data processing method provided in the embodiments of this specification determines the information extraction rules corresponding to each control type and extracts the control information of the corresponding control type from each initial reference image according to the information extraction rules, so that the identity information, atmosphere information, composition information and content information are separated into their respective control channels during the extraction stage.

[0071] Step 204: Obtain a set of visual relationships based on the atmosphere information of the at least one target reference image in each target atmosphere dimension.

[0072] The visual relationship set can be understood as a structured visual relationship set that is decoupled from the specific content, obtained by integrating the atmospheric information of each target reference image in each target atmosphere dimension. This visual relationship can be understood as the interaction between color, light, objects, lens, material, air, movement and space, rather than a single style label.

[0073] Specifically, after obtaining the atmosphere information of each target reference image in each target atmosphere dimension, the atmosphere information of each target reference image is summarized, and then the relationship between each target reference image is compared in each target atmosphere dimension to distinguish consensus information, scope information, conflict information and excluded content, and further compressed to obtain style control anchor points, while marking the dimension-level confidence of each target atmosphere dimension.

[0074] Consensus information can be understood as visual features that recur or are stable and consistent in relationships across multiple target reference images; range information can be understood as acceptable variation ranges that, while differing between different target reference images, still belong to the same visual system; conflict information can be understood as structural contradictions in different target reference images that cannot be simultaneously satisfied in the same target atmosphere dimension; excluded content can be understood as occasional people, clothing, objects, text, signs, or specific locations that must not be transferred from target reference images to target visual data; style control anchors can be understood as control information obtained by compressing the set of visual relationships and that can be received by the visual data generation model; dimensional confidence can be understood as a label for judging the reliability of each target atmosphere dimension, which can be labeled as high, medium, or low.

[0075] In practical applications, when there is only one target reference image, the visual relationship set is mainly formed based on the atmospheric information of the target reference image in each target atmosphere dimension, and the reliability of the judgment in each dimension is reflected by the dimension-level confidence. When there are two or more target reference images, the visual relationship set can establish more reliable consensus information based on the common relationship of multiple target reference images in the same target atmosphere dimension, and classify the differences between them into range information or conflict information.

[0076] The set of visual relationships obtained in this embodiment includes the interactions between color, lighting, composition, optics, atmosphere, movement, material, emotion, and art design. For example, it can transfer relationships such as a cool blue environment with only a few warm-colored real-world lights, reduced reflections in wetlands, and soft, protective highlights on faces, but it will not automatically copy the characters, scenes, or other content contained in the reference image.

[0077] In practice, a set of visual relationships is obtained based on the atmosphere information of each target reference image in each target atmosphere dimension. This integrates the dimensional information scattered in each target reference image and each target atmosphere dimension into a structured set of visual relationships decoupled from the specific content, so that atmosphere control is elevated from raw observation of each image and each dimension to a unified atmosphere basis that can be used to generate requests.

[0078] By obtaining a set of visual relationships based on the atmospheric information of each target reference image in each target atmospheric dimension, the scattered dimensional information is integrated into a set of visual relationships, thereby forming a reusable, transferable atmospheric profile that is decoupled from specific content.

[0079] In one or more embodiments of this specification, when obtaining a visual relationship set based on the atmospheric information of each target reference image in each target atmospheric dimension, a cross-image comparative analysis is specifically performed on the atmospheric information of each target reference image in each target atmospheric dimension to obtain the analysis results. A visual relationship set is then generated based on the analysis results. This process identifies stable relationships, acceptable differences, structural contradictions, and non-transferable content among multiple target reference images in the same target atmospheric dimension. This ensures that the visual relationship set is not a simple averaging of multiple reference images, but rather a structured integration result based on relationship comparison. Specific implementation methods are described below: The step of obtaining a visual relationship set based on the atmospheric information of the at least one target reference image in each target atmospheric dimension includes: When the at least one target reference image is multiple target reference images, cross-image comparative analysis is performed on the atmosphere information of each target reference image in each target atmosphere dimension to obtain analysis results, wherein the analysis results include at least one of consensus information, range information, conflict information and exclusion content; The set of visual relationships is generated based on the analysis results.

[0080] Cross-image comparative analysis can be understood as an observational process that compares the atmospheric information extracted from each target reference image dimension by dimension within the same target atmosphere dimension framework. The comparison objects are whether the visual relationships of each target reference image are consistent in the same target atmosphere dimension, whether they belong to the same visual system, and whether they cannot be satisfied simultaneously. The analysis results can be understood as structured comparative conclusions obtained after cross-image comparative analysis, used to generate a set of visual relationships.

[0081] Consensus information can be understood as visual features that recur or are stable and consistent in relationship among various target reference images; range information can be understood as information within an acceptable range of variation that, although there are differences among different target reference images, still belong to the same visual system; conflict information can be understood as structural contradictions that cannot be simultaneously satisfied in different target reference images on the same target atmosphere dimension; excluded content can be understood as occasional people, clothing, objects, text, signs, or specific locations that must not be transferred from the target reference images to the target visual data.

[0082] Specifically, when performing cross-image comparative analysis of the atmosphere information of each target reference image in each target atmosphere dimension, the atmosphere information of each target reference image in the same target atmosphere dimension is compared dimension by dimension. The visual relationships that are stably shared by multiple target reference images are classified as consensus information, the differences that belong to the same visual system but are allowed to vary are classified as range information, the structural contradictions that cannot be safely averaged or cannot be satisfied simultaneously are classified as conflict information, and faces, clothing, objects, text, signs and precise locations that cannot be migrated are classified as exclusion content. Thus, the analysis results containing consensus information, range information, conflict information and exclusion content are obtained.

[0083] After obtaining the analysis results, a visual relationship set is generated based on the analysis results. Specifically, based on consensus information, the visual relationships in the visual relationship set that should be stably migrated are determined; based on range information, the ranges in the visual relationship set that are allowed to change are determined; based on conflict information, the parts in the visual relationship set that need to be avoided or require user adjudication are determined; based on excluded content, the boundaries of content in the visual relationship set that cannot be migrated are determined; and style control anchors (control information obtained by compressing the visual relationship set and that can be received by the visual data generation model) can be further compressed to obtain style control anchors.

[0084] It should be noted that when there is only one target reference image, cross-image comparison analysis degenerates into judging the confidence level of the atmosphere information of the target reference image in each target atmosphere dimension. If the confidence level of the atmosphere information in a certain target atmosphere dimension exceeds the first confidence threshold (e.g., 0.8), the atmosphere information is classified as consensus information. If the confidence level of the atmosphere information in a certain target atmosphere dimension is between the first confidence threshold (e.g., 0.8) and the second confidence threshold (e.g., 0.5), it is classified as range information. The parts that cannot be satisfied simultaneously with other known constraints are classified as conflict information. Among them, faces, clothing, objects, text, signs and precise locations are classified as exclusion content.

[0085] For example, a user uploads three reference images of a modern urban rainy night scene and requests the generation of a rainy night chase scene. When performing a cross-image comparative analysis of the three reference images' atmospheric information across dimensions such as color and tone, lighting, weather and atmosphere, material and texture, and mood, if the three images are consistent in terms of cool blue dominant color family, bluish shadows, controlled highlights, rain and fog density, and wetland reflection, these visual relationships are categorized as consensus information; if the three images differ in the direction of the main light and the scale of composition but belong to the same visual system, these differences are categorized as range information.

[0086] If one image uses hard top lighting and another uses soft side-backlighting and cannot be safely averaged, this difference in lighting dimension is classified as conflicting information; while if the same actor, the same raincoat, or the same station appears in all three images, these faces, clothing, and precise locations are classified as excluded content.

[0087] When generating a set of visual relationships based on the analysis results, consensus information becomes the core visual relationships that should be stably transferred in the set of visual relationships, range information becomes the range that is allowed to change, conflict information becomes the parts that need to be avoided or need to be decided by the user, and excluded content becomes the content boundaries that cannot be transferred. These can be compressed into style control anchor points for use by the generation model, such as a cool blue environment, a small number of warm-colored real-scene lights, wetland reflections being suppressed, facial highlights being softly protected, rain and fog having near and far layers, and emotions being tense and restrained.

[0088] In practical applications, when conflicting information in the analysis results cannot be safely averaged, the visual relationship set does not forcibly average the conflicting information, but retains the conflicting information for the user to decide or process according to a fixed priority.

[0089] The data processing method provided in the embodiments of this specification obtains analysis results including consensus information, range information, conflict information, and excluded content by performing cross-image comparative analysis on the atmosphere information of each target reference image in each target atmosphere dimension. Based on the analysis results, a visual relationship set is generated, which is based on cross-image relationship comparison rather than simple averaging. This accurately distinguishes stable consensus, permissible changes, structural conflicts, and non-migratable content, providing a reusable, interpretable, and versionable atmosphere control basis for subsequent request generation.

[0090] Step 206: Input the set of visual relationships and the target prompt text into the visual data generation model to generate candidate visual data.

[0091] The target prompt text can be understood as a textual description organized according to the user's request content, used to indicate the subject, action, event, location, shot, etc. that should appear in the visual data to be generated; the visual data generation model can be understood as a generation service or generation model used to render images; and the candidate visual data can be understood as the unverified generation result generated by the visual data generation model based on the set of visual relationships and the target prompt text.

[0092] Specifically, when generating target visual data based on a set of visual relationships, the set of visual relationships and the target prompt text are input into the visual data generation model. In this case, the set of visual relationships is used to provide atmosphere control signals, and the target prompt text is used to provide the content requested by the user.

[0093] The visual data generation model determines the subject, action, event, location, shot, etc. that should appear based on the target prompt text, and determines the visual relationships in the target atmosphere dimensions such as color and tone, lighting, composition, optics and depth of field, weather and air, motion, material and texture, mood tone, and art and production design based on the set of visual relationships, thereby rendering candidate visual data.

[0094] In this embodiment, the visual relationship set and target prompt text are input into the visual data generation model, thereby achieving precise guidance of the generation process in the atmosphere channel through the separated atmosphere control signal, instead of directly inputting the entire reference image or mixed features into the model.

[0095] Using the previous example, a user uploads three reference images of a modern urban rainy night scene and requests the generation of a rainy night chase scene, using a registered character. The visual relationship set includes core visual relationships such as a cool blue environment, a small amount of warm-colored ambient lighting, suppressed reflections from wetlands, softly protected facial highlights, varying depths of rain and fog, and a tense yet restrained emotional state, as well as excluded content.

[0096] When the visual relationship set and target cue text are input into the visual data generation model, the target cue text provides the subject, action, event, location, and shot of the rainy night chase scene, while the visual relationship set provides atmosphere control signals such as cool blue tone, lighting relationship, rain and fog density, wetland reflection, material texture, and mood tone. The visual data generation model generates candidate visual data based on these signals.

[0097] By generating candidate visual data by inputting visual relationship sets and target prompt text into the visual data generation model, the atmosphere control signals and user request content come from their respective control channels before being input into the generation model. This avoids the atmosphere reference from contaminating character identity, body proportions, scene, etc. during the generation stage.

[0098] In one or more embodiments of this specification, when a visual relationship set is obtained, the visual relationship set is saved and a corresponding target set identifier is generated. This allows the visual relationship set to be invoked upon receiving a visual data generation request carrying the target set identifier, and the target visual data to be generated based on the visual relationship set. This avoids re-extracting the atmosphere dimension from the target reference image each time a data generation request is received, thus avoiding the need to do so every time. Specific implementation methods are described below: In the case of generating target visual data based on the visual relationship set, before inputting the visual relationship set and the generated prompt text into the visual data generation model to generate candidate visual data, the method further includes: Save the visual relationship set and generate the target set identifier corresponding to the visual relationship set; The step of inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data includes: Upon receiving a visual data generation request, and if the visual data generation request carries the target set identifier, the visual relationship set is invoked, and the visual relationship set and the target prompt text are input into the visual data generation model to generate candidate visual data.

[0099] The target set identifier can be understood as identification information used to uniquely identify a saved set of visual relationships. This identifier can be used to locate and call the corresponding set of visual relationships in subsequent visual data generation requests. The visual data generation request can be understood as a request message used to trigger the generation of target visual data. This request message may carry the target set identifier to indicate that the saved set of visual relationships should be used in this generation. Calling can be understood as reading the saved set of visual relationships based on the target set identifier carried in the visual data generation request and using it as the basis for atmosphere control in the process of generating target visual data.

[0100] Specifically, upon obtaining a set of visual relationships, the set can be saved, and a target set identifier corresponding to that set can be generated, establishing a correspondence between the visual relationship set and the target set identifier. The saved visual relationship set can contain consensus information, scope information, conflict information, excluded content, style control anchors, and dimensional confidence levels for each target atmosphere dimension, and can also record fields such as the source target reference image and generation time of the visual relationship set.

[0101] When a visual data generation request is received, and the request carries a target set identifier, the saved visual relationship set is called according to the target set identifier to generate target visual data based on the visual relationship set, without needing to re-extract information in the target atmosphere dimension from each target reference image.

[0102] By saving the set of visual relationships, the set of visual relationships can be reused multiple times. The same atmosphere image can be used for different subjects, different scenes or different batches of generation tasks, while maintaining the consistency and traceability of the atmosphere control basis.

[0103] For example, a user uploads three target reference images in the style of a modern urban rainy night and requests the generation of a rainy night chase scene. After obtaining a set of visual relationships from the three target reference images, the set of visual relationships can be saved and a corresponding target set identifier can be generated. For example, the set of visual relationships can be saved as an atmosphere portrait record and an identifier can be assigned.

[0104] If the user then wants to generate another modern urban rainy night confrontation scene with the same atmosphere, the target set identifier is included in the visual data generation request. The system calls the saved visual relationship set based on the target set identifier, inputs the visual relationship set and the new generation prompt text into the visual data generation model, and generates candidate visual data without having to re-extract the atmosphere information of the three target reference images in each target atmosphere dimension.

[0105] The data processing method provided in the embodiments of this specification saves the visual relationship set and generates a corresponding target set identifier, making the visual relationship set a savable, identifiable, reusable, and versionable atmosphere asset, thereby avoiding repeated extraction of atmosphere information and maintaining the consistency and traceability of atmosphere control basis between multiple generation tasks.

[0106] In one or more embodiments of this specification, when at least one target reference image is input into the visual data generation model, if the target reference image contains competing objects, the target reference image can be converted into a text description to prevent other objects from being mistakenly transferred to candidate visual data through image input, while preserving transferable visual relationships in the target reference image. Specific implementation methods are described below: The step of inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data includes: If the generated prompt text contains object description information of the target object, and the target reference image contains other objects different from the target object, the target reference image is converted into a text description, and the text description is integrated into the generated prompt text to obtain the target prompt text, wherein the text description does not contain object description information of the other objects; The visual relationship set and the target prompt text are input into the visual data generation model to generate the candidate visual data.

[0107] The generated prompt text can be understood as constraint text containing the user's request content. If the generated prompt text does not contain object description information of the target object, or if the target object corresponding to the object description information in the generated prompt text is consistent with the object contained in the target reference image, the generated prompt text can be directly used as the target prompt text.

[0108] The target object can be understood as the subject that is explicitly required to appear in the candidate visual data in the generated prompt text, including people, animals or other subjects, without limitation here; the object description information can be understood as the textual information used to describe the target object, including the target object's appearance, identity, actions, posture or clothing, etc.

[0109] Other objects can be understood as objects that appear in the target reference image but are different from the target object, such as actors, passersby, or other figures in the target reference image. These other objects are not required to appear in the candidate visual data by the generated cue text. The text description can be understood as textual information obtained by converting the target reference image into text, used to express transferable visual relationships in the target reference image. The text description does not contain object description information of other objects. The target cue text can be understood as the cue text obtained by integrating the text description into the generated cue text, used as input to the visual data generation model.

[0110] Specifically, when the visual data generation model is a model that generates new images based on reference images, when using the visual data generation model to generate candidate visual data, the target reference image used to provide atmosphere information can be input into the visual data generation model. However, when inputting at least one target reference image into the visual data generation model, it is necessary to first determine whether the generated prompt text contains object description information of the target object, and to determine whether the target reference image contains other objects that are different from the target object.

[0111] If the generated prompt text contains object description information of the target object, meaning the candidate visual data needs to include the target object, but the target reference image contains other objects, it may introduce noise or interference to the generated target object. Therefore, the target reference image cannot be directly input into the visual data generation model as an image containing a face. Instead, the target reference image is converted into a text description. This text description retains transferable atmospheric information from the target reference image, such as color and tone, lighting, optics and depth of field, weather and atmosphere, motion, material and texture, mood tone, and artistic and production design relationships, but does not contain object description information of other objects, nor does it include the identity, appearance, clothing, or posture of other objects.

[0112] By integrating text descriptions into the generated prompt text to obtain the target prompt text, and then inputting the visual relationship set and the target prompt text into the visual data generation model, candidate visual data is generated. In this way, other objects in the target reference image do not continue to participate in the generation as images; instead, the transferable atmospheric information of the target reference image is incorporated into the target prompt text as text descriptions. The visual relationship set continues to serve as the atmospheric control signal, and the target object description information in the generated prompt text continues to serve as the user's requested content.

[0113] For example, a user uploads a target reference image showing actor A standing in a rainy night train station. The reference text describes the image as being used to evoke a rainy night atmosphere. Simultaneously, a prompt text is generated requesting the generation of a scene depicting character B in a modern urban rainy night chase. In this case, the prompt text includes object description information for the target object, character B, while the target reference image contains other objects different from character B, namely actor A.

[0114] By converting the target reference image into a text description, the text description preserves atmospheric relationships such as cool blue tones, rain and fog density, wetland reflection, soft protection of facial highlights, and tense and restrained emotions, but does not include the actor A's identity, appearance, clothing, or posture. This text description is integrated into the generated prompt text to obtain the target prompt text. Then, the visual relationship set and the target prompt text are input into the visual data generation model to generate candidate visual data. The candidate visual data will not copy the actor A's face due to the input of the target reference image.

[0115] The data processing method provided in the embodiments of this specification converts the target reference image into a text description that does not contain description information of other objects and integrates it into the generated prompt text when the generated prompt text contains object description information of the target object and the target reference image contains other objects. This ensures that other objects in the target reference image do not contaminate the identity of the target object in the form of images, while retaining the transferable atmosphere information in the target reference image. Thus, when there are competing objects in the atmosphere reference, both atmosphere transfer and identity isolation are taken into account.

[0116] In one or more embodiments of this specification, upon receiving a visual data generation request and determining that the image generation description in the visual data generation request contains object description information, the target object corresponding to the object description information is determined. Then, a target prompt text is constructed based on the image generation description, the object reference information of the target object, and the generation constraint information, so that the target prompt text has a clear object basis before being input into the visual data generation model. Specific implementation methods are described below: Before inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data, the method further includes: Receive a visual data generation request and obtain the image generation description in the visual data generation request; If the image generation description includes object description information, the target object corresponding to the object description information is obtained, wherein the target object is obtained from a registered object library, extracted from an object reference image, or generated based on the object description information; The target prompt text is constructed based on the image generation description, the object reference information of the target object, and the generation constraint information.

[0117] Among them, the visual data generation request can be understood as the request information used to trigger the generation of target visual data. This request information can carry the target set identifier or the user's description of this generation task; the image generation description can be understood as the text description in the visual data generation request used to express what content the user wants to generate, such as text describing the subject, action, event, location, shot, etc.; the object description information can be understood as the text information used to describe the target object, including the target object's appearance, identity, action, posture, or clothing, etc.

[0118] The target object can be understood as the subject specified by the object description information that needs to appear in the candidate visual data, including people, animals or other subjects; the registered object library can be understood as a registry that stores published object packages. An object package can contain the unique object identifier, version, status, multi-view reference images, etc. of the corresponding object; the object reference image can be understood as a reference image uploaded or specified by the user to provide a basis for the appearance of the target object.

[0119] Object reference information can be understood as the control basis for constraining the appearance and identity of the target object in the candidate visual data, obtained from the parsing of the target object. It includes the appearance features, body proportion features, and corresponding viewpoints of the target object. Generation constraint information can be understood as supplementary information used to constrain the generation process or generation results. It includes information such as lens requirements, scene constraints, identity constraints, body proportion constraints, atmosphere constraints, and exclusion content.

[0120] Specifically, upon receiving a visual data generation request, it is determined whether the image generation description in the request contains object description information. If the image generation description contains object description information, the target object corresponding to that object description information is determined. There are three ways to determine the target object: 1. Obtain from a registered object library: When the object description information matches a registration tag of a published object package, the corresponding object package is loaded, and information such as the person description and multi-view reference images can be used as object reference information. 2. Extract from an object reference image: When the user uploads a reference image of a specified person, the object reference information is parsed from that image. 3. Generate based on object description information: When the user neither specifies a published object nor provides an object reference image, an image corresponding to the target object is generated based on the appearance definition in the object description information, and in this case, the object description information can be used as object reference information.

[0121] After determining the target object and its object reference information, a target prompt text is constructed based on the image generation description, the target object's object reference information, and the generation constraint information. This ensures that the target prompt text contains both the user's request content and the identity and body proportions of the target object, and is subject to the constraints of the generation constraint information.

[0122] For example, a user submits a visual data generation request, where the image generation description is "Character A in a modern urban rainy night chase scene." This image generation description includes object description information, i.e., Character A. If Character A matches a registered tag in the registered object library, the corresponding ready character package for Character A is retrieved from the registered object library, and its character description, multi-view references, and other information are read as object reference information. Based on the image generation description, Character A's object reference information, and generation constraint information, target prompt text is constructed.

[0123] For example, if a user submits a visual data generation request, with the image generation description being "using this person's image in a modern urban rainy night chase scene," and uploads a reference image of the person, then the target object is extracted from the reference image of the person, and information such as appearance features, body proportion features, and corresponding viewpoints are parsed as object reference information. Then, the target prompt text is constructed based on the image generation description, the object reference information, and the generation constraint information.

[0124] When a user submits a visual data generation request, and the image generation description is "a cool and aloof young man chasing a character in a rainy night in a modern city", but no published character is specified or no character reference is provided, the target object can be generated based on the object description information, and then the object reference information can be extracted. Then, the target prompt text can be constructed based on the image generation description, the object reference information, and the generation constraint information.

[0125] The data processing method provided in the embodiments of this specification determines the target object corresponding to the object description information when the image generation description contains object description information, and constructs target prompt text based on the image generation description, the object reference information of the target object, and the generation constraint information, so that the visual data generation request containing the object description obtains a clear and unique target object basis, thereby taking on the identity channel and atmosphere control channel respectively with the visual relationship set, providing the correct object basis for the subsequent input visual data generation model to generate candidate visual data.

[0126] In one or more embodiments of this specification, if the image generation description in the visual data generation request does not contain object description information, a target prompt text is constructed based on the image generation description and generation constraint information. Specific implementation methods are as follows: Before inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data, the method further includes: Receive a visual data generation request and determine the image generation description in the visual data generation request; If the image generation description does not contain object description information, the target prompt text is constructed based on the image generation description and generation constraint information.

[0127] Specifically, when the image generation description does not contain object description information, that is, when the user only describes the scene, atmosphere, event or shot without specifying a specific person, animal or other subject, the target prompt text is constructed based on the image generation description and generation constraint information.

[0128] For example, a user submits a visual data generation request, in which the image generation description is "Rainy night train station, reference to cold blues", without specifying a person. After determining that the image generation description does not contain object description information, a target prompt text is constructed based on the image generation description and generation constraint information. Then, the target set identifier is called to identify the corresponding visual relationship set. The visual relationship set and the target prompt text are input into the visual data generation model to generate candidate visual data, which is an unmanned scene.

[0129] The data processing method provided in the embodiments of this specification constructs target prompt text based on the image generation description and generation constraint information when the image generation description does not contain object description information, so that the generation request without object description can still obtain complete and constrained target prompt text.

[0130] Step 208: Perform content verification on the candidate visual data based on the target prompt text, and perform atmosphere verification on the candidate visual data based on the visual relationship set in the target atmosphere dimension. If both the content verification and the atmosphere verification pass, the candidate visual data is taken as the target visual data.

[0131] Content verification can be understood as a verification process that checks whether candidate visual data meets the user's requested content based on the generated prompt text. The objects of the check include whether the candidate visual data contains the subject, action, event, location, shot, etc. required by the generated prompt text, and whether it has copied the source content that was not requested.

[0132] Atmosphere verification can be understood as a verification process that checks whether candidate visual data meets the atmosphere control requirements based on a set of visual relationships within the target atmosphere dimension. The checks include verifying whether the visual relationships across the target atmosphere dimensions—such as color and tone, lighting, composition, optics and depth of field, weather and atmosphere, motion, materials and textures, mood tone, and art and production design—are consistent with the set of visual relationships. Target visual data can be understood as candidate visual data that has passed both content and atmosphere verification and can be delivered to the client as a result.

[0133] Specifically, the candidate visual data is content-verified based on the generated prompt text, and atmosphere-verified based on the visual relationship set in the target atmosphere dimension. Thus, before the candidate visual data is delivered, it is checked whether the candidate visual data meets the user's requested content and whether it meets the atmosphere control requirements. Only candidate visual data that passes both verifications is accepted and used as the target visual data.

[0134] In practice, when verifying the content of candidate visual data based on the generated prompt text, the candidate visual data is compared with the generated prompt text to check whether the candidate visual data retains the subject, action, event, location, shot and delivery specifications requested by the user, whether the candidate visual data replicates other objects, clothing, objects, text, logos or precise locations in the atmosphere reference, and whether the candidate visual data deviates from the content requested by the user due to style compensation.

[0135] When verifying the atmosphere of candidate visual data based on the visual relationship set in the target atmosphere dimension, the candidate visual data is compared with the visual relationship set, rather than requiring the candidate visual data to copy the content of the target reference image. The candidate visual data is checked item by item according to the target atmosphere dimensions such as color and tone, lighting, composition, optics and depth of field, weather and atmosphere, motion, material and texture, mood tone, and art and production design. The check is to see if the candidate visual data meets the consensus information in the visual relationship set, whether it falls within the scope information, whether it avoids conflicting information, and whether it complies with the exclusion content.

[0136] If both content verification and atmosphere verification pass, the candidate visual data will be used as the target visual data. If either content verification or atmosphere verification fails, the candidate visual data will not be used as the target visual data. Instead, a targeted retry will be initiated, and only one failed relationship will be repaired at a time. Only the failed control channels and relationships will be modified. Passed channels will not be arbitrarily replaced when repairing another channel.

[0137] For example, a user submits a visual data generation request, asking for footage of character A in a modern urban rainy night chase scene, and includes a target set identifier to access a saved set of visual relationships. After the candidate visual data is generated, content verification is performed based on the generation prompt text. This checks whether character A appears in the candidate visual data, whether it is in a modern urban rainy night chase scene, whether it meets the delivery specifications, and whether the candidate visual data replicates actors, raincoats, or train stations from the target reference image.

[0138] In terms of target atmosphere dimensions such as color and tone, lighting, composition, optics and depth of field, weather and atmosphere, movement, material and texture, mood tone, and art and production design, atmosphere verification is conducted based on a set of visual relationships. This checks whether candidate visual data presents visual relationships such as a cool blue environment, a small amount of warm-colored real-scene lighting, suppressed reflections from wetlands, soft and protective highlights on faces, varying depths in rain and fog, and a tense yet restrained mood. If both content verification and atmosphere verification pass, the candidate visual data is adopted as the target visual data.

[0139] In practical applications, after content verification and atmosphere verification are passed, human-scene fusion verification can be carried out to check whether the subject and environment share the lighting model, whether the environment affects the subject, whether there is evidence of contact or occlusion, whether there is obvious cutout halo, and whether the perspective scale is valid. After human-scene fusion verification is passed, the candidate visual data is used as the target visual data.

[0140] It's important to note that passing the atmosphere verification does not guarantee successful human-scene fusion. The former compares candidate visual data with the set of visual relationships, while the latter checks whether the subject and environment share the same physical field. Furthermore, each production run can record identity, body proportions, atmosphere and human-scene fusion scores, hard thresholds, reasons for failure, and which variables were modified in each round, making the acceptance, rejection, or pending manual review status of the target visual data auditable.

[0141] The data processing method provided in the embodiments of this specification verifies the content of candidate visual data based on generated prompt text and verifies the atmosphere of candidate visual data based on visual relationship set in the target atmosphere dimension. When both content verification and atmosphere verification are passed, the candidate visual data is used as target visual data. This allows the release of candidate visual data to be constrained by both user-requested content and atmosphere control requirements, thereby avoiding atmosphere reference pollution of the user scene, preventing unrequested content from being migrated, and providing target visual data with a clear pass status for subsequent human-scene fusion verification and production records.

[0142] In one or more embodiments of this specification, when performing content verification on candidate visual data based on generated prompt text, the content verification of the candidate visual data can be specifically performed based on preset basic scoring dimensions. If the verification results of each basic scoring dimension are passed, the content verification of the candidate visual data is determined to be successful. Specific implementation methods are as follows: The content verification of the candidate visual data based on the generated prompt text includes: Based on preset basic scoring dimensions, the candidate visual data is subjected to content verification, wherein the basic scoring dimensions include at least a text alignment dimension, which is used to check the consistency between the candidate visual data and the target prompt text; If the verification results of each basic scoring dimension are passed, the content verification of the candidate visual data is determined to be successful.

[0143] The basic scoring dimensions can be understood as pre-defined observation directions used to check whether candidate visual data meets the observation direction of the user's requested content expressed in the target prompt text. The basic scoring dimensions are used to cover different aspects of the user's requested content, so that content verification does not rely on a single overall feature. The verification result can be understood as a single conclusion obtained after checking the candidate visual data based on the basic scoring dimensions. The verification result can be marked as pass or fail, or a score or evidence can be given. Passing the content verification can be understood as the candidate visual data being deemed qualified in terms of the user's requested content when the verification results of each basic scoring dimension meet the pass requirements.

[0144] Specifically, when validating candidate visual data based on preset basic scoring dimensions, the basic scoring dimensions can be used to check the consistency between the candidate visual data and the target prompt text. In fact, the basic scoring dimensions can cover the subject, action, event, location, and shot of the content requested by the user, as well as whether the candidate visual data has copied unrequested source content, whether it has deviated from the content requested by the user due to style compensation, and whether it retains the user's scene. Validation results are provided for each basic scoring dimension.

[0145] For example, the text alignment dimension can check whether the candidate visual data accurately presents the content expressed by the target prompt text; the subject dimension can check whether the subject presented in the candidate visual data is correct; and the action dimension can check whether the actions in the candidate visual data match, etc.

[0146] If the verification results of all basic scoring dimensions pass, the content verification of the candidate visual data is determined to be successful; if the verification result of any basic scoring dimension fails, the content verification of the candidate visual data is determined to be unsuccessful, and the candidate visual data will not be used as the target visual data.

[0147] The data processing method provided in the embodiments of this specification verifies the content of candidate visual data based on preset basic scoring dimensions. This allows the content verification to be carried out item by item according to each dimension and to give a verifiable passing conclusion, thereby avoiding content errors and providing a structured basis for determining whether candidate visual data should be used as target visual data.

[0148] In one or more embodiments of this specification, if the candidate visual data contains a target object, then when performing content verification on the candidate visual data, if the verification results of all basic scoring dimensions pass, object verification is performed separately on the target object at the object dimension. If the object verification passes, the content verification of the candidate visual data is determined to be successful. Specific implementation methods are as follows: If the verification results of all basic scoring dimensions pass, determining that the content verification of the candidate visual data has passed also includes: If the candidate visual data contains a target object and the verification results of each basic scoring dimension are all passed, then, in terms of the object dimension, the target object in the candidate visual data is verified. If the object verification passes, the content verification of the candidate visual data is determined to be successful.

[0149] Among them, the object dimension can be understood as the observation direction specifically used to check whether the target object in the candidate visual data meets the requirements; object verification can be understood as the verification process of checking whether the target object in the candidate visual data is consistent with the target object, whether there is identity drift, whether the age category matches, and whether the key structure is visible, based on the object reference information of the target object.

[0150] Specifically, if the candidate visual data contains the target object and the verification results of each basic scoring dimension pass, further object verification is performed on the target object in the candidate visual data at the object dimension. Object verification can be performed based on the object reference information of the target object, checking whether the target object in the candidate visual data is consistent with the object package in the registered object library, or whether it is consistent with the object information extracted from the object reference image, or whether it is consistent with the target object generated based on the object description information; checking whether the target object in the candidate visual data shows drift in key structures such as face shape, eyes, and nose, whether the age category matches, and whether the key structures are visible, etc.

[0151] If object verification passes, the content verification of the candidate visual data is deemed successful; if object verification fails, content verification fails, and the candidate visual data is not considered as target visual data. In other words, when the candidate visual data contains the target object, content verification adds an object-level check to the basic scoring dimensions, ensuring that the target object in the candidate visual data is independently verified at the content level.

[0152] The data processing method provided in the embodiments of this specification adds an independent check at the target object level to the content verification in addition to the basic scoring dimension when the candidate visual data contains the target object. This simultaneously constrains the user's requested content and the identity of the target object at the content level, preventing candidate visual data with identity drift from being used as target visual data.

[0153] In one or more embodiments of this specification, when performing atmosphere verification on candidate visual data, a dimensional score of the candidate visual data on each target atmosphere dimension is calculated, and the atmosphere verification of the candidate visual data is determined based on the dimensional scores and the comprehensively calculated atmosphere score. Specific implementation methods are as follows: On the target atmosphere dimension, atmosphere verification is performed on the candidate visual data based on the visual relationship set, including: For each target atmosphere dimension, the candidate visual data is validated based on the visual relationship set to obtain the dimension score for each target atmosphere dimension. The dimensional scores of each target atmosphere dimension are weighted and calculated to obtain the atmosphere score of the candidate visual data; If the dimensional scores of each target atmosphere dimension reach the corresponding dimensional threshold and the atmosphere score of the candidate visual data reaches the comprehensive threshold, then the atmosphere verification of the candidate visual data is determined to be successful.

[0154] Among them, the dimensional score can be understood as the individual score obtained after checking the candidate visual data according to the visual relationship set on each target atmosphere dimension, which is used to represent the degree of consistency between the candidate visual data and the visual relationship set on the target atmosphere dimension; the weighted calculation can be understood as the process of weighted summation of the dimensional scores of each target atmosphere dimension according to the weights corresponding to each target atmosphere dimension.

[0155] Atmosphere score can be understood as a comprehensive score obtained by weighting the dimensional scores of each target atmosphere dimension, used to represent the consistency between the overall atmosphere of the candidate visual data and the set of visual relationships; dimensional threshold can be understood as a pre-set scoring requirement for a single target atmosphere dimension, which the score of that dimension needs to meet; comprehensive threshold can be understood as a pre-set scoring requirement for the overall atmosphere of the candidate visual data.

[0156] Specifically, when verifying the atmosphere of candidate visual data based on the visual relationship set for each target atmosphere dimension, the candidate visual data and the visual relationship set are compared item by item according to the target atmosphere dimensions such as color and tone, lighting, composition, optics and depth of field, weather and air, motion, material and texture, mood tone, and art and production design, and corresponding dimension scores are obtained for each target atmosphere dimension.

[0157] Subsequently, the dimensional scores of each target atmosphere dimension are weighted and calculated to obtain the atmosphere score of the candidate visual data. The weights can be determined by allocating them according to a set ratio for dimensions such as color and tone, lighting, optics and depth of field, composition, weather and air, motion, material and texture, mood tone, art and production design, etc.

[0158] If the dimensional scores of each target atmosphere dimension reach the corresponding dimensional threshold and the atmosphere score of the candidate visual data reaches the comprehensive threshold, the atmosphere verification of the candidate visual data is determined to be successful; if any target atmosphere dimension is lower than the corresponding dimensional threshold, or the atmosphere score is lower than the comprehensive threshold, the atmosphere verification fails and the candidate visual data is not used as target visual data.

[0159] Of course, when conducting atmosphere verification, the verification criteria can be relaxed according to actual needs. For example, one or more key atmosphere dimensions can be determined from multiple target atmosphere dimensions. If the dimension score of the key atmosphere dimension reaches the corresponding dimension threshold and the atmosphere score of the candidate visual data reaches the comprehensive threshold, the atmosphere verification of the candidate visual data can be determined to be successful.

[0160] Continuing with the previous example, a user submits a visual data generation request, requesting the generation of a scene depicting character A in a rainy night chase in a modern city, along with a target set identifier to access a saved set of visual relationships. After the candidate visual data is generated, it undergoes atmosphere verification based on the set of visual relationships across target atmosphere dimensions such as color and tone, lighting, composition, optics and depth of field, weather and atmosphere, motion, material and texture, mood tone, and art and production design, yielding a dimensional score for each target atmosphere dimension.

[0161] For example, in the color and tone dimension, check whether the cool blue main color family, the blue shadows, and the control of highlights are consistent; in the lighting dimension, check whether the small number of warm-colored real-scene lights, the direction of the main light, and the soft protection of facial highlights are consistent; in the weather and air dimension, check whether the density of rain and fog and the layers of distance are consistent; in the material and texture dimension, check whether the reflection of wetlands is suppressed.

[0162] The dimensional scores of each target atmosphere dimension are weighted and calculated to obtain the atmosphere score of the candidate visual data. If the key dimensions such as color and tone, lighting and mood tone all meet the corresponding dimension thresholds, and other dimensions also meet their respective dimension thresholds, and the atmosphere score meets the comprehensive threshold, then the atmosphere verification of the candidate visual data is determined to be successful; if the lighting dimension is lower than the corresponding dimension threshold, then the atmosphere verification fails.

[0163] In practical applications, passing the atmosphere verification does not mean that human-scene fusion has passed. Atmosphere verification is used to compare candidate visual data with the set of visual relationships, while human-scene fusion is used to check whether the subject and the environment share the same physical field. Based on content verification and atmosphere verification, candidate visual data can be checked on the human-scene fusion dimension. Thus, if all three pass, the candidate visual data can be used as the target visual data.

[0164] The data processing method provided in the embodiments of this specification allows for the evaluation of the atmosphere of candidate visual data by scoring each item according to multiple target atmosphere dimensions. This evaluation, along with the comprehensive score, enables a double check, thereby preventing a single-dimensional failure from being masked by the overall atmosphere. It also provides a structured basis for jointly determining whether candidate visual data should be used as target visual data with content evaluation.

[0165] In one or more embodiments of this specification, if content verification and / or atmosphere verification fail, the dimensions that failed verification are identified and targeted repairs are performed on these dimensions. This ensures that the candidate visual data is adjusted only for the dimensions that failed verification, while the dimensions that passed verification are not arbitrarily altered. Specific implementation methods are described below: After performing content verification on the candidate visual data based on the generated prompt text, and atmosphere verification on the candidate visual data based on the visual relationship set in the target atmosphere dimension, the method further includes: If content verification and / or atmosphere verification fail, identify the dimensions that failed verification and require adjustment. The candidate visual data is adjusted along the dimension to be adjusted to generate new candidate visual data. The new candidate visual data is subjected to content verification and atmosphere verification. If both the content verification and atmosphere verification are passed, the candidate visual data is used as the target visual data.

[0166] Among them, the dimension to be adjusted can be understood as the specific dimension that failed the content verification or atmosphere verification. The dimension to be adjusted can be the basic scoring dimension or object dimension in the content verification, or the target atmosphere dimension in the atmosphere verification. Adjustment can be understood as making targeted modifications to the candidate visual data on the dimension to be adjusted. The modification methods include reorganizing the generation request, adjusting the corresponding control channel, and re-rendering the candidate visual data. The new candidate visual data can be understood as the candidate visual data that has been regenerated after adjusting the candidate visual data on the dimension to be adjusted, and that needs to be re-submitted for content verification and atmosphere verification.

[0167] Specifically, if content verification and / or atmosphere verification fail, first determine the dimensions that failed verification and require adjustment. If content verification fails, the dimensions to be adjusted may include one or more of the following dimensions: subject, action, event, location, shot, or object. If atmosphere verification fails, the dimensions to be adjusted may include one or more of the following: color and tone, lighting, composition, optics and depth of field, weather and atmosphere, movement, material and texture, mood tone, and art and production design.

[0168] When adjusting candidate visual data on the dimensions to be adjusted, the principle of fixing only one failed relationship at a time should be followed. Only the failed control channels and relationships should be modified, and channels that have passed should not be arbitrarily replaced when fixing another channel. For example, when identity fails, match a closer original angle, enlarge and clearly display the face, and reduce occlusion, only correcting the drifting facial relationships; when atmosphere fails, only modify the distorted color, lighting, depth of field, material, or emotion dimensions.

[0169] After adjustment, new candidate visual data is generated. Then, content verification and atmosphere verification are performed again on the new candidate visual data. If both content verification and atmosphere verification pass, the candidate visual data is used as the target visual data. If the new candidate visual data still fails, new dimensions to be adjusted can be determined and adjusted again until it passes or enters manual review.

[0170] In practical applications, after new candidate visual data passes content verification and atmosphere verification, it can continue to undergo human-scene fusion verification to check whether the subject and environment share the lighting model, whether the environment affects the subject, whether there is evidence of contact or occlusion, whether there is obvious cutout halo, and whether the perspective scale is valid. Only after the human-scene fusion verification is passed can the candidate visual data be used as the target visual data. In each round, only which variables are modified can be saved in the log to make the targeted retries auditable.

[0171] The data processing method provided in the embodiments of this specification generates new candidate visual data by adjusting candidate visual data on the dimension to be adjusted, so that the verification failure is isolated to a specific dimension and repaired in a targeted manner according to the univariate principle, thereby avoiding the arbitrary replacement of the already passed channel and improving the reliability of the candidate visual data after re-verification.

[0172] See Figure 3 , Figure 3 A flowchart illustrating the processing procedure of a visual data processing method provided in one embodiment of this specification is shown.

[0173] The user uploads a reference image set, which can be understood as one or more images uploaded by the user to constrain the image to be generated, i.e., at least one initial reference image in the above embodiments.

[0174] Each initial reference image in the reference image set is categorized into five control roles: identity, atmosphere, composition, and content. The identity role defines the identity of a specific person, animal, or other subject; the atmosphere role provides information on color, lighting, optics, weather, motion, texture, mood, or artistic design relationships; the composition role provides information on subject scale, image position, negative space, horizon, depth channels, or occlusion levels; the content role specifies the people, objects, places, clothing, actions, or events that should appear in the target visual data; and the mixed role contains two or more effective control information simultaneously. A single image can have one or more roles, but these must be explicitly separated into individual observations.

[0175] For initial reference images classified as hybrid, they need to be decomposed item by item. That is, the specific control information contained in the reference image needs to be determined and extracted according to the corresponding control type, rather than being inherited as a whole. It should be noted here that even if an atmosphere-type reference image contains a clear human face, it will not acquire identity rights, and an identity-type reference image will not determine the clothing and background of the new scene.

[0176] This decomposition prevents reference images used for atmosphere from being mistakenly identified as authoritative figures, compositional basis, or content sources. After classification and decomposition, the initial reference image with the control type of atmosphere is identified as the target reference image. Subsequently, atmosphere information is extracted only from this target reference image, without obtaining identity information or content designation rights from it.

[0177] After determining the target reference image, a nine-dimensional decomposition is performed on the atmosphere channel. This involves extracting information from each target reference image according to nine target atmosphere dimensions: color, lighting, composition, optical depth of field, weather and atmosphere, motion, material texture, mood, and art design. This yields the dimensional information of each target reference image in each target atmosphere dimension.

[0178] If multiple target reference images exist, cross-image comparative analysis is performed within the same nine-dimensional framework to distinguish which visual relationships are stably shared across multiple reference images, which differences still belong to acceptable variations within the same visual system, which differences belong to structural conflicts that cannot be safely averaged, and which faces, clothing, objects, text, signs, or precise locations are excluded content that cannot be migrated.

[0179] Following cross-image comparative analysis, the atmosphere information is organized into consensus information, scope information, conflict information, excluded content, style atmosphere anchors, and dimensional confidence levels. Consensus information comprises visual features that recur or are consistently stable across multiple reference images; scope information refers to acceptable ranges of variation within the same visual system, despite differences; conflict information identifies structural contradictions in the same dimension that cannot be simultaneously satisfied in different reference images; excluded content includes occasional figures, clothing, objects, text, signs, or specific locations that cannot be transferred from reference images to the new image; style control anchors are control information obtained through atmosphere profile compression and can be received by the visual data generation model; and dimensional confidence levels are categorized as high, medium, or low to indicate the reliability of judgments in each dimension. Based on these organized results, a set of visual relationships, i.e., the atmosphere profile, is generated. This set is decoupled from specific content, retaining only transferable visual relationships.

[0180] After generating the visual relationship set, it is further saved as a versioned atmosphere profile, making it a savable, identifiable, reusable, and versionable style asset. The saved visual relationship set can generate a corresponding target set identifier. When subsequent visual data generation requests include this target set identifier, the saved visual relationship set can be directly called for reuse in image or video generation, without needing to re-extract the atmosphere dimension from each target reference image. This versioning mechanism allows the same atmosphere profile to be used for different subjects, different scenes, or different batches of generation tasks, while maintaining the consistency and traceability of atmosphere control criteria.

[0181] During the request arrangement phase, the target prompt text generated from the input visual data is arranged according to a certain priority. Specifically, in the target prompt text, the user request content takes priority over the person's identity, the person's identity takes priority over the body proportion, the body proportion takes priority over the user's reference atmosphere, the user's reference atmosphere takes priority over the human-scene physical fusion, and the human-scene physical fusion takes priority over decorative visual elements.

[0182] In this embodiment, sufficient materials to support the target angle and identity are selected, instead of inputting all character images, style images, and character swatches into the visual data generation model. If the generated prompt text contains object description information of the target object and the target reference image contains other objects different from the target object, the target reference image is converted into a text description that does not contain object description information of other objects. This text description is then integrated into the generated prompt text to obtain the target prompt text. If the generated prompt text does not contain object description information, the target prompt text is constructed based on the image-generated description and generation constraint information. If the target prompt text contains object description information, the target object corresponding to the object description information is determined. The target object can be obtained from the registered object library, extracted from the object reference image, or generated based on the object description information. The target prompt text is then constructed based on the image-generated description, the target object's object reference information, and generation constraint information. After arrangement, the consensus information and scope information in the user-required subject, action, event, location, shot, character identity authority, atmosphere profile, excluded content, and information shared by the subject and environment such as lighting, air, contact, focus, movement, and perspective are organized as inputs to the visual data generation model.

[0183] The visual data generation model generates candidate visual data based on a set of visual relationships and target cue text. The generated candidate visual data is then validated, including atmosphere validation and content validation. When the candidate visual data contains the target object, validation is performed on the object dimension and on the human-scene fusion aspect.

[0184] Content verification can be performed item by item based on preset basic scoring dimensions. If the verification results of each basic scoring dimension are passed, the content verification is considered to be passed. If the candidate visual data contains the target object and each basic scoring dimension is passed, then object verification is further performed on the object dimension. The object verification checks whether key structures such as face shape, eyes, and nose are drifted, whether the age category matches, and whether the key structures are visible, based on the object reference information of the target object. Only after the object verification is passed is the content verification considered to be passed.

[0185] The atmosphere verification process scores candidate visual data based on a set of visual relationships across nine target atmosphere dimensions, obtaining a dimensional score for each target atmosphere dimension. The dimensional scores are then weighted to obtain the atmosphere score for the candidate visual data. The atmosphere verification is considered successful if the scores for each dimension meet the corresponding dimensional threshold and the overall atmosphere score meets the comprehensive threshold.

[0186] It's important to note that color, lighting, and mood are key dimensions; any one of them falling below a preset threshold results in failure. Passing the atmosphere verification does not guarantee that the human-scene fusion verification is successful. Human-scene fusion is used to check whether the subject and environment are coupled. If the subject and environment are coupled, the human-scene fusion verification is considered successful. Specifically, human-scene fusion checks whether the subject and environment share the same physical field. Therefore, it is also necessary to independently check the lighting continuity, air and weather involvement, contact physics, focus and texture continuity, motion field continuity, perspective and scale, and edges and compositing traces between the subject and environment. This confirms that the subject and environment share a lighting model, that the environment does indeed affect the subject, that there is evidence of contact or occlusion, that there are no obvious matting halos, and that the perspective and scale are correct.

[0187] If validation fails, identify the dimension that failed validation and adjust it accordingly. Adjust the candidate visual data based on this dimension to generate new candidate visual data, and then re-execute validation on the new candidate visual data. Retrying follows the univariate directed retry principle: only one failed channel is repaired at a time, and a passed channel cannot be arbitrarily replaced when repairing another channel. Failed candidate visual data is only used for diagnosis and cannot be used as a reference for the next round. If the new candidate visual data still fails, continue to identify new dimensions to adjust and adjust them again until they pass or manual review is initiated.

[0188] Once all verifications are passed, the candidate visual data is used as the target visual data and returned to the client for the user to view and use.

[0189] The visual data processing method provided in this specification, through controlled role classification, item-by-item decomposition of mixed roles, nine-dimensional atmosphere decomposition, consensus information and conflict organization, versioned atmosphere profiles, dual verification of atmosphere and content, and single-variable targeted retries, completely separates identity, atmosphere, composition, and content that were originally mixed in the same input. This avoids the mis-migrating of characters, clothing, objects, text, logos, or precise locations in the atmosphere reference, and prevents style averaging, identity drift, and human-scene separation when multiple images conflict. In this embodiment, visual relationships are migrated rather than pixels. The atmosphere profile can be saved, identified, reused, and versioned. Failure repair only targets dimensions that have not passed, and channels that have passed are not arbitrarily replaced. The final delivered target visual data has interpretable, auditable, and rollbackable quality evidence. This method can ensure character similarity, body stability, user scene, atmosphere consistency, and human-scene physical fusion in the production of visual content such as images and videos.

[0190] This specification also provides a visual data processing system, which includes a client and a server, wherein... The client is used to send a visual data generation request to the server, wherein the visual data generation request includes at least one target reference image; The server is configured to extract information from at least one target reference image in the target atmosphere dimension to obtain atmosphere information of the at least one target reference image in each target atmosphere dimension; obtain a visual relationship set based on the atmosphere information of the at least one target reference image in each target atmosphere dimension; input the visual relationship set and target prompt text into a visual data generation model to generate candidate visual data; perform content verification on the candidate visual data based on the target prompt text, and perform atmosphere verification on the candidate visual data based on the visual relationship set in the target atmosphere dimension; if both the content verification and the atmosphere verification pass, use the candidate visual data as the target visual data and return the target visual data to the client.

[0191] In practice, after obtaining the visual relationship set, this set can be saved, and a target set identifier corresponding to the visual relationship set can be generated and returned to the client. Therefore, when the client subsequently sends a visual data generation request, it can send the target set identifier instead of at least one target reference image. The server can then use the target set identifier to call the visual relationship set to generate the target visual data. This method avoids generating the visual relationship set every time visual data is requested, saving computational resources.

[0192] For specific implementation methods, please refer to the above discussion, which will not be repeated here.

[0193] Corresponding to the above method embodiments, this specification also provides embodiments of a visual data processing device. Figure 4 A schematic diagram of the structure of a visual data processing apparatus according to one embodiment of this specification is shown. Figure 4 As shown, the device includes: The information extraction module 402 is configured to extract information from at least one target reference image in the target atmosphere dimension to obtain the atmosphere information of the at least one target reference image in each target atmosphere dimension. The relationship acquisition module 404 is configured to obtain a set of visual relationships based on the atmosphere information of the at least one target reference image in each target atmosphere dimension; The image generation module 406 is configured to input the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data when generating target visual data based on the visual relationship set; The image verification module 408 is configured to perform content verification on the candidate visual data based on the target prompt text, and to perform atmosphere verification on the candidate visual data based on the visual relationship set in the target atmosphere dimension. If both the content verification and the atmosphere verification pass, the candidate visual data is used as the target visual data.

[0194] The device further includes: The image acquisition module is configured to acquire at least one initial reference image and a reference text description; determine the control type corresponding to each initial reference image based on the reference text description; and determine the initial reference image whose control type is atmosphere type as the target reference image.

[0195] Optionally, the image generation module 406 is further configured to: If the generated prompt text contains object description information of the target object, and the target reference image contains other objects different from the target object, the target reference image is converted into a text description, and the text description is integrated into the generated prompt text to obtain the target prompt text, wherein the text description does not contain object description information of the other objects; The visual relationship set and the target prompt text are input into the visual data generation model to generate the candidate visual data.

[0196] Optionally, the relationship acquisition module 404 is further configured to: When the at least one target reference image is multiple target reference images, cross-image comparative analysis is performed on the atmosphere information of each target reference image in each target atmosphere dimension to obtain analysis results, wherein the analysis results include at least one of consensus information, range information, conflict information and exclusion content; The set of visual relationships is generated based on the analysis results.

[0197] Optionally, the relationship acquisition module 404 is further configured to: Save the visual relationship set and generate the target set identifier corresponding to the visual relationship set; Upon receiving a visual data generation request, and if the visual data generation request carries the target set identifier, the visual relationship set is invoked, and the visual relationship set and the target prompt text are input into the visual data generation model to generate candidate visual data.

[0198] The device further includes: The prompt generation module is configured to receive a visual data generation request, obtain an image generation description from the visual data generation request; if the image generation description contains object description information, obtain the target object corresponding to the object description information, wherein the target object is obtained from a registered object library, extracted from an object reference image, or generated based on the object description information; and construct the generated prompt text based on the image generation description, the object reference information of the target object, and the generation constraint information.

[0199] Optionally, the prompt generation module is further configured to: Receive a visual data generation request and obtain the image generation description in the visual data generation request; If the image generation description does not contain object description information, the target prompt text is constructed based on the image generation description and generation constraint information.

[0200] Optionally, the image verification module 408 is further configured to: Based on preset basic scoring dimensions, the candidate visual data is subjected to content verification, wherein the basic scoring dimensions include at least a text alignment dimension, which is used to check the consistency between the candidate visual data and the target prompt text; If the verification results of each basic scoring dimension are passed, the content verification of the candidate visual data is determined to be successful.

[0201] Optionally, the image verification module 408 is further configured to: If the candidate visual data contains a target object and the verification results of each basic scoring dimension are all passed, then, in terms of the object dimension, the target object in the candidate visual data is verified. If the object verification passes, the content verification of the candidate visual data is determined to be successful.

[0202] Optionally, the image verification module 408 is further configured to: For each target atmosphere dimension, the candidate visual data is validated based on the visual relationship set to obtain the dimension score for each target atmosphere dimension. The dimensional scores of each target atmosphere dimension are weighted and calculated to obtain the atmosphere score of the candidate visual data; If the dimensional scores of each target atmosphere dimension reach the corresponding dimensional threshold and the atmosphere score of the candidate visual data reaches the comprehensive threshold, then the atmosphere verification of the candidate visual data is determined to be successful.

[0203] The device further includes: The adjustment module is configured to determine the dimensions to be adjusted if the content verification and / or atmosphere verification fail; adjust the candidate visual data on the dimensions to be adjusted to generate new candidate visual data; perform the content verification and atmosphere verification on the new candidate visual data; and if both the content verification and atmosphere verification pass, use the candidate visual data as the target visual data.

[0204] The above is an illustrative scheme of a visual data processing device according to this embodiment. It should be noted that the technical solution of this visual data processing device and the technical solution of the above-described visual data processing method belong to the same concept. For details not described in detail in the technical solution of the visual data processing device, please refer to the description of the technical solution of the above-described visual data processing method.

[0205] Figure 5 A structural block diagram of a computing device 500 according to one embodiment of this specification is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0206] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0207] In one embodiment of this specification, the above-described components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0208] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0209] The processor 520 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described visual data processing method.

[0210] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the visual data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the visual data processing method embodiments.

[0211] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual data processing method.

[0212] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the visual data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the visual data processing method embodiments.

[0213] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual data processing method.

[0214] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-described visual data processing method belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-described visual data processing method.

[0215] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0216] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0217] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0218] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0219] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A visual data processing method, comprising: In the target atmosphere dimension, information is extracted from at least one target reference image to obtain the atmosphere information of the at least one target reference image in each target atmosphere dimension; Based on the atmospheric information of the at least one target reference image in each target atmospheric dimension, a set of visual relationships is obtained; The set of visual relationships and the target prompt text are input into the visual data generation model to generate candidate visual data. The candidate visual data is subjected to content verification based on the target prompt text, and the candidate visual data is subjected to atmosphere verification based on the visual relationship set in the target atmosphere dimension. If both the content verification and the atmosphere verification are passed, the candidate visual data is taken as the target visual data.

2. The visual data processing method according to claim 1, wherein inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data includes: If the generated prompt text contains object description information of the target object, and the target reference image contains other objects different from the target object, the target reference image is converted into a text description, and the text description is integrated into the generated prompt text to obtain the target prompt text, wherein the text description does not contain object description information of the other objects; The visual relationship set and the target prompt text are input into the visual data generation model to generate the candidate visual data.

3. The visual data processing method according to claim 1, wherein obtaining a visual relationship set based on the atmosphere information of the at least one target reference image in each target atmosphere dimension includes: When the at least one target reference image is multiple target reference images, cross-image comparative analysis is performed on the atmosphere information of each target reference image in each target atmosphere dimension to obtain analysis results, wherein the analysis results include at least one of consensus information, range information, conflict information and exclusion content; The set of visual relationships is generated based on the analysis results.

4. The visual data processing method according to claim 1, wherein the content verification of the candidate visual data based on the target prompt text includes: Based on preset basic scoring dimensions, the candidate visual data is subjected to content verification, wherein the basic scoring dimensions include at least a text alignment dimension, which is used to check the consistency between the candidate visual data and the target prompt text; If the verification results of each basic scoring dimension are passed, the content verification of the candidate visual data is determined to be successful.

5. The visual data processing method according to claim 4, further comprising determining that the content verification of the candidate visual data has passed when the verification results of each basic scoring dimension have passed, further comprising: If the candidate visual data contains a target object and the verification results of each basic scoring dimension are all passed, then, in terms of the object dimension, the target object in the candidate visual data is verified. If the object verification passes, the content verification of the candidate visual data is determined to be successful.

6. The visual data processing method according to any one of claims 1-5, wherein, based on the visual relationship set, atmosphere verification is performed on the candidate visual data in the target atmosphere dimension, comprising: At each target atmosphere dimension, the candidate visual data is validated based on the visual relationship set to obtain the dimensional score of the candidate visual data at each target atmosphere dimension. The dimensional scores of each target atmosphere dimension are weighted and calculated to obtain the atmosphere score of the candidate visual data; If the dimensional scores of each target atmosphere dimension reach the corresponding dimensional threshold and the atmosphere score of the candidate visual data reaches the comprehensive threshold, then the atmosphere verification of the candidate visual data is determined to be successful.

7. The visual data processing method according to claim 1, further comprising, before extracting information from at least one target reference image in the target atmosphere dimension to obtain the atmosphere information of the at least one target reference image in each target atmosphere dimension: Obtain at least one initial reference image and a reference text description; Based on the reference text description, determine the control type corresponding to each initial reference image; The initial reference image with the control type of "atmosphere" is determined as the target reference image.

8. The visual data processing method according to claim 1, before inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data, further includes: Save the visual relationship set and generate the target set identifier corresponding to the visual relationship set; The step of inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data includes: Upon receiving a visual data generation request, and if the visual data generation request carries the target set identifier, the visual relationship set is invoked, and the visual relationship set and the target prompt text are input into the visual data generation model to generate candidate visual data.

9. The visual data processing method according to claim 1, before inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data, further comprising: Receive a visual data generation request and obtain the image generation description in the visual data generation request; If the image generation description includes object description information, the target object corresponding to the object description information is obtained, wherein the target object is obtained from a registered object library, extracted from an object reference image, or generated based on the object description information; The target prompt text is constructed based on the image generation description, the object reference information of the target object, and the generation constraint information.

10. The visual data processing method according to claim 1, further comprising, before inputting the visual relationship set and the target prompt text into the visual data generation model to generate candidate visual data: Receive a visual data generation request and obtain the image generation description in the visual data generation request; If the image generation description does not contain object description information, the target prompt text is constructed based on the image generation description and generation constraint information.

11. The visual data processing method according to any one of claims 1-5 and 7-10, wherein after performing content verification on the candidate visual data based on the target prompt text and atmosphere verification on the candidate visual data based on the visual relationship set in the target atmosphere dimension, it further includes: If content verification and / or atmosphere verification fail, identify the dimensions that failed verification and require adjustment. The candidate visual data is adjusted along the dimension to be adjusted to generate new candidate visual data. The new candidate visual data is subjected to content verification and atmosphere verification. If both the content verification and atmosphere verification are passed, the candidate visual data is used as the target visual data.

12. A visual data processing system, comprising a client and a server, wherein, The client is used to send a visual data generation request to the server, wherein the visual data generation request includes at least one target reference image; The server is used to extract information from at least one target reference image in the target atmosphere dimension to obtain the atmosphere information of the at least one target reference image in each target atmosphere dimension. Based on the atmospheric information of the at least one target reference image in each target atmospheric dimension, a set of visual relationships is obtained; The set of visual relationships and the target prompt text are input into the visual data generation model to generate candidate visual data. The candidate visual data is subjected to content verification based on the target prompt text, and the candidate visual data is subjected to atmosphere verification based on the visual relationship set in the target atmosphere dimension. If both the content verification and the atmosphere verification pass, the candidate visual data is used as the target visual data and the target visual data is returned to the client.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the visual data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the visual data processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the visual data processing method according to any one of claims 1 to 11.

Images